Extended stars.dat addons

[fungun]

The reason I am starting this poll, is because another user and i have been having discussions about where I put some of the fictional stars from Star Trek.
His ideas are sound, as far as I know, because I do not use either of the databases from the Motherload.
So If I make an addon from Star Trek and use say a TYC star from one of the other databases for it, how many users would not see it because they're not using that database.
Thanks,
Tim

[ajtribick]

The extended stars.dat databases are based on the first version of the Hipparcos catalog. The version of stars.dat included in 1.6.0 (you can download it from the SVN repository) is based on the new reduction of the Hipparcos data by Floor van Leeuwen, released in 2007. Furthermore the parallaxes used for the additional stars in the extended .dat filess are based on assumptions that may be statistically valid for the general population of stars, but not for individual stars.

[Hungry4info]

So essentially, the extended databases are fictional?

[Reiko]

I use the 2.1C I like have 2 million stars

[selden]

So essentially, the extended databases are fictional?

I would describe them as being "of limited accuracy". The technical term for the way their distances were calculated is "spectrographic distance" or "spectrographic parallax". The error bars are quite large.

[t00fri]

The extended stars.dat databases are based on the first version of the Hipparcos catalog. The version of stars.dat included in 1.6.0 (you can download it from the SVN repository) is based on the new reduction of the Hipparcos data by Floor van Leeuwen, released in 2007. Furthermore the parallaxes used for the additional stars in the extended .dat filess are based on assumptions that may be statistically valid for the general population of stars, but not for individual stars.

Andrew,

I made some cross checks of the contents of stars.txt from your new analysis in SVN.

There I find quite a few stars with distances well above 10000 ly! Immediately, common sense should tell that such figures cannot make much sense physically, no matter what fitting tricks have been used.

Given the general relation

++++++++++++++++++++++++++++++
parallax [mas] = 3261.6348 / distance[ly]
++++++++++++++++++++++++++++++

you have e.g. included in the Celestia distribution stars like HIP 58205, with a distance value as
large as d = 14825.8 ly (!).

This amounts to a parallax of ONLY

0.22 milli arcsec

Looking up the corresponding error in the revaluated HIP data, it is absolutely NO SURPRISE that one finds a nonsensically large uncertainty of

2.29 milli arc secs, i.e 10 times as large as the value itself!

So applying naively Gauss' error propagation formula to this result, the distance that you accepted in stars.text has a HUGE uncertainty...

Fridger

[ajtribick]

I can't check right now but what are the appmags of these stars from Earth? There is a magnitude threshold where stars that are brighter than a certain value are always included to keep Earth's night sky looking ok (this criterion was also used for the old database), but in the old case they had distances set to a certain fixed value... seems that this is not happening even though I thought I'd coded that (either way the result is nonsense, so I'm not too worried if it turns out that this is what's going on).

If these stars are below the Vmag = 6 (I think) threshold then there are serious problems!

[t00fri]

I can't check right now but what are the appmags of these stars from Earth? There is a magnitude threshold where stars that are brighter than a certain value are always included to keep Earth's night sky looking ok (this criterion was also used for the old database), but in the old case they had distances set to a certain fixed value... seems that this is not happening even though I thought I'd coded that (either way the result is nonsense, so I'm not too worried if it turns out that this is what's going on).

If these stars are below the Vmag = 6 (I think) threshold then there are serious problems!

Here is the corresponding line in SVN stars.txt

58205 179.061529756 +17.132589594 14825.772727 10.83 K0

So appMag is 10.83 in this particular case. But I really spotted quite many stars above 10000 ly distance in that file.

Fridger

PS: What I don't understand is this: why an appMag cut should be a decisive criterion for a reasonably well-determined distance value?

++++++++++++++++++++++
If I was involved, I would read out the quoted error on parallax as well in the Perl script and then test on the ratio delta(parallax) / parallax <= O(1)?. Anything else is not physically sensible, unless we invent a method that is able to visualize measurement uncertainties in Celestia!
++++++++++++++++++++++

[ajtribick]

Hmmm... I thought I'd squished this bug. I'll take another look at the code. Regarding the appMag cutoff, that's what was previously done. I did mention it in the devlist discussion...

[t00fri]

Let me detail a bit why a parallax error [tex]\delta(parallax)/parallax > 1[/tex] gives nonsensical results for the distance d of the star, using the general relation

[tex]d =\frac{const.}{parallax \pm \delta(parallax)}[/tex]

According to Gauss' error propagation law, one can easily calculate the resulting uncertainty on the distance, given that on the parallax

[tex]\delta(d) = \pm |\delta(parallax)\, \frac{\partial d}{\partial parallax}|[/tex]

or with the above relation,
[tex]\delta(d) = \pm \frac{\delta(parallax)}{parallax}\, d[/tex]

Hence we can write the distance along with it's uncertainty as

+++++++++++++++++++++++++++++++++++++++++
[tex]d \pm \delta(d) = \left (1 \pm \frac{\delta(parallax)}{parallax}\right )\, d[/tex]
+++++++++++++++++++++++++++++++++++++++++

Since in case of HIP 58205 and similar candidates, [tex]\frac{\delta(parallax)}{parallax}\approx 10 \gg 1[/tex], the resulting distance can be anywhere between

-9 d and +11 d, with d being the center value = 14825.8 ly that you used in star.dat!

+++++++++++++++++++++++++++
Generally, as one can see from these simple considerations: whenever [tex]\frac{\delta(parallax)}{parallax}>1[/tex], the resulting distance can become NEGATIVE, which is utter nonsense!

I think Celestia deserves a star data set that only involves sensible measurements...
+++++++++++++++++++++++++++


Fridger

[ajtribick]

To be honest, I thought I'd set things up to reject stars with parallax error > parallax with extreme prejudice for exactly that reason. Turns out I hadn't and the stars I had tested got rejected for failing a combination of tests. I have now fixed that bit of the code. The question is whether we want to preserve the bright stars with large parallaxes to preserve Earth's sky or not - I'm perfectly happy not to do this but I'd want some input from more people before I inflict yet more large downloads on people who are keeping up to date with the SVN. Also I've converted the script to "use strict" and in the process picked up a couple of minor bugs in the spectral type parser.

[chris]

To be honest, I thought I'd set things up to reject stars with parallax error > parallax with extreme prejudice for exactly that reason. Turns out I hadn't and the stars I had tested got rejected for failing a combination of tests. I have now fixed that bit of the code. The question is whether we want to preserve the bright stars with large parallaxes to preserve Earth's sky or not - I'm perfectly happy not to do this but I'd want some input from more people before I inflict yet more large downloads on people who are keeping up to date with the SVN. Also I've converted the script to "use strict" and in the process picked up a couple of minor bugs in the spectral type parser.

I feel strongly that we shouldn't eliminate naked eye stars from Celestia's default catalog. That doesn't mean they need to be left in stars.dat, however. How many stars with apparent magnitude < 6.0 have parallax error / parallax > 1.0? If there are just a few, is it feasible to create an stc file for these stars, perhaps using spectroscopic distance estimates. This would both keep stars.dat free of extremely questionable data and give us a place to add notes and references for distances based on something other than HIPPARCOS parallax measurements.

--Chris

[t00fri]

To be honest, I thought I'd set things up to reject stars with parallax error > parallax with extreme prejudice for exactly that reason. Turns out I hadn't and the stars I had tested got rejected for failing a combination of tests. I have now fixed that bit of the code. The question is whether we want to preserve the bright stars with large parallaxes to preserve Earth's sky or not - I'm perfectly happy not to do this but I'd want some input from more people before I inflict yet more large downloads on people who are keeping up to date with the SVN. Also I've converted the script to "use strict" and in the process picked up a couple of minor bugs in the spectral type parser.

I feel strongly that we shouldn't eliminate naked eye stars from Celestia's default catalog. That doesn't mean they need to be left in stars.dat, however. How many stars with apparent magnitude < 6.0 have parallax error / parallax > 1.0? If there are just a few, is it feasible to create an stc file for these stars, perhaps using spectroscopic distance estimates. This would both keep stars.dat free of extremely questionable data and give us a place to add notes and references for distances based on something other than HIPPARCOS parallax measurements.

--Chris

My above findings of quite a number of stars with [tex]\delta(parallax)/parallax>>1[/tex] in the re-evaluated Celestia star data base, which triggered this discussion off, makes me wonder why this crucial error ratio has not been systematically examined during the re-analysis. Instead, various untransparent criteria seem to have been implemented such as appMag cuts etc.

Anyway, the stars I looked at, corresponded to largest distances > 10000 ly and typically were not very bright. But this issue is most easy to clear up properly by means of the Perl script...

Fridger

[chris]

To be honest, I thought I'd set things up to reject stars with parallax error > parallax with extreme prejudice for exactly that reason. Turns out I hadn't and the stars I had tested got rejected for failing a combination of tests. I have now fixed that bit of the code. The question is whether we want to preserve the bright stars with large parallaxes to preserve Earth's sky or not - I'm perfectly happy not to do this but I'd want some input from more people before I inflict yet more large downloads on people who are keeping up to date with the SVN. Also I've converted the script to "use strict" and in the process picked up a couple of minor bugs in the spectral type parser.

I feel strongly that we shouldn't eliminate naked eye stars from Celestia's default catalog. That doesn't mean they need to be left in stars.dat, however. How many stars with apparent magnitude < 6.0 have parallax error / parallax > 1.0? If there are just a few, is it feasible to create an stc file for these stars, perhaps using spectroscopic distance estimates. This would both keep stars.dat free of extremely questionable data and give us a place to add notes and references for distances based on something other than HIPPARCOS parallax measurements.

--Chris

My above findings of quite a number of stars with [tex]\delta(parallax)/parallax>>1[/tex] in the re-evaluated Celestia star data base, which triggered this discussion off, makes me wonder why this crucial error ratio has not been systematically examined during the re-analysis. Instead, various untransparent criteria seem to have been implemented such as appMag cuts etc.

It was examined, and there was extensive discussion (I believe in the forum) about what to do with stars that had questionable parallaxes, and what amount of parallax error should cause a star be omitted from stars.dat.

Practicality might still demand something like an absolute magnitude cut. To reuse a favorite example, SIMBAD lists the HIPPARCOS parallax of Deneb as 1.01 mas, and the error as +/- 0.57 mas. It's not bad enough to give a negative distance for Deneb, but it's a very large error. How should this be resolved? Allow very large parallax errors for all stars? Or just for bright ones?

Anyhow, it seems like allowing stars with parallax error > parallax was just a bug and not the intended behavior. The bug should be fixed, but we should also be sure that it won't result in any bright stars getting dropped. If there are bright stars lost, we need to find out which ones and consider adding them in a supplementary .stc file.

--Chris

[t00fri]

...
Anyhow, it seems like allowing stars with parallax error > parallax was just a bug and not the intended behavior. The bug should be fixed, but we should also be sure that it won't result in any bright stars getting dropped. If there are bright stars lost, we need to find out which ones and consider adding them in a supplementary .stc file.

--Chris

I am happy to go along with this

Fridger

[ajtribick]

It'd probably be better to simplify the rejection criteria which are based on the old way of doing it, which isn't particularly well-reasoned. Instead of a "dubiousness score" which is currently used, I'd be perfectly happy to do some straightforward rejection of stars: anything with parallax error < parallax, anything with parallax < some cut off (0.2 mas seems to have been used as a "this parallax is very dubious" cutoff in the old system), anything with large position errors (any suggestions for what would be a reasonable criterion here?). Missing magnitude information is another reason to throw out the star in the current system.

[t00fri]

It'd probably be better to simplify the rejection criteria which are based on the old way of doing it, which isn't particularly well-reasoned. Instead of a "dubiousness score" which is currently used, I'd be perfectly happy to do some straightforward rejection of stars: anything with parallax error < parallax, anything with parallax < some cut off (0.2 mas seems to have been used as a "this parallax is very dubious" cutoff in the old system), anything with large position errors (any suggestions for what would be a reasonable criterion here?). Missing magnitude information is another reason to throw out the star in the current system.

Andrew,

to me this sounds like a much improved set of criteria, except for the typo:

rejection of stars: anything with parallax error > parallax, etc...

Still it would be a useful piece of info, if you could simply print out those stars with parallax error > parallax AND appMag <= 7, say. Just to get a feel about what numbers of stars we are talking...

Another important info would be to sum up the number of "survivors" of these criteria!
With Perl, all this is a 5 minute affair.

Fridger

[ajtribick]

Using the criterion appMag<=6 for compatibility with the currently-used threshold magnitude, here are the list of bright stars with parallax<0.2 and the list of bright stars with parallax error>parallax.

[t00fri]

Using the criterion appMag<=6 for compatibility with the currently-used threshold magnitude, here are the list of bright stars with parallax<0.2 and the list of bright stars with parallax error>parallax.

Thanks, Andrew,

that's useful information! So we have O(40) appMag< 6 stars with [tex]\frac{\delta(parallax)}{parallax}>1[/tex], which is bad. Right now, there are only two options I could think of for these O(40) stars:

• Try and combine the parallax info with the spectroscopic approach (B-V, HR diagram-> absMag-> distance) that was applied by Pascal Hartmann to obtain some distance estimates for the 2 million star data set. http://pascal.hartman.free.fr/
• Use the highly uncertain distances as a "guesstimate", BUT attach a well visible warning flag to the respective star info in some suitable manner.

I agree also that it would be a pity to leave out stars with appMag < 6 from Celestia's star data set.

Fridger

[selden]

Just a comment:

Some of the problematic stars are variables or double stars, which would seem to explain why they had poor parallax measurements. These characteristics could also make it difficult to determine reasonable spectrographic distances.