stars.txt

[ajtribick]

As far as I understand it, CCDM lists both binary stars and optical doubles. Entry in CCDM does not mean the pair is a bound system.

[t00fri]

Note the comment by the most recent change:

Regressed to contain only Hipparcos-derived data

There should be no remaining hand modifications to stars.txt.

--Chris

But can you then explain to me why after that statement and his V 1.4 commits, stars.txt has these entries??

167 : granthutchison 1.4 171 0.54018776 27.08448905 1203.56825 5.79688 G3V

As we have agreed meanwhile (I hope) , the official 1997 HIP catalog has ~40 ly distance instead of the incorrect 1203 ly for HIP 171.

In any case, all this is a most ambiguous and unsatisfactory documentation of scientific data. My saying since a long time.

Bye Fridger

[t00fri]

As far as I understand it, CCDM lists both binary stars and optical doubles. Entry in CCDM does not mean the pair is a bound system.

Chris,

I think that's the catch for why your present code does not provide the correct distance assignments.

Bye Fridger

[chris]

As far as I understand it, CCDM lists both binary stars and optical doubles. Entry in CCDM does not mean the pair is a bound system.

Chris,

I think that's the catch for why your present code does not provide the correct distance assignments.

Agreed--CCDM is apparently not a reliable way to find bound pairs. There may be some other information in the data set that will help us figure out which stars are actually bound and which are merely optical doubles.

--Chris

[t00fri]

As far as I understand it, CCDM lists both binary stars and optical doubles. Entry in CCDM does not mean the pair is a bound system.

Chris,

I think that's the catch for why your present code does not provide the correct distance assignments.

Agreed--CCDM is apparently not a reliable way to find bound pairs. There may be some other information in the data set that will help us figure out which stars are actually bound and which are merely optical doubles.

--Chris

Yes, let's have a look. I'll think more about it.

Bye Fridger

[chris]

Note the comment by the most recent change:

Regressed to contain only Hipparcos-derived data

There should be no remaining hand modifications to stars.txt.

--Chris

But can you then explain to me why after that statement and his V 1.4 commits, stars.txt has these entries??

167 : granthutchison 1.4 171 0.54018776 27.08448905 1203.56825 5.79688 G3V

As we have agreed meanwhile (I hope) , the official 1997 HIP catalog has ~40 ly distance instead of the incorrect 1203 ly for HIP 171.

Grant removed his correction and reverted the distance to the incorrect value generated by buildstarsdb. The correct value is in revised.stc along with all the other corrections to the purely programmatically produced file stars.txt. So the star database used by Celestia is actually in quite good shape, but I agree that it would still be better if we could eliminate some of the errors in stars.txt by fixing buildstarsdb to not adjust distances of optical binaries.

--Chris

[t00fri]

So the star database used by Celestia is actually in quite good shape, but I agree that it would still be better if we could eliminate some of the errors in stars.txt by fixing buildstarsdb to not adjust distances of optical binaries.

--Chris

Well,...

the problem arose since my virtualbins.pl script reads the /incorrect/ stars.txt and thus produces wrong distance values in virtualbins.stc. That's why this discussion started after all. Since in the course of time Grant's philosophy has changed (with poor documentation), it /tacitly/ introduced those errors into my virtualbins.stc updates more recently...

Since I have at least identified TWO such binary systems,
HIP 171 and HIP 85582, where this transfer of errors into virtualbins.stc happens, we should definitely revise buildstarsdb.

For now I have modified visualbins.pl to read the distances from Soederhjelm's paper (instead of stars.txt), which are based on HIP, but include some minor corrections < 10% here and there . Soederhjelm claims that these corrections represent improvements over the original HIP distances.

Bye Fridger

[granthutchison]

Fridger:
The sequence of events with relation to stars.dat and stars.txt is as follows:

1) Stars.dat originally existed in isolation, as the only way of introducing stars into Celestia. To correct errors in stars.dat necessarily required hand-editing of the file, the data structure of which did not accommodate in-file documentation. I of course kept full records of the modifications, and annotated each change on CVS.

2) With the introduction of stc files, it became possible to make corrections outside stars.dat, with suitable annotation within stc files, which is of course much preferrable. At that point I reverted stars.dat and (the new, accompanying) stars.txt to their content as it would be generated by Chris's building code (easily done from my own records). This was the only consistent way to render this large datafile, since I was aware that Chris introduced some data-processing during the build process: the documentation for that is, of course, Chris's code.

Thus there are three levels of documentation: the original Hipparcos data; Chris's code to modify the Hip data to make stars.txt; my fully annotated revised.stc, which modifies stars.txt.

For "data purity", you might have considered using the raw Hip data to generate your binaries files; if using stars.txt, you might have considered checking through revised.stc for necessary corrections. Stars.txt alone was certainly the worst choice as a data source.


Grant

[ajtribick]

Agreed--CCDM is apparently not a reliable way to find bound pairs. There may be some other information in the data set that will help us figure out which stars are actually bound and which are merely optical doubles.

Could proper motion be used as a criterion for deciding whether the stars constitute a true binary? The proper motion is listed in both the Hipparcos catalogue and in CCDM.

[t00fri]

Fridger:
The sequence of events with relation to stars.dat and stars.txt is as follows:

1) Stars.dat originally existed in isolation, as the only way of introducing stars into Celestia. To correct errors in stars.dat necessarily required hand-editing of the file, the data structure of which did not accommodate in-file documentation. I of course kept full records of the modifications, and annotated each change on CVS.

2) With the introduction of stc files, it became possible to make corrections outside stars.dat, with suitable annotation within stc files, which is of course much preferrable. At that point I reverted stars.dat and (the new, accompanying) stars.txt to their content as it would be generated by Chris's building code (easily done from my own records). This was the only consistent way to render this large datafile, since I was aware that Chris introduced some data-processing during the build process: the documentation for that is, of course, Chris's code.

Thus there are three levels of documentation: the original Hipparcos data; Chris's code to modify the Hip data to make stars.txt; my fully annotated revised.stc, which modifies stars.txt.

For "data purity", you might have considered using the raw Hip data to generate your binaries files; if using stars.txt, you might have considered checking through revised.stc for necessary corrections. Stars.txt alone was certainly the worst choice as a data source.


Grant

Grant,

while I always have highly appreciated your careful and most competent work around Celestia, you know well that I am critical since a long time about the poor and unprofessional level/status of documentation of our astronomical data (including tacit modifications of catalog data).

This past discussion was a showcase example of the confusion that tends to arise this way and has cost some very experienced Celestians > 1 day of work to sort it all out and trace the origin of the induced bugs.

+++++++++++++++++++++
I can assure you that to a significant proportion, our untransparent way of documentation puts off numerous professional astronomers, physicists and the like trusting Celestia as an accurate tool of scientific standing.
+++++++++++++++++++++

-- /private/ notes about data and their modifications are just not admissible/desirable as sole means of scientific documentation of so many complex data sets

--your CVS logs like

** Some missing stars from the Bright Star Cat, some spectral & distance corrections

171 0.54018779 27.08449 40.452314 5.79688 G3V
85582 262.3342 29.392614 524.38426 8.98828 K2
distance = right for HIP 171, wrong for HIP 85582!

** A few final revisions that were lost in the stars.dat rebuild

171 0.54018779 27.08449 40.452314 5.79688 G3V
85582 262.3342 29.392614 524.38426 8.98828 K2
distance = right for HIP 171, wrong for HIP 85582!

** Regressed to contain only Hipparcos-derived data

171 0.54018776 27.08448905 1203.56825 5.79688 G3V
85582 262.3342 29.392614 524.38426 8.98828 K2
distance = wrong for HIP 171, wrong for HIP 85582!

are NEITHER available to serious Celestia users NOR do they contain precise information about which stars were actually modified and how/why by what!

Your latter entry is particularly misleading, since the HIP catalog contains the CORRECT distances throughout, while the INCORRECT distances only entered through Chris' inadequate exploitation of the CCDM flags in his starbuilder code. That I would NOT designate as "only Hipparcos-derived"...

C++ code can hardly be seen as a documentation for users. Despite reading C++ fluently, it took even me quite a while to locate the spot where Chris generated the wrong distances from the correct HIP data!

In my view stars.txt should rather be built as /precise/ human-readable image of those star data that Celestia actually uses. That was also the way I interpreted it's purpose, when using its content for visualbins.stc. Any correction files like revised.stc should therefore already be merged into stars.txt.

As is also standard in the global astronomical data bases, Celestia should have a human readable file in the source tarball e.g. called 'changes.txt', where all modifications of the original catalogs are precisely logged and easily available to every one.

Personally, I prefer (commented) PERL scripts that are included in the sources, as you know well. They are easy to read also for people not able to code PERL AND --at the same time-- were used precisely to generate Celestia's ACTUAL data from published catalogs.

Bye Fridger

[ajtribick]

My personal preference would be to use only Hipparcos-derived data in stars.txt/dat and if we want to join systems together as doubles, do that with something like revised.stc, instead of using revised.stc to go back to Hipparcos data when the double star flag is set on systems which are merely optical doubles.

[granthutchison]

I can assure you that to a significant proportion, our untransparent way of documentation puts off numerous professional astronomers, physicists and the like trusting Celestia as an accurate tool of scientific standing.

Then feel free to improve it. I left behind what I believed to be an interim solution, pending your development of a better system.

/private/ notes about data and their modifications are just not admissible/desirable as sole means of scientific documentation of so many complex data sets

Which is why I made them explicit in revised.stc before departing Celestia development. Your objections are to historical problems, which we all agreed were unsatisfactory at the time, and which were subsequently addressed.

... are NEITHER available to serious Celestia users NOR do they contain precise information about which stars were actually modified and how/why by what!

Hence the move to revised.stc once the advent of stc's made it possible to document changes properly. The final note indicates, quite clearly I think, that the previous opaque revisions have been regressed.

Your latter entry is particularly misleading, since the HIP catalog contains the CORRECT distances throughout, while the INCORRECT distances only entered through Chris' inadequate exploitation of the CCDM flags in his starbuilder code. That I would NOT designate as "only Hipparcos-derived"...

Then we differ. Chris derived these figures from Hipparcos. They are Hipparcos-derived. If I had meant to say that these were unchanged Hipparcos data, I would have said just that.

C++ code can hardly be seen as a documentation for users. Despite reading C++ fluently, it took even me quite a while to locate the spot where Chris generated the wrong distances from the correct HIP data!

Then your issue is perhaps with Chris rather than with me, on that point.

In my view stars.txt should rather be built as /precise/ human-readable image of those star data that Celestia actually uses. That was also the way I interpreted it's purpose, when using its content for visualbins.stc. Any correction files like revised.stc should therefore already be merged into stars.txt.

I'm astonished to see you suggest that hand revisions should be incorporated into the "pure" dataset: this is exactly the opposite of what you've been saying for years, and in fact was why I left behind a carefully annotated revised.stc to keep hand-editing separate from automatically generated data.

As is also standard in the global astronomical data bases, Celestia should have a human readable file in the source tarball e.g. called 'changes.txt', where all modifications of the original catalogs are precisely logged and easily available to every one.

Excellent suggestion. (Although I notice you haven't created such a thing.)

I'd suggest the way ahead is for you to work with Chris towards a data format that you find acceptable: it seems to me that that is no closer now than it was two years ago.
But please just give the sniping a rest: it's not my fault that you misunderstood the purpose of stars.txt, which was well-discussed on the Developers' list.

Grant

[t00fri]

/private/ notes about data and their modifications are just not admissible/desirable as sole means of scientific documentation of so many complex data sets

Which is why I made them explicit in revised.stc before departing Celestia development.

While revised.stc represents some improvement, it is certainly lacking any kind of documentation where these new numbers are from and why they have been replaced.

Your latter entry is particularly misleading, since the HIP catalog contains the CORRECT distances throughout, while the INCORRECT distances only entered through Chris' inadequate exploitation of the CCDM flags in his starbuilder code. That I would NOT designate as "only Hipparcos-derived"...

Then we differ. Chris derived these figures from Hipparcos. They are Hipparcos-derived. If I had meant to say that these were unchanged Hipparcos data, I would have said just that.

"Hipparcos-derived" is clearly inadequate here, since Chris' code inferred inadequately /physical/ double star configurations from the CCDM flags also in case of (unbound) /optical/ doubles which was the origin of the incorrect distance assignments of HIP 171 and HIP 85582,...

As you can read earlier on in his posts, he agrees to this.

Under "Hipparcos-derived" I for one understand /correct/ conclusions from the available Hipparcos data NOT /incorrect/ ones. An optical double star is not a proper double star as you know very well...

Are you really defending a procedure, such that

1) the Celestia starbuilding code can assign an unbound binary system incorrectly as a physical double with a corresponding joint distance wrong by a factor ~25 and an entry of these /wrongly generated/ data into stars.txt

2) a corresponding hand corrected entry is subsequently made into another file (revised.stc), to restore what was right ORIGINALLY, i.e.in the published HIP catalog!

3) an /incorrect/ stars.txt is lurking in the sources...as the /only/ coherent, human-readable reference source for Celestia's stars

If you think about it, I am of course not advocating hand corrections of any kind when arguing in favour of ONE CORRECT human-readable star.txt resource in Celestia. It just means I want the starbuilder to merge in the revised.stc content /before/ printing stars.txt not /thereafter/. This clearly requires the existence of an additional proper 'changes.txt' file that contains proper citation of the resources and modified stars involved. This 'changes.txt' file would eventually replace revised.stc, but unlike revised.stc, include proper citations and a short reasoning for the replacements made, just like in case of the professional data bases...

After I had to spend > 1 day of digging through this "documentation jungle" , I really find your final remarks quite displaced. Notably, since it was not my misunderstanding that caused the problem but rather the improper evaluation of the CCDM flags in the starbuilder code.
Also, I simply doubt that you were aware of the origins of those awkward errors before. Chris, for example was NOT, despite being the author of the starbuilder code.

...give the sniping a rest: it's not my fault that you misunderstood the purpose of stars.txt, which was well-discussed on the Developers' list.

Bye Fridger

[ajtribick]

Hmmm... it's probably better now we know what's going on in the backend to try and fix that rather than assign blame to who made it go wrong in the first place. We have a non-ideal situation right now, but it is recoverable.

We basically need to remove the binary star code from whatever it is that generates the star database, regenerate stars.txt and stars.dat and supply a notice explaining the change, and regenerate the binary star databases, again explaining the changes. I guess revised.stc can be significantly shortened since most of the entries in that file are to correct optical doubles.

Once that has been done, that gives us a well-documented starting point from which to build on.

This is surely far better than trying to figure out who made what go wrong when, which is just going to make people feel bad with no benefit to the end-user.

[granthutchison]

While revised.stc represents some improvement, it is certainly lacking any kind of documentation where these new numbers are from and why they have been replaced.

To quote from the opening text of revised.stc:

# Revisions to stars listed in Celestia distribution stars.dat
#
# These consist of improvements or revisions to the Hipparcos
# data from which stars.dat was prepared, the addition of some
# bright stars with incomplete data in Hipparcos, and the correction
# of some parallax errors introduced by Celestia's processing of
# apparently fictitious multiple systems generated by Hipparcos.
# Commonly used sources are listed below, and other sources are
# referenced in the annotations.
#
# NASA's NStars Database of nearby stars:
# http://nstars.arc.nasa.gov/select_fields.cfm
# McCook & Sion's white dwarf database:
# http://www.astronomy.villanova.edu/WDCatalog/index.html
# The RECONS database of nearby stars:
# http://joy.chara.gsu.edu/RECONS/TOP100.htm
# The SIMBAD stellar database:
# http://simbad.u-strasbg.fr/sim-fid.pl

(My bold.)
And the annotation to the first entry:

171 # 85 Peg: restored original Hipparcos parallax

Every single entry is similarly annotated, with a source and reason for replacement.

"Hipparcos-derived" is clearly inadequate here, since Chris' code inferred inadequately /physical/ double star configurations from the CCDM flags also in case of (unbound) /optical/ doubles which was the origin of the incorrect distance assignments of HIP 171 and HIP 85582,...

Then your problem is with how these data are derived from Hipparcos. It is undeniable that they are derived from Hipparcos.

Under "Hipparcos-derived" I for one understand /correct/ conclusions from the available Hipparcos data NOT /incorrect/ ones.

Then you are adding a level of interpretation that is unjustified. The problems with the derivation of stars.dat, in particular with the interpretation of fictitious doubles, has been known and discussed for years.

Are you really defending a procedure, such that ...

Of course not. It's a botch, which awaits a revision to the code that builds stars.dat. It is, however, as clear a botch as I could generate pending a code revision.

After I had to spend > 1 day of digging through this "documentation jungle" , I really find your final remarks quite displaced. Notably, since it was not my misunderstanding that caused the problem but rather the improper evaluation of the CCDM flags in the starbuilder code. Also, I simply doubt that you were aware of the origins of those awkward errors before. Chris, for example was NOT, despite being the author of the starbuilder code.

Of course I didn't know the origin of the problem within the starbuilder code; but I was well aware of its effects, as were many people on this forum, who have reported parallax errors from fictitious doubles on many occasions.
Revised.stc has always been no more than a cludge to undo those effects (among other problems) while awaiting a better solution.

So, once more with feeling: Please stop flailing around allocating blame, and move on with something more constructive.

Grant

PS. Since I doubt you'll follow my advice, I hereby withdraw from the field. Continue as you will.

[t00fri]

Really no intention from my side to snipe or blame anybody.

However, the issue about Celestia's poor scientific documentation level is an old one. It bothers me since years.

It is also affecting Celestia's reputation as a serious /scientific/ visualization tool.

Moreover the situation is highly prone of errors, as the present double star examples showed once more.
And if I have to spend > 1 day of unnecessary digging because of this, I am simply "not amused"

It's not worth going on fighting for an improved documentation plan, once I see that Chris ALWAYS spares out any such resources in his official (binary) versions for Windows, which consitute by far the dominant share of the Celestia distribution...

So all efforts of improving the documentation level along such lines, like adding my PERL extraction scripts etc are effectively useless, since they are never distributed.

I wonder how tiny the fraction of Windows users might be, who would download the Celestia source tarball

Bye Fridger

[bh]

Oh dear!

[chris]

OK. Here's the plan . . .

I'll fix the problem with the star database builder so that it doesn't use CCDM records to incorrectly associate stars that are only optical binaries. We can then remove some of the parallax corrections revised.stc. However, revised.stc will not go away, nor will the changes in it be merged into stars.txt--the whole raison d'etre of this file is to be a human readable version of data derived exclusively from the HIPPARCOS catalog. Revised.stc is the preferred place to introduce corrections from sources other than HIPPARCOS. The annotations in that file serve as adequate documentation for all the changes made.

Another useful project would be to write a document that explains in detail the process used to create the star catalog used by Celestia. This document could also contain a list of all the changes in revised.stc, but I don't think that this is essential: revised.stc is already a human-readable file, and having just a single source for the data means there are no problems keeping two files in sync.

Would this be satisfactory?

--Chris

[chris]

Really no intention from my side to snipe or blame anybody.

That's how it comes across, though--I've felt personally attacked throughout the discussion of the stars data file and binary star orbits, and it's challenging at times to respond to you in a civil and constructive manner. Please think about how you might present your points in a tone less likely to offend.

However, the issue about Celestia's poor scientific documentation level is an old one. It bothers me since years.

It is also affecting Celestia's reputation as a serious /scientific/ visualization tool.

Certainly, there are scientists that are already using Celestia for scientific visualization. I (and others) have heard and answered a lot of specific requests from them. Despite what you imply below, you'll get no disagreement from me that better documentation would improve Celestia.

It's not worth going on fighting for an improved documentation plan, once I see that Chris ALWAYS spares out any such resources in his official (binary) versions for Windows, which consitute by far the dominant share of the Celestia distribution...

So all efforts of improving the documentation level along such lines, like adding my PERL extraction scripts etc are effectively useless, since they are never distributed.

These scripts are all readily available on SourceForge, however. And if you would like to have them included in the Windows distribution of Celestia, please just say that directly rather than making some backhanded remark.

--Chris

[t00fri]

OK. Here's the plan . . .

I'll fix the problem with the star database builder so that it doesn't use CCDM records to incorrectly associate stars that are only optical binaries. We can then remove some of the parallax corrections revised.stc. However, revised.stc will not go away, nor will the changes in it be merged into stars.txt--the whole raison d'etre of this file is to be a human readable version of data derived exclusively from the HIPPARCOS catalog. Revised.stc is the preferred place to introduce corrections from sources other than HIPPARCOS. The annotations in that file serve as adequate documentation for all the changes made.

Another useful project would be to write a document that explains in detail the process used to create the star catalog used by Celestia. This document could also contain a list of all the changes in revised.stc, but I don't think that this is essential: revised.stc is already a human-readable file, and having just a single source for the data means there are no problems keeping two files in sync.

Would this be satisfactory?

--Chris

Chris,

would this mean that stars.txt is going to contain the ORIGINAL 1997 (ESA) Hipparcos data, without any further interference/modification from the starbuilder code??

If correct, then we may of course collect ALL revisions relative to the official HIP data in 'revised.stc' along with comments about references and the reason for modifications.

So far I was against that option, since your code did modify substantially (CCDM...) the HIP input data in stars.txt.

+++++++++++
In parallel, I stongly argue in favor of implementing also into your (binary) Windows distribution a human-readable, minimal documentation set, allowing an exact reconstruction of Celestia's data, in principle!
+++++++++++

Most Windows users will never consider downloading the Celestia tarball, which many will not even know how to unpack. This overwhelming number of users of the binary Win32 Celestia distribution has all rights of being concisely informed about the origin of Celestia's data

For stars and DSO data, with stars.txt REALLY containing the official HIP catalog data and nothing else, these would be sufficient:

1) revised.stc
2) visualbins.pl
3) spectbins.pl

These relatively short files will barely affect the length of the Celestia distribution! Since in such a scheme, 'revised.stc' will strictly correct relative to the published AND cited HIP data, all required information about Celestia stars is accessible this way.

Analogously, the PERL scripts for the generation of the binary orbit data will also refer strictly to published AND referenced catalogs anf thus everything looks "proper".

Without 'nearstars.stc' the situation would be quite simple and well-defined at this point (albeit NOT perfect as we know well). Since the latter file contains an (admittedly useful) /mix/ of interesting objects, we need to further contemplate the future of this file in the context of a probable increase of the number of binary systems, both of /visual/ type and /spectroscopic/ type.

I am well aware of the problematics here. We have discussed it at length with Grant, some time ago...I admit I don't have many new ideas in this context.

I certainly don't want to argue nearstars.stc "away"! It has most illustrative triple systems, for example, that are VERY useful when discussing e.g. a better drawing algorithm of orbits in different frames (how about it?? -;))

Bye Fridger

PS: As to visualbins.stc I now face two alternatives:

a) I may read in besides stars.txt also revised.stc, overlay the corrections and extract RA, DEC and the distance.
The advantage is that we restrict ourselves to the same data as for the HIP stars (RA,Dec, distance).

b) I may strictly use Soederhjelm's distance data which are ALL within 10% of the HIP distances and are argued by him to be more accurate than pure HIP. That's what I did for now, after understanding the reasons for the present incorrect distance data.