stars.txt

[selden]

Thanks!

[chris]

Furthermore, Celestia's README file contains an incorrect description of stars.dat. It looks like it is describing the old file format, except the description is incorrect: the old file format included both the HD and HIP catalogue numbers, while the README file lists only one catalogue number per record. (I remember trying to work out what the new file format was a while back... turns out I did it in 2005)

I've added a page describing the stars.dat file to the Wikibook here.

I don't think that a description of the stars.dat format is even appropriate for the README. Rather than updating it there, we should just remove it. The WikiBook and a README specific to buildstardb are the proper locations for this information.

--Chris

[chris]

Hmm... the code doesn't seem to run correctly on my system: the stellarClass gets output as 4x32-bit integers instead of being packed into a 16-bit integer.

I think I see what happened here: buildstardb.cpp uses the StellarClass type from the main Celestia source. However, StellarClass changed significantly a few years ago. It used to represent stellar classes exactly as they were stored in stars.dat, but no longer. buildstardb.cpp was not modified to account for the change, and no one noticed because buildstardb is run so infrequently. The pack/unpack methods of SpectralClass can be used to compress the spectral type for storage on disk as a 16-bit integer.

In addition, it is clear this is NOT the code that was used to generate the star database, as it still includes the correction for the spectral type of Capella. The current versions of stars.txt and stars.dat do not include this correction (as agreed quite a while back now).

So clearly, the version of buildstardb in the SVN is not the current version!

It is the current version. I think that the spectral type of Capella was changed back to the HIPPARCOS type by hand-editing stars.txt. The SVN log shows a 2005 change by Grant: "Regressed to contain only HIPPARCOS-derived data." The diffs show that this is where the class of Capella was changed from G5III back to the incorrect HIPPARCOS value of M1. For a while, stars.txt was the 'master' star file. But now that we can use stc files to replace stars originally defined in stars.dat, that's no longer necessary or desirable. What we want is a working buildstardb that converts the HIPPARCOS text catalog to a binary stars.dat file.

(Since I don't know what other changes have been made along with the removal of the Capella correction, I can't just remove the relevant lines of code)

EDITED TO ADD: I've sent a PM to Grant about this matter, as he is listed on the SVN as being the author of the stars.dat and stars.txt files.

Good. Hopefully he can confirm my recollections about what happened with stars.dat.

--Chris

[symaski62]

(Since I don't know what other changes have been made along with the removal of the Capella correction, I can't just remove the relevant lines of code)

EDITED TO ADD: I've sent a PM to Grant about this matter, as he is listed on the SVN as being the author of the stars.dat and stars.txt files.

Good. Hopefully he can confirm my recollections about what happened with stars.dat.

--Chris

http://celestia.svn.sourceforge.net/vie ... ortby=date

Code: Select all

------------------------------------------------------
|            |   1.4.1   ||   1.5.1   || 1.6.0 BETA    |
------------------------------------------------------
| stars.dat   |  SVN 2800 || SVN 2800 ||  SVN ????     |
------------------------------------------------------
| stars.txt   |  SVN 2800 || SVN 2800 ||  SVN ????     |
-----------------------------------------------------


[granthutchison]

I'm responding to Andrew's PM.

The "old" version of stars.dat had no corresponding stars.txt. I wrote my own editor to allow hand-editing. At the time, levering revisions into the compressed structure of stars.dat was the only way to make changes to the way Celestia portrayed stars. I maintained my own documentation, because there was no way to document the stars.dat edits within the existing data structure.
Then three things happened:
1) Chris produced a new data structure for stars.dat, interconvertible with the accompanying stars.txt.
2) *.stc files become available, allowing the overwriting of the star definitions that loaded with stars.dat.
3) I retired from Celestia development.

Event 2) provided a much-needed way of publicly documenting my various revisions to the original dataset. Since I'd kept a detailed record of all the changes I'd made, it was easy (but tedious) to produce revised.stc (part of the current distribution) which makes each revision clear, and documents its origin. This seemed to be a Good Thing, and I spent some considerable time on it after my "official" withdrawal.
But what should we do with stars.dat: leave it in its edited state, or restore it to a more pristine version? It seemed to make more sense to put things back the way they should have been, so I undid all my previous changes by hand using stars.txt, and built a new stars.dat. (The obvious alternative route of rebuilding through old and new versions directly from Hipparcos was unavailable to me, since I didn't have the facilities to run Chris's original code.)

Grant

[ajtribick]

Ok, thanks for the info.

So it looks like stars.dat/txt is generated by the output of buildstardb.cpp (less the alteration to Capella). This means it is actually a merged dataset from three catalogues in I/239:

• The Hipparcos Main Catalog (hip_main.dat)
• The main part of Tycho Catalog (tyc_main.dat) - used to replace position/parallax information when error is less on Tycho parameters
• Double and Multiples: Component solutions -COMP (h_dm_com.dat) - used to put components of multiple stars at the same distance

Taking a look at h_dm_com.dat, there are three types of solutions: F, I and L, of which the I-type is listed as "individual parallaxes and linear (relative) motion (possible optical double star)". For doubles of the other types, the parallaxes in the main Hipparcos catalog are already identical, thus there is no need to process this file in the generation of stars.dat.

There's also code to guess spectral types from B-V colour index for stars which don't have this information in the Hipparcos Main Catalog - the source for the values of B-V which correspond to whichever spectral type is not given (this information would be very useful).

So what I propose: remove all entries in revised.stc which are described as "restore original Hipparcos parallax", only use the Hipparcos Main Catalog to generate stars.dat/.txt and move the edits using parameters from the Tycho Catalog and the B-V to spectral type converter to revised.stc.

In addition, the system for actually generating the stars.dat/.txt files requires passing several files through multiple C++ programs, not all of which get built when Make-ing Celestia. Might as well make the whole thing a PERL script while I'm at it.

[ajtribick]

Ok, here's a PERL script that will generate stars.dat and revised.stc from the hip_main.dat and tyc_main.dat files (available here). It is based on the program buildstardb.cpp in the celestia/src directory. Hopefully the comments in the script should allow you to figure out what is going on. The script allows several options to be set at the command line, run with -? for instructions. Without parameters it defaults to a fairly sensible set of parameters.

Some points to note:

• This script does not give the same subset of the Hipparcos catalogue as are in the current stars.dat file.
• The stars.txt file is not generated - is this file really necessary?
• The paths for the various file options are not validated in any way. Be careful.

...script removed for now...

[t00fri]

[*]The stars.txt file is not generated - is this file really necessary?

Certainly, stars.txt and revised.stc are an input for my binary star PERL scripts, in order not to mess around with two slightly different distances for the barycenters! See my respective .pl scripts in src/tools/binaries

If you changed Grant's most recent strategy wrto revised.stc in any way,
please let me know. Otherwise, my scripts will not work properly anymore!

Fridger

[chris]

Ok, here's a PERL script that will generate stars.dat and revised.stc from the hip_main.dat and tyc_main.dat files (available here). It is based on the program buildstardb.cpp in the celestia/src directory. Hopefully the comments in the script should allow you to figure out what is going on. The script allows several options to be set at the command line, run with -? for instructions. Without parameters it defaults to a fairly sensible set of parameters.

Some points to note:

• This script does not give the same subset of the Hipparcos catalogue as are in the current stars.dat file.
• The stars.txt file is not generated - is this file really necessary?
• The paths for the various file options are not validated in any way. Be careful.

Andrew--thanks for your work on this!

Ultimately, it shouldn't be necessary to generate stars.txt--stc files provide us a much better way to make corrections to the Hipparcos data (among other things, they allow comments for documentation.) But, we do need to consider that the binary star Perl scripts currently use stars.txt as an input. They might have to be adapted.

I don't expect to generate the exact same set of stars as the old buildstarsdb program did--we don't want to replicate all the old quirks. But, I'd be interested in knowing just how many differences there are.

--Chris

(By the way, I think the usual capitalization convention is Perl, not PERL--the language name isn't truly an acronym.)

[t00fri]

Ok, here's a PERL script that will generate stars.dat and revised.stc from the hip_main.dat and tyc_main.dat files (available here). It is based on the program buildstardb.cpp in the celestia/src directory. Hopefully the comments in the script should allow you to figure out what is going on. The script allows several options to be set at the command line, run with -? for instructions. Without parameters it defaults to a fairly sensible set of parameters.

Some points to note:

• This script does not give the same subset of the Hipparcos catalogue as are in the current stars.dat file.
• The stars.txt file is not generated - is this file really necessary?
• The paths for the various file options are not validated in any way. Be careful.

Andrew--thanks for your work on this!

Ultimately, it shouldn't be necessary to generate stars.txt--stc files provide us a much better way to make corrections to the Hipparcos data (among other things, they allow comments for documentation.) But, we do need to consider that the binary star Perl scripts currently use stars.txt as an input. They might have to be adapted.

I don't expect to generate the exact same set of stars as the old buildstarsdb program did--we don't want to replicate all the old quirks. But, I'd be interested in knowing just how many differences there are.

--Chris

(By the way, I think the usual capitalization convention is Perl, not PERL--the language name isn't truly an acronym.)

.stc files would be equally OK. The changes in my .pl scripts would be minor. HOWEVER, the corrective actions of revised.stc MUST not change without concise documentation. Otherwise=> chaos

Personally, I would much prefer a .pl script for generating stars.stc that reads in the original HIP/Tycho data AND contains all revisions of the original data in a human readable form (with references!).

Fridger

[ajtribick]

.stc files would be equally OK. The changes in my .pl scripts would be minor. HOWEVER, the corrective actions of revised.stc MUST not change without concise documentation. Otherwise=> chaos

Personally, I would much prefer a .pl script for generating stars.stc that reads in the original HIP/Tycho data AND contains all revisions of the original data in a human readable form (with references!).

Fridger

Out of interest, why not use the Perl unpack() function to read in stars.dat directly?

In any case the script I produced does put documentation into revised.stc (though the default option on the spectral type guesser does not explicitly mark these, specifying -s1 puts all guessed spectral types into revised.stc and leaves them out of stars.dat) - any time a value is changed, e.g. taken from Tycho, a comment gets added to revised.stc. The revised.stc file produced still contains all of Grant's changes which are applied through use of the %handEdits array defined near the top of the script.

The only entries in revised.stc I left out were the ones marked as "restore original Hipparcos parallax" and suchlike, and the entry for HIP 43177 which may or may not belong in extrasolar.stc instead (though following the logic dictated at the top of the extrasolar.stc file, the correction to HIP 43177 probably does belong in revised.stc, and the entry for TW Hya=HIP 53911 in extrasolar.stc should probably get moved to revised.stc).

[t00fri]

.stc files would be equally OK. The changes in my .pl scripts would be minor. HOWEVER, the corrective actions of revised.stc MUST not change without concise documentation. Otherwise=> chaos

Personally, I would much prefer a .pl script for generating stars.stc that reads in the original HIP/Tycho data AND contains all revisions of the original data in a human readable form (with references!).

Fridger

Out of interest, why not use the Perl unpack() function to read in stars.dat directly?

Also that is fine, as long as there is a human readable .pl file containing ALL changes AND corresponding references relative to the published original data. Note we did not have a Perl file before!

Fridger

[ajtribick]

The major lacking reference in the current version of the code is the spectral type guessing routine - I took it from buildstardb.cpp but there is no reference given there for the values used in it to convert B-V to spectral type. (Of course it is quite simple to turn off the spectral type guesser with the command-line option -s0)

[chris]

The major lacking reference in the current version of the code is the spectral type guessing routine - I took it from buildstardb.cpp but there is no reference given there for the values used in it to convert B-V to spectral type. (Of course it is quite simple to turn off the spectral type guesser with the command-line option -s0)

Any reference for the spectral type guesser is long gone; I think it was just based on my own observations of the correlation between B-V and spectral type. I'd say it's not really suitable to use in its current state. The first question is whether we want to use it at all. The problem is that spectral type is currently the only means that Celestia has for determining the star color. Omitting the spectral type guess means that we throw out color information about the star.

As long as we're updating the star catalog generation, it seems like it might be a good time to also discuss what information should be stored with stars. B-V measurements should be available for many more stars than spectral types. Perhaps we should store these as well as the spectral type. The spectral type would be displayed when available, but the B-V color index would be always used to determine the color for star rendering. For compatibility with existing STC files, the B-V value would have to be derived from the spectral type. This could be done more reliably than the inverse problem, since it doesn't involve trying to fill in missing spectral data.

One thing that I'm not sure about is whether reddening from the interstellar medium is accounted for in the HIPPARCOS B-V values and spectral types. If not, the star colors would look correct from Earth but not from other points in space. I suppose that this is really a no-win solution unless we want to actually try and model the distribution of the intermedium

--Chris

[ajtribick]

I guess this is where we start fighting about what the best parameters to use are, and what should get put into revised.stc. This is why I gave the option of either switching the spectral type guesser off (-s0) or moving all the guesses into revised.stc (-s1).

Also, at present the Tycho catalogue is only used as a source for corrections to Hipparcos stars (which get put in revised.stc by default). It would be fairly trivial to produce a binary database/stc file based on the Tycho catalog which includes non-Hipparcos stars. Actually, since Tycho does not include spectral types, either they'd all come out as ? or we'd have to use a spectral type guesser. I've worked up a script that does the tallies, and we'd end up with even more stars: at drop-level 3, 77279 stars pass the inclusion criteria, at drop-levels 2 and 1, the number increases to 493795, while at drop-level 0 no stars get dropped (I'm not particularly comfortable with such inclusiveness myself), which adds 934901 stars. I am NOT advocating putting these in stars.dat, however it might be interesting to consider tycho.dat.

(See the script documentation for what these drop-levels mean)

[chris]

I guess this is where we start fighting about what the best parameters to use are, and what should get put into revised.stc. This is why I gave the option of either switching the spectral type guesser off (-s0) or moving all the guesses into revised.stc (-s1).

Also, at present the Tycho catalogue is only used as a source for corrections to Hipparcos stars (which get put in revised.stc by default). It would be fairly trivial to produce a binary database/stc file based on the Tycho catalog which includes non-Hipparcos stars. Actually, since Tycho does not include spectral types, either they'd all come out as ? or we'd have to use a spectral type guesser. I've worked up a script that does the tallies, and we'd end up with even more stars: at drop-level 3, 77279 stars pass the inclusion criteria, at drop-levels 2 and 1, the number increases to 493795, while at drop-level 0 no stars get dropped (I'm not particularly comfortable with such inclusiveness myself), which adds 934901 stars. I am NOT advocating putting these in stars.dat, however it might be interesting to consider tycho.dat.

(See the script documentation for what these drop-levels mean)

My preference would be to include in stars.dat all the star records generated from purely Hipparcos and Tycho data. Revised.stc would include just those records that use corrections from other catalogs. As long as the procedure for deriving the star records from Tycho and Hipparcos data is well-documented, I don't think it's necessary to put Tycho-derived adjustments in revised.stc.

Something that could be useful is to modify the binary star database format slightly so that there's room for a text comment in the header. When buildstardb.pl generated stars.dat, it would insert information about the program version and command line arguments.

--Chris

[chris]

Also, at present the Tycho catalogue is only used as a source for corrections to Hipparcos stars (which get put in revised.stc by default). It would be fairly trivial to produce a binary database/stc file based on the Tycho catalog which includes non-Hipparcos stars. Actually, since Tycho does not include spectral types, either they'd all come out as ? or we'd have to use a spectral type guesser. I've worked up a script that does the tallies, and we'd end up with even more stars: at drop-level 3, 77279 stars pass the inclusion criteria, at drop-levels 2 and 1, the number increases to 493795, while at drop-level 0 no stars get dropped (I'm not particularly comfortable with such inclusiveness myself), which adds 934901 stars. I am NOT advocating putting these in stars.dat, however it might be interesting to consider tycho.dat.

What parallax and magnitude thresholds are you using to get these counts?

--Chris

[symaski62]

ftp://cdsarc.u-strasbg.fr/pub/cats/I/250 => (990182 stars) 1998

http://cdsweb.u-strasbg.fr/cgi-bin/qcat?I/259 => (2539913 stars) 2000

[ajtribick]

What parallax and magnitude thresholds are you using to get these counts?

I'm using the same thresholds as were in the src/buildstardb.cpp file (and reimplemented in buildstardb.pl):

at drop-level 0, all stars with parallax below 0.4 are edited to have parallax=0.4, and the parallax-error is set to 1000.
at drop-level 1, all stars with parallax below 0.4 and Vmag<6 are edited to have parallax=0.4, and the parallax-error is set to 1000.
at drop-levels 2 and 3, no parallaxes get fixed.
(Note there is no difference in the counts between 1 and 2, since this is non-Hipparcos stars and Hipparcos covered all the bright ones)

In Perl, the function to calculate the "dubiousness" (parameter is a reference to a hash):

Code: Select all

# dubiousness function based on HipparcosStar::analyze() in buildstardb.cpp
sub TestDubious {
   my $ref = shift;
   my %star = %{$ref};
   my $dubious = 0;
   $dubious += 400 if($star{'Vmag'} eq ''); # if there is no magnitude information, we can't use this star
   if($star{'e_Plx'} > -1000) { # skip this step if distance is hand-edited
      if($star{'Plx'} <= 0) {
         $dubious += 400;
      } elsif($star{'Plx'} < 0.2) {
         $dubious += 4;
      } elsif($star{'Plx'} < 0.4) {
         $dubious += 2;
      }
      if($star{'Plx'} <= $star{'e_Plx'}) {
         $dubious += 2;
      } elsif($star{'Plx'} <= (2*$star{'e_Plx'})) {
         $dubious += 1;
      }
   }
   $dubious += 400 if(($star{'RAdeg'} eq '') || ($star{'DEdeg'} eq ''));
   if($star{'e_RAdeg'} >= 1000) {
      $dubious += 4;
   } elsif($star{'e_RAdeg'} > 20) {
      $dubious += 100;
   } elsif($star{'e_RAdeg'} > 15) {
      $dubious += 2;
   } elsif($star{'e_RAdeg'} > 10) {
      $dubious += 1;
   }
   if($star{'e_DEdeg'} >= 1000) {
      $dubious += 4;
   } elsif($star{'e_DEdeg'} > 20) {
      $dubious += 100;
   } elsif($star{'e_DEdeg'} > 15) {
      $dubious += 2;
   } elsif($star{'e_DEdeg'} > 10) {
      $dubious += 1;
   }
   return $dubious;
}

The dubious value is then processed as follows:

Code: Select all

if($dubious <= 3) {
   # good stars always get added
} elsif(($DROP_LEVEL < 3) && (($dubious <= 5) || (($dubious < 400) && (($DROP_LEVEL == 0) || ($stars{$HIP}{'Vmag'} < 6))))) {
   # star gets flagged as dubious
} else {
   # star data unreliable so drop
}

As I said, this is basically just a reimplementation of the function in buildstardb.cpp - if we want a better function to accept/reject stars we should discuss this.

[ajtribick]

Ok progress report on this...

The solution I'm working towards is to implement hip_main.dat, h_dm_com.dat (components which do not get identified as either HIP or TYC stars will have to go in a .stc file) and tyc_main.dat, all of which are from I/239.

I'm also trying to get the spectral type guesser on a more solid scientific footing. I'm considering using the B-V values in this paper, but whether these are appropriate, or if there are better tables linking B-V to spectral type, please let me know!

Rather than using a hard switch between values in HIP and TYC where both catalogues contain information on the same star, I'm now using error-weighted averages. However identifying which TYC star refers to which HIP star (in order to check data/supply missing values) is complicated by multiple systems, which is why I have removed the Perl script from my post upthread. I've got a system which works by using the CCDM component identifiers, which works in most cases, though there are still issues when the HIP entry refers to photocentre or centre-of-mass. The way I'm thinking of doing it is by using the h_dm_com.dat file to identify the alphabetically-first component of the system (usually this will be A, but some CCDM entries contain multiple HIP stars).

It should also be possible to create a TYC-HIP cross-index. If I get the implementation of h_dm_com.dat working, it might also be desirable to have a HIP/TYC-CCDM cross-index too.

Alternatively we can just go with implementing hip_main.dat on its own.