The format of the cross index files

[LukeCEL]

Hi everyone. Unrelated to the other stuff I was posting about, but does anyone know exactly what format the cross index files (hdindex.dat and saoindex.dat) are in? The WikiBooks only has a page about Binary Star Files (stars.dat) I don't really know how to use the tools on Github (such as makexindex.cpp), so that won't really be of any help. Thanks in advance.

[Art Blos]

I join the question (although a little on another occasion). I asked Greg to compile utilities to edit a binary file, but it didn't work out.
If I could translate a binary file stars.dat into a text version and vice versa, "Celestia Origin" would be better. Can anyone make a successful compilation and write a detailed manual, how to use ready-made utilities?

[selden]

The Perl script buildstardb.pl can be used to create stars.dat

The documentation is in the buildstardb.pl Perl script itself. It is used to create stars.dat from the Hipparcos catalog. See https://sourceforge.net/p/celestia/code/HEAD/tree/trunk/celestia/src/tools/stardb/

[gironde]

usually, .dat files are database files. There is not a single program to open them; you need to know the database software for which they were designed.
For hdindex.dat and saoindex.dat we can see with a text editor that the header of the file is CELINDEX. The .dat file reader is inside the Celestia code.

[Janus]

Has anyone looked in tools?

src\tools\stardb to be specific.

celdat2txt.cpp might do what you want.

You can also get format information from

src\tools\xindex

buildxindices.pl to be exact.

Then there is stars.txt, which is used to build stars.dat from.

Somewhere in my stuff I have listings of all the database formats from when I made my own.
I can find or reproduce them if needed.
The formats are all simple.

The stars.dat is just a relative position file, a conversion from RaDecDist to XYZ, then saved in binary compact format to speed loading.
In one of my forks I preserved RaDecDist to work with some custom commands I added for looking stars up.
That preconversion is one of the ways early versions of Celestia brought down loading time.

One of the things I have been working toward is conversion directly in the GPU perframe with controlled framerate.

For those who actually care, here is how stars.dat is created.
Exercpt from buildstardb.pl

Spoiler

Code: Select all

sub WriteDat
{
   my $numStars = keys %stars;
   print "Writing databases...\n";

   print "  Writing binary database to $DAT_PATH\n";
   local(*DATFILE);
   open(DATFILE, '>', $DAT_PATH) or die "ERROR: Could not write to $DAT_PATH\n";
   binmode(DATFILE);
   
   print "  Writing text database to $TXT_PATH\n";
   local(*TXTFILE);
   open(TXTFILE, '>', $TXT_PATH) or die "ERROR: Could not write to $TXT_PATH\n";
   
   # write file header
   print DATFILE pack('a8ccL', 'CELSTARS', 0, 1, $numStars);
   print TXTFILE sprintf("%u\n", $numStars);
   
   # write each star
   foreach my $HIP (sort { $a <=> $b } keys %stars)
   {
      my $dist = PlxToDistance($stars{$HIP}{'Plx'});
      my $theta = $stars{$HIP}{'RArad'} + pi;
      my $phi = $stars{$HIP}{'DErad'} - pi / 2;
      my @xyz = (
          $dist * cos($theta) * sin($phi),
          $dist * cos($phi),
         -$dist * sin($theta) * sin($phi)
      );
      my $xc = $eqToCel[0][0] * $xyz[0] + $eqToCel[1][0] * $xyz[1] + $eqToCel[2][0] * $xyz[2];
      my $yc = $eqToCel[0][1] * $xyz[0] + $eqToCel[1][1] * $xyz[1] + $eqToCel[2][1] * $xyz[2];
      my $zc = $eqToCel[0][2] * $xyz[0] + $eqToCel[1][2] * $xyz[1] + $eqToCel[2][2] * $xyz[2];
      my $absMag = AppMagToAbsMag($stars{$HIP}{'Vmag'}, $stars{$HIP}{'Plx'});
      my $spType = ParseSpType($stars{$HIP}{'SpType'});
      print DATFILE pack('LfffsS', $HIP, $xc, $yc, $zc, $absMag * 256, $spType);
      print TXTFILE sprintf("%u  %.9f %+.9f %.6f %.2f %s\n", $HIP,
                            rad2deg($stars{$HIP}{'RArad'}), rad2deg($stars{$HIP}{'DErad'}),
                       $dist, $stars{$HIP}{'Vmag'}, $stars{$HIP}{'SpType'});
   }
   
   close(DATFILE);
   close(TXTFILE);
   
   print "  Wrote a total of $numStars stars.\n";
}


And here is how the CELINDEX files are created.
buildindices.pl

Spoiler

Code: Select all

#!/usr/bin/perl

# buildcrossidx.pl by Andrew Tribick
# version 1.0 - 2008-08-26

open XIDS, '<', 'crossids.txt';

@HDtoHIP  = ();
@HDlevels = ();
@SAOtoHIP = ();

while($curLine = <XIDS>) {
   chomp $curLine;

   # get Hipparcos designation
   $HIP = '';
   $HIP = $2 if($curLine =~ m/(^|:)HIP ([0-9]+)(:|$)/);
   next if($HIP eq ''); # ignore entries which are not Hipparcos stars

   @dList = split(/:/, $curLine);
   for($i = 0; $i <= $#dList; $i++) {
      # no component identifiers on SAO designations - makes things easy...
      $SAOtoHIP{$1} = $HIP if($dList[$i] =~ m/^SAO ([0-9]+)$/);

      # only use HD designations with component identifiers A,J or none
      if($dList[$i] =~ m/^HD ([0-9]+)([AJ]?)$/) {
         $HDnum = $1;
         $Level = HDlevel($2);
         if(!exists($HDtoHIP{$HDnum})) {
            # if this HD number is not already assigned, add it to list
            $HDtoHIP{$HDnum}  = $HIP;
            $HDlevels{$HDnum} = $Level;
         } elsif($Level > $HDlevels{$HDnum}) {
            # otherwise we prefer A over none over J.
            $HDtoHIP{$HDnum}  = $HIP;
            $HDlevels{$HDnum} = $Level;
         }
      }
   }
}

close XIDS;

# write out HD index file
open HDX,  '>', 'hdxindex.dat';
binmode HDX;
print HDX pack('a8S', 'CELINDEX', 0x0100);
foreach $HD (sort { $a <=> $b } keys %HDtoHIP) {
   print HDX pack('LL', $HD, $HDtoHIP{$HD});
}
close HDX;

# write out SAO index file
open SAOX, '>', 'saoxindex.dat';
binmode SAOX;
print SAOX pack('a8S', 'CELINDEX', 0x0100);
foreach $SAO (sort { $a <=> $b } keys %SAOtoHIP) {
   print SAOX pack('LL', $SAO, $SAOtoHIP{$SAO});
}
close SAOX;

# ---END OF MAIN PROGRAM---

sub HDlevel {
   # return a score based on component identifier
   my $d = shift;
   return 0 if($d eq 'J');
   return 1 if($d eq '');
   return 2 if($d eq 'A');
   return -999;
}


I hope that helps.

Keeping the RaDecDist directly in the db rather than computing it is one of those things I was going to recommend.
However, things stalled out, so I went my own way.

Now that onetwothree is getting things moving, I will bring it up at a later time.


Janus.

[selden]

The star positions in stars.dat are actually in Celestia's internal xyz coordinates, not ra,dec,distance. This reduces the runtime computation needed to determine their positions relative to the current viewpoint.

[LukeCEL]

Somewhere in my stuff I have listings of all the database formats from when I made my own.
I can find or reproduce them if needed.
The formats are all simple.

Sorry, I think there was a misunderstanding. Instead of the tools used to make these files (which have pretty scant documentation anyway), I'm requesting this instead.

[onetwothree]

Hi LukeCeL, xindex.dat file has very simple format:

HEADER, DATA_CHUNKS[]

The header is:
MAGIC, 8 bytes, CELINDEX
VERSION, 2 bytes, 0x0100

A data chunk is:
STARS_DAT_INDEX, 4 bytes
CATALOG_INDEX, 4 bytes

So for entry #1 in stars.dat we have 0x000000D9 (217) in saoxindex.dat and 0x000001A6 (422) in hdxindex.dat.

All numbers are store in Low Endian format.

But instead of editing existing stars.dat and indexes I'd suggest to use a source file stars.txt and build that files using buildstardb.pl (see src/tools/stardb/readme.txt) and buildxindices.pl (see src/tools/xindex/readme.txt).

But be warned that current version of stars.dat doesn't support spectral class Y.

[LukeCEL]

Thanks everyone for the information. I was able to create expanded versions of the hdindex.dat and saoindex.dat, which include HD and SAO numbers for TYC names. However, I guess the information about the cross index files can (and should) go into the Celestia Wikibooks.

[selden]

I've created a "stub" entry in Wikibooks. Please update the page appropriately.

https://en.wikibooks.org/wiki/Celestia/Catalog_Cross_References