External tool for updating the start/planet information

[jdavies]

Hi all,
I'm new to Celestia, and a Java developer ( I haven't done any C++ for over 10 years now ). I'm working on a simple utility that would check multiple site on the web for star information and then update the various .dat files in Celestia in an automated manner. I downloaded the .dat files that contain over 2 millions stars from Celestia Motherlode (http://www.celestiamotherlode.net/creat ... _1.4.0.zip) but given the rate at which new stars are catalogued and especially the new planets, I wanted to an easier way to ensure that my stars.dat (and associated files) are constantly updated with the latest info.

Last night I got started on reading the stars.dat file using a Java program. By and large I can read the file just fine, except for the x, y, z floating point coordinates in the file (everything else reads just fine). I want to double check one of my assumptions here to make sure that I am on the right track.

Using the source code for Celestia 1.5.1, I started by looking at the data/stars.txt file file, which I assume is the flat text source that is used to generate the data/stars.dat file. The first 2 lines of the stars.txt file are:

Code: Select all

112519
1  0.00091185 1.0890133 921.37572 9.09766 F5

Which of course represents the numver of records in the file, and then the first star record itself. If I examine the stars.dat file in a hex editor, I see the following bytes that represent that first star record (skipping over the file header, version, etc.):

Code: Select all

01 00 00 00 65 4d 66 44 0f 7c 80 41 b7 54 fd c0 d7 01 58 03

The bytes 65 4d 66 44 should be for the x coordinate, in little-endian form (Java uses Big-endian, which I account for by flipping the byte order accordingly). This should correspond to the value 0.0091185 in the text file. However, try as I might I cannot trnaslate the bytes 65 4d 66 44 into that floating point number, not even close. Java floats are stored in 64 bits and there is a standard conversion from C/C++ 32 bit floats to Java 64 bit floats as follows:

Code: Select all

int iy = dis.readInt(); // Read the 4 bytes in from the disk file
String hex = Integer.toHexString(iy);  // Debugger shows hex = 65 4d 66 44
int iy2 = Integer.reverseBytes(iy);  // Reverse the byte order to be Java big-endian
String hex2 = Integer.toHexString(iy2); // Make sure the bytes look good, and they do: 44 66 4d 65
float y = Float.intBitsToFloat(iy2); // Convert the integer bits into a Java float. x becomes 6.0623225E22, WAY wrong

Does anyone have any ideas why the floating point numbers are not workign for me? I can read the int16 values just fine for the Abs Mag and the bit-flagged start type / stellar class. Its only these floating point numbers that are a problem for me.

Many thanks in advance!

- Jeff

[selden]

Jeff,

You probably should start from the perl code written to generate stars.dat for Celestia v1.6.
It's available on sourceforge at
http://celestia.svn.sourceforge.net/vie ... dstardb.pl

[jdavies]

Thanks Selden. I'm still learning my way around the project. I really appreciate your guidance.

- Jeff

[selden]

You're very welcome.

[chris]

Also, this page by Andrew Tribick--primary developer of the new star processing scripts--gives a lot of information about the binary star format:

http://en.wikibooks.org/wiki/Celestia/Binary_Star_File

--Chris

[jdavies]

Thanks Chris. I wish I had that site yesterday.

Fortunately, it was fairly easy to figure out be reviewing the source code that read the star.dat file. I'm now looking at the current source code for 1.6. I just sync with the HEAD of the project. Glad to see the project using SVN, I love it.

- Jeff

[t00fri]

Jeff,

how about explaining a bit more about this project of yours. Coding is one aspect but the underlying astrophysical considerations may well be the main issue here.

In view of the scientific standards we are applying for Celestia's database, I am not convinced that your auto-update plans for stars.dat are a good idea. You can always do what you want of course within the framework of add-ons that are disjunct from the main Celestia distribution.

What we really would need is an auto-update and conversion facility for satellite orbital elements from the respective, official data bases. As you surely are aware, statellite orbital elements require almost a daily update if you want to locate the satellites on Earth exactly at the right time and the right place in the sky. The physical reasons for this frequent updating necessity should be obvious, I suppose.

Measurements always have uncertainties. Larger star distances tend to have considerable uncertainties. The distance information is absolutely ESSENTIAL for a 3d visualization like Celestia. It is very easy to estimate what the largest star distance would be for which the error on the parallax measurement would be larger than the parallax itself...The respective distance is surprisingly CLOSE.

Please, could you tell us precisely how you are planning to handle these crucial uncertainties within your auto-update plans. What do you do if you encounter contradictory star distances for the same star from several sources?

The 2 million star database is a good example of a BAD database . It involves lots of incorrect distances that have never been measured but are rough estimates based on shaky assumptions! We would never implement data of such doubtful quality into the official Celestia distribution. Another strict requirement is that of concise documentation of entries. Data that are not yet peer reviewed and published would not be good candidates for Celestia.

Finally, the stars.txt database is used as a source and cross-check for barycenter distances of my data base for binary star orbits[ (visualbins.dsc and spectbins.dsc in Celelstia's data directory). The respective Perl scripts are in src/tools/binaries. I would not want to have this messed up by updated data that do not come from VERY respected, published sources like e.g. Hipparcos or Tycho.

Fridger

[ajtribick]

I strongly suggest that any new utilities should operate directly on stars.dat and not on stars.txt, which is a holdover from the previous star database generator.

[t00fri]

I strongly suggest that any new utilities should operate directly on stars.dat and not on stars.txt, which is a holdover from the previous star database generator.

Fine, but how do you inspect stars.dat visually, e.g. to apply your "common sense" to individual star data or to quickly cross-check an entry vs Simbad ? .

Fridger

[ajtribick]

Fine, but how do you inspect stars.dat visually, e.g. to apply your "common sense" to individual star data or to quickly cross-check an entry vs Simbad ? .

Even if people feel stars.txt is necessary for doing these visual inspections, I am still unconvinced about the necessity to actually distribute the file itself: it is a very large download which serves only to provide a representation of an existing file in a more inefficient format. To save on bandwidth, it would be better to include a program or script to convert stars.dat (or a subset thereof) to whatever representation is thought desirable (e.g. the current stars.txt format, or .stc file format). Such a script could easily be written in Perl.

[chris]

Fine, but how do you inspect stars.dat visually, e.g. to apply your "common sense" to individual star data or to quickly cross-check an entry vs Simbad ? .

Even if people feel stars.txt is necessary for doing these visual inspections, I am still unconvinced about the necessity to actually distribute the file itself: it is a very large download which serves only to provide a representation of an existing file in a more inefficient format. To save on bandwidth, it would be better to include a program or script to convert stars.dat (or a subset thereof) to whatever representation is thought desirable (e.g. the current stars.txt format, or .stc file format). Such a script could easily be written in Perl.

Exactly my feeling. Providing such a script would eliminate any remaining reasons for distributing stars.txt or keeping it in SVN.

--Chris

[t00fri]

Fine, but how do you inspect stars.dat visually, e.g. to apply your "common sense" to individual star data or to quickly cross-check an entry vs Simbad ? .

Even if people feel stars.txt is necessary for doing these visual inspections, I am still unconvinced about the necessity to actually distribute the file itself: it is a very large download which serves only to provide a representation of an existing file in a more inefficient format. To save on bandwidth, it would be better to include a program or script to convert stars.dat (or a subset thereof) to whatever representation is thought desirable (e.g. the current stars.txt format, or .stc file format). Such a script could easily be written in Perl.

Well, stars.text is not THAT big...stars.txt.zip is just 2.5 MB, about like dione.jpg in textures->hires or earth.png in textures -> medres.

Personally, I have no problems whatsoever to deal with the binary stars.dat in my Perl scripts. Yet I think that stars.txt serves it's purpose for many people being less versatile in quickly writing their own few-line Perl script for inspecting the binary stars.dat file! Please don't forget: that --for reasons I cannot understand,-- most Windows (end) users seem to develop an unsurmountable resistance of doing anything with Perl.

Fridger

[t00fri]

Fine, but how do you inspect stars.dat visually, e.g. to apply your "common sense" to individual star data or to quickly cross-check an entry vs Simbad ? .

Even if people feel stars.txt is necessary for doing these visual inspections, I am still unconvinced about the necessity to actually distribute the file itself: it is a very large download which serves only to provide a representation of an existing file in a more inefficient format. To save on bandwidth, it would be better to include a program or script to convert stars.dat (or a subset thereof) to whatever representation is thought desirable (e.g. the current stars.txt format, or .stc file format). Such a script could easily be written in Perl.

Exactly my feeling. Providing such a script would eliminate any remaining reasons for distributing stars.txt or keeping it in SVN.

--Chris

Given this fact that most (end) users have high respect of installing Perl and running Perl scripts, someone should write a Lua script that dumps Celestia's star.dat content in ascii format onto a file (stars.txt) from within the running Celestia.

Then we could indeed stop including stars.txt in the distribution.

For me it would of course be peanuts to switch from using stars.txt to stars.dat in the Perl generation of binary orbit dsc's.

Fridger

[jdavies]

Wow, thanks for all of the great feedback! I think the most important question I need to answer has to do with what I intend for the "addon" to do. First let me clarify that I have no background in Astronomy at all. I have a strong interest in the night sky and I downloaded Celestia in an effort to try and identify "nearby" stars with planets to get some idea of what our stellar neighborhood looks like for the purposes of writing a sci-fi novel. I am a software engineer by trade and I have publlished a technical book (http://www.amazon.com/Definitive-Guide- ... 421&sr=1-1). The novel is a new direction for me.

With that said, please do not assume that I know anything about Astronomy. I assumed (which is always dangerous) that new stars, planets, and satellites are being discovered or placed into orbits fairly frequently. My idea for the addon was to create a simple mechanism, outside of Celestia, to enable the user to update their start information from official sources in a simple GUI driven manner. One person mentioned their amazement at Windows user's lack of willingness to use Perl. UI think that is natural. Not everyone has the time to become well versed in Perl or any other scripting/programming language. Just as Celestia alllows us to see the universe in an easy to use (and beautiful) way, the update utilities for Celestia ought to do the same.

However, examining the Hipparcos catalog I see that it is fairly static. It hasn't been updated since 1999. I understand the reasons why, now that I have visited the site to download the raw data. This raises the first question that I need to answer before proceding with my project:

Is the information for new stars, planets or satellites "officially" published fairly often (i.e. several times a year)?

I understand that there may be a variety of publishers of this informaion, and the tool would have to be able to check a number of different sites to see what has changed. If this is not the case, and this star/planet/satelllite data changes very slowly (i.e. once a year or so), then the need for any automatic update utility is nonexistant.

As for stars.txt, now that I know its history I am in agreement with you that want it's information to be generated dynamically and not distributed statically. The reason I started this thread is beacuse I mistakenly assumed that stars.txt was the source file for stars.dat. Thanks to Selden pointing me in the right direction, I see now that this assumption was wrong and examining the builddbstars.pl script I can plainly see why my code was giving me different numbers. The stars.dat file stores X, Y , Z coordinate information in it where the stars.txt file stores information in RA radians, DE radians (both in degrees) and distance (in ly I assume).

I suppose a better approach would be for me to ask the Celestia community at large for what kind of addon would be more useful. The folks on this thread raised a nuber of issues that I was blissfully unaware of (mostly around the veracity of the data). I'm open to any guidance on how to handle these problems in a manner that is best suited to keeping Celestia as accurate and useful as possible.

Thanks in advance!

- Jeff

[jdavies]

What we really would need is an auto-update and conversion facility for satellite orbital elements from the respective, official data bases. As you surely are aware, statellite orbital elements require almost a daily update if you want to locate the satellites on Earth exactly at the right time and the right place in the sky. The physical reasons for this frequent updating necessity should be obvious, I suppose.

Chris,
Can you point me to these official sites? That sounds like a fairly simple project. My one caveat is that my current intention is to create the update ustillity as a stand-alone program outside of the Celestia code base. I'm writing Java for now since my C++ skills are quite rusty. I have written code in the past that extracts data from public websites, so that part should be easy enough.

- Jeff

[selden]

The "official" site for satellite and space probe orbital elements is JPL's Horizons ephemeris server. It's only as good as the data it's given, though. Too often the responsible research groups fail to provide updates to JPL.

http://ssd.jpl.nasa.gov/?horizons

The orbital elements of the International Space Station are of most interest to everyone, I think.
They change erratically due to many different factors, like dockings by the Progress supply ships
and atmospheric drag, which varies, too.

Current ISS orbital elements, in TLE (Two Line Element) format, are available at
http://celestrak.com/NORAD/elements/stations.txt

http://celestrak.com/NORAD/elements/
has links to others.

An Excel spreadsheet which converts most TLEs into Celestia's SSC format is available at
http://www.lepp.cornell.edu/~seb/celest ... ets.html#3
Hopefully it'd be relatively easy to convert its algorithms.

As Fridger pointed out, you might want to consider learning Lua.
http://www.lua.org/docs.html

That would make it possible for your application to be a script run directly by Celestia.
http://en.wikibooks.org/wiki/Celestia/Celx_Scripting

[ajtribick]

However, examining the Hipparcos catalog I see that it is fairly static. It hasn't been updated since 1999. I understand the reasons why, now that I have visited the site to download the raw data.

Note that the latest versions of stars.dat/stars.txt are based on the new reduction of the Hipparcos data published in 2007 (CDS I/311), though the spectral types are taken from the 1997 version (CDS I/239) as the new version does not include this information.

[t00fri]

What we really would need is an auto-update and conversion facility for satellite orbital elements from the respective, official data bases. As you surely are aware, statellite orbital elements require almost a daily update if you want to locate the satellites on Earth exactly at the right time and the right place in the sky. The physical reasons for this frequent updating necessity should be obvious, I suppose.

Chris,
Can you point me to these official sites? That sounds like a fairly simple project. My one caveat is that my current intention is to create the update ustillity as a stand-alone program outside of the Celestia code base. I'm writing Java for now since my C++ skills are quite rusty. I have written code in the past that extracts data from public websites, so that part should be easy enough.

- Jeff

Now I am really curious: Why do you ask Chris rather than me to point you to the sites in question, after I made that proposal.? Being a Professor in Theoretical Physics and a longstanding co-author of Celestia should be about enough for this "challenging" task... ahem

Anyway, as I see, Selden did an excellent job with the references...


Fridger

[jdavies]

Excellent! Thanks guys.

Thinking a bit more about making the information "readable" to humans, I can envision an application that can read from the Celestia data files (.dat, .ssc, etc) along with the other official sources on the web. The app should also be able to write information out to the Celestia application files (the .dat, .ssc, etc.) along with MS Excel, CSV and XML at the very least, along with a built-in data browser.

I have some concerns about writing this as a script. I have used LUA in the past and I'll see if I can get a CELX script to spawn a local program. If I can, the rest is easy.

- Jeff

[jdavies]

Quick question, the buildstardb.pl script needs two input files. The first is the HIP file which I assume is the one I can FTP from ftp://cdsarc.u-strasbg.fr/cats/I/239 using the file name hip_main.dat.gz (and unzipping it to get to the data file, of course). However, which file is the HIP2 file? The devguide.txt file that comes with celestia says:

* buildstardb.cpp
The program used to munge the HIPPARCOS data set into the star database.
If you wish to munge your own database, you will need to download a copy
of the HIPPARCOS database, it may be found at the URL:
ftp://cdsarc.u-strasbg.fr/cats/I/239
You will need the files hip_main.dat h_dm_com.dat hip_dm_o.da & tyc_main.dat

Which of those other files is the HIP2 file I need to run the buildstardb.pl script successfully? Also, what parameters do you pqass into the buildstarsdb.pl script when you generate the starsdb.dat file for the official distribution? Thanks in advance!

DOH! PLEASE IGNORE THIS

I found the file. Celestia is probably the best documented open source project I've come across. I found the hip2.dat file, properly documented in the buildstarsdb.pl script.