MySQL star database add-on.

[Evil Dr Ganymede]

OK, I had the MySQL Celestia thingie working fine with the default star database. Just now, I tried loading Pascal's big 50MB star.dat file into it and getting it to generate the SQL database for it.

After waiting for about 10 minutes, the DOS screen started to print up lots of error messages. I couldn't cut and paste them unfortunately, but basically it was going through all the columns of the tables (HD, SPECTRAL TYPE, etc) and saying that there was an error loading them up. Then when I fired up the MH Celestia, I found that the program started up, but with only some stars (not sure where it's getting those from, since I renamed the old stars.dat file, and it's not loading up Pascal's big file).

It appears to have died when it was compiling the database. I do currently have a 15,291 KB star.MYD, a 50,721 KB star.MYI file, and a 10 KB star.frm file in the Celestia/dbdata/mh folder.
I'm running this on Windows 2000 (SP2) with 512 MB RAM and 4GB free space on C drive, BTW.

Any clue what's going on? If that's not enough info, let me know and I'll see if I can dig up some more.

[marc]

Show me what is in your *.err file that is in your dbdata directory. Your not running out of disk space?
How much ram do you have?

[marc]

I think Ive found the problem. There is a memory leak with the generation process. I didnt notice this as ive got 512M of ram.
Shit, I should have checked the memory usage more thoroughly. Sorry guys.
Ill try and fix this asap.
In the mean time, if people have less than 256M of ram stick to the standard stars.dat that should work fine.

Im still interested in the above questions Dr Ganymede.

[Evil Dr Ganymede]

Sorry, I added the RAM to my post later on - I've got 512MB, same as you.

The only .err file I have in the dbdata directory has a filesize of 0 KB - there's nothing in it.

But wait. I'm trying it again, reinstalling the MH stuff. Now it seems to be working. It's taking about 10-15 minutes to do each index, but it's working. Maybe I had too many apps open when I tried it last time....
Anyway, seems OK now.

[marc]

False alarm about the memory leak, there isnt one. I was looking at the inMemory usage rather than the total memory+pagefile usage.
Im guessing that when the MySQL server starts up the key buffer is paged out, then as rows are inserted more of the key buffer is need so it is paged in. This makes sense as the MySQL docs mention that the server runs best with no pagefile.
In the process for looking for this phantom bug I made some more changes and managed to halve the memory usage of the generation process. This has a big effect when using the big stars.dat.
Ive also managed to make it a little faster, and generation errors are now logged to a file.

As for the above problems that people had, all I can say is [b]do not run anything else during the installation and generation process[/b.]

[Evil Dr Ganymede]

Well, it successfully compiled the database eventually. But be warned, it takes a long time - it took 4 hours on my Athlon XP 2100+ with 512 MB RAM! The database itself is pretty huge - about 230 MB!

[marc]

Well, it successfully compiled the database eventually. But be warned, it takes a long time - it took 4 hours on my Athlon XP 2100+ with 512 MB RAM! The database itself is pretty huge - about 230 MB!

That performance is really surprising, in fact its quite shit-house (excuse me) . Pascal's 2.1C stars.dat (50M) takes half an hour on my Duron 1GHz with 512M and about 5 minutes on the 2.4GHz P4 with 256M that I have at work.

The size of the database is correct, its that big because:
a, there is a lot more derived information in the database than in stars.dat
b, each column of data is fully indexed (which makes the queries really fast)

ps. in some prevoius posts ive quoted my processor as an Athlon I was a little mixed up it is a Duron.

[marc]

I've been having a play with the new version 0.1.4. Here is a query that modifies the star database:
UPDATE star SET RA = 24*RAND(), declination= 180*RAND()-90, distFromSol= RAND()*5000;
LOAD STARS;

and the resulting pattern of stars. Each links to a larger image (~300k)


http://mostlyharmless.sourceforge.net/screenshots/stardb_update1.jpg


http://mostlyharmless.sourceforge.net/screenshots/stardb_update2.jpg


http://mostlyharmless.sourceforge.net/screenshots/stardb_update3.jpg

I was attempting to randomly distribute the stars to new locations. Its a perfect example to show that the MySQL RAND() function isn't really random.

[Evil Dr Ganymede]

Looks darn purty though

[scienceman]

Hi,

This looks really useful, but all of our MySQL databases are either Linux or Mac OSX - based. How would we go about creating makefiles or configure scripts that would build this, instead of the Windows batch stuff?

[marc]

I have not yet found the time to get celestia (and mh) compiling on on my linux machine, one i do ill let everyone know Linking to the mysql embedded library might be the hardest bit. The two versions of the modified celestia need to be compiled, one for the generation and one that does the fancy stuff. (each has different #defines that are specified in mh_switches.h)
The database server and client is embedded into the modified Celestia. This is done for speed, it will also stops it conflicting with any other existing mysql servers running on the target machine.

[MostlyImpressive]

I have WinXP with Celestia 1.3.0. It works fine!

Just to let you know: I had to delete my old mostlyharmless.exe (in Celestia folder) before installing (otherwise it keeps the old one and it doesnt work).

Thanks!

[MackTuesday]

Is the SQL addon supposed to be able to use the .ssc.gz files created by the System Generator? Because I can't get it to work. I definitely installed into the Celestia root directory and eveything seems to be working fine as far as creating the star system files: I have a Celestia/random/part1 directory filled with 1000 .ssc.gz files, even the auto-generated .html listing habitable systems, it's just that MH doesn't pick them up. Do I need to be using a pre-SQL-addon version of MH?

[marc]

Is the SQL addon supposed to be able to use the .ssc.gz files created by the System Generator? Because I can't get it to work. I definitely installed into the Celestia root directory and eveything seems to be working fine as far as creating the star system files: I have a Celestia/random/part1 directory filled with 1000 .ssc.gz files, even the auto-generated .html listing habitable systems, it's just that MH doesn't pick them up. Do I need to be using a pre-SQL-addon version of MH?

Sorry Mack, the system generator does not work with mh versions 1.0-1.3. (ie the SQL stuff). I think that the version of MH that uses the ssc.gz files was 0.0.5 (there is a readme somewhere that says what goes with what)
Also have a look at this thread.
http://www.celestiaproject.net/forum/viewtopic ... +generator

I know its confusing but it is still all very experimental.
I am re-working the system generation to take advantage of the database programming that I've learned, no more of those stupid ssc.gz files.

Hope this helps.

And thanks for your comments on win98 in the other forum. I should have those issues fixed when i get around to the next release.

[marc]

Some more screenshots for everyones viewing pleasure. (and an excuse to go to the top of the list again)

[Thierry_d]

Could you tell me why MostlyHarmless tries to connect to internet when i run it ?!

[marc]

I can assure you this is unintentional, there is no spyware involved. Its the mySQL dll. Someone else noticed this before, Either the mySQL server is doing a DNS request when it starts up (to localhost). Or it could be that the client and server are communicating via sockets. (which it shouldnt but it might on win98).
Ive disabled the networking and DNS lookup in the next version.

What OS are you using?

[Thierry_d]

My OS: WinXp

Ok let's trust you

BTW is there any sql query which would mark all stars with planetary companions discovered? It could be fun to see where they are...

[marc]

My OS: WinXp

Ok let's trust you

BTW is there any sql query which would mark all stars with planetary companions discovered? It could be fun to see where they are...

Not at the present time, planetary data is not part of the database yet.

[Felix]

Hum..
Does this still work in Celestia 1.3.1? I found it extremely useful and handy when i used it some time ago.. and why is this addon not present within Celestia Motherlode? It belongs there, it should be there... the only way i know to find out a specific group of stars with specific properties out of that 2 million stars .dat file

edit: the first part of my posting just became obsolete, as i just tried it. it works