Loading huge ssc files

[Malenfant]

Just out of curiosity, I tried loading the huge mainbeltasteroids.ssc file at the motherlode, that contains all 230,000 main belt asteroids.

http://celestiamotherlode.net/creators/ ... ds.ssc.zip

First, without any extras loading at all, Celestia takes up 52.3MB of memory according to Task Manager.

Naturally my computer completely failed to load up the whole file in Celestia 1.4.1. But I noticed when I gave up after about 10-15 minutes of letting the CPU chug away that Task Manager said it was still loading up and taking up 150MB of memory and counting... (so how does the memory allocation work? It seems that it doesn't just allocate 94MB to loading the file to memory, but it didn't seem to want to stop allocating memory... should I have just left it for an hour or so?!)

So I tried to cull the file by removing all the asteroids labeled J-something or K-something, so there was now just the asteroids numbered from 00001 to 99999. That dropped it down to 39.9MB, but that didn't load up either (I gave up when it hit 100MB of memory and still rising). Cut it down to 49999 (about 18MB of ssc file), still it's going on and on up to and beyond 100MB of memory after several minutes of chugging away.

I cut it down to 10,000 asteroids (a mere 3.8MB ssc file) and finally it could finish loading up. But looking at how it loaded into memory I noticed that it loaded the initial default sscs very quickly (in a couple of seconds) then spent about 30 seconds stuck on the abridged mainbelt.ssc file, then a couple of seconds loading everything else up, making the screen and creating the graphics and finally opening the program. It ended up taking up 100MB of memory, total loading time of about 40 seconds.

Seems to me that it just slows down horribly when reading large ssc files - even 4MB (which isn't all that big a file really) grinds the loadup process to a halt for a bit. And even loading up the 10k asteroid file, Celestia ran like a dog when I told it to view asteroid orbits.

I guess this is unavoidable without rewriting Celestia's code, but I just thought it was interesting to see where the bottlenecks are. My computer isn't horrifically slow either, it's an Athlon XP 2100+ with 1GB of RAM and a 256MB NVidia Geforce FX5500 card running XP Pro.

There's no real way to trim the ssc file either, short of massively culling the number of asteroids - they've each got pretty much the barest minimum of information to allow them to be shown in Celestia.

[cartrite]

Malenfant wrote:

Naturally my computer completely failed to load up the whole file in Celestia 1.4.1. But I noticed when I gave up after about 10-15 minutes of letting the CPU chug away that Task Manager said it was still loading up and taking up 150MB of memory and counting... (so how does the memory allocation work? It seems that it doesn't just allocate 94MB to loading the file to memory, but it didn't seem to want to stop allocating memory... should I have just left it for an hour or so?!

When you say you gave up, you mean that you killed the process and the splash screen dissappeared? I just tried this with KDE and Gnome. I didn't let it go that long but tried to kill the process on my first try. I used KDE system guard and hit the kill button but Celestia kept loading. I logged out to finnaly kill it. Then I tried with KDE and the same thing happened. But this time I remembered that a popup comes up to ask for conformation. It's default answer is yes so I hit enter. Problem is I was never able to see it. It was covered by the splash screen.
I think windows also asks for conformation. That file is too big and I don't want to full with it any more. Good Luck.

cartrite

[steffens]

The problem here is probably not the filesize - the file could be read in small blocks - but every asteroid that was read from the file has to be stored in some kind of data structure.
I don't have the Celestia sourcecode here right now to have a look at what information is stored for each asteroid, but I guess that it is more than what is written in the ssc file.

steffens

[Malenfant]

When you say you gave up, you mean that you killed the process and the splash screen dissappeared?

Yep. I don't know of any popup window that comes up for confirmation (of what? that you're trying to kill it?) - maybe that's something linux does differently to Windows. I'll have another crack at it tonight though to check.

[Rassilon]

I had the same problem when I was creating the star cluster generator... Break it up into smaller files and it should load up fine...

[Malenfant]

I had the same problem when I was creating the star cluster generator... Break it up into smaller files and it should load up fine...

So I could theoretically split the 94MB file into thirty or so 3 MB files and it'd be able to load them up (it'd explode when I turned on asteroid orbits though ). At least I guess I'd be able to see at what stage of the process it's at when it's loading them. Or would it still take forever to load it?

I guess a binary format for the asteroid belt file would load quicker too?

I'm just curious as to what the entire asteroid belt would look like

[cartrite]

I had a quick look at the top of this file in a text editor last night and I agree that it would probally work better as many small files. You got to remember that each of these 230,000 asteroids has a model and texture. In the code I think the module observer has something to do with what files are loaded and when. When you approach an asteroid (if they are all in the same file), Celestia may try to load all 230,000 models and 230000 textures.

Im not sure how these were sorted? Size could have been one sort option. If size was used then there may be a large number of asteroids loaded even if they are in different regions of the asteroid belt. The file would probally work best if the asteroids were sorted by location and renumbered. Then split into smaller files? Just a guess? Of course all bets are off if you try to activate an orbit for every one. Probally bring your computer to it's knees.

Edit:On the other hand, I think there is a distance factor used when loading the models and textures. So it would probally work best by sorting in a way that seperates them somehow. Mabey size would be a good sort option.


cartrite

[Rassilon]

Put 1000 asteroids in 230 ssc files and you should be ok...

[Malenfant]

Put 1000 asteroids in 230 ssc files and you should be ok...

Might be just a little timeconsuming, that . Unless maybe I can find a filesplitting program...

[Rassilon]

Yeah or write a parser in C... Or wait for someone else to do it.... I think I could implement something like this in the editor I am building... but that doesnt solve your problem at this time...

[t00fri]

It takes only about 5-10 PERL statements to generate a MUCH smaller file containing only asteroids above a certain desired Radius, for example.

It would have been much better to leave the albedo as it was in the published catalog. One could then also use the albedo as a selection criterion and boost the values LATER for better visibility.

The present loading is of course very slow since an efficient octree type culling mechanism is (still) missing for asteroids.

Bye Fridger

[Rassilon]

It takes about 5-10 PERL statements to generate a MUCH smaller file containing only asteroids above a certain desired Radius, for example.

It would have been much better to leave the albedo as it was in the published catalog. One could then also use the albedo as a selection criterion and boost the values LATER for better visibility.

Bye Fridger

What he said...

Generate maybe only a few thousand asteroids with that builder in addons section and hike up the radius and albedo... Also you could overlay a semi-transparent ring model around the sun using dsc code to give it the appearence of an asteroid belt along with the few thousand models I mentioned before...

[Johaen]

... should I have just left it for an hour or so?!

Just to test this, during lunch I DLed the file, put it in Celestia and let it attempt to start up while I went back to work for the rest of the day. I don't know how long it took, but it did eventually load up. Celestia was using 500k or so memory, and was running at .4~.5 FPS. I normally get 15-20 FPS while traveling in the solor system.

Just thought I'd post these results.

[Paolo]

.... I think I could implement something like this in the editor I am building...

Hello Rassilon

Perhaps I've missed something in the past.
Can you please explain this a little bit more?

Kind regards

[Rassilon]

.... I think I could implement something like this in the editor I am building...

Hello Rassilon

Perhaps I've missed something in the past.
Can you please explain this a little bit more?

Kind regards

I will probably add a way to create an asteroid belt randomly similar to the cluster generator... I will also try to mirror what the cluster generator does and have it create star clusters and even galaxy shaped star clusters... if there will ever be a time we can add stars to galaxies in Celestia it will be there...

[Hunter Parasite]

I avoid downloading large files like the asteroid one you're talking about or the 1 or 2 million star expanision. Those, if you have a slow computer, can 'overencumber', or even freeze your computer because of so much data you're loading. It's like force-feeding your dog.