celestia memory management

[phoenix]

hi all,

is there a certain reason that celestia loads all textures into memory if needed but never unloads them until the program exits?

and is it possible to "hotfix" or just workaround the problem and give the celestia core a maximum memory-usage even if it could allocate some more?

or just "simply" unload textures if you get out of visible range... of course its not simple but i think celestia really needs a better memory-management

greets
phoenix

[phoenix]

ok as i expected noone around to answer those questions

maybe i just scan the source for a texture-unload-function, if there is one for loading there _should_ be one for unloading ... i hope...

[selden]

Only Chris Laurel knows the details of why Celestia handles memory management for models and textures the way it does. My suspicision is that it simply isn't done because that's not where Chris' interests lie. He spends his very limited amount of free time on other aspects of the program. I suspect you'll have to add the code to delete objects and all the bookeeping that's needed to keep track of what's in main memory and what's still in the graphic card's memory.

[phoenix]

Only Chris Laurel knows the details of why Celestia handles memory management for models and textures the way it does. My suspicision is that it simply isn't done because that's not where Chris' interests lie. He spends his very limited amount of free time on other aspects of the program. I suspect you'll have to add the code to delete objects and all the bookeeping that's needed to keep track of what's in main memory and what's still in the graphic card's memory.

oh dear ... well then i should finish such a patch somewhen around 2008

but i fear that this can't be done by just patching celestia rather than rewriting the whole core of texture and object-management.

[t00fri]

If you want to economize on memory consumption, you may always use 'virtual textures' (VT's). There indeed the 'central' tile and various connecting tiles are loaded /and/ unloaded according to some transparent algorithm.

This is the preferable setup for Celestia "power users" who use e.g. earth textures of 2.8 GB size...

Did you look at the rspective code? Only the small textures (e.g. of the default distribution) are meant to be loaded in unchopped form.

Bye Fridger

[phoenix]

the reason i came up with this question is that i want to build a galaxy with each star having some planets and moons and hundreds of different textures assigned to them.
so if i fly around from system to system my system-memory fills up with each new texture loaded.
each planet itself will get a 2k texture, that should be no problem even for older systems with less ram. but if you navigate 50 of those planets you end up with your ram filled.

as for VTs, i already used some very large VTs for earth and mars and they never got unloaded. they just fill up my 1 GB of ram until windows starts swapping and everything gets really slow and choppy...

so it would be nice to be able to unload single textures and objects once you get out of visible-range or leave the solar-system.

i'll need some time to locate the proper code and maybe find out how to accomplish this
but i think atm that's way beyond my c++ skills

greets
phoenix

[phoenix]

ok *bump* here... now that chris is back maybe there is a solution on its way?

[Kolano]

I believe that Celestia's memory management is on Chris' list of todo's. I think its been a little while since he had mentioned it though, so perhaps it has slid down his list.

[jnthn]

...perhaps it has slid down his list.

Here's hoping another vote would help it slide back up the list. I've been playing with the new Blue Marble NG at 64K. I can only "browse" a few places around Earth for a few minutes until memory beomes saturated, and I start swapping, as I only have 1 GB of RAM. Did I just say only 1GB of RAM? Ai! What's the world coming to???

Ahem... anyhow, something will really have to be done eventually, won't it? In another few years, maybe we'll have 128K textures. I know there are more glamourous things to work on than memory management, but I believe the current behavior is going to really become a problem as VT's get larger.

JM2C,
Jonathan

[guest jo]

@jnthn: Maybe this is also interesting for u :

http://www.celestiaproject.net/forum/viewtopic ... ed5c25ee7c

J?¶rg [...]

[jnthn]

@jnthn: Maybe this is also interesting for u :

Yes, it is. I have made a reply on that thread. Thanks for the pointer.

Jonathan

Edit: Oops. Deleted the duplicate post.

[phoenix]

a little *bump* from me.

chris, any feedback on this?

[chris]

a little *bump* from me.

chris, any feedback on this?

It's something I'd like to do . . . A basic texture management scheme shouldn't be that difficult, either since all texture loading always goes through the texture manager and texture references are through handles rather than pointers. I don't think I'll get to this for a 'while' though . . . I've simply got too many other things on my plate right now. If anyone else would like to try their hand at improving the texture manager, I'd welcome the help.

--Chris

[phoenix]

well... i tried and horribily failed

i took this as an example:

http://members.iinet.net.au/~cleathley/ ... Loader.htm

but it works quite different than the celestia texmanager.
at least it's on your list even if it takes you a while
btw, keep up the good work the new rendering looks fantastic!

[cartrite]

After playing around with the official 1.4.1 version and an older cvs version from around May of 2006, I seen a vast difference in performance compared to a recent cvs version.
See Here: http://www.celestiaproject.net/forum/viewtopic ... 9400#79400
Given all the new features on the way or 1.5.0, mabey it's time this matter get some serious thought.

I'm going to take a look to see what I can come up with but I'm pretty new to C++ programming.

cartrite

[chris]

After playing around with the official 1.4.1 version and an older cvs version from around May of 2006, I seen a vast difference in performance compared to a recent cvs version.
See Here: http://www.celestiaproject.net/forum/viewtopic ... 9400#79400
Given all the new features on the way or 1.5.0, mabey it's time this matter get some serious thought.

I'm going to take a look to see what I can come up with but I'm pretty new to C++ programming.

I think that the main difference is that the latest CVS version of Celestia ends up loading a lot more virtual texture tiles than 1.4.1. I think I mentioned in another thread that this is caused by the new depth sorting code. I am working on a fix now; it shouldn't be too difficult.

--Chris

[cartrite]

Chris wrote:

I think that the main difference is that the latest CVS version of Celestia ends up loading a lot more virtual texture tiles than 1.4.1. I think I mentioned in another thread that this is caused by the new depth sorting code. I am working on a fix now; it shouldn't be too difficult.

Yeah, I remember. Forgetting the bug with the VT's though, there are a lot of enhancements coming that require extra resources so I was thinking along the lines, every little bit helps.

cartrite

[Fightspit]

Moreover, I don't know if the current version of Celestia (1.4.1 or 1.5.0) can unload previous tiles of VT when I am doing a zoom where new tiles are loaded. I think it can increase performances in Celestia if we have this feature.

[fsgregs]

I sure wish I knew programming, because the sophistication of Celestia and the size of its textures is becoming a real problem for speed. Failure to dump textures after use is forcing a memory swap that slows down systems so much now that I believe it should become a high priority to fix. Not everyone should be expected to buy top-of-the-line Nvidia cards to run what has become the most amazing Astronomy software package in the world (in my humble opinion). An average card and an average system should be enough ... if we can make it so!

We should be doing everything we can to speed up frame rates so that some of the stunning add-ons now available and coming down the pike, do not freeze systems up solid. As I mentioned in another thread, I have a very good Anthon 64 3800+ CPU system with a GeForce 6600 GT card and 1 GB of RAM, and I cannot run the 64k VT Earth texture set. My frame rates drop to less than one as the textures load, then free up again only so long as I don't move to another location. Once I move, my system freezes again for up to 20 seconds, while another VT set loads. I am not even using a spec map or a normal map. That is just the VT texture set, and a night texture. When I add a spec map and normal map, my system freezes solid and I have to reboot the computer.

That is NUTS! If a new approach to memory management can fix this, please ... someone with the C++ programming skill ... work on it and come up with a better way. Celestia has to take its own steps to free up memory.

Thanks in advance.

Frank

[cartrite]

I used to get by with 512mb ram. That being said, that system never filled up the ram with textures even though I was running a 64k VT of earth.. It would fill about 80% and then start filling the swap drive. The swap drive was and still is 4 GB. Yeah I got frame rates that ranged between 15 and nil. But the system kept chugging along.

Fsgregs, the reason your system, and mine probally freeze up when the ram is full is because some of the background processes may be getting put into swap. or the pagefile in Windows case. With this computer I currently have, I haven't figured out a way to set a max limit for all application programs so they don't interfere with the OS processes. Most of the apps that I run behave and never fill the ram to the point of memory overfill but then again they probally don't have too.

The little bar meter in my desktop toolbar on my linux systems seem to suggest that my ram gets completely filled when running celestia before the swap drive is even used. At the point just before it fills completely, I close the program. Some of the 64k textures I created recently would use about half of my swap drive while using Image Magic or the Gimp. Those programs have settings to set the max memory usage though.

I'm wondering if there is a way to set a max memory for Celestia?


cartrite