Handling of duplicate textures AND MODELS in graphics memory

[Chuft-Captain]

On the subject of texture handling in Celestia, does anyone who is familiar with the code know if Celestia does any load optimization when the same texture file is used on multiple meshes (via the SSC "Texture" command.)

eg,

Code: Select all

Object1 "Sol/....etc"
{
Texture "file1.jpg"
}

Object2 "Sol/....etc"
{
Texture "file2.jpg"
}

Object3 "Sol/....etc"
{
Texture "file1.jpg"
}


Note that Object1 and Object3 use the same texture file, so, to state it very simplistically, I would expect that if Object1 was visited first, then file1.jpg would be loaded into graphics memory at that time, then if Object3 later came into the FOV, then the texture file1.jpg would not have to be re-loaded from disk as it has already been instantiated in memory as a result of displaying object1.

My experience in this scenario suggests however that the file is re-instantiated or reloaded for each object (even though they both reference the same file on disk), as I experience what appears to be a delay as the texture is reloaded, as each successive object comes into view.

Can anyone enlighten me as to how this is handled by Celestia? My experience does seem to suggest no optimization is happening in this situation.

Cheers
CC

[selden]

I just spent some time looking through the code. Texture loading is implemented in celengine/texture.cpp

As best I can tell, there is no optimization of texture storage. I didn't find any code testing if a particular file has already been loaded.

[Chuft-Captain]

I just spent some time looking through the code. Texture loading is implemented in celengine/texture.cpp

As best I can tell, there is no optimization of texture storage. I didn't find any code testing if a particular file has already been loaded.

Thanks for confirming that Selden.

I wonder if the same is true of VT's... I assume the same code is used to load them as well, in which case when a low orbit satellite does successive passes over the same region, all tiles in the FOV presumedly will be reloaded from disk, even though they are likely already in either VRAM or RAM.
This does seem a little wasteful when you consider the orders of magnitude difference in access time to VRAM / RAM compared with disk access speed (even with fast SSD's).

Maybe a simple hash table (at the cost of a few bytes per texture) could keep track of which files have been loaded into RAM (and the loadtime), which would also enable some sort of FIFO or LIFO memory management strategy, if desired.

Anyway, that's my suggestion for today.

Regards
CC

[selden]

I just asked Chris about this. He assures me that textures are supposed to be loaded only once. Apparently I overlooked the optimization code. I'll investigate further.

[Chuft-Captain]

Thanks Selden,

I would trust Chris' word on this, so no need for you to investigate further on my behalf, unless you want to satisfy your own curiosity.
I think that the 'chunky' performance which I sometimes see can be put down to just the overall poor performance of my graphics (ie. Whenever a new object with a large number polygons comes into the FOV, it's taking a significant "human length" of time to load and draw that object).
Not surprising really when you consider that I get less than 5 fps, in scenarios where people with 'real' graphics systems are getting > 60. I'm hoping this situation will be resolved by a new system,...hence the benchmarking thread to try and predict if the system I plan to build will be "good enough".

Cheers
CC

[Fenerit]

Maybe the slow-down is due to the time the GPU spend "to feel" that there is a compression to be processed but then the GPU does use the same, first loaded texture...

[selden]

You can see when models and textures are loaded from disk by turning on Celestia's on-screen log file: type a tilde (~) [which is the same as (shift-`) on US keyboards]. On keyboards which support diacritical marks (also known as "dead keys"), you may need to type a tilde followed by a space.

Loading models and textures into the graphics hardware can take a noticeable amount of time, too. While loading a CMOD model is always faster than the corresponding 3DS model, I don't know if the same is true for DDS compared to JPEG or PNG once they've been converted.

[t00fri]

I don't know if the same is true for DDS compared to JPEG or PNG once they've been converted.

I certainly do know the answer here.

If you load a single, medium sized texture like 16k as PNG it takes very long (depending on compression strength!), JPG takes significantly less but with that 16k texture converted to DXT1, say, you won't even note the slightest loading delay! The reason is that graphics cards include hardware support for DXT decompression, unlike decompression of JPEG/PNG.

Given that todays DXT quality has become so good that only special tools can tell the difference to lossless formats, I never understand why so many users still prefer using PNG / JPG, for example, in Celestia.

It is extremely easy to run a small batch job through a monster sized VT distribution, converting PNG tiles to DXT1 afterwards. My latest F-TexTools meanwhile optionally generate directly VT tiles in highest DXT quality. This includes DXT5nm normal maps.

Fridger

[selden]

Fridger,

I'm referring to the loading of the image from main memory into the graphics hardware
when the same texture is used for multiple objects, *not* the loading from disk into main memory.

[t00fri]

Fridger,

I'm referring to the loading of the image from main memory into the graphics hardware
when the same texture is used for multiple objects, *not* the loading from disk into main memory.

No matter whether the graphics card receives the texture data from disk or from main memory, the decompression takes most of the time. The decompression time in the GPU is what counts. It is done by software drivers in case of JPG/PNG and by the card's hardware for DXT.

Fridger

[PlutonianEmpire]

Given that todays DXT quality has become so good that only special tools can tell the difference to lossless formats, I never understand why so few users still prefer using PNG, for example, in Celestia.

Did you mean to say, "so many users" instead of "so few users"?

Anyways...

It is extremely easy to run a small batch job through a monster sized VT distribution, converting PNG tiles to DXT1 afterwards. My latest F-TexTools meanwhile optionally generate directly VT tiles in highest DXT quality. This includes DXT5nm normal maps.

The vast majority of people, myself included, absolutely despise command line interfaces. We simply will not use an application if it does not have a point-and-click and GUI interface, no matter how "easy" it supposedly is. Plain and simple.

[Chuft-Captain]

If you load a single, medium sized texture like 16k as PNG it takes very long (depending on compression strength!), JPG takes significantly less but with that 16k texture converted to DXT1, say, you won't even note the slightest loading delay! The reason is that graphics cards include hardware support for DXT decompression, unlike decompression of JPEG/PNG.

While developing an addon, my preference is to stick with PNG's and JPG's, as they are easily recognized formats which can be manipulated in most of the common image-editing software, however your comments seem to suggest that before publication, I should convert them all to DXT1, for better performance.

It is extremely easy to run a small batch job through a monster sized VT distribution, converting PNG tiles to DXT1 afterwards.

What are the names of the command line tools which will easily do this?
Is the DXT1 format one of those automatically recognized by the <file>.* syntax in an SSC, and, if so, does it's suffix take priority over the others?

Are there any situations in Celestia (or in specific graphics cards) which are incompatible with DXT1 formats? ie. can they be used just as easily on spacecraft meshes as on planetary meshes? ... or are there some limitations I should be aware of?

Loading models and textures into the graphics hardware can take a noticeable amount of time, too. While loading a CMOD model is always faster than the corresponding 3DS model, I don't know if the same is true for DDS compared to JPEG or PNG once they've been converted.

This begs the question I guess, of whether the same intelligent optimizations are done regarding the loading of duplicated models, as is done for textures. I do hope so.

[t00fri]

Given that todays DXT quality has become so good that only special tools can tell the difference to lossless formats, I never understand why so few users still prefer using PNG, for example, in Celestia.

Did you mean to say, "so many users" instead of "so few users"?

Oh yes, that turned "upside down" unfortunately. Of course, In my view more people should use high-quality DXT compressed textures and DXT5nm compressed normalmaps.

Anyways...

It is extremely easy to run a small batch job through a monster sized VT distribution, converting PNG tiles to DXT1 afterwards. My latest F-TexTools meanwhile optionally generate directly VT tiles in highest DXT quality. This includes DXT5nm normal maps.

The vast majority of people, myself included, absolutely despise command line interfaces. We simply will not use an application if it does not have a point-and-click and GUI interface, no matter how "easy" it supposedly is. Plain and simple.

Well, that's of course your decision, which is very hard for me to understand. Meaning that people refuse learning a few simple things about command line use at the expense of depriving themselves of plenty of benefits...

The vast majority of people, myself included, absolutely despise command line interfaces.

Well, simply because the vast majority of people has at best a loose idea about the real power of computing . I can assure you that the vast majority of computer experts uses command line programs whenever there is a good reason to use them. So, how would one convert an ascii-CMOD model into a binary CMOD model for Celestia? Of course with Chris Laurel's respective command line program . All Celestia stars, binaries, globulars, galaxies have been processed with command line programs (Perl scripts), etc.

Fridger

[PlutonianEmpire]

Yeah, I probably should have spoken for myself instead of the "vast majority of people"...

[selden]

If you load a single, medium sized texture like 16k as PNG it takes very long (depending on compression strength!), JPG takes significantly less but with that 16k texture converted to DXT1, say, you won't even note the slightest loading delay! The reason is that graphics cards include hardware support for DXT decompression, unlike decompression of JPEG/PNG.

While developing an addon, my preference is to stick with PNG's and JPG's, as they are easily recognized formats which can be manipulated in most of the common image-editing software, however your comments seem to suggest that before publication, I should convert them all to DXT1, for better performance.

Paint.NET can save images in DXT1, DXT3 and DXT5 formats, with or without mipmaps. Use DXT5 for images which have alpha channels.

It is extremely easy to run a small batch job through a monster sized VT distribution, converting PNG tiles to DXT1 afterwards.

What are the names of the command line tools which will easily do this?
Is the DXT1 format one of those automatically recognized by the <file>.* syntax in an SSC, and, if so, does it's suffix take priority over the others?

It's recognized, but not at the highest priority. Celestia's preference order for the * filetype is png, jpg, dds, dxt5nm, ctx. In other words, you'll have to remove the corresponding png and jpg files from the texture directory.

Are there any situations in Celestia (or in specific graphics cards) which are incompatible with DXT1 formats? ie. can they be used just as easily on spacecraft meshes as on planetary meshes? ... or are there some limitations I should be aware of?

DDS formats are not recognized by the oldest of graphics cards, like those by 3dfx. I doubt there are any of those still being used.

Loading models and textures into the graphics hardware can take a noticeable amount of time, too. While loading a CMOD model is always faster than the corresponding 3DS model, I don't know if the same is true for DDS compared to JPEG or PNG once they've been converted.

This begs the question I guess, of whether the same intelligent optimizations are done regarding the loading of duplicated models, as is done for textures. I do hope so.

Apparently so. Enable the console log to see when they're loaded from disk.

[Fenerit]

While developing an addon, my preference is to stick with PNG's and JPG's, as they are easily recognized formats which can be manipulated in most of the common image-editing software, however your comments seem to suggest that before publication, I should convert them all to DXT1, for better performance.

Paint.NET can save images in DXT1, DXT3 and DXT5 formats, with or without mipmaps. Use DXT5 for images which have alpha channels.

...and GIMP too; through the DDS plug-in.

[t00fri]

...
Paint.NET can save images in DXT1, DXT3 and DXT5 formats, with or without mipmaps. Use DXT5 for images which have alpha channels.
...

...and GIMP too; through the DDS plug-in.

=========================
Forget about that stuff altogether!
=========================

Over years, myself and others have explored the quality status of these "average quality" DXT compressors, with a fairly unsatisfactory result, notably in case of highres VT tiles.

No, I was referring to the signiificant improvements of the DXT compression algorithm that started with the so-called SQUISH library by Simon Brown (a computer scientist)!

http://code.google.com/p/libsquish/

See also Simon's 2006 Blog about his new cluster fit algorithm:
http://www.sjbrown.co.uk/2006/01/19/dxt ... echniques/

++++++++++++++++++++++++++++
Simon Brown's cluster fit algorithm in squish forms the core DXT compression algorithm for the so-called new NVIDIA Texture Tools. That's the tools to be used for Celestia!

Windows (binary):
http://developer.nvidia.com/content/tex ... 208-64-bit

For DXT compression of normalmaps, the dedicated format DXT5nm is to be used and is supported by Celestia(.Sci). Unlike Selden's claim above, DXT3 is the right format for RGBA, while DXT1 is for RGB textures.
+++++++++++++++++++++++++++++

The open source archive

http://code.google.com/p/nvidia-texture-tools/

was long maintained by Ignacio Castano (NVIDIA), who added a host of further improvements. The source code builds in all OS without problems (cmake based!).

+++++++++++++++++++++++++++++
For all DXT compression options, my F-TexTools/ Nmtools also use internally the latest NVIDIA Texture Tool library based on Squish and CUDA support!
+++++++++++++++++++++++++++++

A most important point being that with the CUDA support (<=> super fast code execution in the GPU !) of all more recent NVIDIA graphics cards, these NVIDIA command line tools aquire a large performance boost (depending on the card)!!

You sure know that like the JPG format, DXT compression is NOT a lossless format. But the output quality strongly depends on the algorithm used for that compression. I normally use PNG or TIF as a lossless, mere storage format, but mostly this high quality DXT format for fast and "butter smooth" graphics action in Celestia(.Sci).

@CC:

What are the names of the command line tools which will easily do this?

The NVIDIA texture tools together with a trivial batch script allow you to convert on the fly "monster VT sets (>= 64k)" in PNG format to the appropriate DXT format. If you buy yourself a powerful new card from NVIDIA, these tools will be VERY fast (CUDA!!) despite highest quality.


@PlutonianEmpire: all this beauty and elegance is not for you and other haters of command line tools

Fridger

PS: Note that for DXT conversion of VT tile sets, you only activate mipmaps for the first, level0 set. For all higher levels you do NOT need mipmaps, since the different tile levels already replace the effect of mimpaps. This saves some storage space.

PPS: For a long time I have used a pretty old NVIDIA FX5900 Ultra/256MB without any problems in DXT handling.

[selden]

For DXT compression of normalmaps, the dedicated format DXT5nm is to be used and is supported by Celestia(.Sci). Unlike Selden's claim above, DXT3 is the right format for RGBA, while DXT1 is for RGB textures.

I've come across posts claiming that some game engines have dropped support of DXT3, requiring the use of only DXT5. DXT3 uses specific values for Alpha values, which can result in "banding" in transparency, while DXT5 uses a table lookup, which can result in smoother gradations in transparency. Which is appropriate depends on the usage. DXT3 could result in sharper edges between opaque and transparent regions than DXT5 does, for example, which would be desirable for planetary specular maps. DXT5 seems more appropriate for clouds.

One place this is discussed is http://www.fsdeveloper.com/wiki/index.p ... _explained

Another is in the discussion of Addons for X-Plane at http://wiki.x-plane.com/DDSTool_manual# ... q_alpha.29

[t00fri]

For DXT compression of normalmaps, the dedicated format DXT5nm is to be used and is supported by Celestia(.Sci). Unlike Selden's claim above, DXT3 is the right format for RGBA, while DXT1 is for RGB textures.

I've come across posts claiming that some game engines have dropped support of DXT3, requiring the use of only DXT5. DXT3 uses specific values for Alpha values, which can result in "banding" in transparency, while DXT5 uses a table lookup, which can result in smoother gradations in transparency. Which is appropriate depends on the usage. DXT3 could result in sharper edges between opaque and transparent regions than DXT5 does, for example, which would be desirable for planetary specular maps. DXT5 seems more appropriate for clouds.

One place this is discussed is http://www.fsdeveloper.com/wiki/index.p ... _explained

Another is in the discussion of Addons for X-Plane at http://wiki.x-plane.com/DDSTool_manual# ... q_alpha.29

Selden,

I really don't need these links, since I have converted innumerable RGBA textures to DXT3 and DXT5 with my tools over many years (besides coding these tools ). The correct choice is clearly dictated by the shape of the information that is to reside in the alpha channel. See further below.

++++++++++++++++
The RGB colors are saved identically in DXT3 and DXT5. No discussion about that. There is indeed a difference in the storage of the alpha channel: In DXT3 one only has 4 bit available for alpha, while DXT5 allows for more gradual changes in alpha.
++++++++++++++++

Now lets go to typical alpha channel applications in Celestia:

A very economical and fast method is to add the specular texture as alpha channel on top of the RGB surface texture. In my F-TexTools there is even a special tool for that (tx2rgba).The specular information consists typically only of black regions (non reflective) and white regions (reflective) . For these frequent applications in Celestia a DXT3 encoding is just perfect, since extremely fine graduations are unnecessary. But DXT3 can often handle also grayscale specular textures pretty well.

I would possibly use DXT5 e.g. for making hires clouds in Celestia. For my well-known "2k clouds" that many used with pleasure over years, DXT3 was also adequate. DXT5 does the standard ST3C algorithm on the alpha pixels, which may not be appropriate for the edges between 1.0 alpha and 0.0 alpha. If you use DXT5 for patterns, the shape is not preserved inside a 4x4 pixel block. Etc. So it really depends on the problem what is better.

Fridger