How about dds.gz textures?

[steffens]

Hello Celestians,

I just head an idea I would like to share with you: What do you think about support for textures in gzip-ed DDS format?
The most obvious advantage would be the reduced size of the textures on disk, as DDS textures tend to compress quite well with gzip. They would not be as small as jpg textures, but still a lot of storage space could be saved on.
Modern computers can deflate gzip compressed files easily and in some cases loading the compressed file is even faster than directly loading the uncompressed file from disk. So this might even help reducing loading times of textures!
If I remember correctly, Celestia is already using zlib, so using it for texture loading should not be a major problem.
What do you think?

Regards,
steffens

[rra]

Hi Steffens ,
what you could do for now is using the NTFS compression settings
(assuming you have a NTFS formatted disk) ,
it doesn't compress a lot, but it is fast , lossless and also
transparant to the Celestia program.
Especially normal maps or bump-maps tend to be compressed very well
even with only NTFS compression.


Ren?©

[steffens]

Sorry, I'm using Linux, so this is not an option.
Might still be usefull for Windows users.

steffens

[steffens]

I have made a small patch to test this feature and see, if it's worth a thought. Here are some results:

- Loading gzip-ed DDS textures works (single image as well as VT)
- gzip compression is quite useful: Jestr's Earth Mark II with matching NormalMap and night lights texture takes about 1,7 GB on disk. After compression of all tiles with gzip (default options) only 282 MB are used! That's certainly worth it!
- loading time of texture images is at least not worse than loading DDS files directly. It is not easy to measure loading times for VTs. A test with DonEdward's 8k Mars texture and 16k normal map resulted in about 9 seconds loading time for the DDS texture versus 5 seconds for the compressed files. This indicates that texture loading is actually faster!

Anyone interested?
steffens

[t00fri]

I have made a small patch to test this feature and see, if it's worth a thought. Here are some results:

- Loading gzip-ed DDS textures works (single image as well as VT)
- gzip compression is quite useful: Jestr's Earth Mark II with matching NormalMap and night lights texture takes about 1,7 GB on disk. After compression of all tiles with gzip (default options) only 282 MB are used! That's certainly worth it!
- loading time of texture images is at least not worse than loading DDS files directly. It is not easy to measure loading times for VTs. A test with DonEdward's 8k Mars texture and 16k normal map resulted in about 9 seconds loading time for the DDS texture versus 5 seconds for the compressed files. This indicates that texture loading is actually faster!

Anyone interested?
steffens

The results will be strongly dependent on the power of the CPU and the amount of RAM used, I suspect...

Bye Fridger

[steffens]

The results will be strongly dependent on the power of the CPU and the amount of RAM used, I suspect...

That's certainly true, that's why I was very careful saying only

loading time of texture images is at least not worse than loading DDS files directly

Still, the saved storage space will be independent from CPU and RAM and, as pointed out, quite remarkable.

steffens

[EDIT] I don't know exactly, how much the amount of RAM would affect the loading time. Deflating the file should not take a lot of additional RAM, as not the whole file is deflated at once, but the stream is deflated in small chunks. The buffer for these chunks is the only additional memory used for reading.

[t00fri]

.....
[EDIT] I don't know exactly, how much the amount of RAM would affect the loading time. Deflating the file should not take a lot of additional RAM, as not the whole file is deflated at once, but the stream is deflated in small chunks. The buffer for these chunks is the only additional memory used for reading.

This whole issue is complex it seems. On the one hand a graphics /card/ with 256MB or even 512MB will work much faster with unpacked big chunks of DDS than a 64MB or 128MB card. I tend to increasingly dislike VT's, since even with "large" 2k tiles my superfast HD produces disturbing loading delays (& noise). 0.5k tiles are a pain...modern HD's generally give faster loading of big files (mine: 60MB/sec) than with many small ones...If the tiles have to be unzipped in addition, this will presumably increase the delay for such fast HDs (smaller files load slower per unit of time).

In summary, there are a number of critical hardware factors AND software issues the product of which is not easy to evaluate /in general/.

[steffens]

The good thing is, that the user can decide!
You can test, if your texture will load faster or slower when using gzip compression. You can compress your file and see, how much space you will save. Then you decide, if for the specific texture of interest compression is good or not. The patch does not eliminate pure DDS format. Further, it does not alter the way the graphics card will receive the DDS texture. It will still be the same "unpacked big chunk".

Not everybody has got superfast huge harddisks. Maybe saving 1,4 GB of texture data only for Earth is nothing for you, but I do like it. And at least in my case it even seems to be faster - why would you not think it is worth a try?

If you don't like the idea of it at all, I can live with that - I can always apply my patch to CVS and use it just by myself. Still, maybe someone else would like to give it a try?

steffens

[t00fri]

The good thing is, that the user can decide!
You can test, if your texture will load faster or slower when using gzip compression. You can compress your file and see, how much space you will save. Then you decide, if for the specific texture of interest compression is good or not. The patch does not eliminate pure DDS format. Further, it does not alter the way the graphics card will receive the DDS texture. It will still be the same "unpacked big chunk".

Not everybody has got superfast huge harddisks. Maybe saving 1,4 GB of texture data only for Earth is nothing for you, but I do like it. And at least in my case it even seems to be faster - why would you not think it is worth a try?

If you don't like the idea of it at all, I can live with that - I can always apply my patch to CVS and use it just by myself. Still, maybe someone else would like to give it a try?

steffens

Why don't you send me the patch for a try. I can certainly then report my own impressions. Also Pat Suwalsky may be a good tester for that.

My email address should be easily localizable.

Bye Fridger

[steffens]

I sent you my patch.

steffens

[steffens]

As I haven't got a lot of feedback on this, I think I will just publish the patch here. Everyone interested can apply it and check out, how performance is affected with different kinds of textures.
The patch is minimal by intention. It just replaces the iostream / fstream calls of the DDS loading rouinte with correlating calls to gzopen, gzread and gzclose. The new image file format ".dds.gz" can be used for single textures as well as for VTs.

Unfortunately, I do not have any webspace to put this file on, but as it is rather small I will just paste the diff here. I hope that this will be useful for somebody.

steffens

Code: Select all

diff -u -r ../celestia-cvs/src/celengine/dds.cpp src/celengine/dds.cpp
--- ../celestia-cvs/src/celengine/dds.cpp   2006-03-04 14:45:32.979668344 +0100
+++ src/celengine/dds.cpp   2006-03-04 14:45:49.857102584 +0100
@@ -9,6 +9,7 @@
 
 #include <iostream>
 #include <fstream>
+#include <zlib.h>
 #include <algorithm>
 #include <celutil/debug.h>
 #include <celutil/bytes.h>
@@ -86,24 +87,32 @@
 
 Image* LoadDDSImage(const string& filename)
 {
-    ifstream in(filename.c_str(), ios::in | ios::binary);
-    if (!in.good())
+    // ifstream in(filename.c_str(), ios::in | ios::binary);
+    //
+    // gzopen will open compressed files as well as uncompressed ones
+    gzFile in = gzopen(filename.c_str(), "rb");
+   
+    // if (!in.good())
+    if (in == 0)
     {
         DPRINTF(0, "Error opening DDS texture file %s.\n", filename.c_str());
         return NULL;
     }
 
     char header[4];
-    in.read(header, sizeof header);
+    // in.read(header, sizeof header);
+    int readCount = gzread(in, header, sizeof header);
     if (header[0] != 'D' || header[1] != 'D' ||
         header[2] != 'S' || header[3] != ' ')
     {
         DPRINTF(0, "DDS texture file %s has bad header.\n", filename.c_str());
+        gzclose(in);
         return NULL;
     }
 
     DDSurfaceDesc ddsd;
-    in.read(reinterpret_cast<char*>(&ddsd), sizeof ddsd);
+    // in.read(reinterpret_cast<char*>(&ddsd), sizeof ddsd);
+    readCount = gzread(in, reinterpret_cast<char*>(&ddsd), sizeof ddsd);
     LE_TO_CPU_INT32(ddsd.size, ddsd.size);
     LE_TO_CPU_INT32(ddsd.pitch, ddsd.pitch);
     LE_TO_CPU_INT32(ddsd.width, ddsd.width);
@@ -179,6 +188,7 @@
     {
         DPRINTF(0, "Unsupported format for DDS texture file %s.\n",
                 filename.c_str());
+        gzclose(in);
         return NULL;
     }
 
@@ -189,7 +199,10 @@
         format == GL_COMPRESSED_RGBA_S3TC_DXT5_EXT)
     {
         if (!ExtensionSupported("GL_EXT_texture_compression_s3tc"))
+   {
+            gzclose(in);
             return NULL;
+   }
     }
 
     // TODO: Verify that the reported texture size matches the amount of
@@ -200,10 +213,16 @@
                            (int) ddsd.height,
                            max(ddsd.mipMapLevels, 1u));
     if (img == NULL)
+    {
+        gzclose(in);
         return NULL;
-
-    in.read(reinterpret_cast<char*>(img->getPixels()), img->getSize());
-    if (!in.eof() && !in.good())
+    }
+    // in.read(reinterpret_cast<char*>(img->getPixels()), img->getSize());
+    unsigned len = img->getSize();
+    readCount = gzread(in, reinterpret_cast<char*>(img->getPixels()), len);
+    gzclose(in);
+    // if (!in.eof() && !in.good())
+    if (readCount != len)
     {
         DPRINTF(0, "Failed reading data from DDS texture file %s.\n",
                 filename.c_str());
diff -u -r ../celestia-cvs/src/celutil/filetype.cpp src/celutil/filetype.cpp
--- ../celestia-cvs/src/celutil/filetype.cpp   2006-03-04 14:45:33.547582008 +0100
+++ src/celutil/filetype.cpp   2006-03-04 14:45:49.857102584 +0100
@@ -33,6 +33,7 @@
 static const string CelestiaScriptExt(".clx");
 static const string CelestiaScriptExt2(".celx");
 static const string CelestiaModelExt(".cmod");
+static const string GZExt(".gz");
 
 
 ContentType DetermineFileType(const string& filename)
@@ -75,6 +76,17 @@
         return Content_CelestiaScript;
     else if (compareIgnoringCase(CelestiaModelExt, ext) == 0)
         return Content_CelestiaModel;
-    else
-        return Content_Unknown;
+    else if (compareIgnoringCase(GZExt, ext) == 0)
+    {
+        // a gzip compressed file. currently only dds files
+        // are supported
+        int extPos2 = filename.rfind('.', extPos-1);
+        if (extPos2 == (int)string::npos)
+            return Content_Unknown;
+        ext = string(filename, extPos2, extPos - extPos2);
+        if (compareIgnoringCase(DDSExt, ext) == 0)
+            return Content_DDS;
+    }
+
+    return Content_Unknown;
 }

[Kolano]

I don't compile my own version, but thanks for publishing the patch.

[rra]

Steffens, I really would like to test the behaviour of your patch,
but can't compile either ,
someone care to compile this thing into a windows executable ??

Ren?©

[jnthn]

As I haven't got a lot of feedback on this, I think I will just publish the patch here.

I meant to reply sooner, and never did. Sorry... I should have voiced my support for your idea right away. This is something I've long wished for.

I just checked out the CVS tree, and compiled with your patch. I see no performance slowdown that I can tell.

As Fridger said... I also see annoying pauses during level transitions with VT's. Of course, VT is the only way to load a 64K texture, so I endure the "hitches". Anyhow, I tried a 64K DDS VT Earth texture (512x512 tiles), with a 64K PNG VT normalmap, a 32K PNG specular map, and a 16K PNG nightmap (no cloudmap). The worst "hitch" between levels for me, with that texture setup, is a four second pause upon loading level three. That's with the plain, unzipped DDS Earth VT. On the other hand, with the gzipped VT... I observed the same four second pause between levels two and three. Granted, I'm only timing by the ticks of the clock on the wall, but if there's any difference, it's certainly unnoticable to me.

The only difference I can see is the fact that the the plain DDS VT takes 1.8GB disk space and the gzipped version takes 442MB. That's a big deal to me. I appreciate the work you've put into the patch. Here's hoping the devs will accept it. I think it's a real winner.

ciao,
Jonathan

[steffens]

I see no performance slowdown that I can tell.

Hi Jonathan,
Nice to hear that the patch is working and useful for you.

steffens

[phoenix]

hmm this is a nice patch.
I wonder why it never made it into the official release?
not only because the current lack of use for the zlib ... it's there because libpng requires it but celestia just don't use it.

[steffens]

hmm this is a nice patch.

Thanks!
Yes, I still think it is quite usefull. Without compressing these monster textures I had to clean up my hard disk much more often
Up to now I had no negative experiences with loading times.

steffens

[phoenix]

I have just implemented your nice little patch into my networking-patch which works quite well.

now I have to decide wether I modify all of celestia's source to be able to read .gz files or simply auto-unzip the .gz files after download.

bye
phoenix

[Fightspit]

most recent celestia win32-cvs-build - use at your own risk Wink

Can put you at each time the date of the most recent celestia CVS in your signature

[phoenix]

editing my signature 1 or 2 times a day is too much work
I should probably just link to that directory since listing is allowed there and you can always see the date of the archive

http://www.purebytes.de/stuff/celestia/

and I cannot guarantee that this is really the latest build or even updated daily.
right now I've enough time to make daily builds but that may change in the future.