Tips & Tricks for 16k Monster Textures...

[Guest]

Since quite a while, this weekend I had some spare time again. Here is
a status report of what I did....

1) I had to configure a new Laptop with Windows XP /256MB RAM and a ATI
Radeon 16MB card. Celestia works fine (with just Basic and
Multitexture render paths supported, i.e no bumping and specular
lights...). DXT Format works as well.

2) Next I assigned 4GB of /virtual/ memory on a separate partition
which resulted in the fact that all NVIDIA command line tools
(5.39) now work on /very large/ textures as well (16k no problem!).

3) I finally managed to convert the huge 21k Blue Mable earth texture
to the nice ("Render") color scheme of the SpaceGraphics earth
textures. Here is what I did (hope to upload the stuff very soon to
my TextureFoundry):

a) Rescale the 21k -> 16k with GIMP (cubic interpolation)
b) Load the 16k sea selection channel,=> channel to selection
c) Apply to the sea only:

Hue/Saturation:
-------------------------
Hue: -27
Lightness: +25
Saturation: -64

d) Invert the selection and do (land!)

Hue/Saturation:
-------------------------
Hue: -14
Lightness: +17
Saturation: -20

e) Use select by color, 25% fuzz, 'add' mode and click a few times on
the world's /green/ areas, save selection to channel.

Hue/Saturation:
-------------------------
Hue: -20
Lightness: +9
Saturation: -55

f) Unselect everything and apply

unsharp masking
---------------------------
4.5
0.55
0


Finally, /select "green" channel/ and for the selection apply


unsharp masking
---------------------------
3.5
0.55
0


Hue/Saturation:
-------------------------
Hue: 0
Lightness: +10
Saturation: 0


g) Next, save the results as *.tga file and convert
the texture to dxt1c with 'nomipmap' setting!

h) Apply the 'detach' command line utility to the 8k SG earth8k-SG.dds
which leads to 13 mipmaps in dds format. Renumber them, and add
the 16k dds file as earth16k-SG_00.dds

i) Finally, apply the 'stitch'command which assembles everything into
a earth16k-SG.dds texture with 14 mipmaps identical to the ones in
the 8k SG texture!


That's it.

I now have a most beautiful 16 earth texture that perfectly matches to
the SpaceGraphics Render-earth 8k one.


4) Moreover, I finally solved the old problem that the DXt normal maps tended
to show all sorts of /artifact patterns/: The trick is to add '-dither'
in nvdxt dxt conversion as well as '-24 dxt1c'. It should be
obvious why;-). Moreover, by far the best normal maps are obtained
with the parameter -n4, i.e. 2x2 sampling! I should note that I
always use the 16bit grayscale bumpmap as alpha channel, which has
to be inverted! So here are typical command line parameters (for earth):

nvdxt -file <bump-texture.tga> -alpha -n4 -dither -wrap -scale 10.0
-24 dxt1c


I generated such new "perfect" normal maps for earth, mars and moon
in 2k,4k,8k and partly 16k resolutions.

Bye Fridger

[t00fri]

too bad, I forgot to log in!

Bye Fridger

[Rassilon]

Looks like I have some tinkering to do

However the dither option does it reduce the pixelation on lower resolution dds textures when converted in dxt3 or 5? Also with bumpmaps I suppose this is the case with dxt1c...

[bh]

Blimey... so much to learn...so little time.

[t00fri]

Looks like I have some tinkering to do :)

However the dither option does it reduce the pixelation on lower resolution dds textures when converted in dxt3 or 5? Also with bumpmaps I suppose this is the case with dxt1c...

I have long asked myself why the normal maps tend to be so "rough". The reason is simply that often the default (e.g. Photoshop) is only 8bit encoding/channel! The height information comes as 16bit levels to be entered into the alpha channel, so 8bit accuracy is very bad! With the -dither command 24bit levels are actually emulated in terms of the 16bit ones, i.e. the optimum of smoothness!

Then with the -24 dxt1c option, the dxt encoding is done in 24bit, too.

All artifacts are gone this way and there is no need whatsoever for dxt3 or dxt5 normal map encoding. Note: it's not the compression that created the probs but the lack of smoothness in the /height/ levels...

Bye Fridger

[Rassilon]

So by that I am better off using the gimp vs photoshop...I am supposing the gimp supports higher than 8bit...

But then using photoshop's rough 8bit textures saving into 24 or 32 bit targa should solve this problem? I am guessing no...

Now using dxt1c instead of the lower compressed dxt3 or 5 is the best option for loading speed using the dither and 24bit options...Or if one doesnt care and wants less compression use dxt3 or 5...Ok I think Im good...now to go play

[t00fri]

So by that I am better off using the gimp vs photoshop...I am supposing the gimp supports higher than 8bit...

But then using photoshop's rough 8bit textures saving into 24 or 32 bit targa should solve this problem? I am guessing no...

Now using dxt1c instead of the lower compressed dxt3 or 5 is the best option for loading speed using the dither and 24bit options...Or if one doesnt care and wants less compression use dxt3 or 5...Ok I think Im good...now to go play :mrgreen:

There are /two different/ issues:

1) filling in the 16bit level info into the alpha channel of the bump texture. Truncating this to 8bit would be a bad! In Photoshop you may set it to 16bit, too, but 8bit is the default. So people who do not know will overlook this.

2) Encoding the bump texture into a 24bit normal map in DXT format. Here also 24 bit is important to specify.


Bye Fridger

[Rassilon]

Well following your directions except for the fact I used photoshop instead of the gimp when I made this texture...I have come to the conclusion that its time to learn the gimp better...



Notice the 'cow pastures' across the landscape...Thats my normal map...It still will not get rid of the pixilation using dither...In fact it looks better when I take dither out...On the seas theres this 'dithering' going on similar to the old windows 3.1 dithering used when it itself was only an 8 bit architecture...Spots appear on the oceans using dither...And I assume its photoshop causing this?

Oh well...

[Rassilon]

Doing the 16 bit conversion requires one to save in tiff format...Then composing as instructed in the gimp to an alpha1 tga then saving and composing dds normalmap produces still the 'cow pastures' hmmm ok well photoshop screwed this texture then. Ill have to remake the whole thing and save as 16 bit because I didnt originally...I saved as 32 bit targa yes but I still had photoshop in 8 bit color mode when I did this...

Oh I forgot to mention this is a 4k texture Im working with...

[t00fri]

Below comes a little illustration what I mean with my new 8k normal maps (applied on my new 16k texture): Check your own preferred normal map, whether you see at any kind of light some artifact patterns across the conspicuous plain in the image below? If not, lean back and be happy;-)...

And note even with my old GF2/32MB this 16k setup performs with ~15fps...


Bye Fridger

[jim]

Hi Fridger,

This post comes a little bit late but the reson is known. Now i hope the new system is save enougth to prevent such a attack. I have the same login problem as you therefore i'm not correct logged in.

Your shot looks really great but let me guess this shot was made under linux? I have the suspicion that Windows causes the problems with DDS files. Sometimes i think that Celestia shows under Windows DXT1 files only with 256 colors and DXT3/5 files with 65536 colors. In another post i descriped that the problem is not the conversion because a reconverted DXT1 to TGA has no artifacts.

16bit bump texture? It seams that you use the 21K bumpmap from Blue Mable as source.
But i have a problem with the 16bit alpla channel and the TGA format. All i know is that TGA supports only 8bit per channel. Photoshop can't save 16bit channel pictures to TGA format. Also the most filters doesn't work in Photoshop with 16bit channels. But i'm not sure if the normalmap tool really use the 16bit i don't think so.

I have tested the version 5.41 of the command line tools and they still work not with 16k textures on my system. I think the problem is the memory managment in Win98SE. :-(

Gimp? I would say it's not made for Windows. Ok i have not the latest version but this does not encourage to install a newer version (awkward to use, crashes to often, to slow, some funktions missed,... ). I would not say that Photoshop is perfect but it can do the most things what i want (it works also with 21k textures).

Finally a question have you inverted the red channel of your normal map or has Chris fixed the bug with standard normal maps?

Bye Jens

[Guest]

Jens,

Your shot looks really great but let me guess this shot was made under
linux? I have the suspicion that Windows causes
the problems with DDS files. Sometimes i think that Celestia shows
under Windows DXT1 files only with 256 colors and DXT3/5 files with
65536 colors. In another post i descriped that the problem is not the
conversion because a reconverted DXT1 to TGA has no artifacts.

The bump and main texture files were, indeed made under Linux (, since
GIMP is more powerful than Photoshop for BIG files, if one knows how
to set it up and use it;-)). I also now have a 1 GHz PIII Dell
Latitude 610 Laptop running under Windows XP, SR1, including Photoshop
7.01. I must say that I find Photoshop is really slow compared to
GIMP. Incidentally, the DXT plugin takes "for ever" to load large (>=
8k) DDS files etc. Sure, the HD's of these Laptops are almost "falling asleep"
compared to the super-fast HD in my Desktop machine. There are some
nice new "gadgets" like the "healing" tools but that's about it. Much
is conceptually /very/ similar to GIMP, so I have no problems
whatsoever, using Photoshop.

On Windows ME /and/ Windows XP, SR1, GIMP 1.2.3 is /rock stable/. If it
is not in your case, then something is fishy with your
setup. Definitely. I /never/ had a single crash of GIMP, neither in
Linux nor in Windows!

I am also sure that under Windows, DDS files are displayed in the
correct colour depth, provided, of course, you run Windows in 32bit mode!

I also have explicitly checked that my *.tga files are 16bit.

I have described in detail above, how I proceeded: I start from the 21k
16bit grayscale bump JPG of the BM earth, reduce it to 16k and assemble it as
the 16bit /inverted/ alpha channel in a RGBA setup, with RGB files being
just transparent 16bit grayscale files. The inversion of the alpha channel
just accounts for your red inversion.

It is trivial to calculate that 8 bit bump levels (i.e. 256
subdivisions between sea level and the top of mount Everest only!)
lead to /huge/ random altitude jumps! No wonder that one sees in this
case all sorts of crazy artifacts in flat terrain, like deserts or
valleys (c.f. above)...Many people also do not realize that in
Photoshop one may switch to 16bit encoding/channel, but the default is
8bit! So people who never read manuals will probably have overlooked
this;-).

Bye Fridger

[jim]

Hi Fridger,

Now i will try a newer version of gimp i hope it's as good as you have descriped. I have no problems with Photoshop 6.0 and large textures (21K). Maybe that the 16bit channel feature is not as good included as in version 7. I know how to change the mode but i can't save such a 16bit channel image to TGA (with Photoshop 6).

I am also sure that under Windows, DDS files are displayed in the
correct colour depth, provided, of course, you run Windows in 32bit mode!

I'm absolut sure that my Windows run in 32 bit mode, the screen resolution is 1024x768x32 and i said the Nvidea driver to use 32 bit textures. But i'm also sure that DDS textures use in Celestia on my system not the full color depth. DXT1c looks out only like 8bit or 10bit colors.

I start from the 21k 16bit grayscale bump JPG of the BM earth, ...

Can you tell me an exact adress for this JPG? I have only a link to files in RAW format. Ok i can use this but they are a little bit large for downloading.

It is trivial to calculate that 8 bit bump levels (i.e. 256
subdivisions between sea level and the top of mount Everest only!)
lead to /huge/ random altitude jumps! No wonder that one sees in this
case all sorts of crazy artifacts in flat terrain, like deserts or
valleys (c.f. above)...

Possible that 16bit is a the solution i will test this if i can (with the new Gimp?). But if i save the normal map to PNG instead to DDS the 8bit resolution is enougth and i get no artifacts.

The inversion of the alpha channel just accounts for your red inversion.

Yes the red channel is inverted with this trick but the green channel too and that means your normal map works incorrect at the poles. You can check this very easy with my test bump map. I hope Chris will fix this bug before 1.3 is released.

Bye Jens

[chris]

Hi Fridger,
Yes the red channel is inverted with this trick but the green channel too and that means your normal map works incorrect at the poles. You can check this very easy with my test bump map. I hope Chris will fix this bug before 1.3 is released.

I checked in the fix last night, so it will appear in 1.3.0. The only thing I didn't do was make the fix for non-vertex shader render paths. Does anyone still use the NVIDIA combiner render path without either NV or ARB vertex programs? I'm considering removing it . . .

--Chris