32k Resolution, 'Virtualtex' and Dim Sam;-)

[chris]

I almost forgot . . . in the course of implementing virtual textures I made a very significant optimization that helps even with normal planet textures. When close to the surface of a planet, I've measured a performance improvement of up to 50%. So stop complaining . . .

--Chris

[selden]

Jens,

Neither ImageMagick nor NetPBM support DDS. You still have to use nvdxt or the equivalent.

NetPBM v10.6.1 Windows binaries seem to be available on SourceForge.
See http://sourceforge.net/project/showfiles.php?group_id=23617&release_id=105239

I'm still running an older version (10.5.1) that I downloaded from a site in France. Unfortunately, I can't locate that site using Google, and my notes about the site are at home, and I'm not.

[t00fri]

But using ImageMagick tools to do the editing, the virtual memory on the HD is mainly used

The NetPBM utilities have the advantage that most of them don't even use virtual memory. Whenever possible, they operate on a single scanline at a time.

We should run some comparisons between the NetPBM tools and IM utilities. Speed is an important factor in the tiling game.

The IM convert utility really knows a lot of image manipulation tasks... And most tasks can also be performed at 16bit/channel, i.e. at 48bit RGB level!

Some IM drawbacks I have noticed in the course of time (>10y;-)): Since IM is largely a one man enterprise, in almost every new version, I tend to discover some (not unessential) new bugs. Hmmm. Another thing is that I always feel (since >10y's;-)) that one could have coded all this stuff a lot faster...

I suspect that by "editing," Comoran means "visually fix local problems in the image" rather than "apply color corrections to the entire image." Such things can be painfully slow while one waits for the paint program to page various parts of the picture to and from the disk. The same slowness due to paging happens with nvdxt. This makes it rather tedious to make several iterations of change/view/change/view at high resolutions. (By "view," I mean "Study how it looks in Celestia.")

Unfortunately, tiling is not yet available for Nebula textures.

What I usually do about editing is this: I generate a size reduced version of the monster texture and use IM's display utility to interactively fix the required corrections. Then I switch to 'batch'-editing for the big texture and go out with my wife;-)

Bye Fridger

[Cormoran]

Umm....When I made reference to editing (I think I used the word 'Creating') I was referring to taking very high resolution images and chopping them into the relevant tiles. If the tools mentioned herein can do that then fine, whoopee and other positive exclamations

And Chris, The performance improvement on the textures is very noticeable, once you know its there...I gave my PC a clearout this weekend and thought that was the reason everything was quicker Either that, or I thought my brain's clockspeed was decreased

Anyways, now to create the toolbox thread

Cheers (and thanks for the performance increase, Chris )

Cormoran

[selden]

Jens,

an update on NetPBM binaries for Windows:

The NetPBM homepage at http://netpbm.sourceforge.net/ mentions several places one can get different versions of NetPBM binaries for Windows.

The ones that I've been using are those provided at the very first link: ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/porters/Humblet_Pierre_A/V1.1.

Be sure to read netpbm-10.15-cygwin-1.3-bin.README
before you download netpbm-10.15-cygwin-1.3-bin.tar.gz

It's not a site in France, though, although the person maintaining the port is named Pierre.

[DBrady]

t00fri
From what i read in your posts IM seems to take a good few hours to operate on large textures. From my own experiences with using netbpm with 32k textures it seems to take about 20-30mins(roughly) to do any operation on them, including dicing them into 2k tiles.
the only thing i've used IM for was stitching together two ~10k globe 16bit dem files. This took about an hour if i remember right. Netpbm can do this considerably faster(not 16bit unfortunately) ~5-10mins. Just thought you'd like to know.

Your pic looks good also. Did you use the colour corrections you mentioned in your post? i'll probaly make them to my own 32k texture if they are. Does the uncompressed dxt format eliminate the box artefacts from normal maps. I'm sure it must-just thought i'd ask to be sure, and whats the difference/best uncompressed format to use-there seems to be a few different ones? Can it be done with devIL?

Jim,
Could you send me a copy of your dos script? I'd just like to see its layout.


As for one or two really high resolution tiles. I tried one for my house-apart from being really difficult to find on earth it was also quite distorted from not being in a cylindrical projection and i cant think of a way to get around this. any ideas?

[t00fri]

t00fri
From what i read in your posts IM seems to take a good few hours to operate on large textures. From my own experiences with using netbpm with 32k textures it seems to take about 20-30mins(roughly) to do any operation on them, including dicing them into 2k tiles.
the only thing i've used IM for was stitching together two ~10k globe 16bit dem files. This took about an hour if i remember right. Netpbm can do this considerably faster(not 16bit unfortunately) ~5-10mins. Just thought you'd like to know.

Your pic looks good also. Did you use the colour corrections you mentioned in your post? i'll probaly make them to my own 32k texture if they are. Does the uncompressed dxt format eliminate the box artefacts from normal maps. I'm sure it must-just thought i'd ask to be sure, and whats the difference/best uncompressed format to use-there seems to be a few different ones? Can it be done with devIL?

Jim,
Could you send me a copy of your dos script? I'd just like to see its layout.


As for one or two really high resolution tiles. I tried one for my house-apart from being really difficult to find on earth it was also quite distorted from not being in a cylindrical projection and i cant think of a way to get around this. any ideas?

DBrady:

NetPBM sounds increasingly interesting and I shall definitely run tests, notably since the required operations are really simple. As I said above, I follow the IM development since many years and whenever I try one of the utilities I feel it should run faster...

What again was your setup: CPU, RAM, swap space on HD?

As to color corrections, I have applied some to my 32k and 16k textures, of course. The BM colors are intolerable and do not fit to the colors of the SpaceGraphics textures that I take as a basis for the 4k and 8k resolutions.

I roughly did for levels 2,3 relative to BM colors.

mogrify -modulate 110,65,80 *.tga

i.e. desaturate by 35%, increase brightness by 10% and shift the hue by 20% to the left. This matches the color of the sea to the SG textures. But strictly speaking, the land should be desaturated less which can only be done by means of a spec-type selection. I am experimenting in this direction. IM supports separate selection files...

In view of my oldtimer card with only 32MB of RAM, I gave up on uncompressed DXT formats. I had made experiments with the u888 uncompressed DXT in nvdxt terminology. It is supported in Celestia now.

For normal maps (and nightlights) I use 24bit DXT3 through

nvdxt -file *.tga -24 dxt3

For main textures, I use 8:1 compressed dxt1c

nvdxt -file *.tga -24 dxt1c

24 bit DXT3 is not a bad compromise: much less artifacts than DXT1c but still 4:1 compression. For my super quality Mars normal maps, DXT3 however is also not totally adequate...

Bye Fridger

[jim]

Selden, thanks for the infos.

Jim,
Could you send me a copy of your dos script? I'd just like to see its layout.

Ok but it's very primitive because of the limits of dos. I use Irfanview for image edit.

"build_vt_dds_1k.bat"

Code: Select all

@echo off
set name=earth_spec
set type=png
set out=dxt1c

if exist %name%.%type% goto start
echo error: file not found
exit

:start

:8k
if exist tmp.bmp del tmp.bmp

echo build 8k temporary file

i_view32.exe %name%.%type% /one /convert=tmp.bmp

:m2
if not exist tmp.bmp goto m2

md %name%
md %name%\level2

call crop 0 0 tx_0_0.bmp
call crop 1024 0 tx_1_0.bmp
call crop 2048 0 tx_2_0.bmp
call crop 3072 0 tx_3_0.bmp
call crop 4096 0 tx_4_0.bmp
call crop 5120 0 tx_5_0.bmp
call crop 6144 0 tx_6_0.bmp
call crop 7168 0 tx_7_0.bmp

call crop 0 1024 tx_0_1.bmp
call crop 1024 1024 tx_1_1.bmp
call crop 2048 1024 tx_2_1.bmp
call crop 3072 1024 tx_3_1.bmp
call crop 4096 1024 tx_4_1.bmp
call crop 5120 1024 tx_5_1.bmp
call crop 6144 1024 tx_6_1.bmp
call crop 7168 1024 tx_7_1.bmp

call crop 0 2048 tx_0_2.bmp
call crop 1024 2048 tx_1_2.bmp
call crop 2048 2048 tx_2_2.bmp
call crop 3072 2048 tx_3_2.bmp
call crop 4096 2048 tx_4_2.bmp
call crop 5120 2048 tx_5_2.bmp
call crop 6144 2048 tx_6_2.bmp
call crop 7168 2048 tx_7_2.bmp

call crop 0 3072 tx_0_3.bmp
call crop 1024 3072 tx_1_3.bmp
call crop 2048 3072 tx_2_3.bmp
call crop 3072 3072 tx_3_3.bmp
call crop 4096 3072 tx_4_3.bmp
call crop 5120 3072 tx_5_3.bmp
call crop 6144 3072 tx_6_3.bmp
call crop 7168 3072 tx_7_3.bmp

rem bmp2dxt  tx_*.bmp
nvdxt -file tx_*.bmp -nomipmap -fadetoalpha 0 -%out%
del tx_*.bmp
move tx_*.dds %name%\level2


:4k

if exist tmp.bmp del tmp.bmp

md %name%\level1
echo build 4k temporary file
i_view32.exe %name%.%type% /resample=(4096,2048) /convert=tmp.bmp

:m1
if not exist tmp.bmp goto m1


call crop 0 0 tx_0_0.bmp
call crop 1024 0 tx_1_0.bmp
call crop 2048 0 tx_2_0.bmp
call crop 3072 0 tx_3_0.bmp

call crop 0 1024 tx_0_1.bmp
call crop 1024 1024 tx_1_1.bmp
call crop 2048 1024 tx_2_1.bmp
call crop 3072 1024 tx_3_1.bmp

rem bmp2dxt  tx_*.bmp
nvdxt -file tx_*.bmp -nomipmap -fadetoalpha 0 -%out%
del tx_*.bmp
move tx_*.dds %name%\level1


:2k

if exist tmp.bmp del tmp.bmp

md %name%\level0
echo build 2k temporary file
i_view32.exe %name%.%type% /resample=(2048,1024) /convert=tmp.bmp

:m0
if not exist tmp.bmp goto m0

call crop 0 0 tx_0_0.bmp
call crop 1024 0 tx_1_0.bmp

rem bmp2dxt  tx_*.bmp
nvdxt -file tx_*.bmp -nomipmap -fadetoalpha 0 -%out%
del tx_*.bmp
move tx_*.dds %name%\level0

del tmp.bmp
echo script finished


and the sub script "crop.bat"

Code: Select all

@echo off

echo build %3
i_view32.exe tmp.bmp /one /silent /crop=(%1,%2,1024,1024) /convert=%3

:p1
if not exist %3 goto p1


Bye Jens

[DBrady]

Thank you t00fri-your expertise is much appreciated.

Thanks jim also-i'm not that familiar with does but it seems there are some quite useful commands. I must find a tutorial on dos sometime.

Here's my effort with a single 2k tile at level eight starting with two tiles at level0. The altitude is completely wrong and the Lat/Long vaguely right, but it is an example of what virtual textures can achieve-zooming in all the way to your house! Which how anyone can say is a bad thing, especially when celestia remains backward compatable with all textures created before, is beyond me.

[DBrady(logged out)]

Forgot my specs...
pentium4 2.4ghz
512ddr ram
4096 vitual memory
geforce4 ti4200

[t00fri]

Wow!

See that black cat crossing the street?..Bad luck ahead;-)

Bye Fridger

[t00fri]

Forgot my specs...
pentium4 2.4ghz
512ddr ram
4096 vitual memory
geforce4 ti4200

DBrady:

Well, you are quite a bit faster than me, for arithmetics about a factor >~2.5 (I have a PIII 1Ghz, 512MB/CL2, 5Gb virtual mem)!

Tiling the gigantic 21k x21k BM western half of Earth into 64 2700x2700 tiles took me 3h:20' with virtualtex working with IM's convert. The rest went much faster: resizing of the 64 tiles to 2kx2k, color corrections and DXT conversion with nvdxt , perhaps an hour altogether.

Divide this by 2.5 and you get quite an acceptable time at least for fast Pentium4 users....

Bye Fridger

[wcomer]

If you resize after cropping are there noticable artifacts at the tile edges? I had assumed that it would be necessary to resize first, but if not I'll give that a try. -Walton

[wcomer]

Fridger,

Perhaps the reason that IM takes 3 hours is that witihin the virtualtex script the convert comand is called 64 times. I presume that each time convert is called the entire 21kx21k file is reloaded and the pixel cache is recreated. Is there a way to prevent this redundant work within the CLI calls? What about the API calls?

Walton

[t00fri]

If you resize after cropping are there noticable artifacts at the tile edges? I had assumed that it would be necessary to resize first, but if not I'll give that a try. -Walton

Well, I found surprisingly few artifacts, only 1-2 mild cases. Since I did not manage either to resize the full {21kx21k(west) + 21kx21k (east)} BM *.tif image within my system, I chose to tile first and then resize.

But perhaps resizing of the full images is better with the netPBM utilities. I'll work on that tonight.

Bye Fridger

[Guest]

i dont mind dowloading 1Gb if i get the same result as you. Where's the download link, please ?

[t00fri]

i dont mind dowloading 1Gb if i get the same result as you. Where's the download link, please ?

I appreciate your idealism, but that 1GB has also to be uploaded first to my TextureFoundry site. I suppose 1GB of DXT tiles will be packing to 300-400 MB with zip or rar.

So it's not all that bad.

I could imagine that I might be able to upload this much (on the weekend) but only with a really fast line >= 500-700 KB/sec, say, that gets the job done in a few minutes. Unfortunately, the connection to shatters.net is (probably) much slower (~70KB/sec?). Also, once the stuff is up there (precisely over there;-), on the other side of the globe), massive download would largely paralyze shatters.net....

In any case, we are thinking about solutions....

Bye Fridger

[wcomer]

Hi All,

I've managed to also build a complete set of 512x512 tiles for levels0-5. I'm curious for opinions as to what is the best choice of graphical file format for tiles. I currently built my virtual texture as .png's but Celestia is rendering them very oddly in the midocean regions (I'll try to get an image of that uploaded.) Has anyone else had such problems, if not then perhaps it is a bug specific to using .png's.

I've also put together a script modified from Fridger's virtualtex which allows me to do the entire build starting from the raw tif's in about 3 hours (including resize before crop, and basic color modifications) on a 600MHz P3 (800 MB Ram). Using NetPHB's pamdice function really cuts down on processing time but requires scripting to rename all the slices. Before I upload the script I want to make it slightly more intelligent and get permission from the original author. I think a simple walk through will suffice to allow most users to build these textures for themselves.

Walton

[t00fri]

Hi All,

I've managed to also build a complete set of 512x512 tiles for levels0-5. I'm curious for opinions as to what is the best choice of graphical file format for tiles. I currently built my virtual texture as .png's but Celestia is rendering them very oddly in the midocean regions (I'll try to get an image of that uploaded.) Has anyone else had such problems, if not then perhaps it is a bug specific to using .png's.

I've also put together a script modified from Fridger's virtualtex which allows me to do the entire build starting from the raw tif's in about 3 hours (including resize before crop, and basic color modifications) on a 600MHz P3 (800 MB Ram). Using NetPHB's pamdice function really cuts down on processing time but requires scripting to rename all the slices. Before I upload the script I want to make it slightly more intelligent and get permission from the original author. I think a simple walk through will suffice to allow most users to build these textures for themselves.

Walton

Hi Walton,

I have also rewritten my virtualtex script yesterday to considerably speed up the tiling process. I have not yet published it, since I always like extended testing beforehand...

For large textures and using this special IM option of cropping without quoting offsets, I gained a factor >3.5 which is very nice.

This way all tiles are produced and written in one go and thus I had to to the renaming in the script afterwards.

A remaining problem is to do the necessary color corrections of the large files appropriately by means of scripting.

My present strategy is to also tile a corresponding large spec file (16k, 32k) and use the resulting tiles as (external) selection textures for correcting the sea color differently from the land colors via the -modulate option. The selection mask files will then be read in via the -mask <filename> option.

This appears absolutely essential!

Bye Fridger

[chris]

Hi All,

I've managed to also build a complete set of 512x512 tiles for levels0-5. I'm curious for opinions as to what is the best choice of graphical file format for tiles. I currently built my virtual texture as .png's but Celestia is rendering them very oddly in the midocean regions (I'll try to get an image of that uploaded.) Has anyone else had such problems, if not then perhaps it is a bug specific to using .png's.

I'd be very interested in seeing a screenshot . . . I suspect that the problem has something to do with a messed up alpha channel in the PNGs. DDS/DXT1 seems like it would be the best format--there's some loss of quality because of the compression, but they DDS files are loaded very quickly and occupy the least amount of texture memory.

--Chris