Auto-tiling Textures

[John Van Vliet]

Hi i just skimed the posts so beer with me
Image Magick will ./configure and make right out of the box on cygwin ---
I just installed it in my cyg-gnome and it works fine , a little slow about 1/2
the speed as my win 32 install ( the q16)



but there seams to be a small bug ,i've only looked at level0,and 1
a 1 pixle overlap


see Bugs for details

[t00fri]

Hi John,

Hi i just skimed the posts so beer with me
Image Magick will ./configure and make right out of the box on cygwin ---
I just installed it in my cyg-gnome and it works fine , a little slow about 1/2
the speed as my win 32 install ( the q16)

great to know!

but there seams to be a small bug ,i've only looked at level0,and 1
a 1 pixle overlap


see Bugs for details

Now initially, I can see various possible origins:

1) my script?
2) ImageMagick?
3) Celestia virtual texture's code?

Do you have an idea already from where the 1px overlap has come?
I have read your bug report, too...

The tiles generated by my script definitely have the right size, though...

Did you check already whether the addition of the two tiles you were using gives exactly the original texture?

Bye Fridger

[John Van Vliet]

Hi t00fri sary for the confusion i was not using your script , I just now donloaded it . I used gimp to crop a 4k map to test the vert tex (.ctx)
and adding 1 extra row of pixels at the equater -1024x1025 ,then resizing to 1024x1024 seams to fix it

[John Van Vliet]

your script works GREAT like a charm , once i directed it to /usr/local/bin
so it could find IM . i am very new to cygwin and have been having fun with /usr/bin being read as /bin so i put IM in /usr/local/bin

but there is still the overlap at the equator in level1
I will let you know what level2 and 3 look like later tonight

[t00fri]

your script works GREAT like a charm , once i directed it to /usr/local/bin
so it could find IM . i am very new to cygwin and have been having fun with /usr/bin being read as /bin so i put IM in /usr/local/bin

but there is still the overlap at the equator in level1
I will let you know what level2 and 3 look like later tonight

Hi John,

...fine! It also works great for me;-). Actually, I am working on further support scripts for 'tilers', such that they can go out more often with their girl friends, while the script continues working;-)

I just produced a number of level3 normal map tiles (size 2k) from the 43k (!!) GTOPO30 16bit earth elevation data! Great stuff, notably, if normal-map converted with Chris' 16bit nm16 tool...

Bye Fridger

[DBrady]

As regards your new normal map tiles-you'll probably see join lines if you just ran chris's nm16 tool on each tile individually, at least i did when making a 32k normal map from each individual tile of globe data.
I tried sticking them all together with IM but it required too much memory.
I was able to stick four tiles together though and since the join lines dont seem to be visible on the horizontal joins-only the vertical-making four rows out of the gtopo30/globe tiles and then running them through nm16 should eliminate all but the last join. I havent done this yet as IM is very slow and i havent found the time. (i should really look into cygwin and making scripts!)

Or course you may not have this problem at all in which case you can just ignore this post.

[t00fri]

As regards your new normal map tiles-you'll probably see join lines if you just ran chris's nm16 tool on each tile individually, at least i did when making a 32k normal map from each individual tile of globe data.
I tried sticking them all together with IM but it required too much memory.
I was able to stick four tiles together though and since the join lines dont seem to be visible on the horizontal joins-only the vertical-making four rows out of the gtopo30/globe tiles and then running them through nm16 should eliminate all but the last join. I havent done this yet as IM is very slow and i havent found the time. (i should really look into cygwin and making scripts!)

Or course you may not have this problem at all in which case you can just ignore this post.

Well, I have quite a bit of experience with 16bit/channel image data manipulations and so far did not see any join lines in my results. What is /crucial/ to avoid artifacts, is to rescale the GTOPO data after the normal map has been built with nm16! The rescaling in fact destroys the map's normalization , hence one has to /renormalize/ the normal map afterwards, which I do with the gimp normal map plugin!

I am working on a (more complex) script that does the following:

Suppose I want to cover a given GTOPO tile (i,j)
http://edcdaac.usgs.gov/gtopo30/tile_img.html
with appropriately numbered level3 tiles with some chosen square tile size (e.g. 2k).

The new script loads and uses the minimal set of needed neighboring GTOPO tiles around the one of interest such that the GTOPO tile of interest is completely contained in the generated tx_m_n tiles!

The performed steps are in detail:

Join the loaded minimal set of GTOPO tiles together with IM's 'convert' in 16bit depth. Apply nm16 to this elevation patch. Rescale the resulting 24bit normal map patch by 32k/43200 and renormalize it properly. Determine the corresponding (x0,y0)-offset for the first level3 tile. Generate all level3 tiles which fit into the normal map patch, convert to DXT3 format...DONE

Bye Fridger

[Buzz]

Thanks for posting the tiling script t00fri, it works very well! Now I just have to find out what size works best...

Buzz

[John Van Vliet]

Now that the power is on I was able to play with the tile size
level 0 to 3 at 512
level 0 to 3 at 1024


when celestia firsts starts i like the detail of the larger tiles better
altho when i put 2 1024 tiles in level0 of the 512 set the image of earth was improved a bit


I am still having a problem with using a basesplit otherthan 0
only a 0 basesplit seams to work

[t00fri]

I have upgraded my tiling script virtualtex again today. Now, it's version is 1.02, to be found again here:

http://www.shatters.net/~t00fri/virtualtex

Upon calling

> virtualtex

you get the following (modified) help/usage output:

Code: Select all

Usage: virtualtex [--help|<texture name> <tile size> <tile format>] [e|E|w|W]


The shell script 'virtualtex' is a tool for Celestia > 1.3.0
that supports 'virtual textures'.

The script generates the required tiles tx_i_j of desired (square) size,
<tile size>, in a specified format, <tile format> = png, tga, jpg,...,
from an input texture, <texture name>, in any popular graphics format.

The optional 4th argument e|E|w|W is for the case of tiling
square /e/astern | /w/estern halfes of the full texture separately!

Besides Linux/Unix, the script also runs in a current Cygwin installation
under Windows, ( http://www.cygwin.com ). If the z-shell ('zsh') is unavailable, it also may be executed with the 'bash' shell, by replacing #! /usr/bin/zsh by #! /bin/bash in the first line.

The script assumes that a recent version of the ImageMagick package
( http://www.ImageMagick.org ) is installed (either under Unix/Linux or Windows). The utilities 'convert' and 'identify' of that package are used.

Author: Dr. Fridger Schrempp, fridger.schrempp@desy.de
Version: 1.02, 08/16/03


Note the new, optional 4th argument [e|E|w|W] designating the eastern |western half of (monster) textures that often are distributed as square 21kx21k west and 21kx21k east textures, each being several hundred MB! In case of the eastern half, virtualtex automatically shifts the first index of the generated tx_i_j tiles appropriately!

Please, remember, that virtualtex also works for arbitrary texture sizes that need not be a power of two or have an aspect ratio of 2:1. Given the desired square tile size, remainders are simply discarded...

Bye Fridger

[t00fri]

People who have downloaded virtualtex 1.02 immediately after I have posted the above upgrade announcement, may want to download the script again, since I have made meanwhile a trivial extension:

The optional 4th parameter of virtualtex may now be both capital E|W or small letters e|w. Other input generates an error message.

Bye Fridger

[wcomer]

This is for all those people who, like me, would love to use the virtualtex tool in the windows environment but know very little about ImageMagick or Cygwin.

Code: Select all

Install Cygwin:
 Step 1:
  Download: http://cygwin.com/setup.exe
 Step 2:
  Run setup.exe
   ->Next
   ->Select "Install From Internet"
     Next
   ->Root Directory: "C:\cigwin"
     Install For: "All Users"
     Default Text File Tpe:"Unix"
     Next
   ->Local Package Directory: "C:\......\Desktop"
     Next
   ->Select "Direct Connection"
     Next
   ->Select "ftp://cygwin.mirrors.pair.com" (this worked well for me)
     Next
   ->Now you have a Select Packages Screen.
     Scroll Down and select the "Shells" Category
     Clicking once on Shells should open up a new set of selections.
     Scroll down and click once on the "Skip" label next to the "zsh: The Z-Shell" package name.  The label should have changed to "4.0.7-2".  If you clicked more than once keep clicking on the label and it will cycle back through to "skip"
     In the same way select Editors->nano
     Next
   ->Create icon on Dektop "Check"
     Add icon to Start Menu "Check"
     Finish

Install ImageMagick
 Step 1:
  Download: ftp://ftp.imagemagick.org/pub/ImageMagick/binaries/ImageMagick-5.5.7-Q8-windows-dll.exe
  or Download: ftp://ftp.imagemagick.org/pub/ImageMagick/binaries/ImageMagick-5.5.7-Q8-windows-static.exe
 Step 2:
  Run either .exe file
  These are pretty self-explanatory installations so I will not do a walk through.
  Note 1:
   I installed both (static followed by dll.) However, I'm pretty certain that this was unecesarry and redundant.  Perhaps Selden can clarify.
  Note 2:
   The final screen asks if you want to change you file associations to ImageMagick.  You will probably NOT want to do that (I found this out the hard way.)

Install NetPbm:
 Step 1:
  Download: ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/porters/Humblet_Pierre_A/V1.1/netpbm-10.15-cygwin-1.3-bin.tar.gz
 Step 2:
  Place the downloaded archive into your root cygwin directory.  This is probably c:/cygwin/
 Step 3:
  Install
   -> Run Cygwin
   -> Type "cd ../.." [Enter]
   -> Type "tar xzf netpbm-10.15-cygwin-1.3-bin.tar.gz" [Enter]

Install Virtualtex
 Step 1:
  Download: http://www.shatters.net/~t00fri/virtualtex
  Make sure you do not accidentally save as a .txt file, else just use rename to drop the file extension after downloading.
 Step 2:
  Move "virtualtex" to "C:\cygwin\usr\local\bin"

Example
 Step 1:
  Copy "eros.jpg" from "C:\Program Files\Celestia\textures\medres" to "C:\cygwin\home\Administrator"
  Note 1:
   The destination directory will be based on your windows user name, i.e. "\home\"[Login].
 Step 2:
  Run the cygwin application
  Type "virtualtex eros.jpg 256 jpg" [Enter]
  In the "\home\"[Login] folder there are now 8 new files which are the level1 textures.
 Step 3:
  Move all 8 level1 textures to a eros level1 folder.
 Step 4:
  Rinse and Repeat for level0 files.


I hope this is helpful (I know it would have saved me several hours of trial and error.)

Cheers,
Walton Comer

10/3/03 - Added instructions for installing NetPbm and the nano editor

[t00fri]

Walton,

SUPER!

Thanks a lot for your most useful contribution. I shall try it myself today, since I plan to install Cygwin on my Laptop (XP) for being able to test further scripts also in that environment....

Bye Fridger

[John Van Vliet]

about a week befor the pre 9 came out i but cygwin and ImageMagick on win xp and had no problems . cygwin +gnome both installed fine using the setup.exe , and imagemagick ./configure prefix=/usr/local/bin, make,make install
with no problems
no my only problem is glib for gtk and gimp

[t00fri]

This is for all those people who, like me, would love to use the virtualtex tool in the windows environment but know very little about ImageMagick or Cygwin.

...

Cheers,
Walton Comer

I have just confiremd Walton Comer's Cygwin Installation instruction for my Windows XP Laptop!

It worked without any problems whatsoever and thus -- with a sufficiently fast connection -- the whole Cygwin + IM-Q8-IM-Q16 +virtualtex installation just took a few minutes.

Cygwin for this scripting setup just takes 30-50 MB of space, i.e. the size of a nice Celestia texture;-).

I really like the Cygwin environment on XP, with a nice zsh-terminal that has the feel of Unix/Linux and recognizes also all Windows native (command-line) programs like ImageMagick packages or Perl! All standard unix tools are there, too...

Thanks again Walton!

Everyone should be able to do the installation within a short time.

Bye Fridger

[wcomer]

Fridger,

I'm trying to make complete instructions for building the 32k virtual texture. But I'm running into a few IM problems for the large blue marble .tif's. Both in DOS and in Cygwin, when I try to resize the texture,

Code: Select all

mogrify -resize 16384x16384! land_west.tif


I get,

mogrify: Pixel cache is not open (lan_west.tif).

Currently I have 800MB RAM with 1GB virtual memory. How do I "open" my Pixel cache? Or is the problem due to insufficient memory? Technically doesn't the pixel cache require ~800MB of memory for the 16kx16k image and ~1.3GB for the 21kx21k image, in which case does the entire operation require 2.1GB of free memory?

thanks,
Walton

[t00fri]

Fridger,

I'm trying to make complete instructions for building the 32k virtual texture. But I'm running into a few IM problems for the large blue marble .tif's. Both in DOS and in Cygwin, when I try to resize the texture,

Code: Select all

mogrify -resize 16384x16384! land_west.tif


I get,

mogrify: Pixel cache is not open (lan_west.tif).

Currently I have 800MB RAM with 1GB virtual memory. How do I "open" my Pixel cache? Or is the problem due to insufficient memory? Technically doesn't the pixel cache require ~800MB of memory for the 16kx16k image and ~1.3GB for the 21kx21k image, in which case does the entire operation require 2.1GB of free memory?

thanks,
Walton

The default value of IM's pixel cache is only 80 MB. Hence for the above file, it's much too small. You may increase its size with the option to 80% of your RAM, e.g

-cache 400

in case of 512MB of RAM which should significantly boost the performance.

Bye Fridger

[jim]

Hi Fridger,

I have some trouble with your 'virtualtex' script (v1.03) and Cigwin unter Win98se.
Fist the script works not with the ZShell. I got this error:

Code: Select all

/usr/local/bin/virtualtex:64: parse error near `^M'


I have this shell installed but it's not in "/usr/bin" the files placed in /bin. Maybe that something is wrong with my Cygwin but several new installations brought not differnce.

Now with the Bash-shell (the standard shell of cygwin) the scrip works except on line:

Code: Select all

texturesize=`identify $1|cut -d " " -f 3`


If I replace it by "texturesize=16386x8192" for 16k map then all works fine. Imagmagic5.5.7 Q8 and the 'identify' command is correct installed and was testet on Dos and Cigwin.

It's a pity that 'identify' doesn't work in the script. I want made my own modifications and do really need this command.

For 8k and lower levels I use my own DOS-scipt that build the complete virtual texture with all levels and folders inclusive DDS-conversion in round 15 minutes (dxt1c).

Bye Jens

[t00fri]

Hi Fridger,

I have some trouble with your 'virtualtex' script (v1.03) and Cigwin unter Win98se.
Fist the script works not with the ZShell. I got this error:

Code: Select all

/usr/local/bin/virtualtex:64: parse error near `^M'


Jens, this looks like a Windows 'line ending' problem!
^M = CR! I'll investigate tomorrow night, when I hopefully get a little spare time....

Bye Fridger

[t00fri]

Jens,

I could reproduce your ^M error with virtualtex, but only if I make a mistake!

The mistake is to edit virtualtex e.g. with notepad or any other Windows editor. This will introduce ^M's = \CR at the line endings which the UNIX-emulation CYGWIN cannot digest

So you should always stay withing a given line ending convention: UNIX or DOS and never mix the two. You must therefore use a CYGWIN version of xemacs, vim, ed,...as editors for text files that are running under CYGWIN.

Another option seems to be to set DOS line endings throughout when CYGWIN is installed. I have not tested this, however. In that case you can load virtualtex into Word and save it again, but with Windows/DOS line endings.

Bye Fridger