Confusion about command lines

[danielj]

I don??t understand a thing.You have so much work doing these normal maps.Why do you think could properly makeyour own,with the tutorial?Of course we can,but the result will be far from yours.I think you should make available your normalmaps in some place.It don??t have to be Motherlode...

Using 'nms' from the nmtools package , I generated a high-quality 16k normalmap of Earth from the srtm-ramp2 16bit elevation data.

Next with GIMP, I decomposed the 16k normalmap into R,G,B grayscale layers and subsequently composed a new 16k map, with R->G, G->Alpha, and B, R->black finally. Saved as a PNG file (compression 9) it's ~49.5 MB. The uncompressed PNG is, however 537.7 MB in size!

Next I used my well-proven 'texconvert' tool that I coded long ago, using the DeVIL lib to compress the modified file as a DXT5 which goes quite fast. The NVDXT tools will not work with big size files as you all know. And they are very slow. One needs however really big size to critically evaluate the resulting quality.

The final DXT5 file is 178.9 MB. So relative to the uncompressed PNG, I got a 3.0:1 compression with my DXT5 enconding (why not 4:1?). Due to Mipmaps?

Anyway the result looks quite good and there seem no obvious bugs in Chris' new CVS code of tonight. I'll get back when I have compared it carefully with the corresponding lossless 16k PNG normalmap.



If I wasn't so busy with finishing my CelestialMatters tutorial (at last), I'd quickly code a little additional module ('remapcolors') to be piped after nms or nmstiles from our nmtools package. Then my tiles from my 64k normalmap could be automatically converted to the new compressed DXT5...

Bye Fridger

[t00fri]

I don??t understand a thing.You have so much work doing these normal maps.Why do you think users could properly make their own,with the tutorial? Of course we can, but the result will be far from yours.I think you should make available your normalmaps in some place.It don??t have to be Motherlode...

Daniel,

you just don't want to understand. OK once again in a drastic "Gedankenexperiment":

I deliberately delete my 8GB of 64k normal map tiles and start drinking one beer after the other until my abilities have reduced to just being able to typing a few simple commands into the console, be it from Windows, Mac OS, MacIntel or Linux.

After downloading the 1.2 GB packed original elevation data from the NASA arctic server, I just punch ONE SINGLE command into the console and wait about

10 minutes

on my machine. While waiting, I drink some further beers, of course. . After the command has finished, I have a complete level5 directory filled with 2048 highest quality 1k x 1k normal map tiles, EXACTLY identical to the ones I had deleted.

Can you follow me!?

Note at this stage I am entirely unable to apply any of my image manipulation tricks etc...

After repeating the same 1-command operations a few further minutes for the lower 5 levels, my entire 64k normal map is finished and EXACTLY restored as it was originally.

That's really all there is to do. The enormous speed comes from the fact that these command line tools were custom-coded and optimized to perform a specific task.

Bye Fridger

[danielj]

OK.NOW I understand.But when this tutorial will be available.What program should I use?
I think the biggest problem is to made a normal map using tiles(VT textures),since a 64k "entire" texture map is too big to be open in a program like Adobe Photoshop

I don??t understand a thing.You have so much work doing these normal maps.Why do you think users could properly make their own,with the tutorial? Of course we can, but the result will be far from yours.I think you should make available your normalmaps in some place.It don??t have to be Motherlode...

Daniel,

you just don't want to understand. OK once again in a drastic "Gedankenexperiment":

I deliberately delete my 8GB of 64k normal map tiles and start drinking one beer after the other until my abilities have reduced to just being able to typing a few simple commands into the console, be it from Windows, Mac OS, MacIntel or Linux.

After downloading the 1.2 GB packed original elevation data from the NASA arctic server, I just punch ONE SINGLE command into the console and wait about

10 minutes

on my machine. While waiting, I drink some further beers, of course. . After the command has finished, I have a complete level5 directory filled with 2048 highest quality normal map tiles, EXACTLY identical to the ones I had deleted.

Can you follow me!?

Note at this stage I am entirely unable to apply any of my image manipulation tricks etc...

After repeating the same 1-command operations a few further minutes for the lower 5 levels, my entire 64k normalpap is finished and EXACTLY restored as it was originally.

That's really all there is to do. The enormous speed comes from the fact that these command line tools were custom-coded and optimized to perform a specific task.

Bye Fridger

[Johaen]

Daniel,

you just don't want to understand. OK once again in a drastic "Gedankenexperiment":

I deliberately delete my 8GB of 64k normal map tiles and start drinking one beer after the other until my abilities have reduced to just being able to typing a few simple commands into the console, be it from Windows, Mac OS, MacIntel or Linux.

After downloading the 1.2 GB packed original elevation data from the NASA arctic server, I just punch ONE SINGLE command into the console and wait about

10 minutes

on my machine. While waiting, I drink some further beers, of course. . After the command has finished, I have a complete level5 directory filled with 2048 highest quality normal map tiles, EXACTLY identical to the ones I had deleted.

Can you follow me!?

Note at this stage I am entirely unable to apply any of my image manipulation tricks etc...

After repeating the same 1-command operations a few further minutes for the lower 5 levels, my entire 64k normalpap is finished and EXACTLY restored as it was originally.

That's really all there is to do. The enormous speed comes from the fact that these command line tools were custom-coded and optimized to perform a specific task.

Bye Fridger

ROFL. Drunken "Thought experiment" ftw.

Is that was that "Gedankenexperiment" is supposed to be? That's how google translate translated it.

[Johaen]

OK.NOW I understand.But when this tutorial will be available.What program should I use?

The way Fridger is describing it, I'm guessing either DOS, or some small .exe that he provides.

[selden]

While a few of Fridger's image translation applications are compiled programs, most are perl or zsh scripts, as he's said many times.

Perl and zsh are interpreted textual languages. (zsh also can be used as a command interpreter for interacting directly with Windows or Unix/Linux.) Many different implementations of both are readily available for Windows as well as other operating systems.

You might want to consider installing Cygwin, which implements the Linux development environment under Windows for free. Perl and zsh are optionally included: you have to check the boxes for them in the Cygwin installation window.

http://www.cygwin.com/

[t00fri]

OK.NOW I understand.But when this tutorial will be available.What program should I use?

The way Fridger is describing it, I'm guessing either DOS, or some small .exe that he provides.

There will be a carefully tested distribution of 4 normal map tools ready to go in ANY of the 4 major platforms: Windows, MAC-OSX, Mac Intel and Linux. So no compilation or additional library installation needed! Of course, the package is under the GPL licence and thus naturally also includes the source code.

The "nmtools" package will be available for download (2.4 MB only with ALL binaries and a Windows Setup.exe included) in CelestialMatters, as soon as our site opens for the public. We are confident that this can happen VERY soon. Unfortunately, I only have extended time on weekends to complete the tutorial.

Bye Fridger

[t00fri]

While a few of Fridger's image translation applications are compiled programs, most are perl or zsh scripts, as he's said many times.

Selden,

the nmtools are all in compiled C++ code and run on all platforms.
No scripts involved whatsover. CYGWIN is NOT necessary. They are compiled with VS.net 2003 as native Windows applications, for example.
Only this way, they can be as fast. Also the memory management is very important. Even for the biggest normalmaps, average home computers will work fine.

Bye Fridger

[ANDREA]

After downloading the 1.2 GB packed original elevation data from the NASA arctic server...
Bye Fridger

Hello Fridger, I'm willing to test your improvements on normalmaps as your tests will be finished and your software available for us, so I've gone to the NASA Arctic Region Supercomputing Center pages in order to start downloading the big maps to be modified (with your 10 minutes software! ).
I think this is the page to search for

http://snowy.arsc.alaska.edu/nasa/ndvi/?C=S;O=A

and here I found a lot of maps, but didn't find any 1.2 GB size (neither a couple totalling such a dimension).
Is this page the right one?
Could you please give the file/s names?
Thanks a lot, appreciated
Bye

Andrea

[t00fri]

After downloading the 1.2 GB packed original elevation data from the NASA arctic server...
Bye Fridger

Hello Fridger, I'm willing to test your improvements on normalmaps as your tests will be finished and your software available for us, so I've gone to the NASA Arctic Region Supercomputing Center pages in order to start downloading the big maps to be modified (with your 10 minutes software! ).
I think this is the page to search for

http://snowy.arsc.alaska.edu/nasa/ndvi/?C=S;O=A

and here I found a lot of maps, but didn't find any 1.2 GB size (neither a couple totalling such a dimension).
Is this page the right one?
Could you please give the file/s names?
Thanks a lot, appreciated
Bye

Andrea

Hi Andrea,

thanks for your interest ahead of the official publication of the nmtools. I will be glad to get lots of feedback how things are going in the various platforms for which the tools will be available. We are presently preparing a special discussion column in CelestialMatters.

Here is what you have to download:

http://snowy.arsc.alaska.edu/nasa/topog ... 200.bin.gz

It is a ~85k sized elevation map with (signed) 16bit integers specifying the heights. Indeed as you will notice, it is 1.2 GB in size in gz-packed form. Unpacked it would be a ~8GB MONSTER.

First the 85k file will be rescaled on the 16bit level (!) to the nearest power-of-2 size, i.e. 64k. The result will be fed in all platforms via the elegant "pipe" mechanism into the actual program that produces the optimized normalmap tiles in one step .

One great aspect of the new tools is that you don't even have to unpack the above elevation map in all those platforms that offer gzip-unpacking via STDOUT (standard output redirection). In any Linux system or also in all Macs this is naturally available. In Windows, I am not sure. Certainly it's true for people with a CYGWIN installation on top of XP. Perhaps 'rar' also does the job.

Did I understand you correctly: you havn't gotten the nmtools yet and wait for their publication?

I am VERY confident that this will be really soon. It's essentially because of me, since I am behind finishing the respective tutorial...

Cheers,
Fridger

[ANDREA]

After downloading the 1.2 GB packed original elevation data from the NASA arctic server...
Bye Fridger

Hello Fridger, I'm willing to test your improvements on normalmaps ... Could you please give the file/s names? ...Andrea

Hi Andrea, thanks for your interest ahead of the official publication of the nmtools. I will be glad to get lots of feedback how things are going in the various platforms for which the tools will be available. We are presently preparing a special discussion column in CelestialMatters.
Here is what you have to download:
http://snowy.arsc.alaska.edu/nasa/topog ... 200.bin.gz
It is a ~85k sized elevation map with (signed) 16bit integers specifying the heights. Indeed as you will notice, it is 1.2 GB in size in gz-packed form. Unpacked it would be a ~8GB MONSTER.

Thanks a lot for the address, Fridger, I was on the wrong page.

First the 85k file will be rescaled on the 16bit level (!) to the nearest power-of-2 size, i.e. 64k. The result will be fed in all platforms via the elegant "pipe" mechanism into the actual program that produces the optimized normalmap tiles in one step .
One great aspect of the new tools is that you don't even have to unpack the above elevation map in all those platforms that offer gzip-unpacking via STDOUT (standard output redirection). In any Linux system or also in all Macs this is naturally available. In Windows, I am not sure. Certainly it's true for people with a CYGWIN installation on top of XP. Perhaps 'rar' also does the job.

Yes, WinRar does the job in Win XP.

Did I understand you correctly: you havn't gotten the nmtools yet and wait for their publication? Cheers, Fridger

Surely, I'm not used to pick-up things in other people's pages, except the ones in MotherLode Creators pages, obviously.
So I haven't the nmtools, but as per your information they will be soon available, so downloading now the file I'll be ready to test it.
Thank you for all what you are doing.
Bye

Andrea

[cartrite]

Fridger,
I remember using that file "srtm_ramp2.world.86400x43200.bin.gz" trying to create a spec map a couple of months ago and I found that it had some quality problems and lots of noise ie Antartica, inland lakes and surronding regions, rivers and surrounding regions. Does your tools clean all this?

I wonder, would your tools also work on SRTM3 files? They are much better quality, they cover much of the globe, but they are without oceans. Only coastal waters. Since the oceans are 0x00, they could be filled in.

What about mgs-m-mola-5-megdr-l3-v1/mgsl_300x/meg128/? Would the tools work with these files also?

cartrite

[t00fri]

...
Perhaps 'rar' also does the job.

Yes, WinRar does the job in Win XP.

Wait a second. Sure I know that WinRar can unpack gz-packed files. Since you got a "king size" machine, you best unpack the input file first.

The generally interesting point is, however, whether there is a Windows tool that unpacks gz-format and is able to work with STDIN/STDOUT redirection like it is standard in UNIX/Linux. What is meant here in practice is that one may plug together various programs like "pipe fitttings", such that the output of program1 becomes input of program2 etc. One writes in this case (using the "pipe" symbol '|') in ALL platforms on the console:

program1 < input file | program2 | program3...

A concrete practical example for the nmtools would be

Code: Select all

gzip -dc <srtm_ramp2.world.86400x43200.bin.gz | resc2pow2 86400 1 | nmstiles 5 6378.14 65536 2.5


That would be all what is needed in Linux or Mac-OSX/Intel to generate a complete directory full of 2048 optimized normalmap tiles of size 1k x 1k at level 5 directly from the packed 1.2 GB elevation file that you just downloaded. The whole procedure (on your machine) will not take more than ~15 minutes.

Note gzip -d is the UNIX/Linux command for unpacking .gz files with the standard gzip tool. The parameter -c indicates above that we want to work with STDIN/STDOUT rather than a fixed filename!

So the real question would be whether WinRar would support this in console mode!?

Cheers,
Fridger

[ANDREA]

... Perhaps 'rar' also does the job.

Yes, WinRar does the job in Win XP.

Wait a second. Sure I know that WinRar can unpack gz-packed files. Since you got a "king size" machine, you best unpack the input file first.
The generally interesting point is, however, whether there is a Windows tools that unpacks gz-format and is able to work with STDIN/STDOUT redirection like it is standard in UNIX/Linux. What is meant here in practice is that one may plug together various programs like "pipe fitttings", such that the output of program1 becomes input of program2 etc. Cheers,
Fridger

Sorry, Fridger, but I have no idea if this is available for Win XP.
I'll try to do a search, but probably some Win XP guru here has the right answer.
BTW, the download from NASA's page is just finished, the link is very speedy.
Bye

Andrea

[rra]

The generally interesting point is, however, whether there is a Windows tools that unpacks gz-format and is able to work with STDIN/STDOUT redirection like it is standard in UNIX/Linux. What is meant here in practice is that one may plug together various programs like "pipe fitttings", such that the output of program1 becomes input of program2 etc. Cheers,
Fridger

Sorry, Fridger, but I have no idea if this is available for Win XP. Embarassed

Sure Andrea, this is available , from the command line that is.
You can use > , < or | (stdin,stdout,pipe) commands.
If you install the winRAR-package there will also be a small DOS program called RAR.exe installed which can deal with pipes and stdin or stdout.

Succes ,

Ren?©

[ANDREA]

The generally interesting point is, however, whether there is a Windows tools that unpacks gz-format and is able to work with STDIN/STDOUT redirection like it is standard in UNIX/Linux. What is meant here in practice is that one may plug together various programs like "pipe fitttings", such that the output of program1 becomes input of program2 etc. Cheers,
Fridger
Sorry, Fridger, but I have no idea if this is available for Win XP. Embarassed

Sure Andrea, this is available , from the command line that is.
You can use > , < or | (stdin,stdout,pipe) commands.
If you install the winRAR-package there will also be a small DOS program called RAR.exe installed which can deal with pipes and stdin or stdout.
Succes , Ren?©

Wow Ren??, a solution in real time!
Thanks a lot, appreciated!
Bye

Andrea

[t00fri]

The generally interesting point is, however, whether there is a Windows tools that unpacks gz-format and is able to work with STDIN/STDOUT redirection like it is standard in UNIX/Linux. What is meant here in practice is that one may plug together various programs like "pipe fitttings", such that the output of program1 becomes input of program2 etc. Cheers,
Fridger

Sorry, Fridger, but I have no idea if this is available for Win XP. Embarassed

Sure Andrea, this is available , from the command line that is.
You can use > , < or | (stdin,stdout,pipe) commands.
If you install the winRAR-package there will also be a small DOS program called RAR.exe installed which can deal with pipes and stdin or stdout.

Succes ,

Ren?©

Wow Rene that's important news. Thanks!

Bye Fridger

PS: dealing with 4 platforms at once is quite a deal

[rra]

Bad news: just looked it up on the Rarlabs homepage:
it states:

In addition to WinRAR you will find, in the distributive, the file rar.exe.

It is also a 32-bit RAR version for Windows, but it supports only the command line, text mode interface.

It is useful to call console RAR from BAT and CMD files, to use it in the DOS prompt and so on. It supports a larger number of command line switches and commands when compared with WinRAR, but it does't support ZIP and other formats.

Though common aspects of the command line WinRAR syntax considered in the Command line interface chapter are also true for console RAR, this help file does not contain descriptions of those switches and commands, which are supported only by the console version. You should read the rar.txt file for a detailed description of all console RAR features.

In other words : the commandline version only supports the native RAR compression. So .gz won't be possible this way.

Sorry for my optimism ,

Ren?©

[t00fri]

Good News:

The latest gzip (1.35) may be trivially installed for Win32 with a setup installer. Download from the official GnuWin32 Sourceforge site:

http://gnuwin32.sourceforge.net/packages/gzip.htm

and gzip homepage

http://www.gzip.org/

I tested STDIN/STDOUT redirection and pipe mode under Windos with this native Windows gzip version. Result : Perfect!

That solves the case and gzip can be used universally for all 4 platforms.

Bye Fridger

[t00fri]

A quite different, almost perfect application of Chris' enhanced normal-map code with Phong shading and the new normal-map tools (nmtools) you may find here:

http://www.celestiaproject.net/forum/viewtopic ... 1&start=15

You can see my new 16k normal-map rendering of the sea-floor that is still wet, since I pumped out the water just a few minutes earlier .

Cheers,
Fridger