Chris Normal Map Converter Code (nm16)

[t00fri]

Hey all,

Chris just confirmed that everyone may download his (amazing) 16bit -> 8bit/channel normal map converter code for own experiments.

Here is the download location:

http://www.shatters.net/~claurel/normalmap/

So far it makes use of basic Unix features with respect to input/output. nm16 works with STDIN|STDOUT and thus may be entered into any pipeline just as a 'filter'.

Under Linux, compilation is absolutely trivial.

g++ -O3 nm16.cpp -o nm16

I suggest you install also a recent version of ImageMagick which gives you full 16bit/channel command power!

Look at my explicit examples in the Valles Marineres thread in this department!

Look at his readme.txt!

Input files must be in (16bit/channel) *.raw binary format and carry the ending *.img. Output is *.bmp (8bit/channel) which easily may be converted with various tools into anything you like.

Bye Fridger

[ElPelado]

does it also work in windows?

[jamarsa]

does it also work in windows?

No and Yes. You must compile the source - after a bit of work on it (Fridger said it, I haven't seen it myself) - in Windows to get an executable. Or wait to someone to do it and post a Download URL.

I'll look to compile it in a DOS 32 bit compiler I use, as I haven't Windows C/C++ compiler installed (or perhaps lcc remain installed somewhere in my system?). The code is small, so it wouldn't be a problem (and there are no windows!!).

[abiogenesis]

I downloaded the .cpp file and built it under Windows. The only problem I ran into was that the math.h header wasn't included. After adding the include, it compiled fine. I don't have any source images to try it out but, as far as I know, the stdin/sdiout "piping" tricks work in Windows as well as xNIX. If I'm wrong and they don't, it is a trivial modification to accept the files as command-line parameters.

- a b i o g e n e s i s -

[jamarsa]

I recall there were *certain* issues about the 'binary' option in file handlers, so stdin&stdout wouldn't work very well with newline characters&EOFs ...

In Unix there are no differences - all file channels are binary. That's why there are no problems using them for images.

And yes, it's trivial to make the change to accept filenames as parameters. Only it's uglier, because you have to create temporary files.

[Pixel]

Hello All,

thank to Chris for the usefull tool.
I have ported the utility nm16 under MS Windows and put also some enchancements.

Usage: nm16 <width> <height> <bumpheight> <filter method> <fileIN> <byteorder> <NODATA value> <fileOUT>

- fileIN is the name of the input 16-bit elevation data file (img/dat).
- for byteorder use "MSB" or "INTEL". (MOLA-mars data is MSB. GLOBE-earth data is INTEL).
- the NODATA-value will be replaced by 0(sea level). use 0 to ignore. Earth GLOBE elevation data-set has NODATA value of '-500'!!. it is used to mark samples with no measured height. These are mostly oceans and this value should be zero insted of -500 meters.
- 'fileOUT' is the name of output file in PPM format.

The link:
http://free.bol.bg/pixel/nm16win.zip

here are some shots of Alps with and without NODATA value:
http://free.bol.bg/pixel/alps2.jpg
http://free.bol.bg/pixel/alps1.jpg

also an inversion of the heigh data is no more necessary.

enjoy,
Pixel.

[jim]

I don't have any source images to try it out ...

Here are two links where you can find 16-bit topographic data:

http://wufs.wustl.edu/missions/mgs/mola/megdr.html
http://edcdaac.usgs.gov/gtopo30/gtopo30.html

Bye Jens

[t00fri]

Pixel:

why did you do this in your Windows port of nm16?

94c175
< dx = samples[y][0] - samples[y][x];
---
> dx = samples[y][x] - samples[y][0];
96c177
< dx = samples[y][x + 1] - samples[y][x];
---
> dx = samples[y][x] - samples[y][x + 1];
100c181
< dy = samples[y][x] - samples[y - 1][x];
---
> dy = samples[y - 1][x] - samples[y][x];
102c183
< dy = samples[y + 1][x] - samples[y][x];
---
> dy = samples[y][x] - samples[y + 1][x];

I implemented your additions for Linux, but do not see, why you changed signs of dx, dy above?

Bye Fridger

[Guest]

Pixel:

why did you do this in your Windows port of nm16?

94c175
< dx = samples[y][0] - samples[y][x];
---
> dx = samples[y][x] - samples[y][0];
96c177
< dx = samples[y][x + 1] - samples[y][x];
---
> dx = samples[y][x] - samples[y][x + 1];
100c181
< dy = samples[y][x] - samples[y - 1][x];
---
> dy = samples[y - 1][x] - samples[y][x];
102c183
< dy = samples[y + 1][x] - samples[y][x];
---
> dy = samples[y][x] - samples[y + 1][x];

I implemented your additions for Linux, but do not see, why you changed signs of dx, dy above?

Bye Fridger

I haven't touch that. I am using nv16.cpp from the link you posted here. And it still says: dx = samples[y][x] - samples[y][0];

Pixel.

[t00fri]

I haven't touch that. I am using nv16.cpp from the link you posted here. And it still says: dx = samples[y][x] - samples[y][0];

Pixel.

You are right, I diffed an earlier version of Chris which apparently had the signs reversed...But something does not seem right for Small endians (INTEL). The normal map appears reversed...

Still working...

By Fridger

[t00fri]

Pixel:

you changed the conventions in your readS16() relative to Chris , e.g. for MSB (big endian)

Chris would use in your notation:
return (short) (b) * (1.0f / 65535.0f);

You used:
return (32767-b) * (1.0f / 65535.0f);

which reverts the normal map relative to Chris.

Bye Fridger

[DBrady]

Hi,
i'm having trouble making normal maps from globe data.
i downloaded the 'i' tile, as it was the smallest to try it out, but when i make a normal map from it using pixels version of nm16 there are streak lines all accross it. I renames the file to .raw and used a .bat file with the following commands:

nm16 10800 6000 100 0 i10g.raw intel 0 tile-i.ppm

Is there anything wrong with this?

I downloaded the file in .tgz format and not .gz - are these file the same?

[Guest]

Its alright i solved the problem.
The .gz files do seem to be different from the .tgz ones.

[wcomer]

Hi folks,

I have a question for the experts. I'm thinking about making two modifications to the nm16 tool and I would like feedback.

1) A very minor problem with the current methodology used in calculating the normal map, is that the calculation shifts the the normal surface by 1/2 pixel up and 1/2 pixel left. Instead I was thinking about calculating the partials at the pixels by using a 7x7 pixel filter based on a modified bicubic spline.

Ok, maybe that is a bit of an overkill. But the next modification isn't.
2) I believe that the projection distortion at high altitudes creates a problem with horizontal partials. At higher altitudes the horizontal distance between pixels becomes proportionately smaller. This reduced distance needs to be included in the norm calculation.

What I propose is to calculate the horizontal partials using either chris's method or a higher order filter. Mutiply the horizontal partial by the strech associated with a given altitude. Then run a linear moving average filter (of half width equal to half the strech) across the horizontal partials for a given latitude. Then calculate the norms from this newly filtered horizontal partial and the unfiltered vertical partial. The reason for the moving average is that at very high latitudes a single meter change in altitude could create a nearly vertical norm, unless this is corrected for. Since the topographic data is provided on a 1kmx1km basis, the higher altitudes are already streched out and have reduced informational content. Therefore very litte information is being lost by using the moving average filter.

thoughts?
Walton

[Buzz]

Does anyone have a normalmap tool to share that will faithfully process the GLOBE data? I modified Pixels code to reflect Fridgers remarks, but it does not compile on Linux Maybe there is some platform-specific code in there... I would be happy with either a Windows or Linux tool!

[timcrews]

Buzz:

I downloaded the Windows version from the link in Pixel's post above. I did have trouble with a couple of stalled downloads, but with about ten minutes of persistence I finally got it. I have been using it all day today, but I'm having enough trouble with the other steps in creating a VT normal map, that I'm still not sure if nm16win worked right or not.

If you are unable to get a copy through the link, send me a personal message on-forum and I'll e-mail a copy of what I have.

Tim Crews

[Buzz]

Thanks Timcrews, I managed to download Pixels tool, and it runs fine. At first sight it seems to work well, but at closer inspection it appears it inverts the bumps So I applied Fridgers corrections and tried to compile, but it does not recognize strupr. I found somewhere it should be part of string.h, so I tried if including that helped, but it didn't. Not having any experience with c, I'm stuck. Can anyone help?

[timcrews]

Buzz:

AHHHHH! You are right! The bumps are inverted! That is the very first step in my texture creation process, alas. That means starting all the way over.

I have lots of expertise with C and C++ coding, but none with the theory of normal maps. If you can tell me what needs to change (conceptually) to fix the bump inversion problem, I can modify the C code and produce a new executable. What did Fridger say, specifically, needed to be done?

Tim Crews

[Buzz]

Good to hear you have lots of C experience! Fridger wrote earlier in this thread:

"you changed the conventions in your readS16() relative to Chris , e.g. for MSB (big endian)

Chris would use in your notation:
return (short) (b) * (1.0f / 65535.0f);

You used:
return (32767-b) * (1.0f / 65535.0f);

which reverts the normal map relative to Chris. "

I hope this helps!

[timcrews]

Buzz:

I've made the required modification on my side. It might be a few days before I can spare enough computer time to let all of the scripts run again. When they are done, I will tell you if the problem is fixed. If so, perhaps we can arrange for the new version to be made available on someone's web site. Or I can just keep a copy and e-mail it to people as requested.

Tim