2M of graphics inside! Screenshots of 64K textures

[timcrews]

Hello everyone:

In numerous other threads, I have kept you all posted (possibly a little "overly-posted" ) on my progress bulding my own 64K normal map and 64K surface texture for Earth. So far, I have not posted screen-shots because I didn't have a web page to put them on.

Well, I finally realized that I am entitled to 5MB of web space with my cox.net account. I don't know how to author web pages, but I do at least know how to transfer a bunch of .jpg files up there and then link to them in this forum.

The images below are based on the following virtual textures that I have generated since Saturday:

Non-shaded Blue Marble texture, 64K upscaled from 43K
GLOBE DEM data, 64K upscaled from 43K
32K Land Cover data referenced in my earlier thread

Plus the following two textures that have been available from other members of this forum:

32K spec map
32K night lights texture
32K political borders map

All pictures were taken on June 21 to make it easy to tell that the shadows were doing the right thing.

Here goes. Cross your fingers:

Himalayas in the evening, Blue Marble texture + normal map + specular map + night lights:



Himalayas in the evening, Land Cover texture + normal map:



The following two images show that the normal map isn't inverted. To make it easier to see the normal map, I have removed the surface texture and replaced it with only a medium-gray planet. I left night lights, specular textures, and borders on. I actually think this makes for a fairly informative view:

Himalayas in the morning:



Himalayas in the evening:



#########################

Now for my hometown area, which certainly doesn't look as interesting, either from orbit or from the ground. I hate it here:

Phoenix in the morning, Blue Marble non-shaded, 64K normal map, 32K specular, 32K night lights:



Phoenix, Land Cover map pretty much tells it all. I live in a barren wasteland:



Phoenix, gray surface, morning:



Phoenix, gray surface, evening:



These high-resolution normal maps show something interesting. To the east of Phoenix (easy to find in the evening texture because of the night lights) is a mountain chain, known as the Superstition Mountains. Just to the northeast of those mountains is a perfect circular shape. It sure looks like a crater to me. The lake on the southwest edge of the circle is Roosevelt Lake.

The small mountain going through south Phoenix is South Mountain, the local landmark. Lots of red-blinking antenna on top of it. If I had taken a picture later at night, you'd be able to see a black smudge in the night lights right where the mountain appears. (There's no houses on the mountain, of course.) Neat.

To the west of that is a north-south-running mountain range known as the Estrellas. I drive between the Estrellas and South Mountain quite often on the way to work. So I can tell you, they look right to me.

########################

Now for some more beautiful territory, Northwest Washington state. In these textures you can see the volcanos, such as Mount Rainier.

Washington, Blue Marble texture + 64K normal map + 32K specular + 32K night lights, noon: (so not very many shadows)



In that image the Blue Marble texture looks quite grainy or low-res to me. It is not the normal map, as evidenced by the remaining three images. I wonder if a level-5 tile failed to load?

Land Cover map + 64K normal map, evening:



(I zoomed out a bit so as not to push the 32K resolution of the LandCover map.)

Gray Planet, morning:



Gray Planet, evening:



Look at the shadows on the east and west of the volcanoes for proof that the normal map is OK.

Well, all of those pictures used up 2M of my 5M of web space. I'll also be throwing together a zip file with scripts and stuff, in another post.

Other than that, are there some more test images any of you would like to see?

Thanks and good night,

Tim Crews

[jestr]

Great work with the normal map-it looks fine to me,thanks for the GLOBE link I'm busy trying to make my own big normal map now,but I know this is a minor whinge,the GLOBE data like every other bump map I've found so far seems to indicate very high elevations in Antarctica and Greenland.I know there are mountains in both these regions but it always looks excessively so-anyone know why this is?

[alphap1us]

Well Tim, now I (and you should too) know why the forum has been so slow recently! I can't wait to get my hands on these gorgeous textures. Maybe you can ask someone else from the forum with their own site to host these things for a while. Hey I just had an astounding idea! Could someone collect all the huge files that are associated with Celestia and package them into a torrent? That way we wouldn't need to find space for all this stuff. I would be happy to donate my HD space and bandwidth to keeping that torrent alive for a while. What do you guys think?

[timcrews]

Well Tim, now I (and you should too) know why the forum has been so slow recently!

Actually, the pictures in this link are coming from my web server, not from the shatters.net server. So as far as I know, they do not use shatters.net bandwidth. If they _are_ using shatters.net bandwidth, then I am taking them off, because I have been very frustrated by the poor performance of the forums for the last four days, while I have been researching all of this stuff. Although now, all of a sudden, they are doing better.

Even the raw textures that I downloaded to start with did not come from shatters.net, but from (very fast) NASA servers.

As for hosting the files, I am still pretty convinced that the right way to do things is to provide the scripts (totalling about 1KB!) that create these textures, and then provide instructions for everyone to go get the raw data, and then run my scripts, and Voila! They have the same thing I have.

Here's how much data would have to be transferred to support generation of all of the pictures I posted:

64K surface texture: 1.25GB
64K normal map: 627MB
32K night lights: 140MB
32K spec map: 341MB
32K land cover: 221MB
32K borders: 329KB

This is using mostly PNG textures, because I'm willing to live with the slow load times (and because I haven't figured out how to make good DDS files yet). DDS textures would be larger.

Admittedly, the above files would compress some. But 2GB is a lot of file to download. I know, I know, you're going to say "6 hours is a lot of time for a script to run." It is a tradeoff, for sure.

Tim

[timcrews]

I'm having second thoughts about normal map inversion.

In the Washington and Phoenix pictures, mountains seem to be illuminated on the east side in the morning, and the west side in the evening, as expected.

But in the Himalayas pictures, the opposite seems to be true. The morning picture has mountains brighter on the left than on the right. The evening picture has mountains brighter on the right than on the left.

Am I mis-reading the texture? Aren't those features on the left side, about a third of the way up from the bottom, mountain ranges?

Is there an explanation for how a normal map could do this?

I suppose it is very remotely possible that I processed the two hemispheres with different versions of the nm16 tool, but this seems pretty unlikely since I used a script to do it.

Tim Crews

[jestr]

Yeah I know what you mean.I'm new to normal maps too but when I made a VT normal for Mars,I tried a 'normal' one and an inverted one and even with the flat mars texture neither version seemed totally satisfactory at all distances.Maybe someone else can tell us how it should work ?Jestr

[Buzz]

Hi Tim,

Beautiful pictures, good to be able to see what you have created! About the Himalaya's: I think you may have reversed morning and evening. The mountain shadows match the more illuminated western or eastern part of the globe...

[timcrews]

Sometimes I can be stupid.

I have the "show local time" option set in Celestia. So in the Himalayas, it is showing a morning time (that is, morning in Arizona), but of course the Himalayas are on the other side of the world. It is evening in India when it is morning here, and vice-versa.

Tim

[wcomer]

Tim

What ratio (3x?) are you using for your bumpheight?

Walton

[danielj]

What is your system specs?
Because if this texture are not VT,only very advanced computers can deal with it,not only to create but also to show itself.
Actually I think this texture will only work in a Pentium 4 3.0,1GB or more of RAM,Geforce FX 5800 or 5900,HD 160 gb
Anyway,a system that I can?t afford and I think of you don?t,too.

[granthutchison]

... the GLOBE data like every other bump map I've found so far seems to indicate very high elevations in Antarctica and Greenland.I know there are mountains in both these regions but it always looks excessively so-anyone know why this is?

They're very high places - Antarctica is the highest continent in the world, with a mean elevation of 7000-8000ft. The Greenland icecap rises to over 10000ft.

Grant

[timcrews]

What is your system specs?
Because if this texture are not VT,only very advanced computers can deal with it,not only to create but also to show itself.
Actually I think this texture will only work in a Pentium 4 3.0,1GB or more of RAM,Geforce FX 5800 or 5900,HD 160 gb
Anyway,a system that I can?t afford and I think of you don?t,too.

Danielj:

I'm trying hard to imagine what my motivations for scamming this particular subject would be. Look at my other posts, and imagine what an elaborate waste of time they would be to gain... what?

Anyway, the textures _are_ virtual textures, as documented in several dozen other posts that have been posted over the last five or six days, as alluded to in the first paragraph of this thread.

My system specs are: Athlon 1.2GHz, 1.25G of RAM, nearly a terabyte of hard disk space (of which these textures are using about 3GB now, but more like 20GB of space was required to generate the textures), and a GeForce4 Ti4200 with 64M of RAM, available for $60 at staples. With this simple two-year-old system, I am getting 40fps at the resolutions shown in this thread.

Admittedly, the RAM is a little higher than most people have in their systems, but not much. Besides, the netpbm tools that I used to produce these textures never used even 25% of that RAM. The textures could have been produced with much less RAM. And Celestia isn't using system RAM for these textures.

Your skepticism is misplaced.

Tim

Tim

[timcrews]

Tim

What ratio (3x?) are you using for your bumpheight?

Walton

Walton:

I am not sure what ratio I actually accomplished, because I don't really know the math behind what nm16 does with the bump height parameter that I give it. I specified a value of 100 for the bump height when invoking the nm16 tool. This was suggested as the value to use for the GLOBE DEM data that I was processing, in the readme file or the comments (I don't remember which) that came with nm16. I did not use your new nm16real code to generate the normal maps in this post. I would like to give that a try. I think Buzz is actually doing that very thing right now.

I know you have calculated somewhere (but I can't find it right now) the value that I should have used, and it was five or six orders of magnitude less than 100. If I remember correctly it was somewhere around .001. Since the shadows that I was seeing seemed sane, I assumed there must have been a misunderstanding somewhere, since .001 is a whole lot less than 100.

I am willing to try again with values that you suggest. However, I do not think the dynamic shadows produced by the current normal maps look unrealistic. For example, the shadows produced by the volcanoes in Washington do not extend abnormally east or west, even under extreme sunrise/sunset conditions. They seem to extend as far as the boundaries of the volcanic cones themselves, and no farther. That seems just about like what the real shadows would do.

Tim

[selden]

Tim,

Celestia only darkens areas that have surface normals that point away from the light source. Celestia does not implement "self shading." As a result, it doesn't show shadows extending across the landscape as they should. Maybe someday...

[Mikeydude750]

Wow...talk about mind-blowing textures...

Nice job...now if only I had the computer free long enough to actually run the script, I would be in business...

[timcrews]

Tim,

Celestia only darkens areas that have surface normals that point away from the light source. Celestia does not implement "self shading." As a result, it doesn't show shadows extending across the landscape as they should. Maybe someday...

I wondered about that, since it never did seem like the shadows got long enough in the minutes just before sunset or just after sunrise.

So what _does_ the bump height effect, anyway? Is the pixel darkening a simple on/off when the difference in the angle between the light source and the surface normal exceeds a certain threshold? In that case, it would seem that extreme bump heights would merely have the effect of prematurely darkening certain pixels, as the sun sets, or lightening them too late as the sun rises. This is a different kind of shadow exaggeration than I had considered. It would not actually make the mountains look any taller than they really are, or canyons any deeper than they really are, would it? It would just keep them in shadow longer. That's all just intuition speaking, not mathematical knowledge.

Tim

[Buzz]

Tim,

Fortunately it is not a simple on/off effect, but gradual. A normal vector is a vector perpendicular to the surface at a certain point. The more this surface normal points towards the light source, the more the surface is illuminated. The colour of the normal map encodes the direction of the normal vector (how exactly I do not know yet).

Walton,

Could you tell me what the input requirements for your nm16 tool are? Tim sent it to me. Does it have to be a complete earth map?

[wcomer]

Tim,

The normal map is just a map of unit vectors normal to the surface, where the rgb vector becomes the xyz vector. Usage of the term shadow is somewhat inappropriate, what is really taking place is shading which is a different concept altogether. One simple shading algorithm says that the surface brightness goes as the cosine of the angle between the light source vector and the surface normal vector. By taking advantage of basic geometry, this works out to the dot product of the unit surface normal vector and the unit light source vector. Thus precomputing the normal map is a very nice shortcut to producing basic shading. The bumpheight tells the computer the relationship between the pixel width/height to the bumpmap data. This is necessary in order to compute the unit normal vectors. If you double the bumpheight you effectively double the slopes of the unit normal vectors. This causes a nonlinear effect on the overall shading algorithm. Small difference for large angles (i.e. dusk/dawn) but a linear change for very small angles (i.e. noon.) If you have a normal map that uses 10,000x bumpheight, the effect is that you will have deep shadows at inappropriate times of the day. There are are almost no places on the Earth where large (i.e. visible at greater than 1km resolution) shadows (or in this case heavy shading) occur from say 9AM to 3PM. It simply isn't possible to sustain slopes like that. But I suspect that with the 10,000x normal map you have heavy shading across much of the landscape during those hours.

cheers,
Walton

[timcrews]

Walton:

I understand what the normal maps do now.

When I look at the north-south-running mountain ranges in Celestia on the June 21 equinox (so that shadows are directly east-west), sure enough, you are right that the mountains are only truly shadowless in a brief 15-minute window around noon. For the rest of the day, they are shaded on one side or the other.

As for whether they _should_ be shaded, realistically: Since the sun appears to move 15 degrees every hour on the equinox, then on the equinox you would think that north-south running mountainous slopes up to 15 degrees in pitch would be shaded until one hour after sunrise; 30 degrees two hours after sunrise, etc. So, I think I agree with you with your 9-to-3 estimate, which on the equinox at my latitude would be four hours after/before sunrise, respectively. Looking at South Mountain in the Phoenix area, for example, I can see that it is remaining shaded far too long.

Of course, "realistically" is sort of a stretch here, since we're not really talking about shadows.

As for aesthetics, I wonder if I really want a normal map that produces no visible effects for most of the day, especially since the night/day terminator in Celestia kicks in _so_ early, that the shadows would be almost entirely obscured by general darkness by the time they were present? Even though I agree with your claims, I'm still not sure it's what I want. I know that the Celestia community generally leans towards realism as opposed to aesthetics (that is, when the two conflict.)

Anyway, it doesn't really cost me anything to set my computer to work tonight generating a normal map with .001 as the bump height. I'll post one of the gray-type images sometime tomorrow using this new normal map.

After that, I'll not be online for a week.

Tim

[timcrews]

Walton:

I generated a normal map using .001 as the bump height argument when invoking nm16. (I'm getting good at running the script -- I just kicked it off and three hours later it was done.)

The result is absolutely no visible shading, even under the most extreme sunset conditions. I don't suppose a screen shot would be worthwhile.

On the plus side, it did make for a highly compressible virtual texture!

Tim