Sources for 500m landcover?

[jnthn]

I have made several attempts at creating a 64K specular texture for Earth to go with the BMNG texture @ 64K. So far, nothing has been at all satisfactory. The first thing I tried was the watermask found with the BMNG images,here. I also tried the ndvi images found in the "ndvi" directory of that server. I'm sure that it's an outstanding accomplishment as far as vegetation indices go. As a source of a landmask for Celestia, it's unusable, though, AFAICT. The coastlines are extremely ragged. Islands and inland water bodies are often twice their actual size or more. Where it truly matches the actual features in the BMNG image, it seems to be by accident.

I've tried scaling up some of the available 1Km landcover datasets to 64K. The best I've found, so far, has been the LCC image done at Boston University, which I obtained here. (The two tiff's, under the "high resolution images" heading) Of course, at 1Km, it's no real match for the BMNG image... at least the coastlines are reasonably well aligned.

I've tried upscaling the UMD 1Km image, and the USGS image "grun2_0ll.img.gz", found on the ftp server listed at thispage, but the Boston University image is far superior, IMO.

I've had a look at some of the data from GLCF and the 500m data seems to be from the same source as the landmask that I found with the BMNG images. Same ragged, mismatched coastlines, etc. No good.

<sigh>

So here I am, lost and begging for help... does anyone know of any 500m landcover images besides the sources I've mentioned? I'd be ever grateful for some links to other sources.

Thanks,
Jonathan

[GlobeMaker]

Hi jnthn,

If you are ready to do some programming work, you do not need to find
pre-made images that match your specific needs. You can go right to a
source of detailed topo data and create your own custom files. Here is a
1 km database that you could struggle with :

http://www.ngdc.noaa.gov/mgg/topo/gltiles.html

I have used that data successfully to make 3D models of land
regions. If you need help, let me know.

[cartrite]

Hi Johnathan,
I have also had trouble with creating a spec map for the 64k bmng.
I've looked at the Watermask files at the bmng site and at a quick first glance I thought they may
have been promising. But you are saying they are no good?
When I did my 64k version of the bmng I used the bathy files to change the dark coastal regions to the
same color as the rest of the oceans. Most of the ocean color is 020514 in hex.
The light blue color in the bathy files follows most of the coastlines. Once combined I changed it to 020514.
Then created a spec map and masked out the dark coasts.
My problem was that I only used the October files. Low light in the northern and southern latitudes caused some land to be too close in color to the water and was selected. So I didn't like the results.
If you use the July files to create the spec map north of the equator and the January files to create
the map south of the equator this may work better.
I did some trial color selections using those maps and was able to isolate the water from the land easier. More light.
I haven't tried to create a whole map yet though.

cartrite

[jnthn]

If you are ready to do some programming work,

If you mean actual coding, unfortunately not. The only "programming language" I know is bash shell.

http://www.ngdc.noaa.gov/mgg/topo/gltiles.html

Hrm... but that's topo data. Are you saying I could extract a water mask from that? Coastlines I could see, but how about inland bodies/rivers?

Thanks,
Jonathan

[jnthn]

I've looked at the Watermask files at the bmng site and at a quick first glance I thought they may
have been promising. But you are saying they are no good?

Certainly unacceptable to me. I made a 64K VT from the raw watermask image, and was sorely disappointed when I saw the result. Dark rings around islands, over-wide rivers, etc. (yes, I scaled it to 64K It does match in a lot of areas.) I'll post some screenshots in the morning and let you judge for yourself.

If you use the July files to create the spec map north of the equator and the January files to create
the map south of the equator this may work better.

Heh... that's a lot of imagery to download/store. I've played around with the bathy image for masking out the shallow ocean water on the A1 tile (I'm working on a 64K August texture). I've also experimented with decomposing the BMNG image to RGB channels, and applying different threshold values to the three channels. Thing is, I do want to remove the anomalous dark blue shallow oceans, but I don't want to mask out all of the coastline. A lot of the algae blooms, coloration in bays, etc. are desirable, IMO.

Of course, that's only the oceans. There's still the matter of inland water...

Thanks for the input,
Jonathan

[cartrite]

I guess the only way to do it right is a LOT of image editing. If you use the Gimp you can edit your selections and save them in xcf format. Then you can chip away at it a little at a time. Do a major selection of the coastline. Then switch to subtract selection to leave the areas like algae, etc. LOT of work.
But when you save in xcf format your selections are saved too and you can pick up where you left off. I used the Gimp for quite a while before I became aware of this. I'm not sure if other image editors have this feature.

I never tried to decompose so thats new to me.

cartrite

[jnthn]

Here are a couple of quick screens to illustrate some of the problems with the BMNG watermask (click them for the full res image):

This is the Adriatic Sea with the BMNG spec map at 64K:


And this is the same area with the Boson University LCC map at 32K:


They were each processed in the exact same way. I used pnmscale to downsize (BMNG from 86K to 64K and BU from 43K to 32K), and pnmremap with a two pixel black/white PPM to remap each pixel to full black or white value.

Notice how much overlap there is around the islands with the BMNG spec. Especially, the smaller islands are simply swamped in a sea of dark blue (#020514) which should be specular ocean. And remember... the BU map is only 32K, while the BMNG is 64K and should have twice the resolution!

Oh well, if worse comes to worst, I'll just keep using my BU spec map.

ciao,
Jonathan

[GlobeMaker]

jnthn replied
"If you mean actual coding, unfortunately not. The only "programming language" I know is bash shell."
GlobeMaker wrote:
http://www.ngdc.noaa.gov/mgg/topo/gltiles.html
"Hrm... but that's topo data. Are you saying I could extract a water mask from that? Coastlines I could see, but how about inland bodies/rivers? "

Without writing computer programs, you are limited to specialized data files created
by other people, and limited to software like Gimp. If you could write a
computer program, you could draw upon non-specialized data files to
fulfill the unusual goals which you have set.

I showed you a link to a topo file. Elevations of land are from 0 meters
to 8900 meters. All of the ocean is set to -500 meters, so it would be
easy to make a specular texture image for the oceans. Inland lakes can
also be identified because they are flat. A computer algorithm can
find all of the flat lakes and give them a specular texture value. Rivers
are almost flat, so they can also be identified by a more complicated computer
algorithm. Even though you cannot program, other people can, and you
could collaborate with them, if your project generates enough interest.

[cartrite]

Here is what I got from the technique mentioned above with the Oct. files.
Keep in mind that I only made the mask to "fix" the coastlines. The spec map is from Harold Schmidt which I downloaded from the Motherload. Did you try his spec map.



cartrite

[cartrite]

GlobeMaker,
I agree, programming can open up a whole different world. It's been years since I wrote any programs though. I am looking at writing a python script (program) to export my blender models to cmod. Or writing a program that
can convert a data file such as the megt's of Mars and turn them into a cmod file. That is going to take a LOT of time to learn the ins and outs of Python before I can get started though.
How is your Mars valley model going?

cartrite

[jestr]

I think it is going to be unlikely you will find any specular map which matches the texture exactly,as the maps are going to be compiled at different times of year (even the time of day will have an effect due to high and low tides),rivers and lakes will be swollen at some times and dry others.Of course with the new BMNG textures you have a choice of months so if you can match this with when the spec map was collected it may be close.With my level12 UK tiles I have had to resort to painting it by hand,cheers,Jestr

[jnthn]

The spec map is from Harold Schmidt which I downloaded from the Motherload. Did you try his spec map.

Yes.. I believe it's just a VT of the UMD 1Km map that was the "standard" landcover for the original Blue Marble image. Unfortunately, it seems to contain all the errors and omissions of the original (Pacific islands are missing, no data along the edges, etc) . One of the first VT's I ever made was a composite of the UMD 1Km and the USGS 1Km maps. I used the USGS map to add the missing Pacific islands, and replace the Antarctic. It's OK, and has its adavantages over the Boston U. map (for one, many rivers are better delineated), but overall, I still prefer the Boston U. for its well matched coastlines.

Maybe I'll have to see what I can do with a composite of the UMD and BU images, plus some help from the main BMNG image.

ciao,
Jonathan

[jnthn]

Even though you cannot program, other people can, and you
could collaborate with them, if your project generates enough interest.

I appreciate the encouragement. I'm not sure I have enough interest to invest that much energy in it, myself, though. For now, I believe I'm just going to make do with such skills/materials as I have available.

Thanks,
Jonathan

[jnthn]

I think it is going to be unlikely you will find any specular map which matches the texture exactly

No doubt. I would not expect anything to match exactly. The old 1Km maps all are much better matches than the supposed 500m watermask map, though. Like I said, I don't want to discount the hard work that went into it, and I'm sure the original NDVI is outstanding as a source of vegetation data. As a land mask, it's awful, though.

Of course with the new BMNG textures you have a choice of months so if you can match this with when the spec map was collected it may be close.

If the beaches of the world were often several kilometers wide, I could believe that there may be a matching monthly image.

The overlap is just too huge. No way is there a near match.

With my level12 UK tiles I have had to resort to painting it by hand,cheers,Jestr

I was reading a thread about spec maps on the orbiter forum today. Someone suggested that a small pillow on the desk would help... for the times you bang your head on the desk in frustration.

/me goes looking for a pillow.

cya,
Jonathan

[cartrite]

Hi Johnathan,
What does that area of the Adriatic Sea look like with no spec map? This is not the same shot but close of the same area with no spec map. I also moved forward in time to July so there was more light.



All of this peaked my interest in the BMNG again. I'm going to try to create an ocean mask with the July/Jan BMNG files. If it looks any good I'll let you know.
cartrite

[jnthn]

Hi Johnathan,
What does that area of the Adriatic Sea look like with no spec map?

Here's what I see at 09:00 UTC on June 21 with no spec map:



I turned up the ambient light a good bit to help contrast. That's the completely unretouched BMNG.

I'm going to try to create an ocean mask with the July/Jan BMNG files. If it looks any good I'll let you know.

Please do. I'd like to see how it turns out. The blender to cmod export script sounds interesting as well. I hope you'll decide that it's worth the effort. That's something that's sorely needed, IMO. Do you read the Blender Python forum over at Elysiun? Maybe you could get some pointers from the folks there.

cya,
Jonathan

[jnthn]

If you could write a computer program, you could draw upon non-specialized data files to fulfill the unusual goals which you have set.

I was wondering what could be so unusual about a spec map that makes a good match to the landcover. I think I have confused the topic by speaking of how much ocean color I want to keep. At that point, I was speaking of the main Earth texture, and not of the spec map. The spec (landcover) map must inlcude all of the ocean water, of course.

Just to make sure we're "on the same page."

Jonathan

[cartrite]

As I suspected. After comparing the 2 images, it clearly shows that I cropped out a lot of interesting areas (algae along that peninsula to the lower right to mention one). I'll have to be more careful to include these areas on my 2'nd attempt.

I scanned thru the blender forum a couple of times but never posted any questions. I plan to do this more in the future. Up to now I've been going thru the blender manual. There is a lot I still must learn. My main motivation to do this is..... When I create a model in blender I have to export it to a obj file. Then with Anim8or export it to 3ds. Then convert it to cmod. Then with a text editor combine all the meshes to 1 file. Then change it to binary.
For a model with 256k vertices. Well enough said. I'll come up with something soon I hope.

cartrite

[jnthn]

As I suspected. After comparing the 2 images, it clearly shows that I cropped out a lot of interesting areas

Here's a wider view of the area, before and after applying the mask I created in Gimp.

Before:



And after:



Note that the "after" shot is a 16K DDS VT, and the "before" shot is a 64K PNG VT. Any difference in land shading is attributable to the lower resolution, and the lossyness of DXT.

Anyhow, in case you're interested in trying it that way, here's the method I used:

I used the PNG [A-D][1-2] tiles from:
http://visibleearth.nasa.gov/view_detail.php?id=7119

The images were cut into smaller tiles using pngtopnm and pamdice from the netpbm utilities. There are sixteen tiles for each [A-D][1-2] image at 5400x5400px per tile. The resulting PPM's were then modified in The Gimp, and saved as XCF. Each was modified in the following manner:

Filters -> Colors -> Decompose -> RGB to layers.

Tools -> Color Tools -> Threshold

Threshold the blue channel 0-26

Threshold Green 0-5 Move the green layer to the top, set the layer mode to "darken only," then right click the layer and "merge down." This amount of green threshold clears up much of the dark coloration where shallow water meets deep. Too high a threshold for the Green channel will strip away the "interesting" colors along the coastlines.

Threshold Red 0-2, Move to top, set to darken only -> merge down. Red threshold helps in preserving the land masses intact.

Set image mode to RGB.

Filters -> Color -> Color to alpha. Set black to alpha. Deselect the alpha channel in channels dialog, then bucket fill the image with #020514 (the ocean color in the original image). Now copy and paste this image as a new layer into the original image.

Now repeat for the other 127 tiles. I have 1GB of RAM, and could have made the tiles larger, but chose 5400x5400 to allow lots of "elbow room" for multiple layers, etc. I'm glad I did, too. I still swap sometimes when working on them.

Of course there are problem areas. The worst of it is that this method hardly touches the "scratchy" coloration in the Northernmost oceans, and in the Southern Atlantic to the East of South America. I've played with other threshold values for the various color channels, but it's just going to take a lot of hard work to clean that up, I'm afraid.

HTH,
Jonathan

[cartrite]

Your method is interesting and I will give it a try. I never worked with the channels separatly before so
this will be a new experience. Thank You.
I too cropped the tiles to 5400x5400.
So I have the tiles already.

A more detailed version of my method in case your interested.
First I painted the lightest blue color from the bathy tiles in 2 opposite corners to make sure the selection was 5400x5400.
After that I basicly used the select by color tool. With the bathy files I used theshhold 14. So not all the light blue color was selected. I made sure that all the coastline was selected. What was selected was pasted (After eraseing the paint in the corners) onto the original world (unshaded) file and bucket filled 100% Normal mode with 020514. I deselected it then selected by color with threshold 18. Copied and pasted as new. Then I bucket filled that with lighten only mode at 25% color 142b4b. Then with soft light mode at 25% with color 82b44f.

(I did this to try to isolate the very dark land areas of the northern and southern latitudes from the water. It seemed to work somewhat. There was still alot of areas that looked like land so I erased them. After the following color selection the file was discarded.)

I selected by color again at threshold 10. Copied and pasted as new and did 3 bucket fills at 25%. Lighten only 8264e7, Soft light 82b44f, and Soft light 8264e7. This turned the color of the ocean a blue green color. And all the inland waters that was selected in the process too. After I finished with all 128 tiles, I used Image Magic with commands ("convert -modulate 120,60 a1-0.xcf a1-0.png" (original file with ocean color 020514)) "composite -dissolve 80 ocean-a1-0.xcf a1-0.png x-0.png" for all 128 tiles. This was done with simple bash scripts. Then came the scaling down to 4096x4096 and assembly into 8192x8192. 4 8k's for each of the tiles A1 thru D2. Then VT'd.

Too much? I agree. But the result was a custom BMNG with brighter oceans.
Using the July files for the north and Jan for the south I hope to eliminate some of these steps. I get the feeling that a 50 50 combination (or some other ratio) with your method and the July/Jan files from the north and south, well it may produce a decent ocean texture. Then that could be turned into a spec map and an improved version of a July BMNG. We'll see. Gonna try A1 first. Should know in about a week or two.

cartrite