32k BlueMarble Virtual Earth Texture is now available

[Mario]

Thanks for the info, DBRady. That's exactly what I was trying to understand in order to me the files available in a comprehensive manner for those who wish to download.

This is doable. It's already 2am in here so I'll only be doing this tomorow.
However, mind you the current pack goes up to level 5 (128k texture). Meanwhile I trust it should also be useful if I divide the pack in 5 zips (with .ctx file) each corresponding to one of the resolutions. See below...

I'll be unzipping the individual files and creating a download schema for them. I trust those interested in this selective download have those grids and will know what they are downloading. Right? If not let me know and I'll create the diferent grids (16k and 32k. See below) myself and post them on the webpage for reference.

So, just as to conclude...

- One full download. Just like what we have now. (the JPG version will have to go to make room on the hard drive for the uncompressed pack)

- One selective download with 5 compressed sets. One for each resolution (or level if you may)

- One selective download for for each tile individually. Mind you, I'll only be doing this for the 32k and 16k textures. Above 32 it's just too much work and below 16k the resulting zip file can easily be downloaded from the other download method above.

Mario

[Guest]

Mmm...at least someone knows what they're doing!...still can't see the screen shots (Yes I've refreshed). Can I still download the 90meg version of the 32k...or are you going somewhere else with this?

I don't want to jump into this 'virtual texture' lark before it's ready!!

Yours...a bit confused..but very interested...bh!

[DBrady]

Mario,
By the sound of things the tiles you have are not 2k x 2k at all.
If they go up to level 5 then they are probably 256 pixels x 256 pixels. The highest level will contain the 32k set of tiles ie. level5.

The resolution contained in each level is dependant on what size you make your tiles. I''l assume that most people use a basesplit of zero which means there'll be 2 tiles in level0 like this...

------------
| | |
| | |
------------

With each file 256 pixels square this would make a resolution of 512 x 256(.5k) in level0
If each file was 2k x 2k on the other hand, level0 would have a resolution of 4096 x 2048 in level0.

Each level after this would be double the resolution of the last.
Level1 with 256 x 256 tiles would be...

--------------------------
| | | | |
| | | | |
--------------------------
| | | | |
| | | | |
--------------------------

Each tile being 256 pixels accross this would be a resolution of 1024 x 512(1k).
Again if they were 2k tiles the res would be 8192 x 4096(8k).

In conclusion if your tiles are 256 pixels accross then each level would be as follows...

level0 = .5k
level1 = 1k
level2 = 2k
level2 = 4k
level3 = 8k
level4 = 16k
level5 = 32k

[TERRIER]

After lifting my lower jaw up off the floor, I can eventually say...AWSOME!

Thank-you to everyone involved in making a 32k texture available for the likes of me with a fairly low spec computer.

I can run this as an alternate texture following my shuttle model at an altitude of arouind 300 km at a whopping (for me) 30fps. Although there is no bumpmapping or specular (or arctic ice!), the level of detail is well worth these sacrifices and the download time. [150k broadband 1hr 20 mins]

bh
I highly recommend this (and installing celestia version 1.3.1pre9) and to anyone else thinking about downloading it.

regards

[Mario]

Ok, DBrady. Thanks for the correction.

Thank you all for yur patience. I just host them, I have no graphics design background. Hence, my trouble trying to do this.

I already separated the zip file into 5 blocks, each representing one of the levels. I'll go now and put these ones available for download (the JPG version will be deleted from the server).

These are the sizes and data of the zip files:

Code: Select all

level0.zip - 92 KB    (made of 2 tiles)
level1.zip - 334 KB  (made of 8 tiles)
level2.zip - 1.20 MB (made of 32 tiles)
level3.zip - 4.55 MB (made of 128 tiles
level4.zip - 17.4 MB (made of 512 tiles)
level5.zip - 66.1 MB (made of 2.048 tiles)

Individual tile size for all levels - 192 KB

So, as you can see, only the level5 pack needs to be made available in individual tiles. It won't be an easy task. We are talking about 2.048 tiles and their respective download links. I beg you all to understand I don't have the free time to build this setup.

But I also don't want you to be forced to dowload what you don't want.
So, alongside the gargantuan 90Mb download link, plus the above individual resolutions download links I just described, I'll make it available one last zip file composed of all the level 5 landmasses (including tiny islands) and just one ocean tile.

I gather this should be a very good improvement over the still big 66MB download above. As for othe other levels, I think the download sizes are already manageable.

Mario

[Mario]

Mmm...at least someone knows what they're doing!...still can't see the screen shots (Yes I've refreshed). Can I still download the 90meg version of the 32k...or are you going somewhere else with this?

I don't want to jump into this 'virtual texture' lark before it's ready!!

Yours...a bit confused..but very interested...bh!

It's not more confused to you than it is for me. I'm trying to catch birds with my hands on this one

However, on what concernes you not being able to see the screenshots, it's something I relate to. You do understand you are the only one who report this problem, do you? The screenshots are there. The image placeholders are to the left of the text and when you click these, a new browser window will open reveiling a 1024x768 jpg image.

The page is straight HTML. No fancy bits in it. No java or anything. No crazy proprietary tags. More, the html code even runs on html version 3 browsers.

Mario

[Darkmiss]

Just a quick question, as im still new to virtual textures.

can you use just the level 5 32k set ?
and have a standard single texture for zoomed out views

I thinks thats how it works
If i had, say a 16k standard texture, and when i get close enought to the surface.
the 32k virtual tile would pop up over the top of my 16k single ?

Is this correct ?

[Mario]

The individual resolution sets are now available for download. This cuts the download into more manageable sets.

Only missing the landmasses tileset for the level 5 texture. Already working on it. It's a somewhat lenghty process finding and stripping all the ocean tiles among a 2,048 tileset. So please be patient. I promise I'll make it available still today.

Mario

[wcomer]

Hi all,

Some thoughts on how to reduce the size. There seem to be two issues. The most prominent issue is that the file is too large too download on a dialup. The second issue is that it takes up too much hard drive space when unzipped. I'll address these separately.

1) Splitting the world into a 2x4 grid would be easily manageable for me. It would reduce the downloads to chunks of 12MB each. I suspect this would be manageable on Mario’s end as well. If we base the grid on level2 4x8 then we have 32 packages of 3MB each; however it is becoming unmanageable if done manually. I can automate the construction of these packages down to any level but most casual PC users are not going to recognize a .tar file extension (can an unregistered version of WinZip be used from the dos or Cygwin command line? If so then this isn't really a problem.) My biggest concern with breaking the package up is that in its current form it is very simple to install for new users. If I break it up, then new users will make more mistakes in installation. I'm looking for some kind of consensus amongst the dialup folks as to whether 12MB chunks is an acceptable compromise, if so then I can get this up before the weekend.

2) Currently 925 (about 33%) of the DDS files are empty water. These can easily be removed from the zip file before extraction by sorting according to compression and deleting all the 257 byte files. After this small atolls start to appear. There is really no reason to distribute these 925 files at all but they only add 237kB to the entire zip file. Removing them reduces the extracted file space by 119MB. Now obviously 33% is not very close to the ideal figure of 75% water area. One way to reduce the installation size further would be to reduce the tile size from 512x512 to say 128x128 (a level7 installation.) I would be very surprised if this didn't reduce the installation size by at least another 100MB. We may be able to get the installation to around 120MB or so. What would be useful is if Chris (or someone equally knowledgeable) could comment on whether this would substantially reduce the performance of Celestia or cause major file system or caching problems (we are talking about 43,690 files at level7 of which at least 15,000 would contain land data and have to be installed.) If I get time this weekend I’ll compare level6 and level7 versions to see is there is any appreciable difference between them in minimum installation size (there may not be.) As a final note, while 75% of the earth's surface is water, that is probably not be true on a cylindrical projection due to the asymmetric distribution of landmasses (I would ballpark 66%.)

cheers,
Walton

[selden]

Having a large number of files in the same directory will cause filesystem performance problems. Is that vague enough?

For NTFS (the filesystem normally used for XP) one cause of slowdowns is the list of 8.3 filenames. Updating "last access time" also slows things. For a discussion of these features and how to disable them, see http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/winxppro/reskit/prkc_fil_punq.asp (They mention problems when there are more than 300,000 files, but performance degrades long before that.)

FAT-16 and FAT-32 filesystems are limited to 2^16 -2 (64K) files per directory. See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/supported_file_systems.asp

I haven't seen any benchmarks that mention the number of files in directories. The only ones I've seen generally show NTFS to be 10-15% faster than either of the FAT systems and with less CPU overhead. See http://www.storagereview.com/articles/9807/980721win98.html for one example.

A related issue is that the size of Celestia grows as more tiles are loaded. If it gets too big, it'll page, which will hurt performance seriously. If you're going to be looking at a lot of the surface with high resolution, it's best to get rid of all the other programs and shared libraries that might be sitting in main memory. Unfortunately, rebooting is the most reliable way of eliminating the unnecessary stuff, but closing all the other windows and eliminating any backdrop image can help.

[bh]

Mario...should be getting my hands on this one in a couple of hours from now. Screen shots look superb...sorry I was looking elsewhere...I'm losing it I think.

Regards...bh.

[wcomer]

Great links Selden!

It looks like from the file system perspective we will be OK with FAT32 and NTFS. If we do a level7 directory, it would have 32k files as a full installations. But using a minimum installation where we drop all the redundant water tiles this would be more like 12k files; which should not cause a performance hit from what I've read.

Why does Celestia grow with the number of tiles? The tile size decreases in proportion to the number of tiles shown. So in theory there is no difference between looking at the earth's surface from 800km using 128x128 level7 tiles vs 512x512 level5 tiles vs 2048x2048 level3 tiles. Whereas in practice, I would suspect that the 2048 tiles would require more memory because with the smaller granularity of smaller tiles you don't need as much unused graphics loaded into memory that are falling off the edge of the screen or the horizon. Likely there is another effect taking place that I'm unfamiliar with, which you had in mind.

This definitely requires more study.

cheers,
Walton

[wcomer]

I suppose that this same method for dropping the redundant tiles can be used for all the earth tilesets that may eventually be provided by various sources. For example, the 30k nightmap .gif from bluemarble can be used to create another tileset of which the overwhelming majority of level7 tiles would be redundant (black) and would be easy to sort out using their zip compressed size. I imagine the minimal installation size would drop by 90+% for the nightmap .dds files. This is very similar to the quad-tree compression algorythm but aborted at 128x128 block sizes. The normal maps (66%), specular maps (95+%) and cloudmaps (hopefully) would see improvements from this method as well.

Walton

[Guest]

Darkmiss,
The level5 set cannot be used on its own. A virtual texture is a self contained texture and cannot be 'called' from another texture(eg. earth16k.dds). Your 16k texure must be tiled aswell and put in level4 of the virtual texture folder. also you must have tiles for level0 or nothing will be visible until you get to the altitude where the 16k set will be visible(about 2000km). Without the itermediate levels it will be very blurred between when the level0 tiles go out of focus and the level4 16k tiles kick in.

DBrady

Sl?n

[Mario]

I'm currently sorting out the ocean tiles from the dds version. I'm doing this manually by looking at the JPG version and annotating those tiles with their size above 3.359 bytes (the one I found to be solely water).

From what I've been doing so far (had to leave for a few hours so I'm only at 1/3 yet) the final level 5 compressed zip will be much smaller than the current 66Mb file. I would say somewhere around 25MB. It's still a drain on a dialup. But the server supports resume so I don't think it will be too much of a problem anymore.

Mario

[selden]

Why does Celestia grow with the number of tiles?

Are you asking why I mentioned that?

I just meant "the obvious". (Obvious to me, anyhow )

As you change altitudes, the previously loaded tiles are not discarded. For example, if the earth is rotating beneath you and you change altitudes in various ways, eventually all of the tiles at all resolutions could, in principle, be loaded into memory. I noticed this while playing with Grant's political boundary tiles. Celestia kept growing and then started paging.

The tile size decreases in proportion to the number of tiles shown. So in theory there is no difference between looking at the earth's surface from 800km using 128x128 level7 tiles vs 512x512 level5 tiles vs 2048x2048 level3 tiles.

I'm a little confused. I haven't downloaded the BlueMarble tile set. Are you saying that the tiles you're providing are different sizes at the various levels? I would have expected the individual tiles to be the same size at all levels.

I'm not sure what the optimum would be, but I'd expect an appropriate tile size to be that which would produce about a 1:1 ratio between the sizes of image pixels and screen pixels when the viewpoint is at the lowest altitude used for a given level. (I haven't tried to figure out exactly what that size is. I don't recall what Chris said about how Celestia chooses to go to the next level.) People do run Celestia at different resolutions, of course, so picking the right size is somewhat of a judgement call. You'd need larger tiles (I mean images with more pixels) at a particular level for higher resolution displays.

[jim]

Thank you Mario!

My computer worked last night 8 hours (while I was sleeping) to upload with 3,3 kbyte per seconde (dial up) this nice texture.

Bye Jens

[wcomer]

As you change altitudes, the previously loaded tiles are not discarded. For example, if the earth is rotating beneath you and you change altitudes in various ways, eventually all of the tiles at all resolutions could, in principle, be loaded into memory. I noticed this while playing with Grant's political boundary tiles. Celestia kept growing and then started paging.

This is true but on paper this is a relatively small effect for tiles of size 4096 and smaller. In practice I suspect you would see the oposite trend because you wouldn't load as much unneccessary graphics when using smaller tiles sizes and more importantly because you would lose the redundant water tiles.

By my latest calculations. The entire tileset less the redundant water tiles (-dxt1c .dds) will weigh in as follows:

Code: Select all

Width   Storage
16,384   262 MB
8,192   328 MB
4,096   344 MB
2,048   334 MB
1,024   286 MB
512   231 MB
256   173 MB
128   114 MB


The memory in Celestia required to store the graphics will be in proportion. Without the water effect that would have been:

Code: Select all

Width   Storage
16,384   262 MB
8,192   328 MB
4,096   344 MB
2,048   348 MB
1,024   349 MB
512   349 MB
256   349 MB
128   349 MB


So really the required cached memory is not going to get worse as we use smaller tilesizes

[t00fri]

Walton,

could you just remind me what the tile size of your 32k tileset is? I could not find this important specification on the server where your tiles are for download.

Is it 512x512?

I would like to try such small tiles with my new card, but can barely afford to download the 90MB at such low speed as ~ 110KB/sec (for my standards at least). This badly loads our network for about 15 mins, which I can hardly justify...(at home, I only have 3-4kb/sec;-))

Bye Fridger

[Mario]

It's 512x512 indeed, Fridger.

You can try and download by chunks and leave it downloading overnight. The server supports resume. But meanwhile Walton is working on a smaller size download. It won't be a big improvement but every bit is always good. I will make it available as soon as he finishes it.

Mario