Celestia Forums

Hi all,

my tiling script virtualtex (1.02) has worked through much of the weekend like a work horse;-). I now have a complete Earth installation with 2k tiles up to 32k effective resolution! Exception: virtualtex is still grinding the great 1km/pixel (48k), 16bit grayscale GLOBE-DEM elevation data, so that for now my normal map resolution is still a "low" 16k;-).

The best of all is that virtualtex has recently been the cause for an increased rate of visits at our favorite Chinese place. Note, Hamburg being a big harbour, has lots of most interesting asian food places...

During level3 grinding of virtualtex, for example, we had this excellent meal;-)

-- Appetizer: Dim Sam, steamed balls of marinated pork meat with shrimps;
-- hot pork with mustard vegetable;
-- Satay, an Indonesian national dish with peanut sauce that is also adored by Chinese notably in in the Singapore area...

After returning from this great meal, here is what I got:

Please note that with this amazing resolution, my antique graphics card, GeForce 2 GTS/32MB performs with ~18 fps below, 1600x1200px /24bit !!

Bye Fridger



PS: The problem now is, however, that these tiles are by far too large for download by others (~ 1GB just for Earth!!). Hence everyone has to do them by him/herself! I am working on detailed instruction sets/further scripts, though....

How large, exactly, are these tiles in total?

Mikeydude750 wrote:How large, exactly, are these tiles in total?

OK, here is the detailed account, all my tiles are 2kx2k:



If The GLOBE Data are finished at level3, they will push the normal-map tiles up to > 500 MB, hence, in total we talk about >1GB of DXT tiles, uncompressed....just for Earth.

After Earth, we got to do Mars, the Moon, Europa etc...
I shall never go back to classical textures after this experience;-). Today I cleaned up my archive and deleted many of my single texture creations....New horizons ahead!;-)

On the other hand, much of this stuff is generated completely automatically, once you have the right scripts and the original data (which are also of huuuuuge size). The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

Bye Fridger

Once (just a few months ago) there was another project called the "Blue Marble Viewer". For some reason, the author was having trouble getting funding from NASA to finish it.

I think now we know why...

I wonder if it's too soon to start thinking about distributing high-resolution data files for Celestia on CD-ROM or DVD?

- Hank

I shall never go back to classical textures after this experience;-).

I hope they'll still be available, though, because a lot of them are worthy of use in other programs like Orbiter - for which a specialised format will be useless.

The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

What an ignorant (and elitist) thing to say! Some of our most powerful, intelligent and productive tools here at work use a Windows GUI... the choice of batch script or GUI depends hugely on the problem at hand - and the expertise of the end-user.

Cheers,
Paul

Fridger, your screen shot looks absolutely beautiful!

HankR wrote:I wonder if it's too soon to start thinking about distributing high-resolution data files for Celestia on CD-ROM or DVD?

From the sound of these file sizes, a multi-DVD "pack" would be necessary. A CD only holds up to 700 MB, where a DVD stores up to 4.7 GB.

Fridger, have you tried any of the file compression programs on these files to find out what level of compression can be achieved?

-Don G.

Paul wrote:

I shall never go back to classical textures after this experience;-).

I hope they'll still be available, though, because a lot of them are worthy of use in other programs like Orbiter - for which a specialised format will be useless.

Some, but I had to do substantial cleaning, else ->overflow;-)

The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

What an ignorant (and elitist) thing to say! Some of our most powerful, intelligent and productive tools here at work use a Windows GUI... the choice of batch script or GUI depends hugely on the problem at hand - and the expertise of the end-user.

Cheers,
Paul

When analyzing the wording of my posts critically (as usual), it would be nice, if from time to time you could give me a little bit of credit with respect to the fact that English is not my mother tongue.

Of course, what is meant above is not that the times of
Windows GUI's are generally over, but rather that in this specific application of producing many tiles from huge textures, the GUI method (clicking) has reached its definite limitations!

Not so hard to find out what was really meant I think, before calling me ignorant and elitist, don't you think so?

Bye Fridger

well i use winXP -- I am stuck with win because there some windows programs I NEED -- and have been slowly going more and more to cwgwin
there are some things about XP i like ,the look and gui mostly . and cygwin i am getting more exparence with and i like it

what I want is the best of both win and linux

Post edited because it has become useless drible.
I misunderstood a few things about the virtual textures and have since changed my mind on the subject.

But I still will not be creating any in the future.



Don.

Don,

Don't forget that the tiling facility does not necessarily use more disk space than the full-resolution textures. In fact, it can use much less, if you only install the high resolution images for the areas that you're personally interested in -- like looking into your home's windows from orbit or your favorite crevice in Valles Marineris.

This does suggest that the tiles need to be made available individually, not just as large sets that cover the entire surface of a planet. It also suggests that it'd be nice to have a utility that could slice and dice the Martian surveyor pictures so that the resulting images correspond to appropriate tiles in Celestia.

Hi Don,

Don. Edwards wrote:I don't think any normal Celestia user is going to willing to give up the ability to just point and click with the mouse and grab any planet and manhandle it the way we want. Scripting is fine if you want to go to a specific location but it will never replace the GUI or the mouse. If Celestia went in that direction I would be out of here faster than you can say "bad idea". If this is what we have to expect from the hi-res tiled textures than I can be counted out and I will not be participating in any texture creation that will preclude the use of the most important input device for Celestia. I was seriously considering to start to work on a 32k level texture tile package but if it is going to be bringing Celestia's GUI to its knees than I will not even start the work. I am more than happy with a 16k texture.

Are you perhaps misunderstanding something here? Nobody for sure would want to argue that any of the great GUI features of Celestia or Windows should be abolished!

The whole issue is that the specific task of tile mass production from super hires 32k monster textures involves doing the same operations of selecting, cutting out and correctly numbering tiles many many many times. Thus, this is the ideal application of scripting tools. Fabrication of these tiles by using the GUI e.g. in Photoshop or GIMP is simply out-of-the question. The ImageMackick image manipulation package was concerned with this aspect of batch controlled mass-manipulations on images from the beginning, i.e. since more than 10 years. Now these tools become most useful in the context of tiling, notably if controlled by a powerful unix-type shell, like 'zsh' or 'bash' or by a Perl API.

If you are not interested in 32k resolution, you can ignore all these issues, of course. No changes whatsoever.

However, after I and similarly various other users had installed the first 16k or 32k tiles, the effect was simply so breathtaking that I hardly can imagine going back to the 'classical' single texture scheme.

Clearly, in my case, given my 32MB (!!) oldtimer card, the effect of tiling is particularly gratifying!

We should tackle jointly the
problem of how to handle those large amounts of data for the benefit of other users. Downloading seems to be simply out. CD/DVD is an alternative, of course, but not quite obvious how to handle on a global scale...

Personally, I think that working together on a set of scripts for Cygwin/Linux, would make the production and installation sufficiently easy for most/many interested users.

Quite a few have succeeded already, it seems....

this is a space simulator not an Earth simulator as several members of this board have stated in the past.

No-one involved in tiles would want to see tiles confined to Earth! The great thing about them is that parts of textures with different resolution can be blended in the tiling process. Chris is going to further improve the smoothness of this blending.

Think of Europa for example. There is amazing 8k data, but not for the full surface. You may just use the tiles from those 8k data where available and blend them with the usual 2k or 4k ones. Mars is another great application for tiles. 16k Moon textures etc....

===>
People with older graphics cards can now run 8k, 16k textures at high fps rates. I think this is worth some effort, already!
===>

I would be much more receptive to being able to use 8k clouds maps than to use a 32k texture even in tiled mode. As you get that close to the surface the clouds look absolutely horrible at that resolution. At least an 8k cloudmap could extend the detail down farther.

I still use my 'classic' 2k clouds;-) Sometimes they indeed look rather 'digital'. I agree. But often, they are also still surprisingly good, even with 32k resolution.

Note: clouds tend to cover the surface up, hires surface textures emphasize the surface details and clouds are in the way, be it 2k or 8k;-) I am often tempted to switch the clouds off now and explore that amazing surface detail...But perhaps other people prefer to look at detailed clouds instead...

Bye Fridger

One of the great joys of Celestia is the ability to create your own textures, worlds etc etc ad infinitum.

While the virtual texture concept does sound excellent (and V nice for those with older cards ), so long as we can still use our old fashioned one-shot, single image textures, I think everyone will be happy.

I must have a go at the virtual texture thing at some point, but as Fridger says, creating them in something like photoshop sounds very daunting, if not impossible.

Maybe one of the extremely clever people in here will create a tool to help create such textures.

Personally I'm all for a Celestia Toolbox...apps not part of Celestia that are merely helpmates in the act of creation...we have some already, such as Rassilon's Cluster Generator, and Marc's (sadly unusable with higher versions of Celestia) planetary system generator. I'm working on a few ideas myself, but I'll keep them just for me unless there is a demand.

Maybe I should start a new thread on the Toolbox idea

Cheers,

Cormoran

Cormoran wrote:
...

Maybe one of the extremely clever people in here will create a tool to help create such textures.

...

Cheers,
Cormoran

I am confused what you mean, or did you miss my thread on Auto-tiling by means of my recent script virtualtex?

Virtualtex is a very simple but powerful and convenient command line tool that has meanwhile generated many Gigabytes of texture tiles.


Bye Fridger

t00fri wrote:The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

The scripting is not the problem I wrote my own (DOS) scrip that convert every 8k texture to a virtual texture with 1k tiles. The real problem is that I can't edit large DDS texture (>8k). In any other case I have no problems with Windows. I can handle with Photoshop even 21k textures.

My system: Duron 900, 512mb ddr ram, GF3 64mb, Win98SE

Bye Jens

jim wrote:

t00fri wrote:The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

The scripting is not the problem I wrote my own (DOS) scrip that convert every 8k texture to a virtual texture with 1k tiles. The real problem is that I can't edit large DDS texture (>8k). In any other case I have no problems with Windows. I can handle with Photoshop even 21k textures.

My system: Duron 900, 512mb ddr ram, GF3 64mb, Win98SE

Bye Jens

I am sorry, Jens, my above sentence concerning Windows was badly worded, since I was in a hurry this morning. Sure, I did not want to argue against Windows in general, but only point out that the task of mass production of tiles is a matter for scripting rather than a GUI application.

I am surprised that you cannot edit larger textures than 8k. You have exactly the same amount of RAM as I have. But when the ImageMagick tools are employed for 'batch'-editing, the virtual memory on the HD is mainly used and the texture is not loaded into the RAM!

So for instance, I can edit those gigantic 21k x 21k BM source files of sizes up to 440MB on my system with the ImageMagick tools. No problems whatsoever. Yes, it takes time, since many operations use the HD instead of your RAM. That's why I diverted on Chinese recipees above;-)...

If for example, I want to increase the brightness by 20%, decrease the saturation by 50% and shift the hue by 20% to the left, I run this statement in the shell (==> Cygwin!)

mogrify -modulate 120,50,80 <texture[s]>

This way you can make exactly the same changes to the 128 tiles in the level3 directory:

mogrify -modulate 120,50,80 *.tga


Bye Fridger

Sorry Fridger, my fault (its been a hard day and I have to go work until 8am now).

I'll be looking into that thread in depth if and when I start working on the virtual texture thing.

I still think my other comment re: a Celestia toolbox suite stands. I'm talking about little apps, such as Asteroid belt generators, planet creators etc, and any other stuff anyone can think of.

Cheers,

Cormoran

Fridger wrote: But using ImageMagick tools to do the editing, the virtual memory on the HD is mainly used

The NetPBM utilities have the advantage that most of them don't even use virtual memory. Whenever possible, they operate on a single scanline at a time.

I suspect that by "editing," Comoran means "visually fix local problems in the image" rather than "apply color corrections to the entire image." Such things can be painfully slow while one waits for the paint program to page various parts of the picture to and from the disk. The same slowness due to paging happens with nvdxt. This makes it rather tedious to make several iterations of change/view/change/view at high resolutions. (By "view," I mean "Study how it looks in Celestia.")

Unfortunately, tiling is not yet available for Nebula textures.

t00fri wrote:I am surprised that you cannot edit larger textures than 8k. You have exactly the same amount of RAM as I have. But when the ImageMagick tools are employed for 'batch'-editing, the virtual memory on the HD is mainly used and the texture is not loaded into the RAM!

Sorry. I can't see that ImageMagick supports the DDS format.
http://www.imagemagick.org/www/formats.html

Selden where can I find a 'exe-version' of NetPBM utilities?

Bye Jens

Don. Edwards wrote:This is a space simulator not an Earth simulator as several members of this board have stated in the past. If you want to take Celestia in that direction than fork the project into two separate programs. One as an "Earth" simulator that can take you down to the most detailed surface images and one that returns us to a more pure "Space Simulator" witch was what Celestia is all about. I think we need to get realistic here. Good 16k textures are just now becoming common place and we are stressing are video cards quite nicely now. From everything I am hearing so far this is just going to slow the process down more. I frankly would prefer to see more in the way of Astronomical features added versus seeing the need to see city streets on the surface of the Earth.

Here are just a few dissenting reasons.
1. The files are just going to be unrealistically too large to download.
2. We are looking at the need of possible DVD media for storage and DVD drives are not on all Celestia users’ computers.
3. The use of multiple CDs for storage of the textures in place of DVD.
4. The acceptable slowdown of Celestia responsiveness to input.
In the end I can see that the large texture tiling scheme is going to cost users for media and also make some of the less experienced uses feel they need to move up to much more expensive video hardware when they really don’t have to. You have proved that time and time again.
So until I can test a large texture pack myself you will find me staunchly in the anti-texture tiling camp if there is one. And if there isn't than I will be more than happy to be the only member. I am just hoping that everyone doesn't get carried away by all the glimmer of these super hi-res textures because for most users it will be nothing but a pipe dream.
That’s my opinion on the subject and no one has to agree with me. I know my opinion carries very little weight with the project but I would hope for a sanity check and a simple return to what Celestia was supposed to be at its roots. The best free “SPACE SIMULATOR” available.

I really don't understand the problem . . . For one thing, virtual textures don't slow Celestia down--as Fridger's case shows, they make it possible to display very high-resolution textures even on older hardware. Also, virtual textures are a feature still very much in development. Currently, there's a great deal of work involved in the creation of virtual textures, and a means of distributing them has to be worked out. I see the variable resolution of virtual textures as very important--you can download high resolution data for just the particular regions in which you're interested.

I see virtual textures as an important part of making Celestia the best space simulator around (whether free or commercial) The variable resolution will allow the use of a great deal of high resolution planetary images--they can be placed in context on the appropriate body amid lower resolution data. Not for everyone perhaps, but there's nothing here that threatens Celestia's established purpose as a space simulator. In fact, I'm quite wary about introducing features which would do that.

Virtual textures integrate seamlessly with the rest of Celestia--they work generally as alternate surfaces, specular maps, night maps, etc. (though as Selden mentioned, not yet as nebula textures.) More significant is that there was absolutely no UI change involved with the introduction of virtual textures into Celestia. The fact that the existing UI accomodates the addition of super high res tiled textures so well should make it absolutely clear that I've in no way changed the direction of Celestia with this new feature.

--Chris