32k Resolution, 'Virtualtex' and Dim Sam;-)

General discussion about Celestia that doesn't fit into other forums.
Avatar
Topic author
t00fri
Developer
Posts: 8772
Joined: 29.03.2002
Age: 22
With us: 22 years 8 months
Location: Hamburg, Germany

32k Resolution, 'Virtualtex' and Dim Sam;-)

Post #1by t00fri » 17.08.2003, 18:29

Hi all,

my tiling script virtualtex (1.02) has worked through much of the weekend like a work horse;-). I now have a complete Earth installation with 2k tiles up to 32k effective resolution! Exception: virtualtex is still grinding the great 1km/pixel (48k), 16bit grayscale GLOBE-DEM elevation data, so that for now my normal map resolution is still a "low" 16k;-).

The best of all is that virtualtex has recently been the cause for an increased rate of visits at our favorite Chinese place. Note, Hamburg being a big harbour, has lots of most interesting asian food places...

During level3 grinding of virtualtex, for example, we had this excellent meal;-)

-- Appetizer: Dim Sam, steamed balls of marinated pork meat with shrimps;
-- hot pork with mustard vegetable;
-- Satay, an Indonesian national dish with peanut sauce that is also adored by Chinese notably in in the Singapore area...

After returning from this great meal, here is what I got:

Please note that with this amazing resolution, my antique graphics card, GeForce 2 GTS/32MB performs with ~18 fps below, 1600x1200px /24bit !!

Bye Fridger

Image

PS: The problem now is, however, that these tiles are by far too large for download by others (~ 1GB just for Earth!!). Hence everyone has to do them by him/herself! I am working on detailed instruction sets/further scripts, though....
Last edited by t00fri on 17.08.2003, 19:45, edited 3 times in total.

Mikeydude750
Posts: 169
Joined: 31.01.2002
With us: 22 years 10 months
Location: Wisconsin

Post #2by Mikeydude750 » 17.08.2003, 18:47

How large, exactly, are these tiles in total?

Avatar
Topic author
t00fri
Developer
Posts: 8772
Joined: 29.03.2002
Age: 22
With us: 22 years 8 months
Location: Hamburg, Germany

Post #3by t00fri » 17.08.2003, 19:32

Mikeydude750 wrote:How large, exactly, are these tiles in total?


OK, here is the detailed account, all my tiles are 2kx2k:

Code: Select all

EarthTiles (DXT1c), 32K, level0..3:       339  MB
EarthNightTiles (DXT3), 16K, level0..2:   230  MB
EarthNormalTiles (DXT3, 16K, level0..2:   230  MB
-------------------------------------------------------


If The GLOBE Data are finished at level3, they will push the normal-map tiles up to > 500 MB, hence, in total we talk about >1GB of DXT tiles, uncompressed....just for Earth.

After Earth, we got to do Mars, the Moon, Europa etc...
I shall never go back to classical textures after this experience;-). Today I cleaned up my archive and deleted many of my single texture creations....New horizons ahead!;-)

On the other hand, much of this stuff is generated completely automatically, once you have the right scripts and the original data (which are also of huuuuuge size). The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

Bye Fridger

Avatar
selden
Developer
Posts: 10192
Joined: 04.09.2002
With us: 22 years 3 months
Location: NY, USA

Post #4by selden » 17.08.2003, 20:42

Once (just a few months ago) there was another project called the "Blue Marble Viewer". For some reason, the author was having trouble getting funding from NASA to finish it.

I think now we know why...
Selden

HankR

Post #5by HankR » 18.08.2003, 03:03

I wonder if it's too soon to start thinking about distributing high-resolution data files for Celestia on CD-ROM or DVD?

- Hank

Paul
Posts: 152
Joined: 13.02.2002
With us: 22 years 10 months
Location: Melbourne, Australia

Post #6by Paul » 18.08.2003, 04:06

I shall never go back to classical textures after this experience;-).

I hope they'll still be available, though, because a lot of them are worthy of use in other programs like Orbiter - for which a specialised format will be useless.

The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...


What an ignorant (and elitist) thing to say! Some of our most powerful, intelligent and productive tools here at work use a Windows GUI... the choice of batch script or GUI depends hugely on the problem at hand - and the expertise of the end-user.

Cheers,
Paul

don
Posts: 1709
Joined: 12.07.2003
With us: 21 years 5 months
Location: Colorado, USA (7000 ft)

Post #7by don » 18.08.2003, 05:13

Fridger, your screen shot looks absolutely beautiful!

HankR wrote:I wonder if it's too soon to start thinking about distributing high-resolution data files for Celestia on CD-ROM or DVD?

From the sound of these file sizes, a multi-DVD "pack" would be necessary. A CD only holds up to 700 MB, where a DVD stores up to 4.7 GB.

Fridger, have you tried any of the file compression programs on these files to find out what level of compression can be achieved?

-Don G.

Avatar
Topic author
t00fri
Developer
Posts: 8772
Joined: 29.03.2002
Age: 22
With us: 22 years 8 months
Location: Hamburg, Germany

Post #8by t00fri » 18.08.2003, 07:36

Paul wrote:
I shall never go back to classical textures after this experience;-).

I hope they'll still be available, though, because a lot of them are worthy of use in other programs like Orbiter - for which a specialised format will be useless.

Some, but I had to do substantial cleaning, else ->overflow;-)
The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...


What an ignorant (and elitist) thing to say! Some of our most powerful, intelligent and productive tools here at work use a Windows GUI... the choice of batch script or GUI depends hugely on the problem at hand - and the expertise of the end-user.

Cheers,
Paul


When analyzing the wording of my posts critically (as usual), it would be nice, if from time to time you could give me a little bit of credit with respect to the fact that English is not my mother tongue.

Of course, what is meant above is not that the times of
Windows GUI's are generally over, but rather that in this specific application of producing many tiles from huge textures, the GUI method (clicking) has reached its definite limitations!

Not so hard to find out what was really meant I think, before calling me ignorant and elitist, don't you think so?

Bye Fridger
Last edited by t00fri on 18.08.2003, 20:35, edited 1 time in total.

Avatar
John Van Vliet
Posts: 2944
Joined: 28.08.2002
With us: 22 years 3 months

re

Post #9by John Van Vliet » 18.08.2003, 09:23

well i use winXP -- I am stuck with win because there some windows programs I NEED -- and have been slowly going more and more to cwgwin
there are some things about XP i like ,the look and gui mostly . and cygwin i am getting more exparence with and i like it

what I want is the best of both win and linux

Don. Edwards
Posts: 1510
Joined: 07.09.2002
Age: 59
With us: 22 years 3 months
Location: Albany, Oregon

Post #10by Don. Edwards » 18.08.2003, 09:36

Post edited because it has become useless drible.
I misunderstood a few things about the virtual textures and have since changed my mind on the subject.

But I still will not be creating any in the future.



Don.
Last edited by Don. Edwards on 06.09.2003, 12:23, edited 1 time in total.
I am officially a retired member.
I might answer a PM or a post if its relevant to something.

Ah, never say never!!
Past texture releases, Hmm let me think about it

Thanks for your understanding.

Avatar
selden
Developer
Posts: 10192
Joined: 04.09.2002
With us: 22 years 3 months
Location: NY, USA

Post #11by selden » 18.08.2003, 14:32

Don,

Don't forget that the tiling facility does not necessarily use more disk space than the full-resolution textures. In fact, it can use much less, if you only install the high resolution images for the areas that you're personally interested in -- like looking into your home's windows from orbit :) or your favorite crevice in Valles Marineris.

This does suggest that the tiles need to be made available individually, not just as large sets that cover the entire surface of a planet. It also suggests that it'd be nice to have a utility that could slice and dice the Martian surveyor pictures so that the resulting images correspond to appropriate tiles in Celestia.
Selden

Avatar
Topic author
t00fri
Developer
Posts: 8772
Joined: 29.03.2002
Age: 22
With us: 22 years 8 months
Location: Hamburg, Germany

Post #12by t00fri » 18.08.2003, 16:15

Hi Don,

Don. Edwards wrote:I don't think any normal Celestia user is going to willing to give up the ability to just point and click with the mouse and grab any planet and manhandle it the way we want. Scripting is fine if you want to go to a specific location but it will never replace the GUI or the mouse. If Celestia went in that direction I would be out of here faster than you can say "bad idea". If this is what we have to expect from the hi-res tiled textures than I can be counted out and I will not be participating in any texture creation that will preclude the use of the most important input device for Celestia. I was seriously considering to start to work on a 32k level texture tile package but if it is going to be bringing Celestia's GUI to its knees than I will not even start the work. I am more than happy with a 16k texture.

Are you perhaps misunderstanding something here? Nobody for sure would want to argue that any of the great GUI features of Celestia or Windows should be abolished!

The whole issue is that the specific task of tile mass production from super hires 32k monster textures involves doing the same operations of selecting, cutting out and correctly numbering tiles many many many times. Thus, this is the ideal application of scripting tools. Fabrication of these tiles by using the GUI e.g. in Photoshop or GIMP is simply out-of-the question. The ImageMackick image manipulation package was concerned with this aspect of batch controlled mass-manipulations on images from the beginning, i.e. since more than 10 years. Now these tools become most useful in the context of tiling, notably if controlled by a powerful unix-type shell, like 'zsh' or 'bash' or by a Perl API.

If you are not interested in 32k resolution, you can ignore all these issues, of course. No changes whatsoever.

However, after I and similarly various other users had installed the first 16k or 32k tiles, the effect was simply so breathtaking that I hardly can imagine going back to the 'classical' single texture scheme.

Clearly, in my case, given my 32MB (!!) oldtimer card, the effect of tiling is particularly gratifying!

We should tackle jointly the
problem of how to handle those large amounts of data for the benefit of other users. Downloading seems to be simply out. CD/DVD is an alternative, of course, but not quite obvious how to handle on a global scale...

Personally, I think that working together on a set of scripts for Cygwin/Linux, would make the production and installation sufficiently easy for most/many interested users.

Quite a few have succeeded already, it seems....

this is a space simulator not an Earth simulator as several members of this board have stated in the past.

No-one involved in tiles would want to see tiles confined to Earth! The great thing about them is that parts of textures with different resolution can be blended in the tiling process. Chris is going to further improve the smoothness of this blending.

Think of Europa for example. There is amazing 8k data, but not for the full surface. You may just use the tiles from those 8k data where available and blend them with the usual 2k or 4k ones. Mars is another great application for tiles. 16k Moon textures etc....

===>
People with older graphics cards can now run 8k, 16k textures at high fps rates. I think this is worth some effort, already!
===>

I would be much more receptive to being able to use 8k clouds maps than to use a 32k texture even in tiled mode. As you get that close to the surface the clouds look absolutely horrible at that resolution. At least an 8k cloudmap could extend the detail down farther.


I still use my 'classic' 2k clouds;-) Sometimes they indeed look rather 'digital'. I agree. But often, they are also still surprisingly good, even with 32k resolution.

Note: clouds tend to cover the surface up, hires surface textures emphasize the surface details and clouds are in the way, be it 2k or 8k;-) I am often tempted to switch the clouds off now and explore that amazing surface detail...But perhaps other people prefer to look at detailed clouds instead...

Bye Fridger
Last edited by t00fri on 18.08.2003, 18:09, edited 1 time in total.

Cormoran
Posts: 198
Joined: 28.07.2003
With us: 21 years 4 months
Location: Slartibartfast's Shed, London

Post #13by Cormoran » 18.08.2003, 17:50

One of the great joys of Celestia is the ability to create your own textures, worlds etc etc ad infinitum.

While the virtual texture concept does sound excellent (and V nice for those with older cards :) ), so long as we can still use our old fashioned one-shot, single image textures, I think everyone will be happy.

I must have a go at the virtual texture thing at some point, but as Fridger says, creating them in something like photoshop sounds very daunting, if not impossible.

Maybe one of the extremely clever people in here will create a tool to help create such textures.

Personally I'm all for a Celestia Toolbox...apps not part of Celestia that are merely helpmates in the act of creation...we have some already, such as Rassilon's Cluster Generator, and Marc's (sadly unusable with higher versions of Celestia) planetary system generator. I'm working on a few ideas myself, but I'll keep them just for me unless there is a demand.

Maybe I should start a new thread on the Toolbox idea :)

Cheers,

Cormoran
'...Gold planets, Platinum Planets, Soft rubber planets with lots of earthquakes....' The HitchHikers Guide to the Galaxy, Page 634784, Section 5a. Entry: Magrathea

Avatar
Topic author
t00fri
Developer
Posts: 8772
Joined: 29.03.2002
Age: 22
With us: 22 years 8 months
Location: Hamburg, Germany

Post #14by t00fri » 18.08.2003, 18:04

Cormoran wrote:
...

Maybe one of the extremely clever people in here will create a tool to help create such textures.

...

Cheers,
Cormoran


I am confused what you mean, or did you miss my thread on Auto-tiling by means of my recent script virtualtex?

Virtualtex is a very simple but powerful and convenient command line tool that has meanwhile generated many Gigabytes of texture tiles.


Bye Fridger

jim
Posts: 378
Joined: 14.01.2003
With us: 21 years 11 months
Location: Germany

Post #15by jim » 18.08.2003, 18:05

t00fri wrote:The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...

The scripting is not the problem I wrote my own (DOS) scrip that convert every 8k texture to a virtual texture with 1k tiles. The real problem is that I can't edit large DDS texture (>8k). In any other case I have no problems with Windows. I can handle with Photoshop even 21k textures.

My system: Duron 900, 512mb ddr ram, GF3 64mb, Win98SE

Bye Jens

Avatar
Topic author
t00fri
Developer
Posts: 8772
Joined: 29.03.2002
Age: 22
With us: 22 years 8 months
Location: Hamburg, Germany

Post #16by t00fri » 18.08.2003, 18:13

jim wrote:
t00fri wrote:The Windows fans must reallize that the clicking times are out now, powerful/intelligent batch scripting is in;-)...
The scripting is not the problem I wrote my own (DOS) scrip that convert every 8k texture to a virtual texture with 1k tiles. The real problem is that I can't edit large DDS texture (>8k). In any other case I have no problems with Windows. I can handle with Photoshop even 21k textures.

My system: Duron 900, 512mb ddr ram, GF3 64mb, Win98SE

Bye Jens


I am sorry, Jens, my above sentence concerning Windows was badly worded, since I was in a hurry this morning. Sure, I did not want to argue against Windows in general, but only point out that the task of mass production of tiles is a matter for scripting rather than a GUI application.

I am surprised that you cannot edit larger textures than 8k. You have exactly the same amount of RAM as I have. But when the ImageMagick tools are employed for 'batch'-editing, the virtual memory on the HD is mainly used and the texture is not loaded into the RAM!

So for instance, I can edit those gigantic 21k x 21k BM source files of sizes up to 440MB on my system with the ImageMagick tools. No problems whatsoever. Yes, it takes time, since many operations use the HD instead of your RAM. That's why I diverted on Chinese recipees above;-)...

If for example, I want to increase the brightness by 20%, decrease the saturation by 50% and shift the hue by 20% to the left, I run this statement in the shell (==> Cygwin!)

mogrify -modulate 120,50,80 <texture[s]>

This way you can make exactly the same changes to the 128 tiles in the level3 directory:

mogrify -modulate 120,50,80 *.tga


Bye Fridger
Last edited by t00fri on 18.08.2003, 18:30, edited 2 times in total.

Cormoran
Posts: 198
Joined: 28.07.2003
With us: 21 years 4 months
Location: Slartibartfast's Shed, London

Post #17by Cormoran » 18.08.2003, 18:19

Sorry Fridger, my fault :oops: (its been a hard day and I have to go work until 8am now).

I'll be looking into that thread in depth if and when I start working on the virtual texture thing.

I still think my other comment re: a Celestia toolbox suite stands. I'm talking about little apps, such as Asteroid belt generators, planet creators etc, and any other stuff anyone can think of.

Cheers,

Cormoran
'...Gold planets, Platinum Planets, Soft rubber planets with lots of earthquakes....' The HitchHikers Guide to the Galaxy, Page 634784, Section 5a. Entry: Magrathea

Avatar
selden
Developer
Posts: 10192
Joined: 04.09.2002
With us: 22 years 3 months
Location: NY, USA

Post #18by selden » 18.08.2003, 18:41

Fridger wrote: But using ImageMagick tools to do the editing, the virtual memory on the HD is mainly used


The NetPBM utilities have the advantage that most of them don't even use virtual memory. Whenever possible, they operate on a single scanline at a time.

I suspect that by "editing," Comoran means "visually fix local problems in the image" rather than "apply color corrections to the entire image." Such things can be painfully slow while one waits for the paint program to page various parts of the picture to and from the disk. The same slowness due to paging happens with nvdxt. This makes it rather tedious to make several iterations of change/view/change/view at high resolutions. (By "view," I mean "Study how it looks in Celestia.")

Unfortunately, tiling is not yet available for Nebula textures.
Selden

jim
Posts: 378
Joined: 14.01.2003
With us: 21 years 11 months
Location: Germany

Post #19by jim » 18.08.2003, 18:57

t00fri wrote:I am surprised that you cannot edit larger textures than 8k. You have exactly the same amount of RAM as I have. But when the ImageMagick tools are employed for 'batch'-editing, the virtual memory on the HD is mainly used and the texture is not loaded into the RAM!


Sorry. I can't see that ImageMagick supports the DDS format.
http://www.imagemagick.org/www/formats.html

Selden where can I find a 'exe-version' of NetPBM utilities?

Bye Jens

chris
Site Admin
Posts: 4211
Joined: 28.01.2002
With us: 22 years 10 months
Location: Seattle, Washington, USA

Post #20by chris » 18.08.2003, 19:26

Don. Edwards wrote:This is a space simulator not an Earth simulator as several members of this board have stated in the past. If you want to take Celestia in that direction than fork the project into two separate programs. One as an "Earth" simulator that can take you down to the most detailed surface images and one that returns us to a more pure "Space Simulator" witch was what Celestia is all about. I think we need to get realistic here. Good 16k textures are just now becoming common place and we are stressing are video cards quite nicely now. From everything I am hearing so far this is just going to slow the process down more. I frankly would prefer to see more in the way of Astronomical features added versus seeing the need to see city streets on the surface of the Earth.
Here are just a few dissenting reasons.
1. The files are just going to be unrealistically too large to download.
2. We are looking at the need of possible DVD media for storage and DVD drives are not on all Celestia users’ computers.
3. The use of multiple CDs for storage of the textures in place of DVD.
4. The acceptable slowdown of Celestia responsiveness to input.
In the end I can see that the large texture tiling scheme is going to cost users for media and also make some of the less experienced uses feel they need to move up to much more expensive video hardware when they really don’t have to. You have proved that time and time again.
So until I can test a large texture pack myself you will find me staunchly in the anti-texture tiling camp if there is one. And if there isn't than I will be more than happy to be the only member. I am just hoping that everyone doesn't get carried away by all the glimmer of these super hi-res textures because for most users it will be nothing but a pipe dream.
That’s my opinion on the subject and no one has to agree with me. I know my opinion carries very little weight with the project but I would hope for a sanity check and a simple return to what Celestia was supposed to be at its roots. The best free “SPACE SIMULATOR” available.


I really don't understand the problem . . . For one thing, virtual textures don't slow Celestia down--as Fridger's case shows, they make it possible to display very high-resolution textures even on older hardware. Also, virtual textures are a feature still very much in development. Currently, there's a great deal of work involved in the creation of virtual textures, and a means of distributing them has to be worked out. I see the variable resolution of virtual textures as very important--you can download high resolution data for just the particular regions in which you're interested.

I see virtual textures as an important part of making Celestia the best space simulator around (whether free or commercial) The variable resolution will allow the use of a great deal of high resolution planetary images--they can be placed in context on the appropriate body amid lower resolution data. Not for everyone perhaps, but there's nothing here that threatens Celestia's established purpose as a space simulator. In fact, I'm quite wary about introducing features which would do that.

Virtual textures integrate seamlessly with the rest of Celestia--they work generally as alternate surfaces, specular maps, night maps, etc. (though as Selden mentioned, not yet as nebula textures.) More significant is that there was absolutely no UI change involved with the introduction of virtual textures into Celestia. The fact that the existing UI accomodates the addition of super high res tiled textures so well should make it absolutely clear that I've in no way changed the direction of Celestia with this new feature.

--Chris


Return to “Celestia Users”