Script-fu to create virtual textures.

[Teto]

Hi!
I'm a newbie in Celestia and add-on programs so forgive me if I'm writing stupid things...

So I discovered Celestia last month and I loved it immediately. But I have a problem: My mother board (PC) is a Nvidia, and my graphic card is an ATI. So that's why I think that I had many problems with Hi-res textures (watch the .gif below). I was annoyed, until I discovered that I can use virtual textures for the planets. But sometimes they don't exist, I just can download big size's textures, that I can't use because of the bug. So I wanted to change them in virtual textures but without script or something... It's long or boring. I spent time to find tool which can do that and I've found nothing "as I want".
So I made 2 script-fu for Gimp to make tiles automatically since a big texture (see pictures below). The only thing that is must be made manually is the creating of the sub-paths (i.e yourPlanet/textures/hires/surface/level2 and so on), because script-fu can't (I didn't found the function). Pity But in the package I put an empty tree, so, well...

I discovered yesterday that a program very similar already exists: F-Tex, which uses a good format (but lossy) and seems very useful (I didn't tried it yet).
I want to do 2 scripts more, for configurate .ssc/.ctx files and for special dimensions. If you judge that it's useless, don't hesitate to tell me.
Also, I won't be pissed if you say that my scripts are useless. I prefer passing time on making textures than script-fu that anyone care.

For the moment I post just the interface, to give you an idea. I think you can guess easily understand the pitch. If you are interested, I'll post the script-fu and the manual.

A little question. As I'm a beginner on script-fu, I decided to simplify the program that user can just make 512x512 tiles in .png. Allowing other sizes would be really useful?

Thanks,
Teto.

[t00fri]

Hi!

So I made 2 script-fu for Gimp to make tiles automatically since a big texture (see pictures below). The only thing that is must be made manually is the creating of the sub-paths (i.e yourPlanet/textures/hires/surface/level2 and so on), because script-fu can't (I didn't found the function). Pity But in the package I put an empty tree, so, well...

I discovered yesterday that a program very similar already exists: F-Tex, which uses a good format (but lossy) and seems very useful (I didn't tried it yet).

I am the author of the F-Tex tools (F = Fridger ). There is NO lossy format involved! You are NOT correct. Also my tools are memory optimized. I don't see how you could write script-fu's for >= 64k textures! My tools implement many more optimizations you don't even know of at this time... They are very crucial and I suggest you read about my tools first in our CelestialMatters site..

My F-Tex tools are written in C++ and hence are faster than script-fu by HUGE factors. If you want to wait for many hours in case of >=64k "monster" textures, go ahead with script-fu

Fridger

[Teto]

I am the author of the F-Tex tools (F = Fridger ). There is NO lossy format involved! You are NOT correct. Also my tools are memory optimized. I don't see how you could write script-fu's for >= 64k textures! My tools implement many more optimizations you don't even know of at this time... They are very crucial and I suggest you read about my tools first in our CelestialMatters site..

My F-Tex tools are written in C++ and hence are faster than script-fu by HUGE factors. If you want to wait for many hours in case of >=64k "monster" textures, go ahead with script-fu

Fridger

Ok. I've mistaken with the DDS compression format, DXT1 ~ DXT5

My F-Tex tools are written in C++ and hence are faster than script-fu by HUGE factors.

No kidding?
Yes I'm a noob, a little child compared to many people here. But I'm not stupid, so, please...

For me Celestia is a pleasure, and when I proposed my scripts (little things, for sure) it's not to show that I'm the best of the best programmer (like you, I agree with that, honestly ) but to help the ones who found a beautiful real/fictional level3~4 texture and want to make a virtual texture quickly (but accurate) without too much operations.
I used Script-fu because it's for Gimp (which is free), easy to learn, and available in many systems. And Gimp can make normal maps too.

If you want to wait for many hours in case of >=64k "monster" textures, go ahead with script-fu.

You're saying that Gimp is a slow soft? The developers will be happy to hear that. Seriously, even at level5 Gimp exhausts the memory of my PC, so... But the program is fast, a 4096x2048 texture with surface takes less than 1mn.

At least, I didn't want to make the best program ever made about textures. I'm not a programmer (but I deeply practiced Java). I just wanted to help people who like Celestia but don't want to learn/use 1253 tools to produce/share their unprofessional work. That's all. Now, if I must have a doctorate (or whatever) to discuss friendly with people here, well...

Regards,
teto.

[t00fri]

I am the author of the F-Tex tools (F = Fridger ). There is NO lossy format involved! You are NOT correct. Also my tools are memory optimized. I don't see how you could write script-fu's for >= 64k textures! My tools implement many more optimizations you don't even know of at this time... They are very crucial and I suggest you read about my tools first in our CelestialMatters site..

My F-Tex tools are written in C++ and hence are faster than script-fu by HUGE factors. If you want to wait for many hours in case of >=64k "monster" textures, go ahead with script-fu

Fridger

Ok. I've mistaken with the DDS compression format, DXT1 ~ DXT5

My F-Tex tools are written in C++ and hence are faster than script-fu by HUGE factors.

No kidding?
Yes I'm a noob, a little child compared to many people here. But I'm not stupid, so, please...

For me Celestia is a pleasure, and when I proposed my scripts (little things, for sure) it's not to show that I'm the best of the best programmer (like you, I agree with that, honestly ) but to help the ones who found a beautiful real/fictional level3~4 texture and want to make a virtual texture quickly (but accurate) without too much operations.
I used Script-fu because it's for Gimp (which is free), easy to learn, and available in many systems. And Gimp can make normal maps too.

If you want to wait for many hours in case of >=64k "monster" textures, go ahead with script-fu.

You're saying that Gimp is a slow soft? The developers will be happy to hear that. Seriously, even at level5 Gimp exhausts the memory of my PC, so... But the program is fast, a 4096x2048 texture with surface takes less than 1mn.

At least, I didn't want to make the best program ever made about textures. I'm not a programmer (but I deeply practiced Java). I just wanted to help people who like Celestia but don't want to learn/use 1253 tools to produce/share their unprofessional work. That's all. Now, if I must have a doctorate (or whatever) to discuss friendly with people here, well...

Regards,
teto.

I think you are mostly misunderstanding and your various allusions are completely unnecessary. I don't think I was unfriendly in any way.

Of course a scripting construct like script-fu, fed into the GIMP is less powerful than specially optimized dedicated C++ code. I am exclusively doing my normal graphics work with the GIMP since > 7 years, hence I know pretty well it's very good points but also it's weaknesses. With 3 GB RAM, I can barely handle a semi-large texture like 32768x16384 in the GIMP (no script-fu even), but it is becoming VERY slow for such sizes. In contrast, my F-TexTools handle bigger textures than these in minutes!

I precisely coded the F-TexTools, since GIMP is definitely at it's limits for LARGE textures for which VT's become interesting.

If you consider such small textures as 4096x2048, why do you make virtual textures at all?? I thought you are interested in real big ones, for which VTs are conceived.

I am talking about sizes like 32768 x16384=32k, 65536 x 32768 = 64k and bigger for which one usually considers VTs. For 4096x2048 you can take whatever you like, it doesn't make much of a difference. VT's will be just slower than the normal textures, since there is a lot of additional harddisk traffic for reading in the many VT tiles ...

The main point is that for real big textures you must make sure that the image is read in and processed line-by-line, since a full 64k RGB texture is about 9GB big! This obviously doesn't fit into any memory. Most software including the GIMP read in a texture as a whole and NOT line-by-line.

But as you explained, these issues don't seem relevant for your task. So I will shut up.


Sorry if I misunderstood.

Fridger

[Hungry4info]

Teto, I am interested in the script you're talking about.

(Post scriptum: No offense, t00fri).

[Teto]

I think you are mostly misunderstanding and your various allusions are completely unnecessary. I don't think I was unfriendly in any way.

Hello t00fri.
I think it was the end of the day and you were tired. No problem.

I precisely coded the F-TexTools, since GIMP is definitely at it's limits for LARGE textures for which VT's become interesting.

Yes, of course. With gimp and my script, making a 512x512 png with "level 9 compression" takes about 1 second. It's easy to calculate how much time for n tiles. With high levels that would sure takes a bunch of time!

If you consider such small textures as 4096x2048, why do you make virtual textures at all?? I thought you are interested in real big ones, for which VTs are conceived.

As I said in the first post (watch the picture in), I simply can't use these textures directly. It's my fault, I'm using an ATI graphic card (HD 4850 512M gold sample, a good one). I tried to put it in the "medres" folder, it was worse! So VT is the only solution for me for accurate vision of a planet. And I think I'm not the only one. And for the really big one I'm interested but I'll never have time to do them.
Also, I gave this example because it's with this size that I tried my script. But I can go until 16k without problem (the next level exhausts my memory, I have a 3,2 Gbytes SP3).

The main point is that for real big textures you must make sure that the image is read in and processed line-by-line, since a full 64k RGB texture is about 9GB big! This obviously doesn't fit into any memory. Most software including the GIMP read in a texture as a whole and NOT line-by-line.

And it's a pity. The soft would be just perfect if it would accept a mode "block-by-block", I mean what you see is in memory, the rest is in the hard disk. Much slower but you can have the size you want (inside the hard disk limits...). Or line by line, as you said.

Regards,
Teto.

[t00fri]

Hi Teto,

I precisely coded the F-TexTools, since GIMP is definitely at it's limits for LARGE textures for which VT's become interesting.

Yes, of course. With gimp and my script, making a 512x512 png with "level 9 compression" takes about 1 second. It's easy to calculate how much time for n tiles. With high levels that would sure takes a bunch of time!

This is of course not a LINEAR scaling problem! The point is that usual graphics programs start dumping the image data to the harddisk beyond a limiting size related to your available memory, which is getting VERY slow (compared to doing all manipulations in memory!). The well-known graphics program package 'ImageMagic' is another example: it's very fast (scripting!) for small textures but at larger sizes it is getting MUCH much slower even than the GIMP since the GIMP's memory management is better optimized.

Still, for the GIMP just reading in a 32768x16384 RGB texture with 3GB RAM memory takes at least 15 minutes! Try it.

Now I show you the original output with my F-TexTiles along with the times for

• reading in a 32768x16384 RGB texture ('m46_color_32k.png')
• converting it into a 8bit binary stream
• chopping it into 1024x1024 tiles with proper nomenclature tx_i_j.png
• reconverting the 512 VT tiles into lossless PNG format:

Just a single line of input is enough to get the job done:

Code: Select all

> png2bin < m46_color_32k.png |  txtiles 3 32768 4

[txtiles]: Input file is a 3x8 bit RGB color map: 32768 x 16384

           Generating  512 optimized VT tiles for level 4
           in PNG format,of size from 256 x 1024 to 1024 x 1024

           PNG_compression level = 1 (range: 0..9)

[png2bin]: Converting PNG format to binary output
           bpp: 3  size:  32768 x 16384
bin [  1024 rows of 16384 ->   1.89 s]
tile[    32 VT's of   512 ->   2.44 s]
bin [  2048 rows of 16384 ->   4.86 s]
tile[    64 VT's of   512 ->   7.70 s]
bin [  3072 rows of 16384 ->   7.96 s]
tile[    96 VT's of   512 ->  18.31 s]
bin [  4096 rows of 16384 ->  11.21 s]
tile[   128 VT's of   512 ->  28.91 s]
bin [  5120 rows of 16384 ->  14.60 s]
tile[   160 VT's of   512 ->  39.67 s]
bin [  6144 rows of 16384 ->  17.97 s]
tile[   192 VT's of   512 ->  50.54 s]
bin [  7168 rows of 16384 ->  21.35 s]
tile[   224 VT's of   512 ->  61.54 s]
bin [  8192 rows of 16384 ->  24.77 s]
tile[   256 VT's of   512 ->  72.58 s]
bin [  9216 rows of 16384 ->  28.24 s]
tile[   288 VT's of   512 ->  83.83 s]
bin [ 10240 rows of 16384 ->  31.62 s]
tile[   320 VT's of   512 ->  94.89 s]
bin [ 11264 rows of 16384 ->  34.98 s]
tile[   352 VT's of   512 -> 106.09 s]
bin [ 12288 rows of 16384 ->  38.38 s]
tile[   384 VT's of   512 -> 117.33 s]
bin [ 13312 rows of 16384 ->  41.62 s]
tile[   416 VT's of   512 -> 128.26 s]
bin [ 14336 rows of 16384 ->  44.80 s]
tile[   448 VT's of   512 -> 138.89 s]
bin [ 15360 rows of 16384 ->  47.80 s]
tile[   480 VT's of   512 -> 144.39 s]
bin [ 16384 rows of 16384 ->  49.72 s]
tile[   512 VT's of   512 -> 147.15 s]


So it's altogether just ~ 150 seconds ~ 2.5 minutes with my old desktop machine!
Another crucial design criterion was that also people with MUCH LESS than 3 GB of RAM can handle these big monster textures as well with my tools.

The real challenge however comes with normalmaps (since you mentioned that GIMP plugin). All existing normalmap programs are WRONG, if applied to textures for SPHERICAL bodies (planets etc). And they are just horrible quality since 8bit = 256 graylevels of elevation input is much too jumpy for generating the required smoothness of such normalmaps. So again, please make your own experiences first...

If you consider such small textures as 4096x2048, why do you make virtual textures at all?? I thought you are interested in real big ones, for which VTs are conceived.

As I said in the first post (watch the picture in), I simply can't use these textures directly. It's my fault, I'm using an ATI graphic card (HD 4850 512M gold sample, a good one). I tried to put it in the "medres" folder, it was worse! So VT is the only solution for me for accurate vision of a planet. And I think I'm not the only one. And for the really big one I'm interested but I'll never have time to do them.

Of course we have plenty of experiences here with bugs in ATI drivers. But in MOST cases these problems go away if people update to the latest drivers. I am unaware of such persistent ATI problems as you describe.

Also, I gave this example because it's with this size that I tried my script. But I can go until 16k without problem (the next level exhausts my memory, I have a 3,2 Gbytes SP3).

see my examples above!

The main point is that for real big textures you must make sure that the image is read in and processed line-by-line, since a full 64k RGB texture is about 9GB big! This obviously doesn't fit into any memory. Most software including the GIMP read in a texture as a whole and NOT line-by-line.

And it's a pity. The soft would be just perfect if it would accept a mode "block-by-block", I mean what you see is in memory, the rest is in the hard disk. Much slower but you can have the size you want (inside the hard disk limits...). Or line by line, as you said.

This argument I didn't understand, sorry. With my tools you can have the size you want (as useful for Celestia, of course).

Fridger

[John Van Vliet]

--- edit ---

[duds26]

Teto, have you checked for updates for your graphics card?
Not with windows update but on the site of ATI.

[Teto]

Teto, have you checked for updates for your graphics card?
Not with windows update but on the site of ATI.

Sorry for the delay, but when I read the last post of t00fri, I've understood that this forum was not for a guy like me. Always someone to say that my script-fu sucks and things like: "Why does it exist?", but nobody to help me about my problem with "hi-res". Except you and very few other guys, of course.

I think now that it's not a CG problem, but more a memory ram problem. I have few problems with a game too, that it is so optimized that it doesn't like bad memory cards. So since I can use my private computer, I'll check that, and change them, after. I'm sure that Celestia will work well.

Thanks in any case, duds26.
Teto.

[selden]

Teto,

I'm very sorry if you felt that this forum wasn't for you. It really is supposed to be. Different approaches to the same problem can be very useful.

[duds26]

What are the specifications of your graphics card?
Type, memory?
OS (e.g. Windows Vista)
Just saying it's an ATI card is not enough.
ATI has produced hundreds of different graphics cards

Microsoft distributes drivers for your graphic card and tries to remove all opengl related code.
They often don't succeed in doing it perfectly which making opengl not working properly.
This problem you have could be such an issue that is caused by not working drivers.

Also what version of gimp do you use?
The developers of gimp are working on a new rendering core with advanced memory management and many other features.
Here's the official site of the new rendering kernel: http://gegl.org/

GEGL (Generic Graphics Library) is a graph based image processing framework.

GEGL provides infrastructure to do demand based cached non destructive image editing on larger than RAM buffers.

It's incorporated in the latest version of gimp (gimp 2.6) but the default is that it's turned off.
You can turn it on and see if there is any improvement.

Your idea is not bad at all but you're trying to reinvent the wheel.
Leveraging the already available tools in an integrated manner would still be usefull.
Using script fu for being able to use Fridgers F-Tex tools from the gimp would make your scripts really useful.
Good idea to make that empty tree but that is not the only thing you can do.
Go a bit further and make it able to use some NASA blue marble and proces them to a level x texture with everything on it that celestia needs.
Including the ctx file and folder structure and proper naming, planet and planet's parent.
The interface could use a bit of cleaning up and adding more options would be welcome.
Especially texture size option (for each image)
Some functionality to stitch textures together for making virtual textures from would be nice.
And a calculator that calculates the maximum virtual resolution.
e.g. level 3 texture size tile size: 1024x512 => (2^3)*1024x(2^3)*512 = 8192 x 4096 = 8K pixels wide, maximum virtual resolution: 33.554432 Mpixels
http://www.lns.cornell.edu/~seb/celestia/textures.html#2.5.3

Make some good documentation for every function.
With answers to what the scripts are, how it works, (why it uses F-Tex), what is celestia, what are virtual textures, ...
The average gimp user won't know what celestia is.
What would also be really useful to see is a benchmark, graph about the times.
Comparison between:
F-Tex,
gimp + old render core + script,
gimp + GEGL + script,
gimp + old render core + F-Tex + script,
gimp + GEGL + F-Tex + script

[Teto]

Hi!
Sorry for the delay again, but I'm working far away from home, and I don't have access to the net easily.
I answer point by point to the last (and very interesting) post.

What are the specifications of your graphics card?
Type, memory?
OS (e.g. Windows Vista)
Just saying it's an ATI card is not enough.
ATI has produced hundreds of different graphics cards

For sure. But I wrote fast (I'm late! I'm late!). So:
Mother board: Asus P5N-D
Chipset : nVidia nForce 750i SLI SPP
Processor : Intel Core 2 Quad Q9300 @ 2500 MHz
Memory : 4096 Mo (2 x 2048 DDR2-SDRAM )
GC: ATI Radeon HD 4800 Series --> Gainward 4850HD gold sample
Windows SP3 with automatic updates.
As you can see it's not an old computer. But, my mother board is Nvidia and my GC ATI. It's because I wanted to change my GC few months ago (the old one was too weak for games), and I did it myself, without knowledge of incompatibility problems between the 2 systems. So I think that my problems come from here.
And about the game I talked, it's because there's a problem since I want anti-alaising. With the GC I have, it should not be a problem (my configuration is good), but well... it is.

Also what version of gimp do you use?

Gimp 2.6 with GEGL. And I allow it to use my dual core. For the GEGL, I tried it for other things, but never to create an huge image. I'll try since I have time.

Your idea is not bad at all but you're trying to reinvent the wheel.

You're right. Why to do a new soft.
Why... Why not?
Linux has many distributions, and each year other distrib appear. Why?
I have to confess that I discovered F-Tex during the development of my soft. As I wanted to learn Script-fu (for other things), I figured that creating virtual texture would be a good exercise. After that, I tried to use F-Tex. Awesome! A soft that you spend more time to launch than to have results!

Using script fu for being able to use Fridgers F-Tex tools from the gimp would make your scripts really useful.

I agree with you. And Script-fu seems to be able to use/launch .exe. I have to try it... someday.

Go a bit further and make it able to use some NASA blue marble and proces them to a level x texture with everything on it that celestia needs.
Including the ctx file and folder structure and proper naming, planet and planet's parent.
The interface could use a bit of cleaning up and adding more options would be welcome.
Especially texture size option (for each image)
Some functionality to stitch textures together for making virtual textures from would be nice.
And a calculator that calculates the maximum virtual resolution.

I agree again, but since the very warm welcome from my soft, I'm not very interested to improve it.
About the blue marble I agree, but it's too much time and too hard for me. I'm not a professional (I'm sure that you knew it!).
The script writes the .ctx, with good names. The only thing I wasn't able to do is the creation of the sub-paths: The language is unable to do that.
Especially texture size option (for each image) --> Sorry but I didn't find anywhere the utility to choose the size. What is the difference between a 1024x2048 texture and a 512x512? Memory, speed?
Some functionality to stitch textures together for making virtual textures from would be nice. --> Sorry but I don't understand. You make the whole texture by the Gimp. So why to want to stitch tiles after virtualisation?

Make some good documentation for every function.

Done in french. I just waited a little after posting the script, to be sure that it wasn't a waste of time to make an english version. I was right...

You know what? I'd like to read this answer/advices after my first post, and not borderline responses from someone proclaimed that my work was a waste of time.

But don't worry. I'm working on brushes/gradients/etc to help artists when they're creating planets' textures (or stars, whatever). Don't panic! I won't pollute this forum with them anymore.

Seeya,
Teto.

[selden]

What is the difference between a 1024x2048 texture and a 512x512? Memory, speed?

Yes to both. When fully expanded with an alpha channel, 1024x2048 --> 8MB, which can take a while to load and convert, while 512x512 is only 1MB. If you use DDS DXT format, the texture file will load directly into the graphics card and not need to go through Celestia's format conversion routines.

[t00fri]

Teto,

Especially texture size option (for each image) --> Sorry but I didn't find anywhere the utility to choose the size. What is the difference between a 1024x2048 texture and a 512x512? Memory, speed?

if you start 'txtiles' just by typing its name, a help text is displayed. The first line
defines the basic relation between tilesize, level and width, as it is used in Celestia.
-----------------------------------------------------------------
> txtiles
Usage: txtiles <channels> <width> <level> [<PNG_compression>]

Version 1.0, August 2007, author: F. Schrempp

--------------------------------------------------------------------
The program reads textures in unsigned bpp x 8 bit integer raw format
from STDIN. It outputs VT tiles with many optimizations in PNG format.
--------------------------------------------------------------------

Units : tilesize[pixel] = width/2^(level+1).

Input : Interleaved RGB(A) storage mode ie. RGB(A)RGB(A)RGB(A)...
for RGB (+ alpha) textures
Inputwidth : inputheight = 2 : 1, power-of-two size.
No header.
Default : PNG_compression = Z_BEST_SPEED = 1
: best choice for subsequent DXT compression!

For VT tiles in PNG format, enter PNG_compression = 6..9 (slow!)

For 4 x 8 bit RGB + alpha textures enter channels = 4.
For 3 x 8 bit colored textures enter channels = 3.
For 1 x 8 bit grayscale textures enter channels = 1.
----------------------------------------------------------------------------------

Since you didn't find this relation, I suppose you didn't notice the help that is identically offered for all my tools? That would also explain this statement of yours:

After that, I tried to use F-Tex. Awesome! A soft that you spend more time to launch than to have results!

Incidentally, there are a number of very simple scripts that people have written for my F-TexTools. E.g. for the 65536 x 32768 Moon (Clementine data), 'cartrite's' recent script allows to generate a full 64k VT tileset (2-3 GB!) with all required configuration files in just 8 minutes with a single click (to start the script).

download/file.php?id=1513

Thus...sorry if I find your above "compliment" not very convincing...To balance your judgement a bit, here is another opinion from a happy, non-expert user of my F-TexTools :

viewtopic.php?f=5&t=13776&st=0&sk=t&sd=a&start=17

You probably also never read my step-by-step tutorial about the nmtools, which are designed analogously to the F-TexTools.


There is plenty of reasons for selecting different tile sizes for different computers. 1024x1024 is just a typical default size.

One example:
The smaller the tilesize (512x512), the more your harddisk has to work, since LOTS of small tiles have to be located and loaded at once. If you use a big tile size instead (e.g. 2048x2048), you rather need a card with a comfortable amount of memory. Many harddisks (-> laptops) are reasonably fast for loading a few, big files, but are VERY slow in loading MANY small ones (i.e they have a high seek/access time, since they tend to turn slowly (5400 rpm) to save energy and/or remain quiet)

Finally, let me also point you to this ongoing project of mine:

viewtopic.php?f=5&t=13776

a "CrossPlatform Graphical User Interface (Qt) for my F-TexTools/Nmtools".

Towards the end of my first post in this thread, you can see how the interface looks like (http://www.celestiaproject.net/~t00fri/images/F_tex_ui1.jpg) Besides C++, it is based on the Qt4 graphical toolkit that we also use for Celestia after 1.6. It supports Windows, Linux and MacOS alike...

Fridger