Celestia Forums

Hi all,

I want to expose new code for handling galaxy templates via the standard .BMP format.

My new approach has lots of advantages over Toti's "homemade" .pts format that can only be "pseudo-edited" with his rather cumbersome bmp2pts tool. Worst of all, the .pts format is a vaste of storage as we shall see. This latter aspect is particularly crucial for future developments, like significantly extending the number of templates to include the extended Hubble scheme or also individual color profiles!

++++++++++++++++++++++++++
NOTE that Celestia already contains ALL required code for loading the standard .BMP format that --best of all-- may be edited with EVERY image manipulation software and is a factor ~ 25 more economical as to storage!
++++++++++++++++++++++++++

Why is the .pts format such a vaste? Well, that's easy to see:

Imagine a 128x128 grayscale template image. Each of the 128x128 = 16384 pixels of this image is just characterized by an 8bit integer specifying its brightness. In case of simple (8bit x3) RGB color images, the same scheme applies, except each integer is replaced by three, designating the respective R,G,B colors instead of brightness, only. Usually, there is some header in addition, containing info about the image size, some "magic number" etc...

Instead of just one byte for brightness at position i of the image vector, Toti's .pts format uses 4 floating point numbers to carry the same information: x,y,z, br. It is close to trivial, however, to quickly map the integer brightness value of popular formats like .pgm, .ppm, .bmp, ... at some position i in the vector of length 128x128=16384 to the required 2d float positions (x,z) after loading the image. This mapping has to be done only once after loading, thus costing no frame rate! Similarly the y extension can be easily generated within Celestia without penalty. My new code is of course not restricted to 128x128 sized gray templates...

Here is a comparison of the storage needed of our present .pts files compared to the equivalent .BMP templates:

e,g, S0: S0.pts => 378 KB while S0.bmp => 17.0 KB

So we can have 140 (!) equivalent templates instead of the present 7 for the same storage consumption.


+++++++++++++++++
The greatest advantage is that everyone now can experiment with his custom .BMP templates using GIMP or Photoshop or whatever, exploit layers, standard brightness normalization and all the beloved "luxus" features of these advanced image manipulation programs. A future extension to individual color profiles for DSO's is utterly trivial.
+++++++++++++++++

How can you profit from the new feature?

To incorporate the new format amounted to a thorough recoding of the galaxy.cpp file only and a conversion of the .pts templates to .bmp format. If people are interested, I wrote PERL routines that do these conversions in both directions...The most essential code changes you'll find in the method buildGalacticForms(const std::string& filename).

All this may be downloaded and tried out from my TextureFoundry site:

++++++++++++++++++++++++++++++++++++
http://www.celestiaproject.net/~t00fri/celesti ... plates.zip
++++++++++++++++++++++++++++++++++++

All you got to do is to

throw the 7 .BMP templates into the 'models' directory, to exchange the file galaxy.cpp in the src/celengine directory and rebuild celestia. PLease use the current version 1.4.1 for this or better the latest CVS version.

Let me know how things go. The galaxy display is of the same quality as before.

Enjoy,

Bye Fridger

Fridger,

This looks like it will be a great improvement!
However....

Do you anticipate that it will become possible to use BMP format for surface textures, bumpmaps and other textures in general?

My understanding is that BMP is not officially supported for most of Celestia's Texture declarations. It certainly is not one of the formats that the wildcard texture file specification looks for. Being able to use BMP format for some things and not others will cause confusion and complaints, I fear.

selden wrote:Fridger,

This looks like it will be a great improvement!
However....

Do you anticipate that it will become possible to use BMP format for surface textures, bumpmaps and other textures in general?

My understanding is that BMP is not officially supported for most of Celestia's Texture declarations. It certainly is not one of the formats that the wildcard texture file specification looks for. Being able to use BMP format for some things and not others will cause confusion and complaints, I fear.

Selden,

No, I just want to replace the galaxy template format by a standard one that is much more economic and editable with familiar software. That's all. I never considered to use this format for textures, bumpmaps and the like. The requirements for templates so far are special: we just need to know the points where to put our sprites within the program. That's all. Since .bmp can easily carry color information (and alpha), we can use the same code to administrate /individual/ color profiles for DSO's.

Bye Fridger

In that case, would it be possible to use PNG images instead of BMP? Then the confusion would be avoided, I think.

I seem to have a vague recollection that Celestia's implementation of the BMP reader is incomplete. I fear this might cause problems, too.

selden wrote:In that case, would it be possible to use PNG images instead of BMP? Then the confusion would be avoided, I think.

I seem to have a vague recollection that Celestia's implementation of the BMP reader is incomplete. I fear this might cause problems, too.

What's your problem with BMP? It works for the purpose it is meant for and it's standard in all OS's and lossless. I assure you. Yes, we might also take PNG, of course (see further below ). What is NOT yet implemented into Celestia's BMP loading code is BMP /compression/. Level 9 compressed PNG code would save another factor of two relative to uncompressed .BMP.

But this is all peanuts, since we got all loading routines we need ready for use in Celestia. PNG is of course a close relative to the other mentioned formats. I'll look into the PNG option for "completeness"

Bye Fridger

Will test this as soon as possible, there is an issue with osX building from cvs, Dirkpitt should fix it soon...

My problem with BMP format is the confusion factor. Right now, I can tell people who have problems "BMP is not supported by Celestia. Don't use it."

Having to qualify that with "except in these circumstances" is going to cause lots of grief.

selden wrote:My problem with BMP format is the confusion factor. Right now, I can tell people who have problems "BMP is not supported by Celestia. Don't use it."

Having to qualify that with "except in these circumstances" is going to cause lots of grief.

Could you specify to whom you are telling all this? I can't remember anything about that confusion. People are supposed to use JPG, PNG or DXTx for /textures/. "basta"

BMP format as I use it for the templates, is about the most simple one you can imagine. I actually first did everything for .PGM format which is Portable Graymap Format. IT REALLY DOES NOT MATTER . It's largely the header that differs. I checked the BMP code in Celestia line by line. For what I need it for, it is CORRECT.


Bye Fridger

People often contact me about models that don't work correctly with Celestia -- models that use BMP textures. I really don't want to get into arguments about "but Celestia uses it for some things!" It just causes hard feelings and takes a lot more time to explain.

selden wrote:People often contact me about models that don't work correctly with Celestia -- models that use BMP textures. I really don't want to get into arguments about "but Celestia uses it for some things!" It just causes hard feelings and takes a lot more time to explain.

Tell them they should use the other formats. Probably these people use RLE compressed BMP and never heard about what RLE is But that's really not my problem.

Bye Fridger

Selden,

I just modified my code with a few trivial changes to read PNG templates with compression factor 9! instead of BMP format. Everything looks the same, you got your "non-confusing" PNG format and we all saved another factor of 2 relative to Toti's .pts . I'll upload the .PNG templates and the adapted galaxy.cpp tomorrow...

Here is the final comparison of the /equivalent/ file sizes:



While the default .pts templates use 1604 kB in total, the new, equivalent .png templates only use 41kB, i.e. a factor of 39 LESS...

Bye Fridger

Fridger,

That's great!

And we don't even have to worry about people accidentally saving them in a format that Celestia can't use!

ElChristou wrote:Will test this as soon as possible, there is an issue with osX building from cvs, Dirkpitt should fix it soon...

I'll see if I can bring the fixed OS X files to work on Monday on my iPod and upload them then.

Incidentally ElChristou, doesn't this new PNG-based galaxy template system answer your wishes (at least in part) for an easier way to make galaxies?

dirkpitt wrote:

ElChristou wrote:Will test this as soon as possible, there is an issue with osX building from cvs, Dirkpitt should fix it soon...

I'll see if I can bring the fixed OS X files to work on Monday on my iPod and upload them then.

Incidentally ElChristou, doesn't this new PNG-based galaxy template system answer your wishes (at least in part) for an easier way to make galaxies?

As so often, Chris has not yet reacted at all, but anyhow I shall commit the new PNG stuff to CVS during the weekend. I went on with ElChristou experimenting with new (improved) BMP/PNG templates meanwhile ...

Cheers,
Fridger

dirkpitt wrote:

ElChristou wrote:...Incidentally ElChristou, doesn't this new PNG-based galaxy template system answer your wishes (at least in part) for an easier way to make galaxies?

Yes and... no! in fact the process is a bit easier, but still it's a lot of work between tweaking code versus pixels modif to obtain approximatively what you want... work in progress...

ElChristou wrote:

dirkpitt wrote:

ElChristou wrote:...Incidentally ElChristou, doesn't this new PNG-based galaxy template system answer your wishes (at least in part) for an easier way to make galaxies?

Yes and... no! in fact the process is a bit easier, but still it's a lot of work between tweaking code versus pixels modif to obtain approximatively what you want... work in progress...

ElChristou,

as to the NO of your "yes & no" reply: it is also not easy to make e.g. a good planetary texture or an add-on 3d model..., by far not everyone is able to without considerable know-how. With the templates, there is a fairly unavoidable element in addition, namely randomness that is essential for the natural impression of galaxies. Randomness enters via the sizes and hence the luminosity of the sprites that we associate to all template image points. It also enters to the image points themselves to some degree. We should also not forget that we don't have to design new templates every now and then. Once we have a satisfactory set, that will do for a while

Moreover it is important for a successful template design to understand essentially how the galaxy.cpp code WORKS!

Bye Fridger

Back from vacations...

t00fri wrote: My new approach has lots of advantages over Toti's "homemade" .pts format that can only be "pseudo-edited" with his rather cumbersome bmp2pts tool. Worst of all, the .pts format is a vaste of storage as we shall see. This latter aspect is particularly crucial for future developments, like significantly extending the number of templates to include the extended Hubble scheme or also individual color profiles!

As you know well, minimizing the template size was one of my primary concerns. But as a matter of design I preferred to have a single format for all of the "gaseous" objects: Chris' recently favoured a "volumetric" version of CMOD to get rid of those nasty .pts files once and for all. You can certify this was in my plans...

Your method comes with the cost of a limitation in the z axis: you need a different z-rebuild function for each DSO subtype. This functions must be hardwired in C++, so users could not flexibly create their own templates, unless you use some tricks, like assigning a channel for the missing axis data, etc. Also, to create more detail, you expenses are n?? (which can be somewhat reduced by compression). And models produced by your method are essentially planar, ie: you cannot have two points on the same vertical line, unless you force it with randomization, but then you lose points from other places.

Some months ago I applied a scheme that gave good results: storing a byte for color-index, another for alpha and 3 additional bytes for position per blob. This position was relative to another (ancestor) blob, thus generating a tree-like structure that was used to quickly discard entire branches that were too far away to be visible, providing an efficient LOD culling mechanism. Despite including color-alpha information, files were only a fraction of comparable .pts ones. I compared two images of a .pts and the new format by substracting them and the difference was quite small.

The big advantage (apart from the "builtin" culling thing) was that it allowed expansion of the detail at a logarithmic cost in filesize *and* RAM. Increase in resolution was also entirely local (ie: where detail was needed). It was also designed to minimize stack overhead, jumping over intermediate branch levels if the LOD decided so.
But despite its efficient design it didn't solve the fill-rate problem, which quickly imposes a hard limit on any video card, including the best ones: the really good visuals started at more or less 200,000 blobs, that the (still unoptimized) algorithm handled with good CPU performance. But actually rendering the thing gave less than 2 fps (!) on my TNT2: the thousands of small blobs that make up the fine detail are enough to saturate the card's fill rate. One could possibly avoid rendering all the small structures that are behind larger blobs (losing some visual quality), but to be fast this probably requires hardware occlusion querying, which has its own drawbacks and discards older cards.

With that sort of rigid constrains it is very difficult to make decent progress on this stuff.

Toti wrote:Back from vacations...

t00fri wrote: My new approach has lots of advantages over Toti's "homemade" .pts format that can only be "pseudo-edited" with his rather cumbersome bmp2pts tool. Worst of all, the .pts format is a vaste of storage as we shall see. This latter aspect is particularly crucial for future developments, like significantly extending the number of templates to include the extended Hubble scheme or also individual color profiles!

As you know well, minimizing the template size was one of my primary concerns. But as a matter of design I preferred to have a single format for all of the "gaseous" objects: Chris' recently favoured a "volumetric" version of CMOD to get rid of those nasty .pts files once and for all. You can certify this was in my plans...

Whatever...I was referring above to the present /facts/ (.pts) NOT to any of your plans.

The present switching to PNG is most simple , since all required routines are part of image.cpp, they allow editing of the templates with standard software and save a factor 40 in disk space. Our present rendering structure remains almost untouched for now. In addition as to some RAM saving, I have recently changed the brightness entry vector in the Blobs to 'unsigned char' thus using only 1 byte per entry instead of the 4 bytes (float) as in your original format. The (x,y,z) coordinates might trivially be stored as 2byte int's if necessary, as Chris suggested.

Anyhow, --unlike your philosophy-- mine proceeds in small controllable steps from what we have and like! It uses a standard yet much expandable image format (PNG) from the onset.

Your planned format --whatever it was-- again was 'home-made' and NOT editable without dedicated software, I seem to remember...

Your method comes with the cost of a limitation in the z axis: you need a different z-rebuild function for each DSO subtype.

Not at all. I use a common Gaussian for all spirals which is perfectly adequate, since we don't have any direct /detailed/ observations about individual thickness profiles of the various spiral subtypes. In addition I use one further flattened spheroidal profile for S0 that for all ellipticals E0..E7 will be trivially blown up into axial rotation symmetry via Vec3f(ell, 4.0f * ell, 1.0f) scaling.

That's a most simple, fast and adequate approach. We do not have enough scientific information available to justify more detailed /transverse/ rendering fancyness in mass rendering.

...
Some months ago I applied a scheme that gave good results: storing a byte for color-index, another for alpha and 3 additional bytes for position per blob. This position was relative to another (ancestor) blob, thus generating a tree-like structure that was used to quickly discard entire branches that were too far away to be visible, providing an efficient LOD culling mechanism. Despite including color-alpha information, files were only a fraction of comparable .pts ones. I compared two images of a .pts and the new format by substracting them and the difference was quite small.

The big advantage (apart from the "builtin" culling thing) was that it allowed expansion of the detail at a logarithmic cost in filesize *and* RAM. Increase in resolution was also entirely local (ie: where detail was needed). It was also designed to minimize stack overhead, jumping over intermediate branch levels if the LOD decided so.

Despite all these continuing promises you never managed to show convincing benchmark images that allowed me or anybody else to judge objectively. Instead you proceeded to "scramble" our present joint code, before first convincing the community in concrete terms...

You place much emphasis on possibilities of customization in your approach. As I pointed out repeatedly, this should be preferrably done via add-ons, since the crucial aspect of the mass rendering of DSO's is to handle many 1000's of objects at once and FAST. DSO mass rendering should be based on strictly scientific information available in published catalogs. With these premises, mass rendering is unsuitable for extreme high resolution 3d rendering of selected DSO's, like e.g. here:

from
http://www.celestiaproject.net/forum/viewtopic ... highlight=

++++++++++++++
That amazing 3d quality you will never be able to achieve with "custom templates" whatever their format. Also much of the required information for such images is NOT known on a scientific basis. So better leave this domain to add-on creators!
++++++++++++++

Of course, we should extend (trivially) the Hubble classification and possibly administrate individual color (and alpha?) profiles in the PNG format. The latter is trivially possible and already prepared for in my present code.

Hey guys, what is that? At first sight it sounds like a childish battle kind "you have do this, but my solution is better na na"...

I hope to see join efforts in the topic; I'm not expert but apparently the good point of Fridger's work is low cost in storage, efficiency in rendering.
The good point of Toti's (prime) work was the idea of a tool with various settings (I haven't see the second tool).

To make simple, I will position myself in the comfortable place of Mister Everybody*:
Until now, the bad point of all methods is that for me (Mister Everybody), it's almost impossible to control the process of creating templates.

We must not forget that there is dozens of well know galaxies and I suppose nobody will spend days in creating those custom templates so the process must be democratize to let a large quantity of users work on this and propose their versions.

In my vision, those custom templates could be must more detailed than the generic ones used for mass rendering; this is a point nobody seems to care, because this mean perhaps another method of creation (different from the mass rendering one) that MUST use the same settings in code than the mass rendering one... what a challenge!!

So guys, open mind

Bye.


*Mister Everybody is the typical end user with 90% chance of using a M$ system, who don't want to know what is behind the scene but want an effective and simple method to not having to use his brain too much

Your planned format --whatever it was-- again was 'home-made' and NOT editable without dedicated software, I seem to remember...

Just like the CMOD format, if my memory doesn't fail...

Your method comes with the cost of a limitation in the z axis: you need a different z-rebuild function for each DSO subtype.

Not at all. I use a common Gaussian for all spirals which is perfectly adequate, since we don't have any direct /detailed/ observations about individual thickness profiles of the various spiral subtypes. In addition I use one further flattened spheroidal profile for S0 that for all ellipticals E0..E7 will be trivially blown up into axial rotation symmetry via Vec3f(ell, 4.0f * ell, 1.0f) scaling.

That's a most simple, fast and adequate approach. We do not have enough scientific information available to justify more detailed /transverse/ rendering fancyness in mass rendering.

It's clear that I referred to DSO subtypes, and not galaxies only. You need to expand the z axis using whatever custom method and this is unavoidable. I tend to naturally discard such inflexibilities before coding a single line...

Some months ago I applied a scheme that gave good results: storing a byte for color-index, another for alpha and 3 additional bytes for position per blob. This position was relative to another (ancestor) blob, thus generating a tree-like structure that was used to quickly discard entire branches that were too far away to be visible, providing an efficient LOD culling mechanism. Despite including color-alpha information, files were only a fraction of comparable .pts ones. I compared two images of a .pts and the new format by substracting them and the difference was quite small.

The big advantage (apart from the "builtin" culling thing) was that it allowed expansion of the detail at a logarithmic cost in filesize *and* RAM. Increase in resolution was also entirely local (ie: where detail was needed). It was also designed to minimize stack overhead, jumping over intermediate branch levels if the LOD decided so.

Despite all these continuing promises you never managed to show convincing benchmark images that allowed me or anybody else to judge objectively. Instead you proceeded to "scramble" our present joint code, before first convincing the community in concrete terms...

Read again the sentence; it explains clearly why I didn't publish anything. And how could I show a single benchmark of a new massive reimplementation without big modifications to the code remains an ontologic mistery to me... (that I DO NOT want to discuss again)

You place much emphasis on possibilities of customization in your approach. As I pointed out repeatedly, this should be preferrably done via add-ons, since the crucial aspect of the mass rendering of DSO's is to handle many 1000's of objects at once and FAST. DSO mass rendering should be based on strictly scientific information available in published catalogs. With these premises, mass rendering is unsuitable for extreme high resolution 3d rendering of selected DSO's, like e.g. here:
...
++++++++++++++
That amazing 3d quality you will never be able to achieve with "custom templates" whatever their format. Also much of the required information for such images is NOT known on a scientific basis. So better leave this domain to add-on creators!
++++++++++++++

A well designed approach would satisfy both needs, serving as a common scheme for lower resolution (mass rendering) and higher resolution (custom models). You are constantly placing an arbitrary division which doesn't fit at all. Also, your idea of a good nebulae model and mine are quite different so that image you keep referring to doesn't impress me at all.
On the other hand you keep misunderstanding the whole issue: I always talked about a generic framework to replace the hack of simulating volumes with surfaces (meshes). This ugly trick has simply too many drawbacks to be acceptable.
And about that "amazing quality" you believe unreachable by volumetric techniques, please google a little bit on the subject: there are real time animations out there made with millions of blobs. I normally tend to know what I write about...

So guys, open mind

I just wanted to let people know the weak points of Fridger's approach as well as some considerations that I once took the time to think about. I am not working on this anymore since long, so I don't have any alternative to offer. Certainly we agree that the removal of the .pts files is mandatory. Chris seems to like the new approach, so I am fine with it.