dxt5nm Format bug?

[cartrite]

I am starting a little cloud experiment to see if I can render a more realistic simulation of clouds. Basicly I want to create a simple sphere mesh and use it a number of times with different radii each having a different coudmap. By changing the rotation period, I found I can get them to move relative to the earth's position. What I want to try next is have beginning and ending declarations to vary the opacity so they appear and disappear to simulate changing dew points and low pressure formation during the day.

Anyhow, I was getting poor performance and I tried to create a dds normalmap to use on the main cloudmap to see if I could better it. But I noticed that the normalmap was loading but not working. I created an addon package of my 3dmars model to demonstrate this. If anyone thinks it's needed to show this problem I'll upload it to my site. But this seems easy to reproduce so......?

Here are a couple a screenshots that alerted me to this problem.

This shot shows my png format 4k cloudnormalmap.



And the dds dxt5nm format. The log sats its loading but doesn't seem to be working.



This shot shows my png format mars normapmap loaded onto the default mars sphere.



This shot shows the dds dxt5nm map on the defaut sphere. Not much difference.



This shot shows the png map on the 3d mesh.



And the dds map on the 3dmesh.




Since this has happened to 2 different dxt5nm normalmaps(mars normalmap and cloudnormalmap), it seems easy to reproduce. If it's still needed, I'll upload the demo for this tonight or tommorow. It will contain the mars model, the dds normalmap, and a cel html..

This is using cvs version Oct/04/2006, Installed by my setup.exe

cartrite

[cartrite]

I created a dxt5nm from erosnm.png and got similar results.

With erosnm.dds



With erosnm.png



All these dxt5nm's were created with the gimp's dds plugin including those from my last post.
These 2 screenshots were taken with CVS version Oct/11/06 KDE.

cartrite

[cartrite]

After looking at the code, it looks like this is not a bug. It just wasn't implemented for meshes.
I may be wrong though,

[selden]

Did you remember to create tangents for your CMOD model? That's one of the functions in the cmodfix utility. Tangents are required for normal maps to work on CMOD Mesh objects.

Unfortunately, I think I'm seeing bugs in the DXT5_nm support on both planets and models.

Here's a picture of what happens when I use a simple dxt5_nm normal map:



Areas that look like bumps with the bumpmap and png normalmap (middle and left spheres) look like depressions with the dxt5_nm normalmap (right sphere).

The dxt5 normalmap was created from the png normalmap using nvdxt v8.22 and the command

Code: Select all

nvdxt -quality_highest -file nm.png -nomipmap -dxt5nm 


A zip file containing the bump and normal maps, the command file I used to create them along with the ssc file and an html file with a Cel url is available at
http://www.lepp.cornell.edu/~seb/celest ... ump-nm.zip
(46KB)

.

[cartrite]

Did you remember to create tangents for your CMOD model? That's one of the functions in the cmodfix utility. Tangents are required for normal maps to work on CMOD Mesh objects.

Yes. If you seen the images, you would notice that the normalmap works in png format with meshes.Both eros and next to last mars.
Anyhow, I looked at the code last night. In this part from renderglsl.cpp

Code: Select all

void renderSphere_GLSL

This is what is controling dxt5nm.

Code: Select all

    if (ri.bumpTex != NULL)
    {
        shadprop.texUsage |= ShaderProperties::NormalTexture;
        textures[nTextures++] = ri.bumpTex;
        if (ri.bumpTex->isCompressed())
            shadprop.texUsage |= ShaderProperties::CompressedNormalTexture;
    }


from shadermanager.cpp

Code: Select all

       if (props.texUsage & ShaderProperties::NormalTexture)
        {
            if (props.texUsage & ShaderProperties::CompressedNormalTexture)
            {
                source += "vec3 n;\n";
                source += "n.xy = texture2D(normTex, " + normTexCoord + ".st).ga * 2.0 - vec2(1.0, 1.0);\n";
                source += "n.z = sqrt(1.0 - n.x * n.x - n.y * n.y);\n";
            }
            else
       

In this part, which is the code that renders meshes

Code: Select all

void renderModel_GLSL(Model* model,
                      const RenderInfo& ri,
                      ResourceHandle texOverride,
                      const LightingState& ls,
                      const Atmosphere* atmosphere,
                      float radius,
                      int renderFlags,
                      const Mat4f& planetMat)
{
    glDisable(GL_LIGHTING);

    GLSL_RenderContext rc(ls, radius, planetMat);

    if (renderFlags & Renderer::ShowAtmospheres)
    {
        rc.setAtmosphere(atmosphere);
    }

    // Handle extended material attributes (per model only, not per submesh)
    rc.setLunarLambert(ri.lunarLambert);

    // Handle material override; a texture specified in an ssc file will override
    // all materials specified in the model file.
    if (texOverride != InvalidResource)
    {
        Mesh::Material m;
        m.diffuse = ri.color;
        m.specular = ri.specularColor;
        m.specularPower = ri.specularPower;
        m.maps[Mesh::DiffuseMap] = texOverride;
        rc.makeCurrent(m);
        rc.lock();
    }

    model->render(rc);

    glx::glUseProgramObjectARB(0);
}


// Render the cloud sphere for a world a cloud layer defined
void renderClouds_GLSL(const RenderInfo& ri,

There is no reference to dealing with dxt5nm.
I don't think it is implemented.

selden wrote:

Areas that look like bumps with the bumpmap and png normalmap (middle and left spheres) look like depressions with the dxt5_nm normalmap (right sphere).

These normalmaps are being rendered on a sphere? I'll check it out in windows so I can run the cel. I'm in linux now.

cartrite

[cartrite]

I agree Selden. The dxt5-nm looks inverted. I thought some of the dxt maps I created looked a little funny. It looks like z is being inverted. Somewhere in here?

Code: Select all

source += "n.z = sqrt(1.0 - n.x * n.x - n.y * n.y);\n";

It may also be where the code reads in x and y.
Mabey the channels are being read in the wrong order ie alpha => x y => g.
cartrite

[selden]

My test case uses Celestia's default sphere, not a 3D CMOD mesh. Sorry I wasn't clearer.

[cartrite]

Yeah I seen that when I looked at the ssc. I didn't look at the package till I booted into windows. I agree, there is an inversion problem though. By the way, was that cell addr supposed to take me right there? I had to get there via the command line.
cartrite

[selden]

The cel addr takes me directly to it. However, I've been using the Solar System Barycenter offset that Chris provided a while ago. It hasn't been made the default yet, I think.

Code: Select all

Barycenter "Solar System Barycenter:SSB"
{
        RA 0
        Dec 0
        Distance 0
}

0 "Sol:Sun"
{
        OrbitBarycenter "Solar System Barycenter"
        CustomOrbit "vsop87-sun"

        SpectralType "G2V"
        AbsMag 4.83

        RotationPeriod      609.12  # 25.38 days
        Obliquity             7.25  # correct orientation relative to ecliptic
        EquatorAscendingNode 75.77  #
        RotationOffset       23.00  # standard meridian
}


[cartrite]

That's probally it. I don't think I made that change to my "installed"version. I know I changed the version that I run from the root build directory but not this one that was installed via the setup. Now it works.
cartrite

[cartrite]

Did you remember to create tangents for your CMOD model? That's one of the functions in the cmodfix utility. Tangents are required for normal maps to work on CMOD Mesh objects.

Unfortunately, I think I'm seeing bugs in the DXT5_nm support on both planets and models.

Here's a picture of what happens when I use a simple dxt5_nm normal map:



Areas that look like bumps with the bumpmap and png normalmap (middle and left spheres) look like depressions with the dxt5_nm normalmap (right sphere).

The dxt5 normalmap was created from the png normalmap using nvdxt v8.22 and the command

Code: Select all

nvdxt -quality_highest -file nm.png -nomipmap -dxt5nm 


A zip file containing the bump and normal maps, the command file I used to create them along with the ssc file and an html file with a Cel url is available at
http://www.lepp.cornell.edu/~seb/celest ... ump-nm.zip
(46KB)

.

What I find strange is that when you veiw your screenshot the top looks depressed and the bottom looks right. But when you run it and rotate so the veiw is from behind, the top looks right and the bottom looks depressed.
The shadows are being drawn on different sides depending on the veiw.

cartrite

[cartrite]

Selden,
This shot seems to correct your problem.

I made a different dxt5nm from your png normalmap . Here is the link
It can be found on the Bug page.
It seems to correct the problem. At first my dds map did the same thing as yours but this time I didn't generate mipmaps. For me at least, mipmaps seemed to cause the problem. But you didn't have any mipmaps. The main difference that I seen was you had a different alpha channel image than I had.

I did this when I decompossed your dxt5nm and noticed that it was completely different from the dxt5nm maps I produce with the gimp. Check it out.

No bug

EDIT

My dxt5nm started working after I edited shadermanager.cpp as follows

Code: Select all

if (props.texUsage & ShaderProperties::CompressedNormalTexture)
            {
                source += "vec3 n;\n";
                source += "n.xy = texture2D(normTex, " + normTexCoord + ".st).ga * 2.0 - vec2(1.0, 1.0);\n";
                source += "n.z = sqrt(1.0 - n.x * n.x - n.y * n.y);\n";
            }
            else
            {
                // TODO: normalizing the filtered normal texture value noticeably improves the appearance; add
                // an option for this.
                source += "vec3 n = normalize(texture2D(normTex, " + normTexCoord + ".st).xyz * 2.0 - vec3(1.0, 1.0, 1.0));\n";
                //source += "vec3 n = texture2D(normTex, " + normTexCoord + ".st).xyz * 2.0 - vec3(1.0, 1.0, 1.0);\n";
            }

This was the line 1334. I uncommented

source += "vec3 n = normalize(texture2D(normTex, " + normTexCoord + ".st).xyz * 2.0 - vec3(1.0, 1.0, 1.0));\n";

and commented out 1335

//source += "vec3 n = texture2D(normTex, " + normTexCoord + ".st).xyz * 2.0 - vec3(1.0, 1.0, 1.0);\n";

I rechecked the mipmap normalmap and it worked. So uncommenting the normalization line is what did it. It seems that this problem is ready to be dealt with.


/EDIT

cartrite

[rra]

Cartrite wrote:

I am starting a little cloud experiment to see if I can render a more realistic simulation of clouds. Basicly I want to create a simple sphere mesh and use it a number of times with different radii each having a different coudmap. By changing the rotation period, I found I can get them to move relative to the earth's position.

Cartrite , for the first part of your experiment,
did you only adapt the solarsys.ssc file or maybe more ?
If so, could you show me how your solarsys.ssc looks like
(I mean up to the first part, without the normap-map problems)


Ren?©

[cartrite]

rra, I started a new thread in Users, Cloud Experiment.

[cartrite]

I created a new dxt5nm to see if it would work on the original cvs code for Oct/4/2006 and it did. This time, before I remapped the channels, I ran the gimp's normalmap plugin to normalize only. Then I remapped and saved as dds with dxt5 and mipmaps. That worked with the original code. So this is an important step as Fridger stated many times before.

cartrite

[cartrite]

After reading some of the Nvidia manual on dxt compression and keeping in mind how dxtnm format is working in Celestia, wouldn't it be better to remap a png normalmap to blk=>r, r=>g, blk=>b, g=>a Anyone got any ideas on this one?

[chris]

After reading some of the Nvidia manual on dxt compression and keeping in mind how dxtnm format is working in Celestia, wouldn't it be better to remap a png normalmap to blk=>r, r=>g, blk=>b, g=>a Anyone got any ideas on this one?

That's exactly what Celestia requires: the x component of the normal is in the green channel, the y component is in the alpha channel, and the z component is generated in the shared as sqrt(1-x^2-y^2). I think that Selden is getting some strange results because nvdxt is using a different channel mapping (x component in red channel.)

Fridger has tested dxt5nm support and was getting very nice results; I'm pretty sure that the feature works correctly if you get the channels right.

--Chris

[chris]

To clarify things further:

When creating a dxt5nm texture, you can't just apply DXT5 compression to a PNG texture. You must first remap the color channels as follows:

png red => dxt5nm green
png green => dxt5nm alpha
png blue => discarded
png alpha => discarded

nvdxt does remap channels, but I think that it uses a different mapping than what Celestia expects (not verified--this is based on my recollection of a post by Fridger.) However, the NVIDIA DXT plugin for Photoshop does use a channel mapping compatible with Celestia.

--Chris

[cartrite]

I thought they were discarded. Before I was mapping the blue channel to the blue channel but after reading that Nvidia hardware handles 0's better, I thought it should be better to map the blue channel as black. I just produced a dxt5 map of mars with the blue hannel mapped as black. Here is a screenshot of that map on the 256x128 lores texture.


cartrite

[cartrite]

Chris wrote:

Fridger has tested dxt5nm support and was getting very nice results; I'm pretty sure that the feature works correctly if you get the channels right.

In case you overlooked this I would like to point out that after I remapped Seldens nm.png as
black => dxt red
red png => dxt green
blue png => dxt blue
green png => dxt alpha
Then compressing as dxt5, I got the same depressions Selden was getting with his dds normalmap.

It wasn't till after I normalized only with the Gimps normalmap plugin and then remapped as I shown above, that I got the correct results as shown here.



cartrite