New Cloud Map

[cartrite]

I decided to start a thread documenting the progress and problems I'm having with producing a new cloudmap with MODIS images. This will probably never be a project for the faint of heart. There is a lot of files to download. I haven't downloaded all the files I need yet and I even download some I didn't need. So far I'm pretty sure that the files needed are the 500m MOD02HKM(Terra) and MYD02HKM(Aqua) along with the MOD35_L2 and MOD06_L2 cloud products. The following images show some of the problems I'm having with producing a mosaic with HDFLook. You should be able to notice the tone difference between some of the shots, probably due to the different times of day the images were acquired. Some of low clouds also have a blocky look to them. And there is some noise from missing data in some of the areas being introduced by the cloud masks. This first version is in no way complete. I ran this to get an idea of what else needs to be done.
I'm pretty sure that I'm going to have to import raw unprojected images into ISIS to clean up some noise and tone match the images before projecting them. At that point I'm not sure what to do next. I think I'm going to have to figure out how to export them back to the hdf format and use HDFLOOK to do the mosaics.

Anyhow, just using HDFLOOK, I produced a 500m mosaic with the MOD02HKM files and a 1km cloud mask mosaic using the MOD35_L2 files.
Commands to produce the cloud mask:

Code: Select all

HDFLook band1-mosaic-1strun-linear-topleft.com
HDFLook band1-mosaic-run-topleft-linear.com

HDFLOOK uses these command files (.com) to do the runs.
band1-mosaic-1strun-linear-topleft.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=43200 HeightTo=21600\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file              MOD35_L2.A2008243.1600.005.2008244015648.hdf
select_SDS                 SDSname="Cloud_Mask"
create_MODIS_SDS_Mosaic        FileName="band1-earth-clouds-mask.raw"  Index=6 ClearFile=Yes


band1-mosaic-run-topleft-linear.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=43200 HeightTo=21600\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file      ../IMAGES/MOD35_L2*
select_SDS                 SDSname="Cloud_Mask"
create_MODIS_SDS_Mosaic        FileName="band1-earth-clouds-mask.raw" Index=6 ClearFile=NO


Commands to produce the 500m mosaic:

Code: Select all

HDFLook mosaic-1strun-linear.com
HDFLook mosaic-run-all-linear.com

Command files:
mosaic-1strun-linear.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=86400 HeightTo=43200\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file              MOD02HKM.A2008243.1600.005.2008244014050.hdf
select_SDS                 SDSname="EV_500_RefSB"
create_MODIS_SDS_Mosaic        FileName="earth-clouds"  Index=1 ClearFile=Yes


mosaic-run-all-linear.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=86400 HeightTo=43200\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file      ../IMAGES/MOD02HKM*
select_SDS                 SDSname="EV_500_RefSB"
create_MODIS_SDS_Mosaic        FileName="earth-clouds" Index=1 ClearFile=NO


Here are some of the problem images. I'll post 3 images of the shots I'm somewhat pleased with in the following post.

problems1.jpg

problems2.jpg

problems3.jpg

After the cloud mask mosaic was produced, I had to change the file from 8 bit signed to 8 bit unsigned. I wrote a little program to do that. The reason being was that ISIS3 currently does not import 8 bit signed data. After the conversion, I imported the 2 files into ISIS3. I then ran fx on the 2 images. Here is the output from running the fx program:

Code: Select all

Object = fx
  IsisVersion       = "3.1.17 | 2008-09-10"
  ProgramVersion    = 2008-04-16
  ProgramPath       = /srv/disk5/isis3/isis/bin
  ExecutionDateTime = 2008-10-08T01:58:47
  HostName          = linux-o863
  UserName          = cartrite
  Description       = "Generalized arithmetic operations using multiple cube
                      files"

  Group = UserParameters
    F1       = band1-earth-clouds.cub
    F2       = cloudmask.cub
    TO       = earth-clouds2.cub
    EQUATION = "(f1 * (f2!=127))-128"
    MODE     = CUBES
  End_Group


127 was the value in the cloud mask for non cloudy land. I think. Anyhow, the fx equation used the pixel value from the 500meter mosaic if the cloud mask didn't have a value of 127. This was -1 in the original signed 8 bit file. Subtracting 128 at the end may be unnecessary.

After that I cropped the output 21600x21600 to produce files that are easier to work with and exported them to png format with isis2std.
cartrite

[cartrite]

Here are some of images I'm somewhat pleased with. This is only a 32k cloud map for now. The MOD35_L2 also has a 250 meter cloud mask but I haven't figured out how to extract it yet.

cloudmap1.jpg

cloudmap2.jpg

cloudmap3.jpg

cartrite

[t00fri]

Splendid!

Fridger

[ElChristou]

Here are some of images I'm somewhat pleased with. This is only a 32k cloud map for now...

A nice 32 k would be a huge step from what we have now. Even a 16k would be cool...

[cartrite]

Thanks Fridger and Chris,
Some areas do look good.
If I ever get a good global result here, I'll make a 64k, 32k, 16k, 8k, 4k, etc.
I've already downloaded over 20 gb of data. To do a 250meter map would require more downloaded files. Maybe I'll do a 250m (128k) map after I work out all the kinks. I do have the ability of doing a 64k map with the files I have now though. I would probably be better off doing a 250 meter and then I'll have all sizes. But there is a warning about the quality of the 250 meter cloud mask. I'll need to see it before I can make any decision though. Apparently, it's in the MOD35_L2 files. Just don't know where or how to extract it yet.
cartrite

[John Van Vliet]

--- edit ---

[cartrite]

to bad my nvidia card has a 4k max using CloudSpeed 0
2k max with CloudSpeed 20

Since a majority of the clouds in this map are dependent on the land they form over, setting cloud speed > 0 would not work with this cloud map. As far as max texture size, I'm not running the 32k map in one piece. It's a VT. 1k tiles.
cartrite

[cartrite]

Apparently I goofed!!!!!!
The cloud mask I used was the 250 meter cloud mask (band6). Actually, half of the 250 meter cloud mask. The 250 meter cloud mask uses bands 5 and 6. It is still the same dimensions as the 1km which through me a curve. The 250 meter cloud mask is represented by 1 bit. 0 = cloud. 1 = no cloud. A visible test. So all I need do is create a byte for every bit. 4x4. In other words from band5, bit 1 = pixel 1,1. bit 2 = pixel 1,2. bit 8 = pixel 2,4. etc. From band 6, bit 8 = 4,4. 16 pixels for every byte. I will need to write some code to build this file.

For now, I'm going to try another run with the first 4 bands combined.
That should be a better match for the 1 km data I'm using. Then I'll have to come up with the range to mask the visible 500m data I projected to 32k with fx from ISIS.
cartrite

[cartrite]

I have a similar thread at CM. Today I added a post to that thread that describes a problem I was having producing a rgb texture. I also wanted to add this post but was unable to because CM would not let me post the sections that have C code in it.
Mask creation
I ran HDFLook using some command files to create 2 different cloudmasks. I used the cloud masks to eliminate land values that were not covered by clouds so they would not be in the final cloud texture.The first mask was created with band1. This was a simple incomplete mask that only queried the presence of cloud. The second mask was done with bands 5 and 6. These bands are supposed to represent a 250 meter cloudmask where each bit represents a byte of the mask. To create a 500 meter mask from these files I used bits 5 and 6 from band5 and bits 1 and 2 from band6. These are the center 4 bits in the 16 bit box that each byte from the 2 files represent. Below are the actual command files to produce the data. I was using the ISIS program fx, but that was creating 32bit cube files which were getting too large. So I wrote a couple of simple c programs that look at the data files and create the actual cloudmasks. Here are command files to produce the data for the 2 cloudmasks. The second mask also uses the quality control data that simply flagged if the test was actually done or not done. In other words, if a cloud was present in bands 5 and 6, the bit would equal 0. If a test was actually done the bit in the corresponding quality band would be set to 1. If it was set to 0, no test was done and the program changed the cloud bit to a no cloud.value.
64k-1strun-band1-mosaic-cloudmap-mask.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=65536 HeightTo=32768\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file              MOD35_L2.A2008243.1600.005.2008244015648.hdf
select_SDS                 SDSname="Cloud_Mask"
create_MODIS_SDS_Mosaic        FileName="64k-band1-earth-clouds-mask.raw"  Index=1 ClearFile=Yes


64k-runall-band1-mosaic-cloudmap-mask.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=65536 HeightTo=32768\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file      ../IMAGES/MOD35_L2*
select_SDS                 SDSname="Cloud_Mask"
create_MODIS_SDS_Mosaic        FileName="64k-band1-earth-clouds-mask.raw" Index=1 ClearFile=NO


1strun-band5-mosaic-cloudmap-quality.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=43200 HeightTo=21600\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file              MOD35_L2.A2008243.1600.005.2008244015648.hdf
select_SDS                 SDSname="Quality_Assurance"
create_MODIS_SDS_Mosaic        FileName="band5-earth-clouds-quality.raw" Index=0,0,5 ClearFile=Yes


1strun-band5-mosaic-cloudmap-mask.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=43200 HeightTo=21600\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_input_hdf_file              MOD35_L2.A2008243.1600.005.2008244015648.hdf
select_SDS                 SDSname="Cloud_Mask"
create_MODIS_SDS_Mosaic        FileName="band5-earth-clouds-mask.raw"  Index=5 ClearFile=Yes


Here is the core from the C programs.
cloudmask from bands 5 and 6.

Code: Select all

        for (int x = 0; x < width; x++)
        {
       int r1 = 0;
       int r2 = 0;
       int r3 = 0;
       int r4 = 0;
            unsigned char r;
            unsigned char s;
       r = samples[y][x];
       s = secondfile[x];
       bitset<8> cloudmask = r;
       bitset<8> cloudmask2 = s;
       if (cloudmask[5] == 0)
      r1 = 128;
       else
      r1 = 10;
       if (cloudmask[6] == 0)
      r2 = 128;
       else
      r2 = 10;
       if (cloudmask2[1] == 0)
      r3 = 128;
       else
      r3 = 10;
       if (cloudmask2[2] == 0)
      r4 = 128;
       else
      r4 = 10;

       ro1[x * 2 + 0] = (unsigned char) (r1);
       ro1[x * 2 + 1] = (unsigned char) (r2);
       ro2[x * 2 + 0] = (unsigned char) (r3);
       ro2[x * 2 + 1] = (unsigned char) (r4);

        }

   fwrite(ro1, width * 2, 1, stdout);
   fwrite(ro2, width * 2, 1, stdout);
    }

Quality check for bands 5 and 6.

Code: Select all

        for (int x = 0; x < width; x++)
        {
       int r1 = 0;
       int r2 = 0;
       int r3 = 0;
       int r4 = 0;
            unsigned char r;
            unsigned char s;
       r = samples[y][x];
       s = secondfile[x];
       bitset<8> cloudmask = r;
       bitset<8> cloudmask2 = s;
       if (cloudmask[5] == 1)
      r1 = 128;
       else
      r1 = 10;
       if (cloudmask[6] == 1)
      r2 = 128;
       else
      r2 = 10;
       if (cloudmask2[1] == 1)
      r3 = 128;
       else
      r3 = 10;
       if (cloudmask2[2] == 1)
      r4 = 128;
       else
      r4 = 10;

       ro1[x * 2 + 0] = (unsigned char) (r1);
       ro1[x * 2 + 1] = (unsigned char) (r2);
       ro2[x * 2 + 0] = (unsigned char) (r3);
       ro2[x * 2 + 1] = (unsigned char) (r4);

        }

   fwrite(ro1, width * 2, 1, stdout);
   fwrite(ro2, width * 2, 1, stdout);
    }

cloudmask-check This checks the cloudmask against the quality-check file to see if the cloudtest was done.

Code: Select all

        for (int x = 0; x < width; x++)
        {
       int r1 = 0;
            unsigned char r;
            unsigned char s;
       r = samples[y][x];
       s = secondfile[x];
       if (s == r)
      r1 = 128;
       else
      r1 = 10;

       ro1[x] = (unsigned char) (r1);

        }

   fwrite(ro1, width, 1, stdout);


cloudmask from band1

Code: Select all

        for (int x = 0; x < width; x++)
        {
            unsigned char r;
       r = samples[y][x];
            bitset<8> cloudmask = r;
       if (cloudmask[0] == 1 && cloudmask[1] == 0 && cloudmask[2] == 0)
      r = 128;
       else
      r = 10;
       ro[x] = (unsigned char) (r);
        }

   fwrite(ro, width, 1, stdout);


The actual files I created that had the visual data for clouds were run with these command files. The first run was a band1 mosaic. This had some noise in the form of horizontal lines. The second run done in rgb mode, cleaned up this noise but had many holes. These holes were set to NULL when I imported the files into ISIS. So I was able to use handmos and place the input file below the rgb mosaic and fill in the holes.

rgb-mosaic-1strun-linear.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES
set_input_directory             ../1km-IMAGES

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=65536 HeightTo=32768\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_misc_options        CLOUDSRGBENHANCE=Yes
set_input_hdf_file              MOD02HKM.A2008243.1440.005.2008277185827.hdf
create_MODIS_RGB_Mosaic        FileName="rgb-earth-clouds" ClearFile=Yes  RGBMode="LinearStandard" \
minR=0.0 maxR=20000 minG=0.0 maxG=20000 minB=0.0 maxB=20000


rgb-mosaic-run-all-linear-standard.com

Code: Select all

verbose
clear_data
set_output_directory            ../OUTPUTS/
set_input_directory             ../IMAGES/
set_input_directory             ../1km-IMAGES/

set_projection_to_geometry ProjectionTo="LINEAR" WidthTo=65536 HeightTo=32768\
                        LatitudeMinTo=-90 LatitudeMaxTo=90.0 \
                        LongitudeMinTo=-180.0 LongitudeMaxTo=180.0 \
         CenterLatitudeTo=0 CenterLongitudeTo=0

set_misc_options        CLOUDSRGBENHANCE=Yes
set_input_hdf_file              ../IMAGES/MOD02HKM*
create_MODIS_RGB_Mosaic        FileName="rgb-earth-clouds" ClearFile=NO  RGBMode="LinearStandard" \
minR=0.0 maxR=20000 minG=0.0 maxG=20000 minB=0.0 maxB=20000


I've still been working on these. I discovered a way of cleaning up some of the noise. I had limited success doing a rgb run. Somehow it cleans up some of the horizontal stripes I've seen in certain areas that appear when I do band1. The rgb run produces tiny holes all over though. The holes do have a value of zero which allows me to mosaic the band1 and red channel from the rgb run together. The trick here is to "stretch" the band 1 mosaic with isis2raw so band1 has the same value range as the red channel. Then handmos can be used to fill in all the holes.
The rgb run also allows the image to be brightened or darkened. But an entry for every tile has to be entered in the command file. This will certainly prove to be a tedious affair.
Also due to the polar orbit of the MODIS satellites, there is much overlapping in the northern and southern latitudes. But the worst problem is the gaps at the equator. Most of this can be filled in with the other satellite. I'm using the Terra satellite so I'm going to try to fill the gaps with data from the Aqua satellite.
So far, I've only done a mosaic of band1 and the red channel from the Terra satellite. I haven't wrote the command file to individually change the brightness or darkness levels to blend in shots taken at different light levels. I also didn't try to blend the changes in cloud positions. But here are a few screen shots. Even in this raw state, some close ups look pretty good. Some images can be seen at CM. http://forum.celestialmatters.org/viewt ... =5756#5756

cartrite