Page 1 of 2

NightLight: 64k VT earth night texture

Posted: 13.09.2016, 11:23
by nardo
i've edited the 2012 VIIRS map to appear dimmer, blacker, and more realistic, then tiled it with 2k tiles. below, i've put the full add-on along with screenshots and a 4k preview. i really liked the look so i've decided to finalize and share.

this is my first add-on, so please give suggestions. if you need it in a different tile size, i'll be happy to upload another version.
screenshot1.jpg
view from iss orbit

screenshot2.jpg
view from space (using jestr's JMII texture and blue marble clouds)


edit: now available as dds as well as jpg

edit: dds version now available in 1k tiles as well as 2k tiles

Posted: 13.09.2016, 12:29
by FarGetaNik
Good one, thanks for sharing! I also found the night texture to be too bright, just compare it to theese images of Rosetta:
http://www.esa.int/Our_Activities/Operations/Cities_at_night_Extraordinary_Rosetta_images
Night lights would probably be invisible to the naked eye. They become visible only on long-exposure photographs. I also never got why to display a blue surface, I mean the ground doesn't glow right? Your version is a lot more realistic. It might be still too bright, but they should remain visible, so it's just fine. I'm not sure if the lights are too yellowish, but there might not even be true-color data. Where did you get the data from? I only found a 16k texture online.

Right and... virtual textures are incompatible with cloud shadows but that's a different issue :zombie:

Posted: 13.09.2016, 12:33
by Goofy
Hi nardo.
Could you please upload a single level6 2K image of a well know place, e.g. a big city, in order to evaluate the correspondence with space taken images?
At a first glance, images look a bit blurry and excessively dim, in my opinion. :wink:
Thank you.
Bye
Goofy :smile:

Posted: 13.09.2016, 12:44
by nardo
FarGetaNik
thank you! i got the full (54k x 27k) image by putting together the tiles here: http://earthobservatory.nasa.gov/Features/NightLights/page3.php

then, i tried to match observations from the ISS since that's what i was most interested in. the readme shows the steps i took. however, my computer can barely hold half of the full res image in gimp so i simplified the process by making it grayscale first then adding the hue later. this might be why it seems too yellow. still, i think it's pretty close to the true color images. they're definitely brighter than life, but as you said, they should stay visible. besides, that's what ctrl+l is for.

as for the cloud shadows, is that an issue on the night side of earth?

Added after 24 minutes 29 seconds:
Goofy
sure thing. here's a couple from the highest level. the blurriness is either exposure or the fact that i'm making a 64k image out of a 54k one. the dimness is intentional but i have a brighter version saved just in case.

Posted: 13.09.2016, 13:17
by FarGetaNik
Thanks for the link! Yeah my laptop also can barely handle such huge textures, my limit is somewhere at 20-25k, then each step takes about an hour to process and Gimp will likely crash.

nardo wrote:besides, that's what ctrl+l is for.
So that's what it is good for? :think: makes sense, but I still don't like exaggerated brightness.

nardo wrote:as for the cloud shadows, is that an issue on the night side of earth?
No but when using a virtual texture Celestia completely refuses to render cloud shadows, so when using your addon they're gone for good. But I fixed this by making your texture an alternative surface, so I can switch back and forth when I need the one or other :wink:

Posted: 13.09.2016, 13:21
by nardo
oh so that's why mine always freak out. all my textures are VTs lol.

Posted: 13.09.2016, 13:40
by FarGetaNik
Then how do you create VTs?

Posted: 13.09.2016, 13:46
by nardo
i meant the cloud shadows always freak out. my laptop can handle up to a 32k square in gimp, so i can process a 64k image in two parts. just verrrryy slowwlllyyy. anything else i have to do from the command line. it's a step up from my last laptop which could barely even open an 8k.

Posted: 13.09.2016, 14:38
by FarGetaNik
nardo wrote:i meant the cloud shadows always freak out.
Why is that? Don't you like the effect or are they also causing bugs? I experience black planet surfaces because of them sometimes.

I meant if all your textures are VT, how do you split them up into tiles? The ftex tool? It didn't work for me, that's why I'm asking. Doing that manually would be insane.

nardo wrote:my laptop can handle up to a 32k square in gimp, so i can process a 64k image in two parts. just verrrryy slowwlllyyy. anything else i have to do from the command line
Then you have a better computer than me. I could not even load the texture you posted a link to... I hope the new version of gimp keeps its promise and works much faster. But I will need a new computer to really think about processing such huge textures.

Posted: 13.09.2016, 15:03
by nardo
how do i tile them? there's an old shell script for imagemagick by t00fri called virtualtex that tiles everything for me. i just have to make the textures and resize them for each level, then it takes care of the rest. it was a pain getting cygwin to work on my laptop and from what i've read it's an outdated method but it works fine for me.

Code: Select all

#! /bin/bash
if [ $# -lt 3 -o "$1" = "--help" ]; then
  echo
  echo 'Usage: virtualtex [--help | <texture name> <tile size> <tile format>] [e|E|w|W]'
  echo
  echo
  echo The shell script \'virtualtex\' is a tool for Celestia \> 1.3.0
  echo that supports \'virtual textures\'.
  echo
  echo The script generates the required tiles tx_i_j of \
       desired \(square\) size, 
  echo \<tile size\>, in a specified format, \<tile format\> = \
       png, tga, jpg,...,
  echo from an input texture, \<texture name\>, in any popular graphics format.
  echo
  echo The optional 4th argument e\|E\|w\|W is for the case of tiling 
  echo square /e/astern \| /w/estern halfes of the full texture separately!
  echo 
  echo Besides Linux/Unix, the script also runs in a current Cygwin \
       installation
  echo under Windows, \( http://www.cygwin.com \). If the z-shell \(\'zsh\'\) \
       is unavailable,
  echo it also may be executed with the \'bash\' shell, by replacing \
       \#\! /usr/bin/zsh by
  echo \#\! /bin/bash in the first line.
  echo
  echo The script assumes that a recent version \(\>= 6.1.8\) of the ImageMagick package
  echo \( http://www.ImageMagick.org \) is installed \
       \(either under Unix/Linux or Windows\).
  echo The utilities \'convert\' and \'identify\' of that package are used.
  echo
  echo You may increase the pixel cache size \$maxmem from the 80 MB default
  echo value to e.g. 80\% of your RAM size within the script with an editor.
  echo This will speed up the performance of \'virtualtex\' considerably.
  echo
  echo On a PIII/512MB RAM the tiling of a 16k x 8k texture into
  echo 32 \(2k x 2k\) tiles now only takes 15 minutes with a Linux OS!
  echo
  echo Author: Dr. Fridger Schrempp, fridger.schrempp@desy.de
  echo Version: 1.04, 02/15/05
  echo 
else
maxmem=80
texturesize=`identify -format "%wx%h" $1`
texturewidth=`identify -format "%w" $1`
textureheight=`identify -format "%h" $1`
tilesize=$2
tileformat="$3"
echo
echo "Texture size = " $texturewidth "x" $textureheight "tilesize = " $tilesize
echo "Number of tiles =" $(( ((texturewidth/tilesize)) * ((textureheight/tilesize)) ))
echo "Image format of tiles:" $tileformat
echo
echo "Tile: "
echo

convert -cache $maxmem -crop ${tilesize}x${tilesize} $1 out%d.$tileformat


j=0
while (( j * tilesize + tilesize <=  textureheight )); do
    ((offy = j * tilesize))
    if [ $# -eq 4 ]; then
   if [ "$4" = "e" -o "$4" = "E" ]; then
       ioff=$(( texturewidth/tilesize ))
   elif [ "$4" = "w" -o "$4" = "W" ]; then
       ioff=0
        else
            echo
            echo "*** Incorrect 4th parameter! ***"
            echo
            return
        fi
    fi
    i=0
    while (( i * tilesize + tilesize <= texturewidth )); do
        ((offx = i * tilesize))
        echo "tx_"$((i + ioff))"_"${j}":  x-offset:" $((offx + ioff * tilesize)) "y-offset:" $offy
    mv out$((i+ ((texturewidth/tilesize)) * j)).$tileformat tx_$((i + ioff))_${j}.$tileformat
        ((i++))
    done
    ((j++))
done
fi

(thank you John VV for uploading it to celestialmatters)
i just type "NightLight32k.png 2048 jpg" and boom, hundreds of tiles pour out.

Posted: 13.09.2016, 16:17
by FarGetaNik
nardo wrote:it was a pain getting cygwin to work on my laptop and from what i've read it's an outdated method but it works fine for me
Yes I was failing at cygwin also. And I didn't got imagemagick to work. If it's outdated then is there an alternative?

Posted: 13.09.2016, 16:22
by Goofy
Hi nardo.
In my life I travelled by air for almost one million miles, so I had the possibility to see a lot of times lighted nightscapes.
I can say for sure that, if there are no clouds and no haze, fog or whatelse between your aircraft and the ground, what you can see is absolutely closer to the right image than to the left one.
I modified only contrast and luminosity, nothing else.
BTW, reducing such a way luminosity and contrast, you killed resolutiom feeling too, so that it looks like a smaller image.
Just my opinion, obviously. :wink:
My little cent.
Click on the image to see it full size.
Bye
Goofy :smile:

tx_22_5-MODDED.jpg
On left your version, on right mine....


Added after 34 minutes 31 seconds:
Hi nardo.
I just downloaded your zip, to clarify a doubt I had looking at its very small dimensions (only 56MB for a 64K VT, while we are used to hundred MBs size of Motherlode's VTs!).
So I found the reason of this: it's up to Level4 instead of Level6, so really the VT is a 16K, not a 64K. :eek:
Am I missing something?
Thank you.
Goofy :smile:

Posted: 13.09.2016, 17:29
by FarGetaNik
Goofy wrote:So I found the reason of this: it's up to Level4 instead of Level6, so really the VT is a 16K, not a 64K. :eek:
Am I missing something?

I checked and found the resoluton WAY better than the 8k night texture I use (the 16k versions lets Celestia crash) so it can't be just a 16k. I checked and found the tile size to be 2048, so level0 is already 4k. So 1 is 8k, 2 is 16k, 3 is 32k and 4 is 64k. I guess the low file size is due to the fact that it is jpg and the large black areas are easy to compress. That lets me wondering if png would be even more effective compressing the vast oceans... or you could just through out every tile that contains nothing but black, Celestia doesn't care, I tried that with a specular map, it worked just fine.
But one more thing, I found tile size of 512 a bit unhandy, since you'll end up with thousands of tiles and the Celestia constantly has to load different ones. But 2048 pixel might be a bit too much, but it loads smoothly when a jpg as used here, so it's just fine. But a 2k tile in png might cause some performance problems maybe...

Added after 5 minutes 48 seconds:
Goofy wrote:In my life I travelled by air for almost one million miles, so I had the possibility to see a lot of times lighted nightscapes.
I can say for sure that, if there are no clouds and no haze, fog or whatelse between your aircraft and the ground, what you can see is absolutely closer to the right image than to the left one.
I modified only contrast and luminosity, nothing else.
BTW, reducing such a way luminosity and contrast, you killed resolutiom feeling too, so that it looks like a smaller image.
Just my opinion, obviously. :wink:

Ok that's your opinion, but let me explain my thoughts on that, don't take that personally. Our eyes are good in adaption brighness, so the city lights just appear bright at night. I think the link I posted proves the brightnes is nothing compared to the sunlit surface of the Earth. In the picture you posted it might seem blurrier, but that's most likely due to the contrast difference between the images. When in Celestia and adapting to the brighness of the lights, you might just get the same apparent sharpness. And obviosly stretching contrast doesn't add real detail in the image but only amplyfies the details already present.

Posted: 13.09.2016, 20:15
by nardo
Goofy
what FarGetaNik said. however, if you can tell me exactly how you made your version (ie which numbers you used for lum and contast), i'll give you the 64k texture for it. maybe we can call it NightBright.

FarGetaNik
i'll see if my laptop can make a 1k tile version and post it here. pretty sure that's the same tile size as Jestr's texture, which even my last laptop could handle. perhaps we can call it NightLite.

Posted: 13.09.2016, 20:41
by FarGetaNik
nardo wrote:FarGetaNik
i'll see if my laptop can make a 1k tile version and post it here. pretty sure that's the same tile size as Jestr's texture, which even my last laptop could handle. perhaps we can call it NightLite.

Actually your version works surprisingly fine! It would be an issue if using png I guess.

Posted: 13.09.2016, 20:46
by selden
You might want to consider using DDS format for the final VT instead of PNG. That way the tiles load directly into graphics memory without having to be expanded to full size. DDS is lossy, so you don't want to use it for your intermediate format, of course.

Posted: 14.09.2016, 03:25
by John Van Vliet
i think you might find that using 512x512 tile sizes works best or 1024x1024

2k tiles are a bit big

also why start with the bluemarble jpg's that are way OVER exposed they look like the top 10% are set to 255

NOAA has the monthly data
the June 2016 "VCMSLCFG" data
http://ngdc.noaa.gov/eog/viirs/download_monthly.html

my home town area " Metro Detroit Area "
the viits data and the above jpg in the leve4 folder
Image Image

long Island and NY.NY
Image

now the monthly data needs a bit of work ( mainly near the poles ) and it is black and white 32 bit float tif's

Posted: 14.09.2016, 03:39
by nardo
Selden
the dds version is being tiled as i type this.

John
WHOAH! thanks for this. they're definitely overexposed but now i have something new to play with.

Posted: 14.09.2016, 08:22
by FarGetaNik
John Van Vliet wrote:NOAA has the monthly data
the June 2016 "VCMSLCFG" data
http://ngdc.noaa.gov/eog/viirs/download_monthly.html

Wow they are huge! One part is more than a GB! :eek: Would be great if someone could process this.
I agree they seem a bit overexposed, but it's even worse with what I got to work with. And I knew this was black and white.

Posted: 14.09.2016, 20:34
by John Van Vliet
Would be great if someone could process this.
i did a few years ago

http://forum.celestialmatters.org/viewtopic.php?f=9&t=606
http://forum.celestialmatters.org/viewtopic.php?f=9&t=451

the links are dead but i can reupload them

one is here
the 8 bit/Chanel red,green,blue colorized "64k.EarthLightsRGB.1.7.14.ppm.7z"

https://drive.google.com/file/d/0B6ZYAd08tZL-Wjg0a2pma2xUQmM/view?usp=sharing