Celestia Forums

Hey Chirs

Just wanted to run this by you to see if I can persuade you to change the directory structure before it gets out of hand. I think most people will like and accept the proposed structure change.

Proposed New Directory Structure

Ok there is a lot of information here. I will start off first with the NEW x_universe.ssc files. These files are a simple pointer files that will define the available universes people can select from in the future (this has been a frequent request, but you know that) it will allow you to turn on and off the universes by a simple click, or possible even just part of them if you want the granularity. Sticking with the basic structure of the scc, the universe file would look something like this;

Proposed x_universe.ssc files


Zipped Universes**
The zipped universes are a simple idea to save space and make installation of universes easier. The basic structure and rules apply as listed above; the only difference is that the universes are zipped into a single package.

Proposed New Directory Structure Continued

As for changes to some of the subdirectories in the current version of Celestia. I think you can get rid of the high, medium, and low image quality directories in the new format and just rely on the developers just to name the images for the different resolutions differently.

A good example is the high rez images of earth. There are three different files for it each with its own name;
earth8k-BM-col-bmp-um.dds
earth4k-BM-col-bmp-um.dds
earth2k-BM-col-bmp-um.dds
so I think by putting them in different directories kind of useless to me.

Final Notes.
If you like this idea or don't understand something let me know. Other peoples constructive ideas accepted too! I don't mind making the changes, even if I have to do it myself, as long as its sooner rather then later.

-Matt-

Matt,

Obviously you've been thinking about the details longer than I have, but let me do some thining "out loud". This isn't intended to be a polished response.

As I understand the problem, it's that celestia currently looks in a single tree of directories for all the files it needs. When people add packages (which can be as simple as a single orbit or as complicated as a whole universe of stars and planets) they currently have to put them all in Celestia's existing directories. When there are name conflicts, then one has to rename one of the files or edit the conflicting files, making one's own changes.

As a result, when a new release of Celestia becomes available,, or a new release of one of the add-on packages, all that work of merging files and resolving name conflicts has to be done all over again.

It seems to me, conceptually, at least, one way to resolve thie issue would be for each add-on package to replicate (the necessary parts of) Celestia's directory tree structure for its own directory tree.

Within the package's directory tree, it would use appropriate names for the various elements of its design. (Some would be the same as Celestia's files. e.g. solarsys.ssc. Within them the add-on package migh make only one change or addition (e.g. just providing the name of a replacement Earth cloud texture.) Only that element would be used to update Celestia's internal database.)

A change to Celestia would be to provide a way to provide a pointer to the package's directory tree (e.g. either in celestia.cfg or in a file explicitly used for listing addon packages, similar to what you've suggested). When Celestia starts up it could read the list of package directory trees, loading them in order, starting with its own.
That way when there are conflicts, the last add-on package's contents (like the texture mentioned above) would override whatever previous conflicting definitions might exist.

(An initial simplified implementation might just load them all in order, replacing already loaded elements by the ones encountered later. A later optimization could be to pre-scan all of the directories and definition files [i.e. the .ssc and/or .xmlf iles] and load only the "last" object when there are conflicts.)

This suggestion has the advantage, if one can call it that, that Chris doesn't have to reorganize the existing tree at all. In principle, only one file would have to be added or changed in Celestia's directories: the list of the roots of addon package directory trees -- and those add-on directory trees could be anywhere on the user's system. (even on a CD, for example.)

What do you think? Is this in conflict with the method you've worked out? Or am I restating it in somewhat different terms?

I agree it would be nice for customizations to be able to specify only the changes (e.g. new cloud texture), without having to duplicate other parts of the definition. Also, it would be nice to be able to apply customizations on the fly, without having to restart Celestia. The "last loaded" (override) paradigm, with scriptable loading, seems like a possible approach. For file-based items (e.g. textures) it seems preferable to use customization scripts with different file names, rather than having to rename or copy files to specified names. For convenience in packaging customizations, filenames could be specified relative to the directory of the load script. The script writer would be responsible for organizing the files within the package as seems appropriate. The initial root load script could be invoked in the Celestia startup script. Alternatively, at startup Celestia might search for package directories and invoke their load scripts automatically. Just a thought...

- Hank

Selden, No you are more or less right on. There is only two sight difference form the way I see it and the way I think you stated it.

The first difference would be with having to enter the celestia.cfg file. My idea was just to say any file in this root directory of the data would be the universe file, and each file would have everything it need to run that universe. This would require an upfront scan for detecting those file when Celestia first launches. (no biggy I think)

Second difference would be the removal of the different resolution directories in place of a common texture directory and just rely on the developing staff to take care of naming textures that or different quality to different names.

HankR

You have it right on

HankR,

I limited the scope of my suggestions because I was hoping to minimize the actual source code changes, and thus have a chance of it being implemented easily=quickly. A list of directory roots (not necessarily in celestia.cfg) and identical directory structures under them means that (in principle) one might be able to implement it by just looking for one more initialization file and if found, loop through the object setup code once for each listed directory. I'm sure there are assumptions in the initialization code that'd make even this approach slightly difficult, though.

I believe that if you look at the Coding for the systems in Celesia you will find that there is no reference for the path for different textures

Example:
LowTexture "earth.png"
HighTexture "earth8k.dds"

So knowing that, its something that just have to be updated in the program itself. As for textures conflicts , looking at them there are only about 40 conflicts, witch could be fixed in a day by just renaming them.

A better system for organizing directories is certainly needed . . .

First of all, I'd like to make the celestia.cfg file more flexible. It will have the usual lines saying which star, star name, and constellation databases to load. The SolarSystemCatalogs and ExtrasDirectory lines will be replaced with a DataDirectories directive that specifies a list of directories to search for .ssc files. The order is important for overrides. To mimic the current behavior of Celestia, you'd use:

DataDirectories [ "data" "extras" ]

All subdirectories will also be searched. Also, Celestia will accept the name of a config file on the command line . . . You'll be able to make shortcuts (or shell scripts) that call Celestia with different command lines for different 'universes.' The is almost exactly what Selden described.

Overrides will only work on a per-object granularity . . . Allowing per-field overrides would be convenient in some cases, but I think it would generate complex dependencies that could make managing Celestia add-ons a huge headache. I feel that the usefulness of per-field overrides is overestimated . . . In practice, it's unlikely that a planet modified by a bunch of different .ssc files will look that good.

The textures directory is completely out of control . . . There are two things that need to be fixed:

- Textures should live in the same directory as the .ssc file in which their referenced (they're not required to be in the same directory, but this will be the common case for add-ons.)
- The texture resolution situation needs to be sorted out. It should be possible to install a high-resolution texture pack without touching any .ssc files. And the separate lores/medres/hires directories are no longer practical if textures are in the same place as .ssc files.

I'm reluctant to encode the resolution in the texture name, but other approaches have drawbacks as well. Entering the names of all three textures resolutions in the .ssc file like this:

Texture [ "mars-low.jpg" "mars.jpg" "mars-high.dds" ]

. . . seems like a pain. But then again, most add-on creators only provide a single resolution. This scheme also fails to accomodate different file extensions. An alternative is to omit the extensions altogether and let Celestia decide the priority for .jpg, .png, .dds, etc.

Zipped packages would be great to have, but I want to focus on getting the directories sorted out first.

--Chris

Ok Chris let me try to explain what I was trying to do with my version of the directory structure. I was trying to make it so the user did not have to modify any of the files to install the updated universes, and in such there is only two way to accomplish this. First way is to somehow merge the new data into a single file, or the second way, the creation new file type to handle the universes nothing but the universes. This is what I was trying to accomplish by labeling the proposed “x_universe.ssc” files. The universe files would be placed in there own directory so when Celestia loads, it just has to read all the contents of the directory to get all the universes.

About the universe files:

Each universe file would include all the required parts to make that universe work. The best example that I can think of is the fan Star Trek Universe. Star Trek Universe would utilize the known universe and perhaps even the extended universe, in addition to its own universe. So the Star Trek Universe is really three universes compiled into a single one during the launch of Celestia. This will cut down the amount of redundant data immensely, because it’s pulling already existing data. If there is any redundant data, say two night maps for earth, the real one and the Star Trek one. Witch one has priority is simply decided by which one comes first in the universe file. As an added benefit to this structure, if the first choice of data is somehow bad, it defaults to the next in line. Or in this case is the Star Trek night map is bad, Celestia truncates it and used the real one instead.

Ok I hope that makes sense.

- Textures should live in the same directory as the .ssc file in which their referenced (they're not required to be in the same directory, but this will be the common case for add-ons.)

I don’t necessarily agree with this, but can understand where you are coming from. To keep the structure as clean as possible I would make a distinction between the different file types and assign default directories for those file.

With this in mind I imagined that textures have there own directory and models get there own etc... What you can do to make Celestia a little more intelligent is to come up with a definable hierarchy of sorts that can be set above the defaults.

Example, in one of the .scc files or in a new one, state.

“hierarchystructure”
}
Directory_one
Directory_two
Directory_three
Directory_four
Etc...
}

This simple trick will let the user to define directories that they want to go first. If not Celestia will default to, in this order, textures, models, then alphabetically all the remaining directories. If a directory has subdirectories, then those would be loaded alphabetically also, and so on. (I hope that makes sense)

- The texture resolution situation needs to be sorted out. It should be possible to install a high-resolution texture pack without touching any .ssc files. And the separate lores/medres/hires directories are no longer practical if textures are in the same place as .ssc files.

Ok this is a tough one, and I can understand this also, but lets see if I can come up with a solution. First let me state that I am in favor of getting rid of the lores/medres/hires directories because of what you say later on “most add-on creators only provide a single resolution.”

I'm reluctant to encode the resolution in the texture name, but other approaches have drawbacks as well. Entering the names of all three textures resolutions in the .ssc file like this:

Texture [ "mars-low.jpg" "mars.jpg" "mars-high.dds" ]

Here is what I would do. It will take a little encoding upfront on your part, but I think it’s the best middle ground, unfortunately I don’t think there is a one solution fits all answer. I would just state in the code for the resolution planets like this;

Texture-Low “mercury.jpg” (this is for Low quality textures)
Texture “mercury.png” (this is for medium quality textures, and is also the default)
Texture-High “mercury.dds” (this is for the highest quality textures)

Yes unfortunately the person will have to edit the files, but I don’t know any other way of doing it simply. If you do keep the lores/medres/hires directories, then this could also become confusing because of dup file name.

. . . seems like a pain. But then again, most add-on creators only provide a single resolution.

Again this was the reason for just getting rid of the lores/medres/hires

This scheme also fails to accommodate different file extensions. An alternative is to omit the extensions altogether and let Celestia decide the priority for .jpg, .png, .dds, etc.

As for priority go from best to worse formats, so dds, png, gif, jpg. This would really only apply if you wanted to allow the developer to not have to enter the file extension for the images for the give resolution.

Update 5min later:

Come to think of it there might be a solution that will work that is even better. It would require creating a new program, a system editor, and that editor would allow you to import the updated textures.. This is a lot of upfront work, but it kills two birds with one stone. There is now a system editor, and now people don’t have to manually edit the files. Solution solved! Maybe i can get the mostlyharmless crew to work on it. It dosent have to be anything elaborate.

Im drawing on prevoius comments here, and what i think would be nice.

This is not a soloution for people updating their sol textures but may help with the adding and removing addons.

The main changes would have to do with how celestia reads the extras directory, and deciding which files in the extras directory to load,.

have a directroy structure like this.



When celestia starts up it looks in the extras directory for any extra directories (addons).

A window similar to this will then show in the options menu. (also it could be set up to pop up at the very start of celestia if any of the directories under extras have changed from the last run.)



Then the user can select which addons to implement in celestia, with the changes coming into effect the next time that celestia is run.

Thus with a well desigend addon the average user only has to run the addons' selfextracting install, start celestia, check a box, then restart celestia.
to disable the addon just uncheck the box.


Sorry Matt, but im not so keen on the system editor. Another program wont make it any simpler to the average user.

I like the idea of keeping the low, med and high resoloution directories (for the default stuff not the addons) and doing away with specifying the extension for the textures in the ssc files.
Then someone downloads the highres dds texture package. puts them in the highres directory (should be empty) then changes one of the options in celestia from med res to high res. No ssc file modification needed.

chris wrote:A better system for organizing directories is certainly needed . . .

First of all, I'd like to make the celestia.cfg file more flexible. It will have the usual lines saying which star, star name, and constellation databases to load. The SolarSystemCatalogs and ExtrasDirectory lines will be replaced with a DataDirectories directive that specifies a list of directories to search for .ssc files. The order is important for overrides. To mimic the current behavior of Celestia, you'd use:

DataDirectories [ "data" "extras" ]

All subdirectories will also be searched. Also, Celestia will accept the name of a config file on the command line . . . You'll be able to make shortcuts (or shell scripts) that call Celestia with different command lines for different 'universes.' The is almost exactly what Selden described.

Overrides will only work on a per-object granularity . . . Allowing per-field overrides would be convenient in some cases, but I think it would generate complex dependencies that could make managing Celestia add-ons a huge headache. I feel that the usefulness of per-field overrides is overestimated . . . In practice, it's unlikely that a planet modified by a bunch of different .ssc files will look that good.

The textures directory is completely out of control . . . There are two things that need to be fixed:

- Textures should live in the same directory as the .ssc file in which their referenced (they're not required to be in the same directory, but this will be the common case for add-ons.)
- The texture resolution situation needs to be sorted out. It should be possible to install a high-resolution texture pack without touching any .ssc files. And the separate lores/medres/hires directories are no longer practical if textures are in the same place as .ssc files.

I'm reluctant to encode the resolution in the texture name, but other approaches have drawbacks as well. Entering the names of all three textures resolutions in the .ssc file like this:

Texture [ "mars-low.jpg" "mars.jpg" "mars-high.dds" ]

. . . seems like a pain. But then again, most add-on creators only provide a single resolution. This scheme also fails to accomodate different file extensions. An alternative is to omit the extensions altogether and let Celestia decide the priority for .jpg, .png, .dds, etc.

Zipped packages would be great to have, but I want to focus on getting the directories sorted out first.

--Chris

the only problem i see with that is a lot of addons would need to be modifyed as far as models and texture directories

Marc, this is what I had in mind for the universe selector system.


I was also hoping to get rid of the "extras" directory as well. With what is in it now, it’s pointless because in it is valid, known data in the solar system. I consider all the new star catalog extras, but it goes under the data directory. So where do you draw the line? On one else is using it. So I say consolidate it.

But to be fare, I did make a mistake on my original structure.



Generic gets its own directoy because it oly contains Generic stuff that can be used by all universes.

Multiple directory for images (reasons against)
Here is something I found out about the use of textures in general. Celestia as I understand it, uses simple logic to find out what textures to use, it checks the high resolution directory to see if the file is there, if its not it goes to the medium resolution directory and if doesn’t find it, it then checks the low resolution directory. Celestia will also check to see if that image can fit into the video cards memory. (This is the only valid reason for keeping the multiple directories, but the largest image is see in the lores directory is only 5.3KB uncompressed so unless some one is trying to used Celestia on a video card that has 512KB video ram or less. It’s pointless to have them.) The largest image uncompressed in the medres directory is still only 4.2MB uncompressed. . Do you ever go and ask your self. “gee I wish I could see that low res image of the planet again” didn’t think so. So why keep two versions?

I agree with removing the extension as a pre requirement. And the solution is still simple if there are two images in the directory named the same (not including the extention), and don’t know with one to use. Simply check the size (the pixel height and width) of the image and come up with your (high, medium, and low) that way

Finally with the current version and possibly the 1.2.5 version as well. Unless you specify the higher resolution images by editing the .scc file anyway, it is never used. So it’s still pointless to have separate directories. The best solution I think would be to just allow users to install the highest resolution images they can into a common directory, and when Celestia comes to an images that the video card cannot use, it would scale it down automatically, to maximize the image for that card. This solves my problem or having multiple directories and multiple images of the same planet, and everyone else’s problems of not being able to get the highest quality textures they can use for their card. I would make the high, medium, and low valid options, but no way force people to use those directory.

As for the system editor. i was just hopeing for a tool that i could build systems with. The replace the texture option was optional.

MKruer,

The way you've been writing about your proposed changes, it sounds as if you are planning to write the code yourself. Is this true? That would be the best way to be sure it's implemented the way you've designed it. Of course, you also need to persuade Chris that it'll fit into his plans for Celestia and will scale well.

If that's what you're planning, then I think you really need to be running the latest 1.2.5 prerelease (it's the code that's currently on Sourceforge) and be familiar with how textures, for example, actually are handled -- and understand the code that does it. Misunderstanding the current features, and why they are that way, can result in substantial confusion.

For example, currently if there are textures present in all three texture directories, Celestia 1.2.5pre4 uses the higher of medres and lores. It only uses hires when it's explicitly selected with the "<shift>R" command. I know this because of my experiences while testing the textures I/ve been creating from the published maps from the various Lunar and Martian surveyor missions. I copy them to the hires directory and start Celestia. It comes up displaying the medres texture. To see the ones I put into the hires directory, I have to type <shift>R.

Also, there is a very good reason for using the smallest acceptable textures: Celestia starts significantly faster.

The way I have been trying to talk, is I would like to see these changes implemented before everything gets so complex, so out or order that Celestia MOD development grinds to a halt because everyone is using the same file names, and no one know where to place anything. Yes I do program, but I’m not that good. (That is not to say I am not structured) Why would I want to pollute the code that is Celestia

Misunderstanding the current features, and why they are that way, can result in substantial confusion.

That’s what you are here for, to explain that which I don’t understand.

At this point, I am will to coincide the (hi, med, low) just to push though the larger directory change (that was the major point to start)

Anyway I guess people are still having trouble understanding how and why I would structure the “new” directory’s the way I did. My Hierarchal schema is real simple Universe first, followed by system. If it ever got to the point where we have information on the whole universe, it would go Universes, megaclusters, galaxies, clusters, groups, stars. The directory structure simply goes from the largest item to the smallest item. (I don’t know how to make it any simpler)
I think that if you look at it. The new structure keeps the existing one in check, and refines it more. I does however call for the creation of the so called “universe.ssc” files to be placed in the universe directory.( I will have to post a revised version of that file later tonight)

*Extending the future! (Planning Ahead)

I would even consider implementing a “system” directory under the star directory to place the planets into. Here is the reason. When scripting becomes available for objects, most likely to keep the individual .ssc file as simple as possible, you would segment the information on then into smaller .ssc files. (The inside of the .ssc for the planets you would have the scripting information.) By placing all the new files into a system directory you are keeping the structure as clean as possible.

As for running the latest release, yes I am running the latest prerelease 1.2.5 pre 5! (There was the patch for the Radeon Cards and the ring shadows)

** Zip files maintain the same structure internally as the directory structure outside

Comments? Question? Flames?

MKruer wrote:Marc, this is what I had in mind for the universe selector system.

Although its a nice idea its pretty complex. Keep it simple i reckon.

Also whats the point of being able to load compressed addons? Most of the image files are compressed already, and you dont want to have to wait around for a dds file to uncompress.

marc wrote:

MKruer wrote:Marc, this is what I had in mind for the universe selector system.

Although its a nice idea its pretty complex. Keep it simple i reckon.

I don’t know what to say. The concept is very simply. It is like a directory structure, but you get to choose to make certain items available. Items that you don’t want just uncheck them. And because its Hierarchal, if you decides not to have the universe shown, then it intelligently removes the child item. This is about as simple as it gets while maintaining the ability to be granular, down to the systems level.

marc wrote:Also whats the point of being able to load compressed addons? Most of the image files are compressed already, and you dont want to have to wait around for a dds file to uncompress.

Compression become quite important if you have many ssc files. A great example is the System Generator http://www.shatters.net/forum/viewtopic.php?t=678&postdays=0&postorder=&start=15
10gigs down to 2gigs. I would say Compression is well worth it. Compression was not designed to compress textures with higher compression.

Good discussion, very usefull, bt it makes me wonder weather or not this should be part of a release like 1.2.5.

reorganising things is good, but I would think that this would either be a change implemented in 1.3 (if that's being planned at all), or 2.0 (same question).

I'm not saying it should not be in 1.2.5 just saying that one may wonder...

MKruer wrote:Marc, this is what I had in mind for the universe selector system.

If you create a title structure for ssc files that wouldnt be too difficult to code...

Name could be picked with
"Sol" "Known Universe"
{ [maybe some variables]
}
on top