Feature Request: User data files (solarsysu.ssc, etc.)

[don]

For many users, updating Celestia can be a nightmare, because they have modified the base Celestia data files, such as solarsys.ssc, to meet the requirements of an add-on, or to change a texture, or any number of other reasons.

When a new version of Celestia becomes available, they need to first save their modified data files, install the new version, and then manually go through their old files, one line at a time, to add their previous changes to the new data files. For a user who has a LOT of add-ons, this process can be painful and time-consuming.

Suggestion / Request for Enhancement ...

Define a second set of data file names for each of the user-modifiable data files. This would allow the user to enter their changes ONCE, into these files. The Celestia distribution would include these text files, and they would contain only the necessary identifier information, but no data. The user would add data as required / necessary instead of changing the base data files.

When installing an update, the install process would detect if any of these files already exist, and if so, NOT copy the skeleton secondary files from the distribution. This provides for an instant update, even when multiple add-ons are installed!

When run, Celestia would first read the base data files, then read these secondary *user* data files, and REPLACE or APPEND the information in them, to the information it already read from the base data files.

When a line in a base data file is to be commented out, the user would copy the existing line to the secondary data file and place the comment marker in front of that line. When Celestia sees a comment marker in a secondary data file, it checks the data it already got from the base data file and if it exists, removes / clears it.

Or, Celestia would read both, the base and secondary files, merging them before it actually sets data values.

The secondary data file names could simply have a "u" appended to them or have a new file extension, such as solarsysu.ssc or solarsys.usr.

Thoughts? Comments?

-Don G.

[Bob Hegwood]

Mr. G...

For what it's worth, I *like* the request. As you've probably guessed by now, I just can't keep my hands off of the solarsys.ssc file.

Thanks, Bob

[don]

Mr. H.,

You're not the only one Bob.

Any developers hanging out here?

Good idea? Bad idea?

Easy/hard to implement?

[Harry]

I think it would be easier to enhance the curent process of reading configuration files. You can already have your changed definitions in the extras-directory, but they don't replace the original defintion, but add a new body (for solarsystem-objects). AFAIK it should be easy to either replace any already exisiting bodies when a body with the same name is found in a non-standard config-file, or add a new config-option "Replace false/true" which determines whether already existing bodies are replaced or a new one with the same name is added (I guess this is needed for some object with different settings for different times, i.e. to use Beginning and Ending).

Then the order in which config-files are read must be defined and documented, and it would be nice to have a extras-directory in the personal-directory of the user (i.e. /home for Linux, c:\documents ...\... for Windows).

If I understand the current code correctly, this should be much easier to implement, without losing much of your proposed functionality.

Any thoughts?

Harald

[Bob Hegwood]

AFAIK it should be easy to either replace any already exisiting bodies when a body with the same name is found in a non-standard config-file, or add a new config-option "Replace false/true" which determines whether already existing bodies are replaced or a new one with the same name is added (I guess this is needed for some object with different settings for different times, i.e. to use Beginning and Ending).

Any thoughts?

Just one Harry...

What the hell does AFAIK mean?

Sorry... Dense Bob

You also lost me when you started talking about "Reading configuration files." Celestia reads configuration files when you install it? News to me. I thought it just wrote whatever it wanted to on your hard drive. If you haven't saved your current configuration in another directory, your screwed.

I thought Don was suggesting that Celestia could be released with an UPDATE option, rather than a complete install?

Thanks - I *know* I'm slow - Bob

[selden]

Bob,

I'm not Harry, but what the heck...

AFAIK = As Far As I Know

He's using "configuration file" in a very generic way and is mostly referring to Celestia's Catalog files (SSC, DSC, STC) which are used at runtime, not at install time. When it starts up, Celestia currently adds all of the objects that it encounters in catalogs to its internal database. It doesn't care if they have the same name, they're still separate objects.

I agree that a "Replace true" (that defaults to false) would be a very useful addition to the logic of all of the catalog files. Of course, a corresponding message has to be written to the "console log" whenever it's actually used, saying exactly what object from what Catalog was replaced, along with what object from what Catalog is replacing it. This preservation of the Catalog's file specification may require some additional record keeping that is not currently being done.

[Harry]

Just one Harry...

What the hell does AFAIK mean?

As Far As I Know.

You also lost me when you started talking about "Reading configuration files." Celestia reads configuration files when you install it? News to me. I thought it just wrote whatever it wanted to on your hard drive. If you haven't saved your current configuration in another directory, your screwed.

I thought Don was suggesting that Celestia could be released with an UPDATE option, rather than a complete install?

Right, I wasn't thinking at all about the installation, but about the normal program-start - my idea is somewhat different than Don's.

What I would like to see is a seperation between the standard data-files like solarsys.ssc (which would get reinstalled with every new version of celestia), and any changes a user or a addon would need (which would be in files in the extras-directory, which should survive a new install).

Assume you wanted to have a different definition of earth, instead of the standard one. IMHO (In My Humble Opinion ) you should be able to add a file my_earth.ssc in the extras-directory, which would contain your preferred earth-defintion, like this:

Code: Select all

"Earth" "Sol"
{
        Replace "true"
        Radius 7000.0  # example: increase radius
        Texture "earth.*"
        NightTexture "earthnight.*"
        [...]
}


Now when celestia is starting, it would read the normal solarsys.ssc and create an object for earth, with the found parameters. Then it would check your extras-directory, and find your new definition.
Currently, celestia adds another earth with your parameters. I propose it should (whenever the parameter Replace is "true") replace the old definition with the new one.

I hope I expressed my thoughts better this time

Harald

[Bob Hegwood]

Thanks Selden and thank *you* Harry...

I've learned a brand new acronym.

Sorry, I *know* I'm slow, but your explanation this time made sense. Sounds like a good idea.

JWICAUWYTA - "Just Wish I Could Always Understand What You're Talking About."

Take care, Bob

[don]

Howdy Gents,

Thank you for your thoughts. This is a tough one because, as Harald wrote, an existing object can be created many times, some data can be added/changed via .ssc files in the extras directory, etc.

My intentions for this idea are:

1. Allow add-on creators an EASY way to add / change data file (.ssc, .xyz, .etc.) contents.

2. Allow the user to make personal modifications to data files that will remain intact when Celestia is upgraded -- without having to do any manual editing.

3. Hopefully come up with an idea that is easy to implement.

Harald, I had considered your idea first, which would be the easiest for existing add-ons. However, it creates the problem of knowing which files get read in what order (as you mentioned), and makes it difficult to know what information from what files will overwrite previous information. This could make it difficult to debug an add-on installation gone bad.

Your suggestion for a "replace" flag is a good one, and will definately be needed.

My second idea was to simply add sequential numbers to all added data files, regardless of where they were located, such as solarsys1.ssc, solarsys2.ssc, etc. This would provide a pre-defined "load order" to Celestia. However, if you have 20 or 30 add-on sub-directories in your extras directory, these could be difficult to number and maintain.

Thus my third idea was born ... to define one "user's" data file for each of the modifiable data files.

If this is too difficult to implement, here's a fourth idea:

Define a directory for each of the user-modifiable data files AND add sequential numbering to the file names as they are added to this directory. For example, when an add-on requires a change to the solarsys.ssc file, its included solarsys.ssc file is copied to the solarsysdata directory, then the user adds a sequential number to it so it gets loaded in the proper order.

However, this idea creates a documentation problem also. It would be difficult to know which data file belongs to which add-on, without actually displaying the file to read its internal comments.

So, I think I'm still leaning towards having one Celestia data file and one user data file, for each of the user-modifiable data files. Yes, this will require the user to edit the required file(s) when adding an add-on, but IMHO, would be the easiest to implement and for the user to maintain.

There has got to be a good idea for this, somewhere out there, just waiting to be born.

... and it would be nice to have a extras-directory in the personal-directory of the user (i.e. /home for Linux, c:\documents ...\... for Windows).

Would this really be of any help? I ask because I have multiple Celestia directories (like many other users), for different purposes, and I would personally *want* to keep extras associated with each directory right there.

Cheers,

-Don G.

[Christophe]

Instead of allowing catalog files in the 'extras' directory to redefine objects, I'd rather allow existing objects to be modified. For example in solarsys.ssc we currently have:

Code: Select all

"Jupiter" "Sol"
{
        Texture "Jupiter.*"
        HazeColor [ 0.4 0.45 0.5 ]
        HazeDensity 0.3

        Radius 71492 # equatorial
        Oblateness 0.0649

        CustomOrbit "vsop87-jupiter"
        EllipticalOrbit
        {
        Period           11.8622
        SemiMajorAxis     5.2034
        Eccentricity      0.0484
        Inclination       1.3053
        AscendingNode   100.556
        LongOfPericenter 14.7539
        MeanLongitude    34.404
        }

        Atmosphere {
                Height 300
                Lower [ 0.7 0.6 0.5 ]
                Upper [ 0.5 0.45 0.4 ]
                Sky [ 0.8 0.8 0.5 ]
                Sunset [ 1.0 0.6 0.2 ]
        }

        RotationPeriod         9.927953 # System II (for GRS)
        # RotationPeriod       9.92425  # System III (radio emissions)
        Obliquity              2.222461 # 1.82246 # 2.22246
        EquatorAscendingNode -22.203    # -1.87785 # -22.203
        RotationOffset        16        # for correct location of GRS
        # RotationOffset     305.40     # correct System III prime meridian

        Albedo 0.51
}

An add-on should be able to add a ring system by providing two files:

Code: Select all

extras/jupiter-rings.ssc
extras/textures/medres/JupiterRings.png


With jupiter-rings.ssc:

Code: Select all

"Jupiter" "Sol"
{
   Rings {
            Inner 92000
            Outer 221000
            Texture "JupiterRings.png"
        }
}

That way solarsys.ssc updates work without having to mess with your installed add-ons and they don't affect them either.

As for the loading order, files should be loaded according to their last modification date. The last installed/modified add-on gets loaded last.

Regarding the location of the extras directory, extra 'extras' directories can be specified in the celestia.cfg file (ExtraDirectories). It'd be nice to have it default to [ "extras" "~/celestia/extras" ] instead of the current [ "extras" ]. Currently absolute paths and ~ extension do not work, so it's not just a matter of changing celestia.cfg.

[Harry]

Harald, I had considered your idea first, which would be the easiest for existing add-ons. However, it creates the problem of knowing which files get read in what order (as you mentioned), and makes it difficult to know what information from what files will overwrite previous information. This could make it difficult to debug an add-on installation gone bad.

I admit this is a problem. But I think this is generally a problem with addons: they can have conflicting settings and may not work together well. If such a conflict exists, the user has to resolve it, which should not be more difficult than creating a "solarsysu.ssc" in the first place (did I overlook something here?).

My second idea was to simply add sequential numbers to all added data files, regardless of where they were located, such as solarsys1.ssc, solarsys2.ssc, etc. This would provide a pre-defined "load order" to Celestia. However, if you have 20 or 30 add-on sub-directories in your extras directory, these could be difficult to number and maintain.

I guess the easiest solution would be to introduce a purely alphabetical sort-order, which would result in mysolarsys.ssc to be read after Addon1\foo.ssc, but before ZZZ\addon.ssc. If you wanted to enforce a particular sort-order, you would have to rename the files or directories, like 01_ZZZ\, 02_Addon1\, 03_mysolarsys.ssc.

But as long as addons dont conflict (i.e. add new bodies, or change different ones), this should not be needed.

Thus my third idea was born ... to define one "user's" data file for each of the modifiable data files.

I still would like to see a solution where each addon would be contained in its own directory (within extras), without the need to modify or add any other files (at least as long as there are no conflicts).

Define a directory for each of the user-modifiable data files AND add sequential numbering to the file names as they are added to this directory.[...]

Same problem as above (i.e. needs user-interaction).

So, I think I'm still leaning towards having one Celestia data file and one user data file, for each of the user-modifiable data files. Yes, this will require the user to edit the required file(s) when adding an add-on, but IMHO, would be the easiest to implement and for the user to maintain.

Note that you can have something like that as a special case of the method I tried to describe You could easily combine all changes in one file "extras\solarsys.ssc" (I am currently only thinking of ssc-files).

... and it would be nice to have a extras-directory in the personal-directory of the user (i.e. /home for Linux, c:\documents ...\... for Windows).

Would this really be of any help? I ask because I have multiple Celestia directories (like many other users), for different purposes, and I would personally *want* to keep extras associated with each directory right there.

Of course you shouldn't have to use it, but in many cases this would be nice. This directory would be common to all installations and safe from overwriting. And on some systems a normal user doesn't have the right to change files in "c:\program files" (he actually shouldn't have the rights - this is enforced more strictly on Linux-systems). Right now I am having problems specifying non-relative extras-directories, but I am not sure why.

Harald

[selden]

Christophe,

It seems to me that your proposal (as stated) would require the elimination of the current feature which permits multiple objects to be loaded which have the same name. I suspect that will break some Addons. Having the keyword "Replace true" in your proposed SSC files would eliminate my objection.

I think the ability to replace (parts of) objects should be available in all types of Catalog files, though, not just SSC files. Avoid the special cases if you can. They'll only add more confusion and complaints.

My understanding is that catalogs currently are loaded in alphabetical order, at least when they're in the same directory. I suppose this might change if multi-threading of file loading ever is implemented.

[Harry]

Instead of allowing catalog files in the 'extras' directory to redefine objects, I'd rather allow existing objects to be modified.

I agree. Maybe there should be three possibilities:
1. adding a new object, keeping the old one
2. replacing an object (if it already existed)
3. overriding settings of an already existing object

Of course mixing 1 with 2/3 could quickly lead to problems Which object is replaced/altered when multiple objects with the same name (but possibly different Beginning/Ending values) already exist?

As for the loading order, files should be loaded according to their last modification date. The last installed/modified add-on gets loaded last.

I disagree, because this can be confusing to the user - making a simple change would change the load-order. If you copy your extras-directory, you could end with two files/directories having equal modification date, or if you extract a freshly downloaded addon, the ZIP-program may restore the original modification-date and your new new addon gets loaded first.

Harald

[don]

After giving this some more thought, I think I'm leaning towards Harald's / my first idea, that would keep every add-on completely self-contained in it's own extras sub-directory, for easy addition and removal. The user can still have their own personal data files (ie. solarsysuser.ssc) stored in the extras directory which would not get overwritten.

In order to allow an add-on to completely replace the base object specification, I also think Harald's idea of adding a "Replace" (true / false) flag is a good one.

In my original message, I also provided a means to comment out specific lines in a data file. For example, the BumpMap entries for Mercury because bumpmaps don't work right on my ATI card. Is there an easier way to implement this?

-Don G.

[Harry]

In my original message, I also provided a means to comment out specific lines in a data file. For example, the BumpMap entries for Mercury because bumpmaps don't work right on my ATI card. Is there an easier way to implement this?

It should suffice to override this with an invalid texture (or the empty string) as bumbmap, shouldn't it? But this has to be checked for all possible settings, like Rings, Atmospheres etc.

Harald

[Christophe]

Instead of allowing catalog files in the 'extras' directory to redefine objects, I'd rather allow existing objects to be modified.

I agree. Maybe there should be three possibilities:
1. adding a new object, keeping the old one
2. replacing an object (if it already existed)
3. overriding settings of an already existing object

Of course mixing 1 with 2/3 could quickly lead to problems :( Which object is replaced/altered when multiple objects with the same name (but possibly different Beginning/Ending values) already exist?

That could be done on a per-object basis:

Code: Select all

"Jupiter" "Sol"
{
   Mode "override"
   Rings {
            Inner 92000
            Outer 221000
            Texture "JupiterRings.png"
        }
}

Mode having one of the following values: add, replace, override. add being the default.

As for the loading order, files should be loaded according to their last modification date. The last installed/modified add-on gets loaded last.

I disagree, because this can be confusing to the user - making a simple change would change the load-order. If you copy your extras-directory, you could end with two files/directories having equal modification date, or if you extract a freshly downloaded addon, the ZIP-program may restore the original modification-date and your new new addon gets loaded first.

That's a good point, but I don't like the explicit name ordering convention either as it requires an extra installation step. I don't think that's workable without an add-on management interface.

Most users are already confused by the simple current behaviour, we can only add more complexity.

Maybe we could have Celestia keep a list of installed add-ons (extras subdirectories only), and provide a simple interface to change the loading order. Newly installed add-ons are always loaded last, but that order can be changed by the user.

[don]

It should suffice to override this with an invalid texture (or the empty string) as bumbmap, shouldn't it? But this has to be checked for all possible settings, like Rings, Atmospheres etc.

In the case of a filename, yes, maybe a non-existent filename would work. But what about the other related bumpmap settings? There are many settings in an .ssc file that are not filenames, and no filenames at all in .xyz files. So I think there still needs to be some way to comment out a line in any existing data file --AND-- be able to change it's content as well.

As another example, you want to make some corrections in an .xyz file. How is that done now? Editing the base file, right? Celestia should provide some way to allow corrections and additions to be made to *any* of the base data files, without editing the base file itself.

-Don G.

[don]

Mode having one of the following values: add, replace, override. add being the default.

Going back to Harald's initial Replace=true/false ... This is on a complete-object basis, not an individual line basis, which is what the default function would be (override this setting). IMHO, having *three* different values (add, replace and override) would make it more confusing.

... but I don't like the explicit name ordering convention either as it requires an extra installation step. I don't think that's workable without an add-on management interface.

The whole intent of my idea was to provide something EASY between now and the time the add-on manager stuff is actually ready.

Most users are already confused by the simple current behaviour, we can only add more complexity.

Remembering back to the first few months of my running Celestia and trying to figure out what is supposed to go where (relating to add-ons), I can say that the fault lies with the add-on developers, and not the user. Some add-ons have very good installation instructions, while others simply provide you with a zip of raw files, that you need to figure out what goes where. We can't expect a new user to understand everything about Celestia's files or how they are organized. Which is the whole reason for having an add-on manager which is a bit in the future.

Maybe we could have Celestia keep a list of installed add-ons (extras subdirectories only), and provide a simple interface to change the loading order. Newly installed add-ons are always loaded last, but that order can be changed by the user.

Is there such a thing as a "simple" interface in any windowed GUI? You mean something like a quick-and-dirty Java applet, since it needs to be cross-platform?

-Don G.

[Christophe]

It should suffice to override this with an invalid texture (or the empty string) as bumbmap, shouldn't it? But this has to be checked for all possible settings, like Rings, Atmospheres etc.

In the case of a filename, yes, maybe a non-existent filename would work. But what about the other related bumpmap settings? There are many settings in an .ssc file that are not filenames, and no filenames at all in .xyz files. So I think there still needs to be some way to comment out a line in any existing data file --AND-- be able to change it's content as well.

As another example, you want to make some corrections in an .xyz file. How is that done now? Editing the base file, right? Celestia should provide some way to allow corrections and additions to be made to *any* of the base data files, without editing the base file itself.

I really don't think allowing modification of base files is a good solution. They shouldn't be modified unless by knowledgeable users, automating patching of those files is really asking for trouble. You will have no easy way of upgrading unless working out the patch from the original file and re-applying to the new version: very messy and many chances to get conflicts when applying the patch anyway, so in practical terms user intervention will be required.

Supporting an 'override' mode for ssc files is just a way to provide an automatic patch system, so it won't be any different from modifying the original file except it's done in a robust way which makes upgrading possible.

To solve your 'commenting out' problem it may be necessary to extend the ssc syntax a bit (add a default or undef value), but it's still better than direct edition of solarsys.ssc.

[Christophe]

Mode having one of the following values: add, replace, override. add being the default.

Going back to Harald's initial Replace=true/false ... This is on a complete-object basis, not an individual line basis, which is what the default function would be (override this setting). IMHO, having *three* different values (add, replace and override) would make it more confusing.

Since it's directly in the add-on ssc file and not 'burried' in celestia.cfg, I think it's in fact less confusing (but maybe that's just me).

... but I don't like the explicit name ordering convention either as it requires an extra installation step. I don't think that's workable without an add-on management interface.

The whole intent of my idea was to provide something EASY between now and the time the add-on manager stuff is actually ready.

I'd rather go directly to something extendable to a full blown add-on manager than having to throw this away and restart from scratch in six months.

Most users are already confused by the simple current behaviour, we can only add more complexity.

Remembering back to the first few months of my running Celestia and trying to figure out what is supposed to go where (relating to add-ons), I can say that the fault lies with the add-on developers, and not the user. Some add-ons have very good installation instructions, while others simply provide you with a zip of raw files, that you need to figure out what goes where. We can't expect a new user to understand everything about Celestia's files or how they are organized. Which is the whole reason for having an add-on manager :) which is a bit in the future.

Yes, but I think the current mixture of supported add-on formats is the main culprit, we should only support extras sub-directories and drop the older standard.

Maybe we could have Celestia keep a list of installed add-ons (extras subdirectories only), and provide a simple interface to change the loading order. Newly installed add-ons are always loaded last, but that order can be changed by the user.

Is there such a thing as a "simple" interface in any windowed GUI? You mean something like a quick-and-dirty Java applet, since it needs to be cross-platform?

It wouldn't have to be a full GUI at first. A simple text file listing the add-on directories in their loading order would be a first step. We could then provide an interface directly in Celestia to edit it (a simple list view with up/down arrows). Later on that could be extended to allow enable/disable addition/removal of add-ons, display of manifest information, etc...