Celestia Forums

. . . This message is a combination of messages from the Users forum . . .

(Note: Times are GMT/UTC - 7 hours)

---< Begin re-located, previous messages >---

FROM: Christophe / Tue Oct 28, 2003 3:25 pm



I think that has already been discussed in the forums too.

The basic plan is to define a package format for add-ons, very simply that would be a zip file including a defined folder hierarchy (depending on the add-on type) and a manifest file.

The manifest file would hold information like the add-on type, author, release date, changelog, objects provided, dependencies on the Celestia version or on other add-ons or on objects, screenshot, and so on.

We'd then have a central web repository for those add-ons, very similar (in my mind) to KDE-Look. Add-on creators would simply upload their zip files, and the database would be directly updated from the manifest file (no need to fill in a form).

The next step would be to make celestia able to connect to this site to download new add-ons or updated add-ons, eventually even dealing with dependencies by itself.


--------------------------------------------------------------------------------
FROM: don / Tue Oct 28, 2003 4:34 pm

Howdy Christophe,

Your description is very good, but it only describes a central repository of add-ons and that their format should be pre-defined. It does not describe anything that I believe would be included in an "Add-On Manager", such as many programs use now.

In my mind, an Add-On Manager for Celestia would be no more than a list window that should be able to do the following things:

* List all existing and locally available (on disk) add-ons, with whatever other information (button) on them is required.

* The list display should include a button to connect the users machine to the Add-Ons repository, obtain a list of currently available add-ons and their descriptions, then list them with the users currently installed or on-disk add-ons (showing location of each -- installed, local disk, repository, etc.).

* The list display should provide a button for repository entries, that would open a web page describing the specific Add-On, along with pictures, from the repository site.

* The list display should provide a button that allows the user to install or remove any add-on, whether it is installed, stored on local disk or stored on the repository. This function will also be required to modify any existing files, such as solarsys.ssc, as necessary (described in the required Add-On manifest file), for both, the add and remove functions.

To me, this is a manager of Add-Ons <smile>. The repository and connect-to-repository features can be added later. A pre-defined file format and required content list (such as the manifest file) for add-ons can certainly be created now. When existing add-ons are modified to meet this requirement, they would automatically be eligible for automated Celestia Add-On Management (add/remove).

The manager part should be easy enough to code, but obviously will require a programmer to code it <smile>. It might be much more difficult to get everyone to agree on the file contents and format of the manifest file, ZIP or TAR file, etc., and then to find somewhere to host an all-encompassing Add-On repository.


--------------------------------------------------------------------------------
FROM: Christophe / Tue Oct 28, 2003 5:30 pm

Of course that's what is intended.


--------------------------------------------------------------------------------
FROM: chris / Wed Oct 29, 2003 3:31 am

Regarding the management of add-ons . . .

I think that creating a central add-on repository with generous amounts of disk space and bandwidth would be the single most useful thing for Celestia users right now.

For a package format, I favor ZIP because of its ubiquity. It would be helpful to define custom file extension like .celz, because you don't want to associate Celestia with all .zip files. The manifest included in the package would have the name of the add-on, the author, a description, and some other yet-to-be-determined data. I think everyone agrees on these basics.

Celestia should include code to manage local add-ons and to get additions and updates from the central repository.

And it should all be as slick as Apple's iTunes


--------------------------------------------------------------------------------
FROM: Christophe / Wed Oct 29, 2003 4:04 am



The road is straight but the slope is strong.


--------------------------------------------------------------------------------
FROM: don / Wed Oct 29, 2003 11:19 am

Howdy Christophe,



CAUTION! 10% DOWN-SLOPE beyond next crest!

Are you ready? Weeeeeeeeee . . .

There MUST be some GUI (list window) programmers out there, yes?

And someone who knows how to work with ZIP files, from a coding standpoint?

And someone who knows SQL and XML, for the repository.

So, it's just a matter of getting these folks what they need (tools, encouragement, a means to work together, etc.), so they can work on Celestia and its sub-projects.


--------------------------------------------------------------------------------
FROM: don / Wed Oct 29, 2003 11:34 am

Howdy Chris,



Does ZIP have an API so files can be extracted from within program code?




Why would there need to be an association? The "manager" would look for ZIP files in a sub-directory within Celestia.



Yep, plus...

* Version Number
* SSC mods (for default/existing objects)



Haven't seen that one yet. Have my own radio station on my disk (21,000+ MIDI files, 7,600+ MP3s), mostly 50's thru 70's -- not to mention a 200 CD carousel.

Have a great day!


--------------------------------------------------------------------------------
FROM: chris / Wed Oct 29, 2003 12:48 pm



It would be nice to be able to click on a package file--or a link to one in a web page--and have Celestia automatically pop up a package installation dialog. This is not an essential feature, but I do think that it'd be convenient.



Yes . . .



I'm not sure how to handle add-ons that modify an existing .ssc file. My temptation is to disallow them, as they make keeping track of versions a complete mess. If an add-on has to modify solarsys.ssc or extrasolar.ssc, you'll have to install it manually.


--------------------------------------------------------------------------------
FROM: Christophe / Wed Oct 29, 2003 1:11 pm



zlib allows to work with files inside zipfiles as if they where directly in the filesystem. No problem here.



I do. I even do that for a living occasionaly but it is a lot of work. Nothing difficult, but that's probably something like one man month. Fitting that into one's spare time and not losing interest along the way is not an easy task.

That's why I think finding an existing code base is a requirement for this to succeed, development time would probably be brought done to a lot more reasonable one man week.


--------------------------------------------------------------------------------
FROM: Christophe / Wed Oct 29, 2003 1:30 pm



I think we need some way to overwrite the default ssc files from the extra directory. That would allow to replace textures from an add-on without having to manualy chage the 'Alternate texture'.


--------------------------------------------------------------------------------
FROM: don / Thu Oct 30, 2003 2:22 am



Okay, that makes sense.




I would think that the first thing most Celestia users will want to do is install higher resolution textures for our own Solar System objects -- all of which reside in the solarsys.ssc file. Maybe Christophe's idea about a .SSC file within the extras folder/sub-folders overriding the default .SSC file or auto-adding as an Alternate Texture would work better?

To modify the .SSC file shouldn't be too bad. For adding or removing an Add-On, a template type of entry could be added to the .SSC file for each line modified, such as:



This way, a human will know what was done, and when, if the file is looked at. Then, to reverse the modification when an Add-On is removed is just a matter of searching for it's name, removing the added line of code, uncommenting the <commented entry> (original entry), and removing the Add-On section comments. Not sure what kind of start-up hit Celestia would take with all the added comment lines.

Maybe not so obvious, the manifest would include instructions such as:
Old line: Texture "earth.*"
New line: Texture "earth.dds"

Just one idea.


--------------------------------------------------------------------------------
FROM: don / Thu Oct 30, 2003 2:43 am



Good to know Christophe. Thanks!




I am merely assuming that SQL (whatever brand/version) would be best for the database, and that XML would be used for remote control. Maybe there are better things to use?

No, creating the repository code would not be an "easy" spare-time task for one person, and would require lots of continued encouragement from other members.

But, this doesn't have to be a one-person job, and you don't have to have any C++ experience either. Maybe several people could work on different parts of this sub-project, to keep each individuals time to a minimum and "interest level" high? Does anyone else know:

* SQL (for the database part)
* XML (for remote control stuff)
* Web forms (CGI -- perl, php, other - for the UI)
* Other things that would be useful for the repository site

The Celestia Add-On Repository web site is more interactive web site design and coding, than anything else. However, the Add-On Manager side of this sub-project, built into Celestia, will probably be a lot of C++ coding.




Agreed. But finding an open source code base might be more difficult / time consuming than just writing it. I've tried to do some preliminary searches and the only ones I found were retail versions.


--------------------------------------------------------------------------------
FROM: Paolo / Thu Oct 30, 2003 2:49 am

Hey guys this thread is extremely interesting, almost all my suggestions and feature requests as exposed in thread:
http://ennui.shatters.net/forum/viewtopic.php?t=2752
and in my website:
http://members.xoom.virgilio.it/pangeli70/My%20Celestia/Group%204.htm
will be implemented.

but...

please may I say one thing ,

one ,

just one,

only one.

Before starting with add-on management development should be better to release 1.3.1.
Don't you agree?

(Last edited by Paolo on Thu Oct 30, 2003 3:04 am; edited 1 time in total )


--------------------------------------------------------------------------------
FROM: Paolo / Thu Oct 30, 2003 3:04 am

I would like add another idea to this brainstorming about the add-on management.
In order to solve some problems about coherence of modifications of system data and add-on data, probably and add-on editor for ssc, xyz and dat file should be very useful.

Many folks have tried to do this in the past.
An integrated solution or a separate editor should solve definitively the problem.

This should be another main task indeed.


---< End of re-located, previous messages >---

Howdy Paolo,

Thank you for pointing out this previous discussion about a Celestia Add-On Manager:
. . . http://ennui.shatters.net/forum/viewtopic.php?t=2752.

And for the link to your "feature requests related to Celestia's Add-On management." page.

Paolo wrote:Before starting with add-on management development should be better to release 1.3.1.
Don't you agree?

This thread is only a discussion Paolo. I'm sure Chris would like to get a final release of 1.3.1 out as much as all of us <smile>.

Paolo wrote:... an add-on editor for ssc, xyz and dat file should be very useful.

No doubt. But this thread is only discussing the idea of an Add-On Manager, not a custom-built editor for Celestia data files.

-Don G.

don wrote:I am merely assuming that SQL (whatever brand/version) would be best for the database, and that XML would be used for remote control. Maybe there are better things to use?

LAMP (Linux, Apache, MySQL, PHP/Perl) is the way to go.

XML is probably not necessary. A simple text file is enough for the manifest.

don wrote:But, this doesn't have to be a one-person job, and you don't have to have any C++ experience either. Maybe several people could work on different parts of this sub-project, to keep each individuals time to a minimum and "interest level" high? Does anyone else know:

* SQL (for the database part)
* XML (for remote control stuff)
* Web forms (CGI -- perl, php, other - for the UI)
* Other things that would be useful for the repository site

I think it's more difficult to have several people working on a web site than on a C/C++ project, especially when face to face discussion is impossible.

Most of the work is on what you call Web Forms. Setting up a database is a matter of minutes, the conception stage takes a bit more time...

don wrote:The Celestia Add-On Repository web site is more interactive web site design and coding, than anything else. However, the Add-On Manager side of this sub-project, built into Celestia, will probably be a lot of C++ coding.

Yes, and once a package format has been agreed upon, they can be developped independantly. I think setting up the web site is the most urgent thing, addition of package management functionality to Celestia can be done later.

don wrote:Agreed. But finding an open source code base might be more difficult / time consuming than just writing it. I've tried to do some preliminary searches and the only ones I found were retail versions.

I've already contacted the KDE-Look webmaster, he thinks his code is not clean enough to be released to the world.

There are some other OpenSource oriented sites with similar functionality we could contact too (like http://art.gnome.org/).

IMHO we should at first define how addons should work before thinking too much about a graphical manager. Self-contained addons itself would be a great improvement for installation, which then could be a simple copy to the right directory (like extras).

My ideas for an addon-system (using the things said above):

Addons should be able to contain any file that is typically modified/replaced now, which includes textures, star- and galaxy-databases, planets, spaceships, locations, ... They should not (physically) modify each other or the base-installation, i.e. completely independent.

As there probably will be many addons with different version of the same file, a mechanism to resolve conflicts is needed.

Simple solution: sort addons by some key, and use always the file defined in the "highest" addon, or if not in there, use the next lower one - only one addon defines the whole file.

More difficult: first parse file of base-installation, then read first addon, which can override or add single entries of the already read configuration. Then proceed with second addon etc. Thus it would be possible to add for instance a new moon/comet in one addon and replace planet-textures in another, or have two non-conflicting texture-sets installed (how are conflicting definitions solved right now, or can't they happen?).

They should be easy to activate or deactivate (needed for any UI-manager), even while celestia is running. Problem is: what happens, when the currently selected object (star, planet, ...) only exists in an addon that gets deactivated?

Just some ideas...

IMHO, no addon system would be useful without a way to manage the data currently stored in SSC files. My recommendations:

- Store ALL data in the new addon data package format. ( ie zip files ) Remove the textures, extras, and models directories ( shaders too? ) Like files would then be grouped together per addon package. For example, the default solarsystem could have the SSC file, models, and all textures in "default.zip".

- Break up solarsys.ssc into seperate files... one for each body, and give them seperate addon packages. ( ie. mars.zip, phobos.zip, sol.zip, iss.zip ) Be sure all relevant textures and models are included.

- Celestia, upon startup, searches through the 'data' directory, including subdirectories for addon files. A config or ini file, generated by the package manager, tells Celestia which ones to load.

- Each addon's SSC should be VERY specific about the hierarchy involved for it's objects. For example, phobos.zip would list it's body as "sol/mars/phobos" so that Celestia knows right where it goes. To prevent problems with load ordering ( ie. What if Phobos is loaded before Mars? ), the package manager should sort it's object/addon list before writing it to the config file, so that Sol is always loaded before Mars, and that before Phobos. ( An alternative could have Celestia scan the config file for an addon that will solve it's dependancy problem, and load it first )

- The package manager is responsable for stopping incompatable packages from conflicting. If the user clicks to use the new 'Mars16k' addon, package manager should peek inside it's SSC, get a list of all objects, and compare it with other, currently selected objects. If a duplicate is found, the user is notified and is instructed to choose one.

- Celestia, when instructed by the package manager, should be able to dump the universe and load again from scratch, using the updated config file.

This, IMHO, is the only way to solve the addon problem. It's complicated from the programmer's point of view, but then most sane GUIs are.

Troy,

The package format is for storing in a repository, once you download them they have to be installed. It is be out of the question for Celestia to store everything in zip files (think huge textures).

Besides that is not required for package management to work.

The way the extra directory works at the moment is all we need (a few extensions are probably needed, but the basics are right). When you install a new add-on package it is extracted in the extra directory where it has its own sub-directory. To un-install simply delete that directory.

Of course to have a true package management we need some dependency checking and resolution in addition, but that could be left aside for a first version.

Then a nice user friendly GUI to manage installed packages is the cherry on the cake.

Hi Christophe,

I agree zip may not be the best solution. My plan could even work with each package unzipped into it's own subfolder.

In any case, I'm sure you can agree that no user should have to manually edit a text file ( read: *.ssc ) to get an addon to work, nor should they have to worry about the newest release OVERWRITING the changes they made. My plan outlined above provides a solution to these problems.

Harry wrote:As there probably will be many addons with different version of the same file, a mechanism to resolve conflicts is needed.

Simple solution: sort addons by some key, and use always the file defined in the "highest" addon, or if not in there, use the next lower one - only one addon defines the whole file.

I agree, some kind of automatic priority system would be nice, but I'm not sure how to implement it.

Christophe wrote:It is be out of the question for Celestia to store everything in zip files (think huge textures).

As you can use zip without compression only as a container, even huge textures could be feasible. Would need some testing to find out whether speed would suffer too much. Personally I would prefer addon as one file, but everything in one directory is good too.

Harald

Hi All.

I think that many ideas was submitted in this thread.
Perhaps is arrived the time that the Developer (Christophe) who is in charge for this job (Add-on manager) posts a new thread with all the specifications that he will implement.
Before starting the coding job this specifications list should be suitable for further updates and corrections.

Acting in this way:
#1 All the developers that would help Christophe, will know exactly what will be the objectives of the first and eventually next versions of the add-on manager.
#2 All the users will know exactly what they should expect to have in the next Celestia version that will contain the add-on manager.
#3 All the add-on creators will know what changes will be needed to comply with the new standard (if a new standard will be released).

I suggest to create a new LIB to collect all the new classes.
I'd call it celaddman (Celestia Add-On Manager).

Bye - Paolo

Howdy All,

LOTS of good ideas to thrash around!

Christophe: I was thinking XML (at the Repository end) for Repository access from the Celestia code. Glad to hear there are some possibilities for the beginning Add-On Manager code-base (Celestia side). Are you considering doing the coding for this project -- being the "project lead"?

Harald: <Self-contained addons ...> This is how most current add-ons work now, for object outside the solar system. Inside *requires* manual modification of the solarsys.ssc file. Will you help with any part of this project Harald?

Troy: I agree with Christophe that the existing folder structure will work fine, once Add-On *requirements* are defined. This would not require a new/separate config file just for keeping track of add-ons. If they are Zipped, they are INactive. If they are not Zipped, then they are Active. Will you help with any part of this project Troy?

Paolo: Will you help with any part of this project Paolo? Yes, with all the ideas being generated, we need some way to keep track of them all.

Is it possible to use SourceForge (or something else) to allow multiple people to work on a single text file, adding, changing, etc.? If so, the next message contains a starter design document.

Personally, I think an analysis/design document like this is REQUIRED in order that all functions be listed, agreed upon, assigned a priority (release version), who is assigned to it, AND so that new folks can SEE it and say, "hey, I can do part XYZ." and get involved <smile>.

Lets keep the ideas flowing, and make some decisions as we go, okay?

-Don G.

Here is a STARTER Analysis/Design document for...

don wrote:Here is a STARTER Analysis/Design document for...

Great work don.

I'll post my comments soon.

I don't think a web forum is the best place for this kind of discussions. Long message editing is a real pain, I think a mailling list or newsgroup would be more approriate.

Great job on the design doc, don!

I've been working hard to get OpenGL working under my development environment. Once that is fixed I'll be happy to help with both the KDE gui for the addon manager and the new Celestia backend code that would be required.

I agree with Christophe that this isn't the best way to discuss the finer details of the plan. Perhaps we should all join the devel mailing list? ( Should we get Chris' blessings first? )

Hello,
Great project!
If my small test of catalogue can help, do not hesitate with you to be useful about it!
(I know: somme of links aren't working: i have no time to update them, sorry)

Your project interests me much, since it is -in more modest for mine- in this direction than I will wish to transform my catalogue into true illustrated repertory site of addons, with explanations more detailed than those of my current listing, etc etc. I do not have unfortunately time to devote to it in this moment, nor in the next weeks, because I must move soon...

Jeam, as usual, sorry for my english... but you know that

Hi Don
Great work!!!!!

Christophe wrote:I don't think a web forum is the best place for this kind of discussions. Long message editing is a real pain, I think a mailling list or newsgroup would be more approriate.

I agree with Christophe.
Posting on the development mailing list is not submitted to access restrictions. If Chris agrees we should use it.

I suggest to place the file with all the specs in the sourceforce CVS.

Don wrote:Paolo: Will you help with any part of this project Paolo?

I don't think that I will be able to help very mouch in this phase. I think my C++ knowledge is still not adequate. But obviously I will follow all the discussion and the developement process.

Bye - Paolo

Other 2 suggestions.

The file with the add-on manager specifications :
#1 Must have a date and a version number.
#2 ALL the contained items must be numbered.

So any suggestion, doubt or question about an item should be correctly addressed referring precisely to it.

Bye - Paolo

Paolo wrote:I agree with Christophe.
Posting on the development mailing list is not submitted to access restrictions. If Chris agrees we should use it.

I'd rather keep this separated from the Celestia core development mailling list since this will be orthogonal at least for a while.

I've set up a newsgroup which we can start using right away.

Paolo wrote:I suggest to place the file with all the specs in the sourceforce CVS.

That will be done once they are finalized.

What if someone could make a simple program that does this, you wold open it befor celestia, and select what add ons you want, and dont want, and what the program would do is make a seporate temp folder, and moov the add ons you dont want to this temp folder, and add ons you do want out of the temp folder, and into the "extras" folder.

What do you think?

I've been following this thread for some time now (to the best of my ability, anyway), and I have one question, would this be a stand alone application, or would it be merged with celestia itsself?