Texture, script, add-on installer?

[ajmas]

Having spent much time manually installing textures, I was wondering whether maybe there might be a better way. What I was thinking is possibly packing things in a standard format that could then be read by an installer tool or loaded by Celestia which would then handle putting things in the right place. Possible approach could be using a zip file with a modified extension ('.cel' for example), so it could be associated with Celestia, and then have a machine parseable manifest containing the relative paths for the installation files and possibly specifying any dependencies.

That is the idea anyhow. I would like to get some other points of view before going too far.

[selden]

How do you envision a texture installer being used? I mean, what procedures would the Addon creator use to create a package and then the user to install it?

My own Celestia installations are sufficiently idiosyncratic that I'm not sure I'd trust someone else's preconceived notions about how the files are organized. That's one of the advantages of a zip file. The Addon creator can organize the files so that the zip file can just be extracted into the extras folder and everything will just work, or the user can manually put the files where they'll work best.

However, one of the issues that the MotherLode has is that many Addon creators simply do not follow the guidelines for making an Addon work on multiple platforms. A utility which checks and enforces good Addon packaging practices while generating a zip file could be quite helpful in that regard.

Bear in mind that the packager should run on all of the architectures supported by Celestia. Writing it in Lua as a CelX script might be a start in meeting that requirement.

A warning: a few years ago there was a very long discussion of what a Celestia Addon Manager should look like, including a formal description of its features, but it finally got so complicated that none of the people involved were willing to invest the time in actually implementing it. (Unfortunately the News server where the discussion took place is no longer available.)

i.e. Keep It Simple.

[ajmas]

Keeping it simple is my intention, as is trying to make the user experience simpler. This is part of the reason for reusing the zip container as the packaging format, but with a different extension. This approach has already been used by Java and Open Office, amongst others. Ideally the archive would respect the structure of the extras folder and the 'manifest' file would describe the author, the package and its contents - at the moment I am thinking of an xml format (but I am not committed to it), because as much as some people hate it, at least it enforces structure and there are libraries in various languages for it.

With regards to the MotherLode by having all the necessary information in a parsable format the upload script would be able to extract the necessary information for cataloging it properly and flagging any inconsistencies and rejecting as appropriate.

I am going to look at Adium on the Mac to see how it approaches add-ons.

With regards to the previous discussion, we can start with a simple concept and have it evolve over time.

[John Van Vliet]

--- edit ---

[Guckytos]

The idea with a changed ZIP-container is not too bad. I can see some benefits there.

But I would make it simpler, at least as a first stage to test this idea out.

• The ZIP-container should contain all the necessary files and folders for the addon
• There should be a manifest file in the top folder of the addon
• This manifest file should contain information about the addon
• The container then has to be placed by the user in his Extras-directory
• Celestia starts, finds the container, reads the manifest and checks the addon
• Celestia throws an error message if the container is corrupt on screen (like "addon xyz is broken" and makes a detailed error output file in the Extras directory --> This way the startup console log could be less cluttered if you have a lot of addons)
• If the container is okay, Celestia loads the addon

And this same idea could also be used for scripts. At least the part with the manifest file. Or it could be a part of the script header.

• Script is put into scripts folder
• Celestia reads manifest
• Celestia displays in the run scripts dialogue the title of the script and has an option to show further info on the script. (right click, submenu?)

This would make script and addon management a lot easier. It should be really thought about putting something like this into Celestia.

My 2 cents.

Regards,

Guckytos

[John Van Vliet]

--- edit ---

[Guckytos]

Hi John,

there seems to be some kind of misunderstanding here about what I am talking about.

Your comments are completely valid, if we are talking about a ZIP file that is downloaded by the user and then extracted in the Extras directory.

But what I am talking about is to have a Container format that does not need to be unpacked in the Extras directory.
The container itself is the complete addon with all necessary files in it.

If you view my post with that in mind the requirements that I posted make sense (I hope).

lets see

The ZIP-container should contain all the necessary files and folders for the addon

this it already a REQUIREMENT ( with the normal exceptions )

# There should be a manifest file in the top folder of the addon
#This manifest file should contain information about the addon

like the required README file

More or less, but the exception that Celestia refuses (or could be made to refuse) to load the addon if it is missing.

The container then has to be placed by the user in his Extras-directory

addons do , just unzip them INTO the root celestia folder or into the extras as is stated in the README file

As I wrote above, this container should not need to be unzipped but Celestia reads it directly. Okay, there would need to be tests if there's impact on speed of Celestia by this.

Celestia starts, finds the container, reads the manifest and checks the addon

celestia dose check the files in celestia/extras & etras-normal, and any that are listed in the "celestia.cfg" file

Celestia throws an error message if the container is corrupt on screen

it dose , that is how i check add ons using ( <shift> + <~> )

* If the container is okay, Celestia loads the addon

well celestia dose run unless the add on causes the program to have a fatal crash

Those points also would have to be lookend at with the idea in mind that the container IS the addon. <shift> + <~> is sure a good thing, but does not always help, especially if a lot of addons are installed and all messages are going past. Okay it stops at a critical error.
But with the other check you would also i.e. look if all textures are there and if not throw an error with details what is missing.

Hope that clarifies my idea further.

[Guckytos]

Post Scriptum:
I moved the post to here as this is the better place for it.

Post Scriptum 2:
This idea of mine could be enhanced further:
This form of packaging would make it easy to have Celestia create and maintain a list of installed addons.
This list could then be displayed somewhere (under File menu/Addon Management?).
Each addon in this list could then have a (check box) showing the status of the addon (Activated/Loaded or Deactivated/Unloaded).
How about it exactly works would have to be decided. A real time loading/unloading of addons would be really nice. But it could also be solved the way that you select the addon changes and then tell Celestia to restart.

Just some further ideas on this.

[ajmas]

I feel this should be split into two tasks: add-on design and add-on management. The idea being that in the short term the goal of simply being able to download and add-on and put in one folder without needing to understand the instructions in the zip (when they are present or clear).

I was thinking of what information would be contained in the 'machine parsable' description file, trying to provide a structure that doesn't fall apart in certain edge cases and that also could be used to populate a site such as the "Celestia Motherlode". Below is a first draft just to see what needs changing and whether this seems like a suitable direction:

Code: Select all


<celestia-addon version="add-on-xml-version">
   <info publication="20090707102100">
      <author>madhatter</author>
           <version>2.0</version>

      <license name="" url="">text if non-standard</license>
   
      <summary>Short one sentence summary</summary>
      <description homepage="">Details about the plugin supporting html
          tags</description>

           <!--  intended to label add-ons that requires an Internet connection
              to function. Think real weather information, real satellite positioning or a map server
                similar to that used by Google Earth, as possibilities -->
           <online-content>true</true>

      <time-frame>past|future|real-time|simulation|fictional</time-frame>
      <!-- Additional info about the time frame. In case of fictional
         then the name of the name of the 'universe' is used, such as
         'Star Wars' -->
      <time-frame-info>...</time-frame-info>
      
      <!-- the type of object represented by this add-on. none would be for scripts
           or add-on that don't represent a specific objet. -->
      <object-type>asteroid|comet|galaxy|nebulae|none|planet|satellite|spacecraft|other..</object-type>
      
      <!-- The name of the object as used by Celestia. English name or catalog ID. Should we provide option for providing both? -->
      <object-name>Mars</object-name>
   </info>
   
   <!-- Names of data types provided by plugin. Allowing for an add-on to
        be a hybrid. Type names would be 'standardised' so that they can
        properly be grouped together with other add-ons offering the same
        type.
   <data-types>
       <type name="data"/>
       <type name="surface map"/>       
   </data-types>
     
</celestia-addon>



[duds26]

What you want to have is a uniform system of describing addons.

Your use of zip archive is not bad, but I find it restricting.

If you just use txt files that say that the folder they are in is an addon that should be fine too.

Let those have a special extension e.g.: *.celaddon

A few important notes with designing such informational structures.

1) Have a version number that says what version of addon structure you use. (Not from the addon but from how you want the information of an addon present itself, thus your format.
Thus having a addonVersion and a addonFormatVersion
2) Avoid using comments for information
3) use folders
4) Don't add things that can be found by looking into the addons content, presenting this information in a clear way is the task of a graphical user interface

With five, i would say the time frame itself would be in the xyz and other files.
Don't do the names of data types, because its useless. This can be done with an algorithm automatically without needing an extra place to describe it by looking into the addons content.


The version thing would also allow future compatibility with zipping, tarring and other things.
As well as allow you to break the names with new revisions, without breaking old addons.

I would strongly suggest your focus on Celestia itself.

Celestia checks for a few preprogrammed directories.
It would be handy if you could add multiple paths and folders yourself.

Also it would be handy on a per Celestia basis (thus not in the addon information) to enable and disable certain addons.
(Default for new addons would be enable)

[duds26]

For presenting information about an addon that can be used in celestia motherload.

I would suggest some sort of html with metadata.
This way you could have something the author fills in.
That can be used unchanged on the motherload and local computer.

With xml in html you can add the remaining things that aren't part of showing information.

Here is some information:
http://poeticcode.wordpress.com/2007/10/24/inlineembedded-xml-in-html/
Nog een voorbeeld: http://www.expertrating.com/courseware/XMLCourse/XML-Embedding-HTML-8.asp
Informatie van de w3c: http://www.w3.org/TR/NOTE-xh

You could use html for stuff that's being viewed with a template and adding the rest in the template with an xml data island.
This way the xml won't be displayed in the webpage for viewing but is still in the same file.
Just embed the xml inline.

[ajtribick]

From the comments in this thread about add-on load order, might be useful to be able to specify order of .stc and .ssc files.

Personally I'd be wary of using XML for such a file. I'd think it probably would be better to stick to the existing syntax, if only because it means we don't have to add another parser to the code. I certainly don't think it would be a bad idea to consider migrating to something a bit more standardised, e.g. JSON (which isn't a million miles from the current syntax, although the lack of comments is annoying) for the data files as this could have benefits for writing utility applications, but that's a sufficiently breaking change that almost certainly we'd have to wait for Celestia 2.0 (whenever that might happen).

[duds26]

Using a (x)html file for presenting information that also can be used on motherload.
While having an .celaddon with simple syntax for the rest then?

[ajmas]

Although I don't mind if another data markup is used, I thought about XML for several reasons:
- schemas can be used for validating the format
- namespaces can be used for adding non-standard info in a standardised way, without breaking the existing parsing of the file
- JavaScript can parse XML
- it can converted to HTML directly, by the browser with stylesheet

The issue I would have with JSON is as the name suggests it is very Javascript oriented.

Maybe we could use the same document structure as already used by celestia, though we need to think of what we need from the data format.

When I drew up my example above I was thinking of several things:
- what information is needed by Celestia to manage the plugin properly
- what information is needed to catalogue the plugin, such in Motherlode, while providing a certain flexibility and taking into account the sort if plugins that could be developed.
- what information is needed to contact the developer

If the developer can achieve all the tasks in one file it would be handy. It would also open the doors to Celestia being able to provide access to a catalogue of available plugins directly. Maybe we could focus in point 1 (above), but accept input for the other points.

[ajtribick]

JSON isn't JavaScript specific, it seems to arise from a variety of languages of which JavaScript is just one. Certainly parsers exist for most languages out there, including C++ and Perl. XML on the other hand is a fairly heavy file format to work with, personally I generally find it less readable and easier to work with (less having to worry about namespace URLs for starters). Schemas are nice but probably not essential, and whether the benefits of schema validation outweigh the cons of the XML format is another matter...

For example, your XML file rewritten as JSON (date format may need some kind of alteration but this is a fairly quick comparison)

Code: Select all

{
   "version": "add-on-json-version",
   "info": {
      "publication": "2009-07-07 10:21:00",
      "author": "madhatter",
      "version": "2.0",
      "license": {
         "name": "name",
         "url": "http://www.example.com",
         "text": "text"
      },
      "summary": "brief summary",
      "description": {
         "homepage": "http://www.example.com",
         "text": "text"
      },
      "online-content": "true",
      "time-frame": [ "past", "future", "real-time", "simulation", "fictional" ],
      "time-frame-info": "info",
      "object-type": [ "asteroid", "comet", "galaxy", "nebulae", "none", "planet", "satellite", "spacecraft", "other" ],
      "object-name": "Mars"
   },
   "data-types": [ "data", "surface map" ]
}

Certainly I find this syntax makes it a lot easier to see what is going on than XML does. Going from that to something like what we have in the Celestia parser is pretty much a matter of removing the colons and commas and unquoting the property names (probably we'd have to add something to handle arrays of strings, but that shouldn't be too problematic):

Code: Select all

AddOn {
   Version "add-on-json-version"
   Info {
      Publication "2009-07-07 10:21:00"
      Author "madhatter"
      Version "2.0"
      License {
         Name "name"
         Url "http://www.example.com"
         Text "text"
      }
      Summary "brief summary"
      Description {
         Homepage "http://www.example.com"
         Text "text"
      }
      OnlineContent true
      TimeFrame [ "past" "future" "real-time" "simulation" "fictional" ]
      TimeFrameInfo "info"
      ObjectType [ "asteroid" "comet" "galaxy" "nebulae" "none" "planet" "satellite" "spacecraft" "other" ]
      ObjectName "Mars"
   }
   DataTypes [ "data" "surface map" ]
}

[Osprey]

I want to inform all about my proto celpkg manager writen on tcl/tk, I was wrote core 1 year ago (as test application when i learn TCL language), being still unfinished it can download (also mirrors available too), extract, patch and install files or configure celestia.cfg, or lua files (very and very crap, but work). I think I need rewrite it, because here is many crap code that hard for support.

Below example of install script for one of addons preganareted by my robot ( that download all addons from celestiamotherload and it is over of 29G data)

Code: Select all

# -*-coding: utf-8; mode: tcl -*-
# Created by robot ./motherload-gen-index.pl
$addon "2010 Exoplanets"
$category {Extrasolar/Stars} {Experimental/Robot/Extrasolar/Stars}
$version {1.0}
$created 2010-04-02
$modified 2010-04-17
$description 19 new exoplanets discovered in early 2010 (Jan-March 2010).
$www "http://www.celestiamotherlode.net/catalog/show_addon_details.php?addon_id=1450"
$license "Public Domain"
$author {Brad}
$distfile {
    -name "2010_Exoplanets.zip" -size 13845 -md5 d4e64e1f73dc14caceb78e34f06af919
    -url http://www.celestiamotherlode.net/creators/Brad/2010_Exoplanets.zip }
$unpack {
    -file "2010_Exoplanets.zip" -type zip
    -dir "extras/2010 Exoplanets" }
$screenshot {
    http://www.celestiamotherlode.net/catalog/images/screenshots/various/extrasolar_stars_2010_Exoplanets_1__.jpg
    http://www.celestiamotherlode.net/catalog/images/thumbs/various/extrasolar_stars_2010_Exoplanets_1__.jpg }
$install
    {"~" -skip}
    {"Thumbs.db" -skip}
    {}
$end


And some font configure script example:

Code: Select all

# -*- coding: utf-8; mode: tcl -*-
$addon "Font Configure"
$description Setup fonts for different languages.
$version 1.0
$category configure
$author Osprey
$maintainer Osprey
$created 2010-06-07
$modified 2010-09-02
$choice {"lang" "Русский" "Target language font" {"Українська" "Русский" "Български" "???" "??" "English \"clean\" font"}}
$options {bigfont
    yes "Prefer big fonts instead small"}
# title font
$xpatch {if {$opt(lang) == "Українська"}} {
    { -file celestia.cfg -body {script variable {Configuration TitleFont} set {"sansbold20_uk.txf"}} } }
$xpatch {if {$opt(lang) == "Русский"}} {
    { -file celestia.cfg -body {script variable {Configuration TitleFont} set {"sansbold20_ru.txf"}} } }
$xpatch {if {$opt(lang) == "Български"}} {
    { -file celestia.cfg -body {script variable {Configuration TitleFont} set {"sansbold20_bg.txf"}} } }
$xpatch {if {$opt(lang) == "???"}} {
    { -file celestia.cfg -body {script variable {Configuration TitleFont} set {"sansbold20_ja.txf"}} } }
$xpatch {if {$opt(lang) == "??"}} {
    { -file celestia.cfg -body {script variable {Configuration TitleFont} set {"sansbold20_zh_CN.txf"}} } }
$xpatch {if {$opt(lang) == "English \"clean\" font"}} {
    { -file celestia.cfg -body {script variable {Configuration TitleFont} set {"cleanbold16.txf"}} } }
# small font
$xpatch {if {!$opt(bigfont) && $opt(lang) == "Українська"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans12_uk.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans12_uk.txf"}} }
}
$xpatch {if {!$opt(bigfont) && $opt(lang) == "Русский"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans12_ru.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans12_ru.txf"}} }
}
$xpatch {if {!$opt(bigfont) && $opt(lang) == "Български"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans12_bg.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans12_bg.txf"}} }
}
$xpatch {if {!$opt(bigfont) && $opt(lang) == "???"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans12_ja.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans12_ja.txf"}} }
}
$xpatch {if {!$opt(bigfont) && $opt(lang) == "??"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans12_zh_CN.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans12_zh_CN.txf"}} }
}
$xpatch {if {!$opt(bigfont) && $opt(lang) == "English \"clean\" font"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"clean12.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"clean12.txf"}} }
}
# font big
$xpatch {if {$opt(bigfont) && $opt(lang) == "Українська"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans14_uk.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans14_uk.txf"}} }
}
$xpatch {if {$opt(bigfont) && $opt(lang) == "Русский"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans14_ru.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans14_ru.txf"}} }
}
$xpatch {if {$opt(bigfont) && $opt(lang) == "Български"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans14_bg.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans14_bg.txf"}} }
}
$xpatch {if {$opt(bigfont) && $opt(lang) == "???"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans14_ja.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans14_ja.txf"}} }
}
$xpatch {if {$opt(bigfont) && $opt(lang) == "??"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"sans14_zh_CN.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"sans14_zh_CN.txf"}} }
}
$xpatch {if {$opt(bigfont) && $opt(lang) == "English \"clean\" font"}} {
    { -file celestia.cfg -body {script variable {Configuration Font} set {"clean16.txf"}} }
    { -file celestia.cfg -body {script variable {Configuration LabelFont} set {"clean16.txf"}} }
}


My opinion about my pkg manager:
+ Crossplatform;
+ Can setup several celestia's data directories, and launch celestia by short key;
+ Almost all install scripts for addons hosted in celestiamotherload may be generated by simple robot;
+ Dependencies support.
+ Options. User may configure for download Hi or low res texture pack for example.
+ Configure lua file (add in lua file "require" and array edit).

- It use many tools (yeah crappy unix way) wget, patch, tar, unzip and other.
- tcl/tk. Fast develop but really not easy to support.

Currently I freeze develop, and next 1-2 weeks I cannot to return. Bu I write here when I prepare windows binaries and several test suit. (aprox ~2-3 weeks)

https://github.com/osv/celpkg

[ajmas]

The advantage with the JSON or XML formats is that they are both well known formats and they provide a tree like structure, which is useful for grouping information.

In reality though, the effort should be less on the file format, but the type of data that goes into it, allowing it to be indexed and also so that Celestia can painlessly install it, without the need of a manual user step. Please could you focus on that. Once we have done that, the format of the file is a luxury.

Edit: I think I jumped the gun a little. I'll take a look at celpkg and see how well it works and what needs improving.

[Osprey]

In any case for it be great to make celestia read zip files like it in quake for .pak files `cause several addons may contain thousands files.

[Osprey]

Eh, I need more time to finish all I'm planning. Currently I create ports for ~100 add-ons (celestiamotherload contain over 1000) for my celpkg (Almost all extrasolar category done).

Downloads:
Unix celpkg-unix-1.0.3.tar.gz (you need install tcl/tk and wget)

Windows: celpkg-pack-1.0.3.exe

Some screenshot:


Install one of the core pkg - celestia current, celestia 16, or celestia 15 before other (add-ons that require celestia.cfg for patching or other files will install as dependencies latest version i.e. "celestia current" but you may wish celestia 1.5 for example). You need original binaries of celestia (soon I prepare windows binary add-on so you don't need to have installed celestia in your system) to launch celestia by C-x C-l key (long keys like in emacs, first Crt+X than Ctr+L) or, if you wish, use command line --dir option.

Run `wish8.5 celpkgk.tcl --configure' to reconfigure your profile (Windows: start -> progr.. -> celpkg -> Celpkg configure).
Here no documentation about format of "index" file which is used for ports, but not sure that someone will have much time to help me port other 900 add-ons.

[ajmas]

On MacOS X I get the following error:

Code: Select all

$ wish8.5 celpkgk.tcl
Error in startup script: can't find package BWidget
    while executing
"package require BWidget"
    (file "celpkgk.tcl" line 12)


What is generally the right way to install this? I will look in the meantime.