Using Lua for Celestia Data Definition

[hank]

On another thread, I mentioned the possibility of using Lua for data definition in Celestia. This suggestion was prompted by Paolo's proposal for a revision of the existing data definition file format:

Going further the new object model should be implemented through a new data format.

I would like to discard the XML. It is powerful but too verbose. I would like that the current data format should change slightly the syntax.

So instead than:

Code: Select all

"Ceres" "Sol"
{
   Class "asteroid"
   Texture "asteroid.jpg"
   Radius 487.5       
   Oblateness 0.068
   ...
}

...


Galaxy "NGC 1:UGC 57:MCG 4-1-25:ZWG 477.54"
{
        Type  "Sb"
        RA            0.1208
        Dec          27.7089
        Distance   2.073e+08  # method: T-F
        ...
}

...

Location "Aix" "Sol/Gaspra"
{
    LongLat [ -156 47 0 ]
    Size 0.6
    Type "AA"
}

...




That requirese different file formats and different file extensions I would like to see something like:

Code: Select all


Asteroid
{
   GUID [ 1236 6585 546a c00a ]
   ParentGUID [ 1256 8474 1565 aa45  ]                  #"Sol"
   Name "Ceres"

   Texture "asteroid.jpg"
   Radius 487.5       
   Oblateness 0.068
   ...
}

...

Galaxy
{
        GUID [ 1325 135c 1325 dd23 ]
        ParentGUID [0000 0000 0000 0000]      # "Universe"
        Name "NGC 1:UGC 57:MCG 4-1-25:ZWG 477.54"
 
        Type  "Sb"
        RA            0.1208
        Dec          27.7089
        Distance   2.073e+08 
        ...
}

...

Location
{
    GUID [ 1356 1258 4785 aae2]
    ParentGUID [ 1256 1588 4589 1556 ]       # "Sol/Gaspra"
    Name "Aix"
 
    LongLat [ -156 47 0 ]
    Size 0.6
    Type "AA"
}




Using Lua, Paolo's example might look like this:

Code: Select all


Asteroid
{
   GUID = "1236 6585 546a c00a" ;
   ParentGUID = "1256 8474 1565 aa45" ;                  -- "Sol"
   Name = "Ceres" ;

   Texture = "asteroid.jpg" ;
   Radius = 487.5 ;     
   Oblateness = 0.068 ;
   ...
}

...

Galaxy
{
        GUID = "1325 135c 1325 dd23" ;
        ParentGUID = "0000 0000 0000 0000" ;      -- "Universe"
        Name = "NGC 1:UGC 57:MCG 4-1-25:ZWG 477.54" ;
 
        Type  = "Sb" ;
        RA     =     0.1208 ;
        Dec    =   27.7089 ;
        Distance  = 2.073e+08 ; 
        ...
}

...

Location
{
    GUID = "1356 1258 4785 aae2" ;
    ParentGUID = "1256 1588 4589 1556" ;       -- "Sol/Gaspra"
    Name = "Aix" ;
 
    LongLat = { -156, 47, 0 } ;
    Size = 0.6 ;
    Type = "AA" ;
}




The syntax changes are fairly minor. Attributes and their values are separated by an equals sign, attribute value pairs are separated by semicolons, square brackets are replaced by curly braces (and the items within separated by commas), and double hyphens are used for comments. (I also used strings for the GUIDs, but that is optional.)

This is a valid chunk of Lua code. Each construct (Asteroid, Galaxy, Location) is a Lua function call, consisting of the function name followed by a table constructor. The table constructor is just a list of key value pairs enclosed in braces.

This approach makes it easy to add new constructs, simply by adding new Lua functions. Thus, another of Paolo's examples:

Try to imagine what you should do using a unified data format with something like this:

Code: Select all


File
{
    ...
}



It is the end of the proliferation of different folders and different Celestia installations for the management of the add-ons!

With the Lua approach, this feature could be added simply by implementing a Lua function named "File". And this would not require a change to the compiled code.

Because of the power of Lua programming, the possibilities are endless.

- Hank

[Paolo]

Now I can see!

Well! It is quite interesting. I like it very much.

Do you think that this will be implemented in a near future?

[hank]

Do you think that this will be implemented in a near future?

I really don't know. This is the first it's been discussed, and I have no idea what Chris might think about it.

However, I believe the current built-in parser builds a C++ hash table with the key-value pairs and then calls a C++ function that uses it to build the internal C++ data structures. So it would be a relatively straight-forward modification to take the key-value pairs from a Lua table instead, I think.

I'm sure Chris could do it if he thought it was a good idea.

- Hank

[hank]

One advantage of using Lua for data definition in Celestia is that all the facilities of Lua are available at no extra charge. Here's another of Paolo's examples:

Code: Select all

Planet
{
?  ?  ...
?  ?  ParentGUID [??]?  # Sol
?  ?  Name "Earth"
?  ?  ...
}

Satellite
{
?  ?  ...
?  ?  ParentGUID [??]?  # Earth
?  ?  Name "Moon"
?  ?  ...
}

Ship
{
?  ?  ...
?  ?  ParentGUID [??]?  # Moon
?  ?  Name "Apollo11"
?  ?  ...
}


Since Lua supports variables, you could easily do the following:

Code: Select all

SolGUID = "1232 1527 38b2 c0af" ;

EarthGUID = "1236 6585 546a c00a" ;
Planet
{
?  ?  GUID = EarthGUID ;
?  ?  ParentGUID = SolGUID ;?  
?  ?  Name = "Earth";
?  ?  ...
}

MoonGUID = "1256 8474 1565 aa45";
Satellite
{
?  ?  GUID = MoonGUID ;
?  ?  ParentGUID = EarthGUID ;
?  ?  Name = "Moon" ;
?  ?  ...
}

Ship
{
?  ?  ...
?  ?  ParentGUID = MoonGUID ;
?  ?  Name = "Apollo11" ;
?  ?  ...
}


You could also set up the GUIDs in a Lua table indexed by name or path, or provide a function to look up GUIDs using name or path, etc.

- Hank

[ElChristou]

Chris is flying to Bolivia for a 2 or 3 weeks treck in the Ande, so we will have to wait a bit for his opinion...

[t00fri]

Chris is flying to Bolivia for a 2 or 3 weeks treck in the Ande, so we will have to wait a bit for his opinion...

Tomorrow

Incidentally, did you all notice that Selden's name color changed from green to orange ?


Bye Fridger

[Paolo]

Hank

There's one thing that I can't understand.
How the objects are loaded form the unified Lua data format?

Will Celestia run a LUA script at startup (that perhaps will be specified in the confing file), that will use the LUA parser for the data loading?

If so can you post an example that shows what instructions this script will contain?

[hank]

As another possibility, any entries in the Lua table used to define an object which are not directly mapped to the internal Celestia data structure could be retained in a Lua table associated with the object. This table could be accessed by Lua scripts written to use the additional entries for any purpose.

- Hank

[hank]

Will Celestia run a LUA script at startup (that perhaps will be specified in the confing file), that will use the LUA parser for the data loading?

If so can you post an example that shows what instructions this script will contain?

Yes, at startup Celestia would run a Lua script specified in the config file. This script would load the data. It would do this ultimately by invoking built-in code to build the internal data structure. What the script does specifically is configurable. In other words, you can write the script to do whatever you want it to do. One option would be for the script to call a built-in routine that loads the data just the way Celestia does now. This would provide backward compatibility, but allow interim development and experimentation with more advanced Lua-based data definition techniques.

- Hank

[Paolo]

Hank

I got your idea now.

I think that you in the development team should discuss a lot about this.
IMO it is a very good approach to the unified data file format and should open other very interesting opportunities in add-on management even if you won't plan to introduce the usage of GUIDs... by now.

So we will wait until Chris will return from his holiday.

Kind regards

[chris]

I've been following both this and Paolo's thread with interest, but I've been too busy with trip preparations and work to comment much.

There are some compelling things about using Lua code as a data format, but I'm concerned that it may offer *too much* flexibility in the file format, turning it into essentially a write-only format. Once you introduce control structures into your data format, it becomes impossible to read an arbitrary file of that format, modify it, then write it out again. We've already compromised the SSC format somewhat with the Modify feature for object definitions. I think this is a worthwhile discussion to be having, but at this point I tend to favor a pure data format.

--Chris

[ajtribick]

There are some compelling things about using Lua code as a data format, but I'm concerned that it may offer *too much* flexibility in the file format, turning it into essentially a write-only format. Once you introduce control structures into your data format, it becomes impossible to read an arbitrary file of that format, modify it, then write it out again. We've already compromised the SSC format somewhat with the Modify feature for object definitions. I think this is a worthwhile discussion to be having, but at this point I tend to favor a pure data format.

This is a good point here, I was going to ask how Lua would handle Modify/Replace (and also maybe Delete, which would be a useful feature to implement at some stage).

Adding further programming features beyond these would make writing third-party applications which take .ssc files as input much harder. If you consider what I was doing for the backend to my EGP add-on, I basically had to extract star information from stars.dat and the various .stc files, and parse the extrasolar.ssc file.

With the current format, I only had to worry about coding for Replace operations (default behaviour for .stc files), since the extrasolar.ssc file doesn't contain Modify directives. If we were using Lua, I would have had to incorporate potentially a full Lua parser, which seems like overkill for the application I was writing.

Also, do we really need all this GUID/ParentGUID stuff? We seem to have a fairly adequate system as it is to define parent objects which is less verbose than writing GUID=foo ParentGUID=bar and doesn't have loads of hex all over the place (which I find decidedly offputting and unattractive). The proposed inclusion of all this GUID stuff in the file seems to open us to the possibility that two completely unrelated add-ons may end up using the same hex codes, so a galaxy add-on may break a planetary system. I don't want to have to download every add-on out there or look up a list of who's using what GUIDs to make sure my GUIDs are unique. GUIDs should probably be internal and invisible to end-users.

The other issue is of course security. I don't want to worry about whether a .ssc file contains code that may wipe out my computer.

[hank]

I've been following both this and Paolo's thread with interest, but I've been too busy with trip preparations and work to comment much.

Chris,

Have a great trip. I'll continue trying to persuade you when you return...

- Hank

[Telepath]

The proposed inclusion of all this GUID stuff in the file seems to open us to the possibility that two completely unrelated add-ons may end up using the same hex codes, so a galaxy add-on may break a planetary system.
I don't want to have to download every add-on out there or look up a list of who's using what GUIDs to make sure my GUIDs are unique. GUIDs should probably be internal and invisible to end-users.

Actually, this is exactly what properly generated GUID's are designed to avoid.

This link: http://en.wikipedia.org/wiki/Universall ... Identifier
explains quite well about how UUID's/GUID's work.
Hope this helps.

I do agree that they should not appear in external interfaces such as SSC or XML files, but should be internal to Celestia only.

[hank]

This link: http://en.wikipedia.org/wiki/Universall ... Identifier
explains quite well about how UUID's/GUID's work.
Hope this helps.

I do agree that they should not appear in external interfaces such as SSC or XML files, but should be internal to Celestia only.

I'm not necessarily advocating GUIDs here. I just included them in my Lua example because Paolo had used them. The issues pro and con re GUIDs should be discussed here.

- Hank

[Telepath]

I'm not necessarily advocating GUIDs here. I just included them in my Lua example because Paolo had used them. The issues pro and con re GUIDs should be discussed here.

Thanks Hank, I've actually already posted the link on Paolo's thread. I posted it here as well to try and help clear up some mis-conceptions apparent in this thread.

No intention to turn this thread into a discussion on GUID's... just trying to help Chaos Syndrome understand them as he commented on them above.