Unified Data Format for new object model

[Paolo]

I don't know if some developers will be involved in this discussion, but I hope so.

Sooner or later Celestia will arrive to a major release. For the release 2.0 I would like that this issue will be discussed and implemented.

In my vision each Celestia object (each) must have two additional properties. The GUID (Global unique identifier) and the Parent ID (The GUID of the parent). Associating these two properties with the Type of the object will be possible to manage the Celestia's object model as a Relational database.

So if someone of you has experienced for a while about database implementation you can only start to imagine the number of features that should be added to Celestia.

Implementing these propertis with a convenient datatype such i64 the total amount of overhead in terms of additional memory should be almost negligible.

What do you think? Can you imagine the new possibilities? Can you see some unwanted drawbacks?

Kind regards

[Paolo]

Going further the new object model should be implemented through a new data format.

I would like to discard the XML. It is powerful but too verbose. I would like that the current data format should change slightly the syntax.

So instead than:

Code: Select all

"Ceres" "Sol"
{
   Class "asteroid"
   Texture "asteroid.jpg"
   Radius 487.5       
   Oblateness 0.068
   ...
}

...


Galaxy "NGC 1:UGC 57:MCG 4-1-25:ZWG 477.54"
{
        Type  "Sb"
        RA            0.1208
        Dec          27.7089
        Distance   2.073e+08  # method: T-F
        ...
}

...

Location "Aix" "Sol/Gaspra"
{
    LongLat [ -156 47 0 ]
    Size 0.6
    Type "AA"
}

...




That requirese different file formats and different file extensions I would like to see something like:

Code: Select all


Asteroid
{
   GUID [ 1236 6585 546a c00a ]
   ParentGUID [ 1256 8474 1565 aa45  ]                  #"Sol"
   Name "Ceres"

   Texture "asteroid.jpg"
   Radius 487.5       
   Oblateness 0.068
   ...
}

...

Galaxy
{
        GUID [ 1325 135c 1325 dd23 ]
        ParentGUID [0000 0000 0000 0000]      # "Universe"
        Name "NGC 1:UGC 57:MCG 4-1-25:ZWG 477.54"
 
        Type  "Sb"
        RA            0.1208
        Dec          27.7089
        Distance   2.073e+08 
        ...
}

...

Location
{
    GUID [ 1356 1258 4785 aae2]
    ParentGUID [ 1256 1588 4589 1556 ]       # "Sol/Gaspra"
    Name "Aix"
 
    LongLat [ -156 47 0 ]
    Size 0.6
    Type "AA"
}




For the XYZ paths I would like to see something like a key framing animation like this:

Code: Select all


Path
{
    GUID [5468 5656 4486 4654]
    ParentGUID [4464 4548 6532 1354 ]    #cassini
    ReferenceGUID [1325 1256 1325 1153]    # sol
    Begin [2005 01 01 22 15 12]   # YMDhms format
    End [2005 12 31 24 00 00]   # YMDhms format

     PathItems
     [
          [1 2005 01 01 22 15 12 138181017 56155827 1471] # (Key type position) Y M D h m s X Y Z
          [2 2005 01 01 22 15 12 1.0 1.0 1.0 1.0] # (Key type orientation quaternion) Y M D h m s X Y Z U
          [1 2005 01 05 12 03 00 94043905 117712356 1603761]
          ...
          [2 2005 01 05 12 03 00 0.25 0.55 0.83 1.0]
          ...
          [1 2005 12 31 24 00 00 -12572111 136402314 2836001]
     ]

}



So it should be possible to create different path segments for Cassini using different frame of references, flybys will be always perfect!

Not very difficult to do isnt'it? So why not thinking about?

Is it necessary to brake the backward compatibility with the current data model? Yes but why not? Converting the current Add-ons in the new format is trivial.

Is it necessary to study a good strategy for the GUID definition? Yes I'm issuing this thread to discuss about this.

But this is only the begining.

Try to imagine what you should do using a unified data format with something like this:

Code: Select all


File
{
    ...
}



It is the end of the proliferation of different folders and different Celestia installations for the management of the add-ons!


Kind regards

[Paolo]

Returning on the new xyz unified format take a look at this:

Code: Select all

Path
{
    GUID [...]
    ParentGUID [... ]    # Opportunity
    ReferenceGUID [...]    # Gusev crater Location
    Begin [...]   # YMDhms format
    End [...]   # YMDhms format

     PathItems
     [
          [1 ... ] # (Key type position) Y M D h m s X Y Z
          [2 ... ] # (Key type orientation quaternion) Y M D h m s X Y Z U
          [1 ... ]
          ...
          [2 ... ]
          ...
          [1 ... ]
     ]

}


It should be the path of a rover on a planet surface starting from the specified location.

The same mechanism should be used for the path of one arm of the antenna of a probe that is changing its orientation during a flyby .

It si difficult to imagine all the possibilities.

[ElChristou]

Paolo, this sounds quite interesting but because it's a quite advanced proposition, why don't you point the dev team over your thread by a quick message to the dev list? (don't forget some of them don't use the forum)

[Paolo]

Hi ElChristou

I've subscribed the Celestia developer mailing list from 2002 but I've never posted on it.
I'm the promoter of the a new developer forum for the dev team.
Thank you for the suggestion but IMHO I should be not very coherent.
Moreover Fridger has said that on the forums there aren't so many interesting discussions!
I hope that the developers will come out to discuss here even if the argument is a bit advanced.

Kind regards

[Malenfant]

Not that I'm a developer, but to me this looks a bit too esoteric and not user-friendly for human input? I wouldn't fancy typing in streams of numbers in the GUID fields every time I wanted to make an object in an add-on... (maybe that's because I have no idea what the numbers in the GUID field mean...?)

[steffens]

My imagination is not that good as well, so could you please try to explain some examples of possible uses of these GUIDs?
The advantages should be quite strong because I think having to use GUIDs (globally unique IDs) would most probably complicate addon creation.

steffens

[Paolo]

Not that I'm a developer, but to me this looks a bit too esoteric and not user-friendly for human input? I wouldn't fancy typing in streams of numbers in the GUID fields every time I wanted to make an object in an add-on... (maybe that's because I have no idea what the numbers in the GUID field mean...?)

Good point Malenfant.

The GUID is a common argument in databases and in windows development. The M$ Windows object model that at present time is still relying heavily on OLE is based upon concepts like this. Every windows user can find the GUIDs of the OLE Server applications browsing the registry. Moreover in windows they are called UUIDs Universal Unique IDentifiers and in the development tools there are specific programs for their generation.

In our case they will be simply codes. In my previous posts I've supposed a long integer number in Hex format for convenience in memory consumption and for further fast sorting and indexing. But it should be used everything else that is unique. So if the names of objects are unique it should be used something like this.

Code: Select all


Asteroid
{
   GUID "Ceres"
   ParentGUID "Sol"
   Name "Ceres"

   Texture "asteroid.jpg"
   Radius 487.5       
   Oblateness 0.068
   ...
}



But this will inhibit another gracious thing like this:

Code: Select all


Language
{
    GUID [XYZQW]
    Name "Italiano"
}

LanguageItem
{
    GUID [??]
    ParentGUID[??]   #italiano
    ReferenceGUID[??]  # Sol Sun
    Text "Sole"
}

LanguageItem
{
    GUID [??]
    ParentGUID[??]   #italiano
    ReferenceGUID[??]  # Earth
    Text "Terra"
}

...



That should allow to translate the text associated to every object contained in Celestia.

As best option I suggest that the GUID is generated through an appropriate Editor.

[Telepath]

Paolo,

This looks like an interesting idea, and I think I see what you're trying to do, however I believe the GUID's should be treated as technical keys internal to Celestia (ie. not visible to user).

For example, to return the name of the object and the name of it's parent in a relational query would require something like:

Code: Select all

select x.objectname, y.objectname
from object x, object y
where x.PARENTGUID = y.GUID


Now as users we shouldn't actually need to know the values of the GUID's (as they are generated by the RDBMS) and only used for relational joins as above.

Another <candidate key> perhaps would be best for users/addon writers to use.

Does this make sense to you?

I haven't seen the Celestia codebase, however your suggestion implies to me that Celestia in future would have to have an inbuilt (or be distributed with a) RDBMS such as mySQL?

Is this what you anticipate?

[Telepath]

Paolo,

Your last post (while I was typing) probably answered at least one of my questions.

[Paolo]

My imagination is not that good as well, so could you please try to explain some examples of possible uses of these GUIDs?
The advantages should be quite strong because I think having to use GUIDs (globally unique IDs) would most probably complicate addon creation.

steffens

Hello Steffens
Please don't feel angry! I'm sure that your immagination is quite good.
If you are not a programmer these things should sound a bit tricky and difficult to imagine. But I think that for the Celestia developers these things are quite clear.

I hope that this feature will improve very much add on creation because add-on management and In-Place add-on editor will be more easy to implement.

[Telepath]

One of the things that I find a little frustating is that in SSC's files, parent relationships have to be defined as follows:

Code: Select all

"Earth" "Sol"
{
.
}

"Moon" "Sol/Earth"
{
..
}

"Apollo 11" "Sol/Earth/Moon"
{
...
}

If your suggestion would allow this to be:

Code: Select all

"Earth" "Sol"
{
.
}

"Moon" "Earth"
{
..
}

"Apollo 11" "Moon"
{
...
}

then it would be a good improvement.

[Paolo]

Paolo,

This looks like an interesting idea, and I think I see what you're trying to do, however I believe the GUID's should be treated as technical keys internal to Celestia (ie. not visible to user).

For example, to return the name of the object and the name of it's parent in a relational query would require something like:

select x.objectname, y.objectname
from object x, object y
where x.PARENTGUID = y.GUID

Now as users we shouldn't actually need to know the values of the GUID's (as they are generated by the RDBMS) and only used for relational joins as above.

Another <candidate key> perhaps would be best for users/script writers to see.

Does this make sense to you?

I haven't seen the Celestia codebase, however your suggestion implies to me that Celestia in future would have to have an inbuilt (or be distributed with a) RDBMS such as mySQL?

Is this what you anticipate?

Hello Telepath

Of course some kind of SQL interface should be implemented as well.

The GUIDs should be generated of course internally by Celestia, and the whole thing should be completely transparent to the add-on creators.
But this would inhibit the development of add-on management.

This point is quite further advanced. Imagine something like this.

Code: Select all


AddOn
{
    GUID [AAAABBBB]  #### Watch this
    ParentGUID [??]

    Author "Telepath"
    CreationDate [2006 06 01 21 12 00]
    Version 1
    Name "The beautifulest Celestia Add On"
    Type 2  # Fictional #### Watch this
}

File
{
   GUID [??]
   ParentGUID[AAAABBBB]  ##### Watch this  The add-on declaration
   Name "telepath01.ucf"
}

Planet
{
   GUID [??]
   ParentGUID [??]   #star
   AddOnGUID [AAAABBBB]  #### Watch this  The add-on declaration
   ...
}



It should be possible to identify where every object contained in Celestia comes from, who is the author and if it is fictional or not!

So this should be the base both for add-on management and the extensive Scientifical and Professional use of Celestia.

The use of an RDBMS as add'on editor should be interesting. Implementing it in M$ Access for instance and creating an export utility should be not very difficult.

[Paolo]

One of the things that I find a little frustating is that in SSC's files, parent relationships have to be defined as follows:

Code: Select all

...


If your suggestion would allow this to be:

Code: Select all

...


then it would be a good improvement.

It would be something like.

Code: Select all

Planet
{
    ...
    ParentGUID [??]  # Sol
    Name "Earth"
    ...
}

Satellite
{
    ...
    ParentGUID [??]  # Earth
    Name "Moon"
    ...
}

Ship
{
    ...
    ParentGUID [??]  # Moon
    Name "Apollo11"
    ...
}


[Paolo]

I haven't seen the Celestia codebase, however your suggestion implies to me that Celestia in future would have to have an inbuilt (or be distributed with a) RDBMS such as mySQL?

Is this what you anticipate?

No in the past MostlyHarmless project has faced this issue.
The problem is that MySQL or any other external RDBMS solution performs quite bad associated with Celestia, or better Celestia performance decreases too much.

In my proposal Celestia should be a 3D user interface framework to a cosmological/astronomical/astronautical database that can simulate the space/time (without gravitational effects) with the greatest accuracy and visual effects exactly like now but with a better data format.

[ajtribick]

The add-on idea is quite a good one (I think), but how would you handle an add-on that puts fictional planets around a real star? In your system this would require two parents for the object: one to refer to the star, another to refer to the add-on. Similarly if an add-on moves a star around (e.g. converts it into a barycentre system, etc)

[Telepath]

It would be something like.

Code: Select all

Planet
{
    ...
    ParentGUID [??]  # Sol
    Name "Earth"

    ...

Ship
{
    ...
    ParentGUID [??]  # Moon
    Name "Apollo11"
    ...
}


Yes, I understood that from your original suggestion, however I don't agree that GUID's should be written into an SSC file like this.
One of the advantages [EDIT: or perhaps disadvantages] of using GUID's of course is that Addon's and Objects, files, etc from different creators could have identical names without risk of conflicts.

I think if you want to go down this route, then you need to go the whole way and replace configuration files altogether with a RDBMS based data-structure with a GUI (and perhaps for advanced designers, a programming and.or scripting interface), and THAT IS NOT A SMALL CHANGE.
(ie. Addon's would effectively be designed using an IDE which is knowledgeable of the meta-data)

The various attributes defined for objects in the current SSC's would become real entities and attributes in the meta-data.

eg.

Code: Select all

ENTITY: CelestiaObject
{
     GUID,
     parentGUID,
     Name,
     ClassGUID,
     ElipticalOrbitGUID,
     Obliquity,
     EquatorAscendingNode,
     PrecessionRate,
     Radius,
     Albedo,
     RotationPeriod,
     ...
     etc
}

ENTITY: EllipticalOrbit
{
    GUID,
    parentGUID,
    Period,
    SemiMajorAxis,
    Eccentricity,
    MeanLongitude,
    ...
   etc
}


Of course I haven't looked at the code, but I'm sure that Celestia will already have something along the lines of these data-structures internally, and it seems that what you're suggesting is the ability to define them offline (in a database, rather than files?) and the addition of GUID's to the existing internal data-structures. So Celestia on startup, would load it's internal structures from a database repository (maintained by a GUI) instead of from files.

To install someone elses addon would require populating database tables instead of copying files.

Is this what you had in mind, or is this much more?

Perhaps you just want to add GUID's to Celestia's intenal structures?

[Telepath]

The problem is that MySQL or any other external RDBMS solution performs quite bad associated with Celestia, or better Celestia performance decreases too much.

I didn't imagine you meant an 'online' database. Merely a repository of addons (as just described in my last post). -- to be loaded at startup, and executed in memory as at present.

[Paolo]

The add-on idea is quite a good one (I think), but how would you handle an add-on that puts fictional planets around a real star? In your system this would require two parents for the object: one to refer to the star, another to refer to the add-on. Similarly if an add-on moves a star around (e.g. converts it into a barycentre system, etc)

My solution will avoid the problem at all. The add-on that will place the fictional planets around the real star will use the GUID of the real star.

Of course the whole question relies upon this: you have to know the GUID of the star in particular and the GUIDs of all the other objects in general.

This is why Telepath is suggesting the usage of some kind of tool that should allow to manage this easily. Doing the things by hand right now will be more difficult.

Moreover if the GUIDs are created and associated carefully will be avoided (or better will be easier to solve) the problems associated to multiple definitions of the same object.
Celestia at load time should log or prompt the user for object redefinitiion and so allow the overwriting or not.

By now in this discussion I can see at least one drawback. The size of the stars definition file (now stars.dat) will be quite a lot larger.

[Telepath]

Celestia at load time should log or prompt the user for object redefinitiion and so allow the overwriting or not.

Surely this shouldn't be nescessary if you use GUIDs. The whole point of using GUID's is that they are almost certainly guaranteed to be Globally Unique. (At least if generated by a decent algorithm)

Sorry, I'm going to have to go now, but will try to catch up with the thread later.