Unified Data Format for new object model

[Telepath]

There is a danger in discussing all these different issues, that we lose focus on what Paolo's original suggestion implies.
IMO what Paolo's original post is proposing, amounts to a rationalisation of Celestia's internal data-structures.

In my vision each Celestia object (each) must have two additional properties. The GUID (Global unique identifier) and the Parent ID (The GUID of the parent). Associating these two properties with the Type of the object will be possible to manage the Celestia's object model as a Relational database.

I agree with Paolo, that all the other things, XML vs attribute vs LUA, GUI interface, etc -- none of these are nescessarily excluded if Paolo's suggestion is implemented.
So the question of which external syntax is best, or which should be done, can be deferred. (In the long run, any or all could be used depending on individual preferences as they are only different syntaxes, but all must representative of the internal structures.


The key question Paolo has posed is "what are the pro's and con's of introducing GUID's into the internal data-structures", regardless of how, or even IF, they (GUIDs) are represented in SSC's,XML, etc.

AM I wrong??

I suggest the other issues can sort themselves out in due course...

[Paolo]

The key question Paolo has posed is "what are the pro's and con's of introducing GUID's into the internal data-structures", regardless of how, or even IF, they (GUIDs) are represented in SSC's,XML, etc.

AM I wrong??

I suggest the other issues can sort themselves out in due course...

Hello Telepath

You are perfectly RIGHT!

Sometimes I think that my English is too poor because I feel that I'm not understood.

There tends to be a fair bit of inertia when someone proposes something new in anything... it's easy for people to say "what we currently have works well enough", it's up to the visionary to present his ideas in such a way as to overcome that inertia .

This evening after work I'll try to do my best in order to start listing some of the features that should be more easy to implement in Celestia adding this GUID approach.

But this is not a monologue, so I would prefer that someone else should envisage the advantages or the disadvanteges.

Kind regards

[hank]

From these words I can imagine that the mainstream development of Celestia is moving towards the abandon of the GUI part in order to focus almost exclusively on the CelestiaCore and LUA.

CelestiaCore object will expose its capabilities through LUA, so Celestia "Mods" will be more easy to implement.

IMHO I think that this is not what the Celestia community is looking for, but on the other side it is not in conflict with the argument of this thread.

Paolo,

The suggestion regarding Lua is my own and does not indicate any movement by the mainstream development team. And the idea is not to abandon the GUI, but to allow it to be more easily be modified and enhanced. This is intended not only to allow custom application development, and discourage forking of the Celestia codebase, but also to accelerate mainstream enhancement of Celestia by allowing prototyping and implementation of new features using Lua.

Different parts of the Celestia community are looking for different things. If Lua allows users to more easily add the features they want, and share them with other users, without requiring the involvement of the core developer team, everyone benefits. And I mentioned it as a possible mechanism for implementing some of the proposals being discussed on this thread.

So I can guess that you are sugesting that in order to get flexibility the new unified data format (XML or different) will be handled directly through LUA.

IMHO first of all LUA will perform worse than Celestia itself in loading data from files, on the other side its difficult to me to imagine an application that runs scripts to load its data. It should make sense if the application is a LUA script itself.

Beside this my proposal implies the brake of the backward compatibility of the data formats. So none flexibility is required at all, since at athe end LUA scripts should have to load data calling core functions, why not use C++ directly like now?

What I'm suggesting is that using Lua for data loading would provide great flexibility. Lua could be used to implement loading from XML, MySQL, custom format data files, internet sites, etc. etc. It could also be used to control which files (data sources) are loaded and in what order, e.g. for an addon management scheme. Data loading from Lua could also allow data to be loaded from user selected scripts during a session, as well as at startup.

The performance tradeoff is an issue that has to be evaluated. But even in cases where performance is not completely satisfactory, it could well be sufficient for prototyping purposes.

Lua is much easier to learn and use than C++, and does not require direct integration into the compiled codebase. The hope would be that Lua would facilitate the contribution (not just discussion) of enhancements to Celestia by a larger part of the community. And since features implemented with Lua could be mixed and matched by users, they would not require agreement by everyone to be used by those who want them.

But these are marginal implementative considerations. I would like to point out the discussion back again on the database aspect and on the advantages of this approach.

Kind regards

Sorry Paolo, I did not intend to hijack your thread.

- Hank

[t00fri]

I think Hank is advocating and following up a most important expansion concept for Celestia.

Notably would it allow (like PERL does, too) users to experiment directly (without compilation) with sophisticated extensions... The example of "Real Time Celestia" from a thread by 'tec' will be an important benchmark case for this philosophy.

Bye Fridger

[Paolo]

Sorry Paolo, I did not intend to hijack your thread.

- Hank

Hank
Don't worry, I did not felt "hijacked" at all.
It is a discussion. I was only trying to keep the discussion focused.

Kind regards

[Telepath]

Hello,

As there seems to have been a little confusion about just what GUID's are. This link: http://en.wikipedia.org/wiki/Universall ... Identifier gives a reasonable explanation about what they are.

Hope this helps.

[ajtribick]

My point is, why make an add-on creator have to deal with the GUIDs at all?

While it is all very well to say that you should make sure you generate GUIDs in the proper way, consider it from the perspective of someone who doesn't really know much programming but wants to create an add-on. Bear in mind that such users exist and we shouldn't exclude them from being able to create add-ons.

Personally I think the current system of pathnames works quite well for the most part, and typing out "Sol/Saturn/Mimas" is a lot less verbose than, say,

Code: Select all

GUID=[392A FCE0 0348 2477]
ParentGUID=[0184 37FC 371A 9CCB]
Name="Mimas"

which hides the relationship away in a meaningless (to humans) string of digits. I can see this trend reaching the point where we need to make custom editors to handle all this coder stuff.

Basically, the object catalogues should be human-readable and human-writeable. As they stand, it is fairly easy to take a look at a .ssc and see what body the object is orbiting and what it is called, without having to cross-reference arbitrary hex strings!

In addition, if, say, during development of the add-on, the author decides to move an object from being a planet to being a satellite (say), in the current system this requires just changing the pathname in an obvious and intuitive manner. Under the explicit-GUID system this will probably require scrolling through the code to find the unmemorable GUID to copy and paste into position.

I agree that there are some limitations of the current system (e.g. objects which have a / in their names), but putting GUIDs in the data files is, in my opinion, the wrong way to go.

I would like to find a compromise. Instead than generate the GUIDs with an algorithm I would like that these GUIDs will be provided by some sort of Web repository. So an Add-On creator shoud ask for instance for 10 Thousands of GUIDs that will be reserved for him. So he will know (and every one else) that the GUIDs form 25629000000 to 25629100000 are already in use.

This makes the add-on creation process rather intimidating don't you think? I personally have started developing several add-ons and later abandoned them - since I know that I rarely get to the completion stage I would have less incentive to register for GUIDs.

And who would run this web repository? From my experience with the Motherlode, updates can take several weeks. If a new add-on author who isn't entirely sure if they'll get their project to completion (bear in mind that with explicit GUIDs in the data-files, there would be no way of starting to create the system without using some numbers for the GUIDs) emails the GUID repository for GUIDs and doesn't get a response back for weeks, what will happen? I think I know the answer - they would get put off creating an add-on.

What happens when someone doesn't know about this GUID repository (I suspect quite a few people learn .ssc mainly by looking through the example files) and posts an add-on on these forums? Since they haven't used the repository, they may be accidentally infringing on some author's GUID range. Then they have all this confusing and intimidating material posted at them through the forums (I'm hesitant to use the word "flaming" here), and they get put off.

Do we really want to head in this direction?

Just my two pennies.

[Paolo]

Well well well
the discussion is going to move in the right direction. Thank you chaos syndrome, hank, Telepath and you all for the contribution!

PS (I hate nicknamens)

[Paolo]

My point is, why make an add-on creator have to deal with the GUIDs at all?

Because GUIDs will add to Celestia more power, more capabilities, more robustness, more trustability and at the and the control over the source of the data, that from a scientifical/professional point of view is the most important thing af all.

It should boost Celestia over the amateur usage. Imagine what kind of improvements should have the project if a team of professionals will be involved in helping the current dev team. The advantages will be for everyone of us that are enjoining this wonderful software.

EG. if at NASA will start to use Celestia I think that they would like very much to be sure where the data comes from. Not if the data comes from inside NASA or not. If they will use their custom versions of SSC, DSC, XYZ, DAT files to replace the standard ones and won't play with Motherlode, there would be no reason to doubt about the correctness of the data.

Imagine instead that at NASA there will be several copies of Celestia running on several Workstations. All the users of those Workstations will work to prepare several data files in several versions. I hope that they will be wyse enough to use some sort of version control system e.g. using a CVS shared repository.

Beside this I can imagine that in such professional environment they willl work with huge datasets to simulate missions and so. In this scenario they will work with different datasets on different workstations in different times.
Some datasets at least will be Work In Progress and will need some sort of approval.
They won't work everytime and everywhere with the entire collection of approved datasets.

In this scenario traceability of data becomes very important. And here the GUIDs can play a foundamental role.

But this is a very advanced and visionary example. Perhaps Rocketman should say his opinion.

[Paolo]

While it is all very well to say that you should make sure you generate GUIDs in the proper way, consider it from the perspective of someone who doesn't really know much programming but wants to create an add-on. Bear in mind that such users exist and we shouldn't exclude them from being able to create add-ons.

Personally I think the current system of pathnames works quite well for the most part, and typing out "Sol/Saturn/Mimas" is a lot less verbose than, say,

Code: Select all

GUID=[392A FCE0 0348 2477]
ParentGUID=[0184 37FC 371A 9CCB]
Name="Mimas"

which hides the relationship away in a meaningless (to humans) string of digits. I can see this trend reaching the point where we need to make custom editors to handle all this coder stuff.

Basically, the object catalogues should be human-readable and human-writeable. As they stand, it is fairly easy to take a look at a .ssc and see what body the object is orbiting and what it is called, without having to cross-reference arbitrary hex strings!

Ok right point.

We should say that add-on creation will be more difficult. But right now does not require programming capabilities.
I dont' want to exclude anyone. Instead an add-on editor or some sort of development kit for add-on will be necessary and perhaps it will be manadatory to develop this editor to hide the complexity of GUID usage.
Using this editor everyone even childs will be able to manage add-on cretion even if the data will be stored in binary format.

Be aware we are not talking about things that will happen within few months. I've suggested this for a major Celestia release (eg. version 2.0). At current development speed I can bet on that it would take years before we will see something real if this discussion will end up with a positive decision.

We are just discussing. You are right. Hand writing of add-ons will be more difficutl or even impossible.

[t00fri]

...
Using this editor everyone even childs will be able to manage add-on cretion even if the data will be stored in binary format.
...
.

Paulo,

how about backward compatibility of such a new GUID scheme with the hundreds of existing add-ons? Do you plan to throw these away? Many former add-on creators are not active anymore and hence hard to reach now.

Bye Fridger

[Paolo]

I agree that there are some limitations of the current system (e.g. objects which have a / in their names), but putting GUIDs in the data files is, in my opinion, the wrong way to go.

I would like to find a compromise. Instead than generate the GUIDs with an algorithm I would like that these GUIDs will be provided by some sort of Web repository. So an Add-On creator shoud ask for instance for 10 Thousands of GUIDs that will be reserved for him. So he will know (and every one else) that the GUIDs form 25629000000 to 25629100000 are already in use.

This makes the add-on creation process rather intimidating don't you think? I personally have started developing several add-ons and later abandoned them - since I know that I rarely get to the completion stage I would have less incentive to register for GUIDs.

And who would run this web repository? From my experience with the Motherlode, updates can take several weeks. If a new add-on author who isn't entirely sure if they'll get their project to completion (bear in mind that with explicit GUIDs in the data-files, there would be no way of starting to create the system without using some numbers for the GUIDs) emails the GUID repository for GUIDs and doesn't get a response back for weeks, what will happen? I think I know the answer - they would get put off creating an add-on.

What happens when someone doesn't know about this GUID repository (I suspect quite a few people learn .ssc mainly by looking through the example files) and posts an add-on on these forums? Since they haven't used the repository, they may be accidentally infringing on some author's GUID range. Then they have all this confusing and intimidating material posted at them through the forums (I'm hesitant to use the word "flaming" here), and they get put off.

Do we really want to head in this direction?

Just my two pennies.

In order to explain better my suggestion about GUID assignment, the idea is the following. An Apache website with MySQL, PHP. A MySQL database with a few tables inside. A php web interface where a user has to register as add-on developer (This is already present in motherlode).

In another page a user should ask for GUID ranges. In order to avoid that someone will waste large sets of GUIDs, a limited amount of those will be available for single operation (eg. 1000, 10000).

Php forms will store the data in the MySQL database and generate a range of GUIDs. If the GUID will contain the ID code of the user it will be more difficult to mix GUID provenience.

Of course some kind of GUID hacking will be possible if someone will excerpt the user code and the ranges from GUIDs contained in the deliverd add-ons. But somewhere we have to stop. If not even encryption of add-on won't be effective since Celestia is open source.

If this operation is performed through an Add-on manger integrated in the add-on editor discussed before, that will connect automatically to the MySQL database, the operation should be completely transparent. Moreover the MySQL should be the database of all the add-on all-over the world.

BTW the editor tool should be useful in order to allow the association of GUIDs to large datasets, such galaxies catalogs, asteroids catalogs or stars databases.

A readme file should explain about the add-on creation procedure.

I don't think that add-on creators will put off if they'll find a few of instructions and suggestions.

Kind regards

[Paolo]

...
Using this editor everyone even childs will be able to manage add-on cretion even if the data will be stored in binary format.
...
.

Paulo,

how about backward compatibility of such a new GUID scheme with the hundreds of existing add-ons? Do you plan to throw these away? Many former add-on creators are not active anymore and hence hard to reach now.

Bye Fridger

This should be a real problem.

The only solution that I can see is that the add-on editor should contain some sort of migration tool, that in my first suggestion (a slight modification of the current hashed formats) or even using XML or Lua should be almost straightforward.
And that some volunteer (accordingly with copyrights issues if present) should make the conversion, taking the ownership of the data and perhaps adding to the files a declaration that the original creator was someone else. This should be the minimum.
Perhaps some kind of license about Celestia data usage should be advisable. It was discussed before, but I don't want to make a drift of this discussion.

[t00fri]

...
Using this editor everyone even childs will be able to manage add-on cretion even if the data will be stored in binary format.
...
.

Paulo,

how about backward compatibility of such a new GUID scheme with the hundreds of existing add-ons? Do you plan to throw these away? Many former add-on creators are not active anymore and hence hard to reach now.

Bye Fridger

This should be a real problem.

The only solution that I can see is that the add-on editor should contain some sort of migration tool, that in my first suggestion (a slight modification of the current hashed formats) or even using XML or Lua should be almost straightforward.
And that some volunteer (accordingly with copyrights issues if present) should make the conversion.

But you and I also KNOW this is never going to happen....

Bye Fridger

[Paolo]

...

But you and I also KNOW this is never going to happen....

Bye Fridger

Why not?
If an add-on is really important and very popular I think that it will be possible to find a volunteer for the conversion.

I can see copyright issues only about textures and perhaps models. If so the add-on will be re-prepared by someone else if possible.

I don't think that adding a GUID to the SSC declaration of a model and its textures without permission would configure a copyright brakedown even if the copyright is on the whole content of the Zip file, and even if this upgrade is not declared anywhere. But I'm not a lawyer, so I won't bet on it.

[t00fri]

...

But you and I also KNOW this is never going to happen....

Bye Fridger

Why not?
If an add-on is really important and very popular I think that it will be possible to find a volunteer for the conversion.

You may not know this, but I have been spending LOTS of time with people using totally outdated galaxy add-ons that their creators simply did not even care to adapt to the present core code requirements.

Since many Motherlode users download before reading what they are going to get, even now we got a problem with outdated add-ons that nobody cares to update!

For example: Older galaxy/nebula add-ons don't have a magnitude entry and thus will appear way too bright with the new code. It's often written in the readme that few care to read.

READING hurts!

Bye Fridger

[Paolo]

Fridger

Thank you for this contribution.

What you've pointed out is one of the reasons why I don't use Motherlode even if it now represents one of the main added values of the Celestia project.

The initial idea of a centralized repository was mine indeed (if you search deeply in these forums you'll find it). But in origin I've foreseen some kind of approval procedure for the data and some kind of direct patroling from the official development team in order to avoid problems like the ones pointed by you.

Imagine what should happen if Vincent/Tec semi-forks will produce add-ons with different data formats and if those will be published on Motherlode! BTW Motherlode is cited almost everywhere as the place where to find data for Celestia!

As I've already said before add-on manager IS REQUIRED now, not to allow newbies to manage the upload of correct file names (other problem) but to ensure correctness and trust in Celestia data.

The GUID approach is the first thing that should allow to do this. It will allow to create a real garrisoned and shared add-on web repository (see the previous post about the GUID generation through PHP-MySQL), where an add-on manager (even external to Celestia and even written in Java/Lua/Visual basic) will be able to get the Celestia data with confidence.

If an add-on will be not up-to date according with the current Celestia features, the dev-team (that is patroling the repository) will mark it as uncompatible starting from a defined Celestia version. So the users all around the world wll run their add-on manger that will tell them. [Hey man! the galaxies add-on (version XX prepared by XX published in date XX under the XX usage license), that yu've installed in date XX is obsolete for the current Celestia version that you are running. Unfortunately there isn't yet an upadate version compatible. Do you want to DELETE this add-on from your HD? Yes/No"]

PS1.
For all the ones that since now has not yet envisioned the possibilities of a new data model based upon GUIDs and a new unified data format: are you starting to see some of the advantages?

PS2.
For everyone. Please if you are seeing other problems contrbute to the discussion! If you'll point out something that will demonstrate the my idea is fallacious/is misleading will save our time!

Kind regards

[ElChristou]

...
Using this editor everyone even childs will be able to manage add-on cretion even if the data will be stored in binary format.
...
.

Paulo,

how about backward compatibility of such a new GUID scheme with the hundreds of existing add-ons? Do you plan to throw these away? Many former add-on creators are not active anymore and hence hard to reach now.

Bye Fridger

This is a real problem and because of this any eventual change in the Celestia structure is impossible because it will break all existing addons...

Shall we take this in account? I would say no because this mean a real impediment for futur dev...

Many addons only work after 1.3.2, so I suppose we can deal again with this situation in the futur.

If people are still active they can upgrade their addons, if not any user who really like something will be free to upgrade it (most addons are not copyrighted).

[Paolo]

...
Using this editor everyone even childs will be able to manage add-on cretion even if the data will be stored in binary format.
...
.

Paulo,

how about backward compatibility of such a new GUID scheme with the hundreds of existing add-ons? Do you plan to throw these away? Many former add-on creators are not active anymore and hence hard to reach now.

Bye Fridger

This is a real problem and because of this any eventual change in the Celestia structure is impossible because it will break all existing addons...

I've considered the break of the backward compatiblity from the beginning. In the Lua data format thread hank is showing how a compatibility should be kept.

In any case my suggestion is: in order to keep this unuseful compatilbiity the core code should be more larger, complex and difficult to maintain. So it is better to make a break.

In my vision if the LUA approach won't be accepted, and will be chosed a pure data unified format, there should be only one parser in core code that will perform the objects loading.

The parser will pass the loaded data (now in hashed format) to a factory object that will create the objects dynamically and that will store those objects in some sort of global STL::map or Octree, where the keys will be the association of datatypes codes and GUIDs. Will be the factory object that will ensure that the GUIDs are really unique and that double definitions of objects are not present in the provided datasets.

This should allow to extend the octree culling to all the visual objects (like the asteroids, see the problem of the huge asteroids SSC in another thread.

Moreover it will be possible to consider this map/octree as the internal relational database of Celestia and execute on it very fast queries through the STL algorithms.

I don't think that there will be performance loss. Instead it is possible that there should be some kind of overall performance improvement in Celestia.

Here it is required the contribution of the developers for further comments.

Kind regards

[tech2000]

Hi, for what I understand it really opens up alot of cool possibilities (refering to Paolos advanced and visionary example.)

Bye, Anders