Unified Data Format for new object model

[ajtribick]

My solution will avoid the problem at all. The add-on that will place the fictional planets around the real star will use the GUID of the real star.

So the add-on definition would refer to the star? That still leaves the problem of an add-on which alters multiple objects in multiple star systems (such as my planet textures add-on, before it was incorporated into the distribution).

[Paolo]

I think if you want to go down this route, then you need to go the whole way and replace configuration files altogether with a RDBMS based data-structure with a GUI (and perhaps for advanced designers, a programming and.or scripting interface), and THAT IS NOT A SMALL CHANGE.
(ie. Addon's would effectively be designed using an IDE which is knowledgeable of the meta-data)

This is not a small change this is a very big one. It represents a correction of the original basic design. It's a looooooot of work in terms of coding activity. So this is why in my first post I've suggested this for a major release (2.0 or 3.0 or so).

To install someone elses addon would require populating database tables instead of copying files.

Is this what you had in mind, or is this much more?

Perhaps you just want to add GUID's to Celestia's intenal structures?

We are discussing. I hope that the developers will join in the discussion and will say something. Even that these things are impossible.

In my vision I wpuld like to add GUIDs to the internal structures. The final users will continue to download add-ons as files.

Perhaps add-on developers will need at least some kind of developmenbt Toolkit for the GUID management, or better they will need for an editor that should connect to an internet repository that should store and generate the GUIDs globally perhaps using php and MySQL.

[Paolo]

My solution will avoid the problem at all. The add-on that will place the fictional planets around the real star will use the GUID of the real star.

So the add-on definition would refer to the star? That still leaves the problem of an add-on which alters multiple objects in multiple star systems (such as my planet textures add-on, before it was incorporated into the distribution).

The textures point is one thing that we haven't touched yet.

This is a real problem that was discussed a long in the past.

Imagine to write something like this

Code: Select all


Texture
{
    GUID [??]
    ParentGUID [???]                  ## The planet, the body, whatever else
    ParentType "Planet"              ## This should help but is not strictly necessary
    File "myAddOn/myNewTexture.jpg"  ## The name should include the path
    Type 2                                 ## Base texture, clouds, specular, heigh map
    Alternate 1                           ## The alternative index
    ...
}



If Celestia at load time finds that a Texture of this type with this alternate level was already defined for this ParentGUID (because for instance already present in the standard package) should prompt the user for replacement.

Of course in this scenario the Celestia GUI should have some kind of flag that allows to accept all the replacements.

[Paolo]

Celestia at load time should log or prompt the user for object redefinitiion and so allow the overwriting or not.

Surely this shouldn't be nescessary if you use GUIDs. The whole point of using GUID's is that they are almost certainly guaranteed to be Globally Unique. (At least if generated by a decent algorithm)

Sorry, I'm going to have to go now, but will try to catch up with the thread later.

I agree in general but this means that the GUIDs must be UUIDs.

In order to have UUIDs generated by algorithms we should fall in the M$ strategy where the ID are quite long and complex.

I would like to find a compromise. Instead than generate the GUIDs with an algorithm I would like that these GUIDs will be provided by some sort of Web repository. So an Add-On creator shoud ask for instance for 10 Thousands of GUIDs that will be reserved for him. So he will know (and every one else) that the GUIDs form 25629000000 to 25629100000 are already in use.

[Paolo]

I have to go now too.
I'll try to catch the discussion later.

Kind regards.

[t00fri]

Paolo,

I think XML is the better scheme. Really many modern programs are using XML, really. I wouldn't want to "pollute " *.ssc DATA files with GUID information which is an administrative type of label.

Probably Christophe is the local expert in this sort of stuff.

Cheers,
Fridger

[ajtribick]

I think XML is the better scheme. Really many modern programs are using XML, really. I wouldn't want to "pollute " *.ssc DATA files with GUID information which is an administrative type of label.

I agree with you here, XML also has the advantage of being much easier to write utilities that operate on the data files: back when I was maintaining the extrasolar planets textures add-on I had to code a parser for .ssc files myself - it would have been a lot easier if the files had been in XML format*.

Also, while XML is quite verbose, it is full of repeating character sequences (i.e. the tags) which compress very well, so this shouldn't affect the distribution size too much.

EDIT:

* this is because I use a programming language which comes with a built-in XML parser. Probably XML isn't very nice if you have to build an XML parser from scratch.

[hank]

I'm not sure it's a good idea to distract developers with requests for comment on design ideas for advanced feature requests. It's fine for non-developers to discuss ideas here, of course, but work on more near-term bug fixes and feature enhancements are probably a better use of our very limited developer resources at this point. Also, it might be helpful to have high-level requirements, implementation scoping and priorities agreed on before getting into design alternatives.

That said, I'd like to briefly mention the possibilty of using Lua for Celestia data definition files. I think something very close to the existing syntax would be possible with Lua, and because Lua is a user extensible programming language it would be possible to experiment with many potential enhancements via Lua programming.

- Hank

[Paolo]

Paolo,

I think XML is the better scheme. Really many modern programs are using XML, really. I wouldn't want to "pollute " *.ssc DATA files with GUID information which is an administrative type of label.

...
Fridger

Hello Fridger

I don't like XML very much but, OK. It does not matter. The data format is only one aspect. If in the future the Celestia data format will move on XML of course it will be fine for me. It has many advantages like described by Chaos Syndrome.

Beside this, is the concept behind that is more important: the data model. The data format is a consequence.

By now I can't see any alternative to the GUID file "pollution" if you want to get easily all the improvements that I've suggested. But perhaps I'm wrong.

Kind regards.

[Paolo]

I'm not sure it's a good idea to distract developers with requests for comment on design ideas for advanced feature requests. It's fine for non-developers to discuss ideas here, of course, but work on more near-term bug fixes and feature enhancements are probably a better use of our very limited developer resources at this point. Also, it might be helpful to have high-level requirements, implementation scoping and priorities agreed on before getting into design alternatives.

...

- Hank

Hello Hank

Sincerely I don't understand your position.
I appreciate your comment. I thank you for spending a bit of time in reading this thread. But its a long time that I can't see anywhere any discussion about "agreements on high-level requirements, implementation scoping and priorities". Perhaps you are exchanging Private E-Mail each other. I hope so.

I think that the last public discussion was started on the developer mailing list 9th July 2004 by Chris. The title was Celestia 2.0 and was quite interesting.

I think that discussions about design alternatives can keep the development alive. If no-one discusses about advanced features requests who will put them in the list of the development priorities?
Everyone knows that bug-fixing is not much interesting. Instead knowing that after bug fixing a new exciting and challenging feature will be implemented IMHO should help and motivate the entire community.

Kind regards

[Malenfant]

Surely developers who are interested in this sort of thing can comment on them here, whereas those who are interested in the bug fixes and other things they consider more important can just ignore it and continue doing that? Just because a thread is here doesn't mean that anyone is being forced to comment on it, right?

I certainly don't see any reason why discussion should be discouraged just because it's not something that some developers think has high priority. If it's the case that a developer's time is so precious that he shouldn't be 'distracted' by these threads then by the same token he probably shouldn't be spending any time reading the forums at all! But I don't think that talking about possibilities can do any harm .

[Paolo]

That said, I'd like to briefly mention the possibilty of using Lua for Celestia data definition files. I think something very close to the existing syntax would be possible with Lua, and because Lua is a user extensible programming language it would be possible to experiment with many potential enhancements via Lua programming.

- Hank

Hank

Please be aware that I don't want to be sarcastic. But I want to express my doubts.

I've noticed that in the last months some efforts in the (very scarse) development activity was aimed to improve the interface between LUA and Celestia. Now even with an event handler!
I agree that this adds a lot of power and flexibility to Celestia.

But the questions are: is it really useful? how many users should be able to write LUA scripts for Celestia? Why an average or intermediate or advanced Celestia user (not a newbie) should write or use LUA scripts? Why in the current distribution there is a so small collection of scripts almost unuseful?

E.G. WinCVS uses Pyhton instead than LUA. A series of useful automation scripts are included in the program and are integrated directly in the GUI. Having an advanced scripting language makes sense. But for Celestia? Scripting is used almost only for creation of automated demos in teaching activities. Was LUA necessary for this? Ok it allows a better approach using an extensible OOP language with integrated garbage collector instead than a collection of cel commands, and so and so.

Beside this if using LUA will be possible to manage some kind of XML data format in order to implement a data model that should allow all the things that I've pointed out in this thread, of course it is welcome!

[Paolo]

Hey guys COME ON!

Have you switched the brain off?

Am I speeking Greek?

Is it possible that you are not able to see the enormous potential of what I'm pointing out?

Of course if there are drawbacks that I'm missing please tell me!

If you like we can use an approach a bit mode scientifical.
I'm trying to formalize an idea, a theory that should sound:

A data model similar to a Relationa Database with a unified data format should allow to get very important improvements to Celestia. The easiest way to acheive this is through the suggestion that is in the first post of this thread.

Please let discuss about this! Post your opinions and your comments! Try to dismantle and refute this theory. On my side I'll try to prove that is correct! So perhaps in a future the development team will consider it for a real implementation.

[hank]

Hello Hank

Sincerely I don't understand your position.
I appreciate your comment. I thank you for spending a bit of time in reading this thread. But its a long time that I can't see anywhere any discussion about "agreements on high-level requirements, implementation scoping and priorities". Perhaps you are exchanging Private E-Mail each other. I hope so.

I think that the last public discussion was started on the developer mailing list 9th July 2004 by Chris. The title was Celestia 2.0 and was quite interesting.

I think that discussions about design alternatives can keep the development alive. If no-one discusses about advanced features requests who will put them in the list of the development priorities?
Everyone knows that bug-fixing is not much interesting. Instead knowing that after bug fixing a new exciting and challenging feature will be implemented IMHO should help and motivate the entire community.

Kind regards

Hi Paolo,

I haven't been active as a developer for some time. I don't presume to speak for the active developers. But it seems clear from the mailing list that the developers are working on bug fixes and new features to the extent that their time permits. There is no shortage of ideas for new features. Rather, there is a shortage of developers to design and implement them.

In some instances, people not on the developer team have contributed new features. But even when they have provided working code, it has proven difficult to get their contributions integrated into the mainline codebase. This is due partly to the limited developer time available for integration, as well as lack of prior design coordination, also a result of limited developer time.

The developers generally work on what interests them, with review and approval by Chris. There is no overall list of development priorities, although some developers may have lists of what they plan to work on. Chris, of course, knows what he plans to include in the immediate uncoming releases. He sometimes describes what his plans are and asks for feedback when he wants it. But there isn't really any long term development planning that I know of, probably because the developers already have their hands full.

It doesn't seem very likely to me that the developers will have time for design discussions of possible long-term future enhancements as long as there are more immediate development tasks at hand. And to the extent that it might distract them from their current work, that would just further delay longer-term development.

- Hank

[Malenfant]

I think the key thing would be to demonstrate how much more useful this could be than the existing XML Format. So far you've shown us how you think it could work, but apparently it's not been that persuasive... so if you can show us some specific examples /case studies in which your database/GUID proposal would be demonstrably better than XML then you'll probably have more luck...

There tends to be a fair bit of inertia when someone proposes something new in anything... it's easy for people to say "what we currently have works well enough", it's up to the visionary to present his ideas in such a way as to overcome that inertia .

[Paolo]

I think the key thing would be to demonstrate how much more useful this could be than the existing XML Format. So far you've shown us how you think it could work, but apparently it's not been that persuasive... so if you can show us some specific examples /case studies in which your database/GUID proposal would be demonstrably better than XML then you'll probably have more luck...

Malenfant

Perhaps I'm wrong but I think that I was not able to explain myself.
Even if you use XML as data format you will need some kind of GUID as tag field in order to allow unique identification of the Celestia objects.
The point is not using or not XML as language for the data description. XML should be fine so instead than my initial proposal:

Code: Select all

Texture
{
    GUID [??]
    ParentGUID [???]                  ## The planet, the body, whatever else
    ParentType "Planet"              ## This should help but is not strictly necessary
    File "myAddOn/myNewTexture.jpg"  ## The name should include the path
    Type 2                                 ## Base texture, clouds, specular, heigh map
    Alternate 1                           ## The alternative index
    ...
}


You ll'have something like.

Code: Select all

<Texture>

    <GUID> ??? </GUID>
    <ParentGUID> ??? </ParentGUID>
    <ParentType>Planet</ParentType>
    <File>myAddOn/myNewTexture.jpg</file>
    <Type>2</Type>
    <Alternate> 1 </Alternate>
    ...
</texture>


With an XML schema that describes the types of the tagged fileds.

This is not the point.
I think that if you spend 5 minutes to read again the posts in this thread and youve tried to write an add-on everything will be more clear.

Now is very late Tomorrow I'll try to explain and show some examples.

[hank]

Please be aware that I don't want to be sarcastic. But I want to express my doubts.

I've noticed that in the last months some efforts in the (very scarse) development activity was aimed to improve the interface between LUA and Celestia. Now even with an event handler!
I agree that this adds a lot of power and flexibility to Celestia.

But the questions are: is it really useful? how many users should be able to write LUA scripts for Celestia? Why an average or intermediate or advanced Celestia user (not a newbie) should write or use LUA scripts? Why in the current distribution there is a so small collection of scripts almost unuseful?

E.G. WinCVS uses Pyhton instead than LUA. A series of useful automation scripts are included in the program and are integrated directly in the GUI. Having an advanced scripting language makes sense. But for Celestia? Scripting is used almost only for creation of automated demos in teaching activities. Was LUA necessary for this? Ok it allows a better approach using an extensible OOP language with integrated garbage collector instead than a collection of cel commands, and so and so.

Beside this if using LUA will be possible to manage some kind of XML data format in order to implement a data model that should allow all the things that I've pointed out in this thread, of course it is welcome!

I'm not so much interested in Lua for simple user scripting, although it's of course very useful for that. Instead, I'm interested in using Lua for customization and extension of Celestia. I believe Lua could be used to allow advanced users to implement enhanced or customized versions of Celestia which could be distributed as addons, without the need for a non-standard version of the base program. The advantage of using Lua for Celestia data loading is that it is extensible, allowing functions to be added as desired to load data from XML, MySQL, or any custom format.

- Hank

[Paolo]

I would like to find a compromise. Instead than generate the GUIDs with an algorithm I would like that these GUIDs will be provided by some sort of Web repository. So an Add-On creator shoud ask for instance for 10 Thousands of GUIDs that will be reserved for him. So he will know (and every one else) that the GUIDs form 25629000000 to 25629100000 are already in use.

Another idea about this issue should be using an IP4 or IP6 strategy, and then the add-on creators should request GUIDs to a shared entity like now internet users asks to the NIC for reservation of public internet IP addresses.

An IP4 like strategy should allow to manage about 4,3 Billions of GUIDs.
An IP6 like strategy should allow to manage about 281,5 Trilions of GUIDs.

I think that even in a far Future the Celestia comunity won't ever be able to manage such a so large set of cosmological/astronomical/astronautical related objects and data.

Using this approach some sets of GUIDs IP like address ranges should be reserved to official releases, official data, fictional data and so on.

Using a combined strategy Object Type + GUID IP4 Style for the identification of objects should be more complex but should be even more flexible without overloading the memory usage.

Kind regards.

[ajtribick]

While this may be getting too much into specifics, I personally prefer using attributes rather than element content for structured data, since you don't end up with loads of close tags all over the place.

i.e.

Code: Select all

...
<SemimajorAxis val="2" unit="AU" />
<Eccentricity val="0.2" />
<MeanAnomaly val="124" unit="deg" />
...


rather than

Code: Select all

...
<SemimajorAxis unit="AU">2</SemimajorAxis>
<Eccentricity>0.2</Eccentricity>
<MeanAnomaly unit="deg">124</MeanAnomaly>
...


[Paolo]

Please be aware that I don't want to be sarcastic. But I want to express my doubts.

...

Beside this if using LUA will be possible to manage some kind of XML data format in order to implement a data model that should allow all the things that I've pointed out in this thread, of course it is welcome!

I'm not so much interested in Lua for simple user scripting, although it's of course very useful for that. Instead, I'm interested in using Lua for customization and extension of Celestia. I believe Lua could be used to allow advanced users to implement enhanced or customized versions of Celestia which could be distributed as addons, without the need for a non-standard version of the base program. The advantage of using Lua for Celestia data loading is that it is extensible, allowing functions to be added as desired to load data from XML, MySQL, or any custom format.

- Hank

Hello Hank

From these words I can imagine that the mainstream development of Celestia is moving towards the abandon of the GUI part in order to focus almost exclusively on the CelestiaCore and LUA.

CelestiaCore object will expose its capabilities through LUA, so Celestia "Mods" will be more easy to implement.

IMHO I think that this is not what the Celestia community is looking for, but on the other side it is not in conflict with the argument of this thread.

So I can guess that you are sugesting that in order to get flexibility the new unified data format (XML or different) will be handled directly through LUA.
IMHO first of all LUA will perform worse than Celestia itself in loading data from files, on the other side its difficult to me to imagine an application that runs scripts to load its data. It should make sense if the application is a LUA script itself.

Beside this my proposal implies the brake of the backward compatibility of the data formats. So none flexibility is required at all, since at athe end LUA scripts should have to load data calling core functions, why not use C++ directly like now?

But these are marginal implementative considerations. I would like to point out the discussion back again on the database aspect and on the advantages of this approach.

Kind regards