Page 1 of 1

Parser conflict resolutions

Posted: 02.04.2008, 20:48
by ajtribick
At present Celestia does not provide an easy way to override definitions of multiple star systems, etc., which gets in the way of implementing large catalogues of objects. If the two catalogues define the same object, the result is often two versions of the same system being displayed, which is not desirable.

There should be some way to cleanly handle this situation: for example in the situation of Alpha Aurigae, the solutions given in papers by Pourbaix and Hummel et al. differ, but at present it would be difficult to implement the Hummel et al. solution because of the Pourbaix catalogue implemented in the default datafiles. It is also impossible to switch between the two solutions in Celestia.

Some way of grouping together definitions for multiple stars would be useful, perhaps also some way of identifying what the data source used is. Any suggestions on how to achieve this, how interactions between conflicting definitions should work?

Bear in mind that there might be cases where different solutions involve different numbers of stars (including ones which may involve different numbers of stars with differing HIP numbers), and that for some purposes it might be good to switch to the basic HIP catalogue.

(This kind of feature might also be useful for exoplanetary systems - there are a couple out there for which several very different solutions have been published)

Re: Parser conflict resolutions

Posted: 02.04.2008, 21:07
by t00fri
Andrew,

why don't you spell it out clearly: if Celestia is to visualize TRUE measurements (e.g. of binary orbits), like any measurement in Nature, there will be uncertainties attached to the published orbit solutions. Sometimes these are very large. If we want to stay on a scientific fundament, we DO NEED a way of accounting (visually) for such uncertainties. Otherwise it will remain rather meaningless and intrinsically inconsistent what is displayed on the canvas.

You mentioned already typical systematical effects, namely that the nature of the orbit solution might even be ambiguous... Such a situation clearly precludes any simple averaging procedures.

Since a large portion of these uncertainties are systematical in nature, we are NOT allowed to add the individual measuements in square as we would do in case of statistical uncertainties and an underlying Normal distribution.

Every physics student knows this (I suppose ;-) ).

F.

Re: Parser conflict resolutions

Posted: 02.04.2008, 22:17
by chris
ajtribick wrote:At present Celestia does not provide an easy way to override definitions of multiple star systems, etc., which gets in the way of implementing large catalogues of objects. If the two catalogues define the same object, the result is often two versions of the same system being displayed, which is not desirable.

It seems like a big part of the challenge will be determining when two objects from different catalogs are actually the same star. To my knowledge, there's no universal indexing scheme for stars. Celestia has until this point relied up HIPPARCOS numbers, with any stars not in that catalog identified by strings that may or may not refer to a star from another in use.

As for switching between catalogs, that will be challenging to do without restarting Celestia, since the stars get arranged into octrees for fast visibility determination. The octree may need to be rebuilt when a star position or magnitude changes, or when a star is added or removed. Any internal star reference in Celestia must be updated when the octree is rebuilt--it's quite a bookkeeping problem. But, I suppose before getting too far into the technical challenges, we should think mostly about how star catalogs would ideally be handled. In that spirit, let me ask this: what star catalogs do you want to use? How big are they? What sort of data do they contain? It seems to me that there are a limited number of catalogs that are appropriate, given any catalog for Celestia must contain distance information. So perhaps we don't need to design a solution that can accommodate a hundred thousand dynamic stars without compromising rendering performance. The more we can constrain the problem, the better.

--Chris

Re: Parser conflict resolutions

Posted: 03.04.2008, 14:26
by ajtribick
I think as a first step, getting some mechanism to override multiple systems cleanly would be a start (including the possibility of adding/deleting stars), never mind switching or finding a way to somehow graphically represent uncertainties.

Re: Parser conflict resolutions

Posted: 03.04.2008, 14:31
by t00fri
ajtribick wrote:I think as a first step, getting some mechanism to override multiple systems cleanly would be a start (including the possibility of adding/deleting stars), never mind switching or finding a way to somehow graphically represent uncertainties.

Could you explain more concretely what you mean with "overriding multiple systems cleanly"?

F.

Re: Parser conflict resolutions

Posted: 03.04.2008, 15:44
by ajtribick
t00fri wrote:Could you explain more concretely what you mean with "overriding multiple systems cleanly"?
A way of overriding that does not leave multiple copies of the system (or bits of it) all over the place.

As you yourself have stated, you've had to comment out bits of your binaries files to avoid conflicts with Grant's. If instead a clean override system was in place, then the parser could replace one set of systems for another (presumably this would be decided by loading order, unless some other switching mechanism is provided).

For example, if I wanted to make an add-on based on Hummel's model of Alpha Aurigae, I don't want to have to put in the installation instructions (which no-one reads anyway) that you have to go into one of the files in the data directory and comment out the Pourbaix model, because even if the user in question read the instructions, they would still have to remember to uncomment it if they uninstalled the add on.

Alternatively consider a situation where the model used has different numbers of components, e.g. the Xi Ursae Majoris (=Alula Australis), for which details about the B subsystem are still uncertain. There's no way to override a multiple system with a version with fewer components.

So it would be a replace mechanism that removed the definitions of all the components of a multiple star system and replaced them with a new set of definitions.

Re: Parser conflict resolutions

Posted: 03.04.2008, 16:41
by t00fri
ajtribick wrote:
t00fri wrote:Could you explain more concretely what you mean with "overriding multiple systems cleanly"?
A way of overriding that does not leave multiple copies of the system (or bits of it) all over the place.

As you yourself have stated, you've had to comment out bits of your binaries files to avoid conflicts with Grant's. If instead a clean override system was in place, then the parser could replace one set of systems for another (presumably this would be decided by loading order, unless some other switching mechanism is provided).

Certainly, but this is NOT a job for the computer to decide in general.

Substitution of certain star data by others is a non-trivial physics/astrophysics issue! That's why the published input data files were selected by me (an experienced (astro) particle physicist... ;-) ) and NOT by a computer parser!

The discussion with Grant mainly goes about syntax and data base selection criteria, but NOT about replacing data for a star system. Grant is himself very experienced in these matters and his data files do reflect that experience.

But even if you want to ignore my choices, I don't see the issue really: all relevant data files are ascii files. Users (like yourself) who know what they are doing, can simply override entries they don't fancy whatever the reasons.

Since moreover there might not be enough informations available for each system in form of alternative names, the stupid computer parser might simply not be able to recognize that two systems are identical. I guess Chris alluded to this already yesterday. The HIP number might simply not be enough in all cases.

So what do you have in mind here? If I had seen an unambiguous method in this respect, I'd implemented it since a LOOOOONG time.

For example, if I wanted to make an add-on based on Hummel's model of Alpha Aurigae, I don't want to have to put in the installation instructions (which no-one reads anyway) that you have to go into one of the files in the data directory and comment out the Pourbaix model, because even if the user in question read the instructions, they would still have to remember to uncomment it if they uninstalled the add on.

That's easy and not worth any further unreliable automatisms: just supply the modified Pourbaix listing (or a part thereof) with your add-on in the /extras/ dir! It's the standard Celestia way for overriding and it does work. We don't need to talk about this sort triviality, do we? The whole spectbins.stc file is only 40k ;-) . Future extensions will hardly be orders of magnitude bigger...

As a serious add-on creator you will anyway have to write an astrophysical justification in the README, why you did erase the published Pourbaix data point!


Alternatively consider a situation where the model used has different numbers of components, e.g. the Xi Ursae Majoris (=Alula Australis), for which details about the B subsystem are still uncertain. There's no way to override a multiple system with a version with fewer components.

So it would be a replace mechanism that removed the definitions of all the components of a multiple star system and replaced them with a new set of definitions.

Same as above. Since this is a rather infrequent case, do it by hand if you please... and place your modified data files into the /extras/ directory.

Anyway, I am not ready to hand the decision of what are the preferred multiple systems over to a computer. If you and other people want to do this, I will definitely not go along with this.

F.

Re: Parser conflict resolutions

Posted: 03.04.2008, 17:35
by ajtribick
I'm not saying the computer parser should be making the decision. The decision is at the user level, in which add-ons they decide to install.

Unfortunately the Celestia limitations prevent a user-friendly way of installing such an add-on (which might provide a more in-depth analysis or revised parameters from the catalogue files currently used). You and I might be perfectly comfortable with editing a bunch of text files to resolve the conflicting datafiles, but it detracts from the professionalism of the add-on and makes installation/uninstallation needlessly complex.

What I'm proposing is something where you can group star definitions, e.g.

multipleSystem {
starDefinition1version1
starDefinition2version1
...
}

...and then in some other file...

multipleSystem {
starDefinition1version2
starDefinition2version2
...
}

with some form of identifier in the multipleSystem code to handle whether or not these refer to the same thing. It would be the same kind of thing that we already do with .stc files in replacing individual stars, but done at the level of a multiple star system.

Out of interest how do you propose to proceed on the binary stars issue? You've already stated that you don't want to go further with binary stars implementation because of conflicts with Grant's files and you don't want to hand-edit the files to avoid conflicts or bring in other data sources.

Re: Parser conflict resolutions

Posted: 03.04.2008, 18:09
by t00fri
Andrew,

in my view this all looks VERY untransparent and hard to realize in a /physically/ sensible way. Why should we encourage users to fiddle around with star definitions in such an intricate ad hoc way, without even taking into account measurement uncertainties! Honestly you are really the first one with this sort of request. So far, we are just talking about possibly modifying 1-2 binary star systems among ~200, where the literature admits sensible other alternatives.

I think it is MUCH simpler and conforms with the other Celestia prescriptions, to fully exploit the overriding possibilities in the extras directory for doing add-ons.

At least in my view, there are MANY burning issues with higher priority...

As to the discussion with Grant, we seem again to be effectively stuck.
He wrote this 1.5-line reply
granthutchison wrote:I'm happy for any and all of "my" stc files to be edited to reflect any convention that is generally adopted.

I won't fiddle around in his nearstars file for sure. What would be needed is a /published/ set of catalogs where ALL the multiple star data are listed that he used. Then one could discuss jointly about a more compatible and expandable definition of multiple star data files in Celestia and I could write a /well documented/ PERL script that reads out these data from the publications into the resulting celestia files.

F.

Re: Parser conflict resolutions

Posted: 03.04.2008, 22:00
by chris
I like the direction that Andrew's proposal is taking, and the philosophy behind it seems sound: users should be able to easily switch between stellar data sources by selecting which catalogs should be loaded. The only problem with it that I see is that it would take some work to implement, and I'm not convinced that it's worth the effort. On this matter, I'm sympathetic to Fridger's point of view: how much more star data can we hope to get for Celestia (before the GAIA data set is ready)? If it's just a matter of adding a few binary systems here and there, then the effort to carefully tune the star catalogs seems less than the effort involved in reworking the star parser. If there's a lot more star data waiting to be integrated into Celestia, then implementing a scheme like Andrew proposed becomes a lot more attractive. Interested in a coding project, Andrew?

--Chris

Re: Parser conflict resolutions

Posted: 03.04.2008, 22:36
by selden
Have any professional catalogs been published which provide spectrographic distances? That's essentially the technique which was used by Pascal Hartman to create the 1 million and 2 million star databases available on the MotherLode.

Spectrographic distances are more unreliable than direct parallax measurements like those done by Hipparcos, but would potentially provide a substantial enhancement to the base version of Celestia.

Re: Parser conflict resolutions

Posted: 03.04.2008, 22:57
by t00fri
chris wrote:I like the direction that Andrew's proposal is taking, and the philosophy behind it seems sound: users should be able to easily switch between stellar data sources by selecting which catalogs should be loaded.

While I am fully sympathetic to a switching mechanism between /published/ entire catalogs, this is NOT what Andrew proposed above. He proposed to be able to switch between individual multiple stars from /different/ sources, ad libitum, so to speak! This is NOT scientifically sound and will inevitably lead to a "tohuwabohu" at the documentation front and otherwise... Then we will be loosing what I have been striving for during the last 5+ years: namely to create a fully documented, reproducable and thus transparent basis for Celestia's input data.

The number of multiple star systems we can furthermore expect in the future depends STRONGLY on the amount of additional assumptions we will be ready to make about unknown parameters.

I think at the level of what I found acceptable so far, we have largely exhausted the available resources. The main problem are certainly NOT the distance measurements that typically refer to the barycenter. We actually need a FULL well-determined set of 5-7 orbit parameters per multiple star! If that set is incomplete, typically several equally good but physically different solutions arise in the orbit fits. That sort of ambiguity makes the result worthless. Up to now I have eliminated such and other related cases. This has cooked down e.g the number of admissible visual binary entries from ~ 370 to 163!

I some orbit parameter is lacking, this has much worse consequences on the reliability of the solution than a lacking apparent magnitude of the secondary or an unknown color class.

F.

Re: Parser conflict resolutions

Posted: 04.04.2008, 17:16
by ajtribick
chris wrote:Interested in a coding project, Andrew?
I'd probably be able to take a look at it, but only after I've finished this year of university! And only if this isn't going to cause major controversy amongst the rest of the developers. As it is, I see the entire project would be objectionable.

Evidently this thread is not the place for it, but it seems to me we urgently need a discussion of data philosophy.

Re: Parser conflict resolutions

Posted: 04.04.2008, 19:18
by granthutchison
t00fri wrote:As to the discussion with Grant, we seem again to be effectively stuck.
He wrote this 1.5-line reply
granthutchison wrote:I'm happy for any and all of "my" stc files to be edited to reflect any convention that is generally adopted.
You're not stuck. But maybe I wasn't clear.

You can edit any and all of "my" stc files, any way you like; which may include removing all or part of a file for reasons of data consistency. I have absolutely no opposition to any changes to any star files ... so long as they don't result in the sort of "conflict debris" that Andrew has already flagged, which would of course be an embarassment for all concerned.

Grant

Re: Parser conflict resolutions

Posted: 04.04.2008, 19:39
by t00fri
ajtribick wrote:
chris wrote:Interested in a coding project, Andrew?
I'd probably be able to take a look at it, but only after I've finished this year of university!

So what are you studying, if I may ask ? I hope it's kind of useful for Celestia ;-) .

F.

Re: Parser conflict resolutions

Posted: 04.04.2008, 19:50
by ajtribick
t00fri wrote:While I am fully sympathetic to a switching mechanism between /published/ entire catalogs, this is NOT what Andrew proposed above. He proposed to be able to switch between individual multiple stars from /different/ sources, ad libitum, so to speak! This is NOT scientifically sound and will inevitably lead to a "tohuwabohu" at the documentation front and otherwise... Then we will be loosing what I have been striving for during the last 5+ years: namely to create a fully documented, reproducable and thus transparent basis for Celestia's input data.
Whether it is scientifically sound or not surely depends on what we want to do with the data. For example, would including into Celestia data from, say, this paper announcing the direct detection of the secondary star in the gamma Cephei system be scientifically sound? If I am going to do a comparison of various binary orbits, then sure, including data from such papers is going to be dubious rather than including large catalogues based on some standardised methodology. Unfortunately, as far as I can tell, the significant discovery that the secondary star of gamma Cephei has been directly detected hasn't managed to make it into one of these catalogues. If on the other hand my objective is to view an up-to-date version of the gamma Cephei system, then the situation is surely different? That's why I suggested the switching option. How would you resolve this situation?

t00fri wrote:So what are you studying, if I may ask ?
You mean you haven't checked the "About Us" page at Celestial Matters? ;)

Re: Parser conflict resolutions

Posted: 05.04.2008, 09:35
by t00fri
ajtribick wrote:
t00fri wrote:While I am fully sympathetic to a switching mechanism between /published/ entire catalogs, this is NOT what Andrew proposed above. He proposed to be able to switch between individual multiple stars from /different/ sources, ad libitum, so to speak! This is NOT scientifically sound and will inevitably lead to a "tohuwabohu" at the documentation front and otherwise... Then we will be loosing what I have been striving for during the last 5+ years: namely to create a fully documented, reproducable and thus transparent basis for Celestia's input data.
Whether it is scientifically sound or not surely depends on what we want to do with the data. For example, would including into Celestia data from, say, this paper announcing the direct detection of the secondary star in the gamma Cephei system be scientifically sound?
While I can see what is in the back of your mind, let me expose and justify my attitude in this respect:
  • The first consideration is that I am thinking in terms of the official distribution of Celestia that addresses people with vastly different degrees of knowledge in scientific matters. For many, too much flexibility in the data input creates a tendency of making things unpleasantly complex and thus hard to handle.

    My great concern over the years was to also provide SERIOUS scientific level information to those who are unable to judge the soundness of the data for themselves.

  • Your example can be scientifically sound at the level of the tiny subset of users with a solid scientific training! People, who do know how to fold in systematical uncertainties of the measuring apparatus and how different measurements have to be compared and interpreted.

    Celestia, however, is presently unable to visualize any kind of uncertainties of particular measurements, hence your example is not very illuminating from a scientific point of view.

  • Another main point is that in most cases, one cannot simply add in partial new discovery-level results (like the mass ratio in your example) to existing orbit parameter solutions. A completely new orbit fit would be required, and not surprisingly, the paper you quoted did NOT attempt this. This task is clearly beyond the ability of practically ALL users. Without refitting, the results can be highly misleading and thus they are NOT sound! This fact strongly argues against an offiicial support of such a "quick shot" attitude.

  • Specifically for multiple stars, disentangling proper motion and orbital motion effects and extracting the required large set of orbit parameters in multiparameter least square fits, is a highly non-trivial task. It is a wise scientific attitude to be somewhat conservative here!

  • I do favor to implement a comparison facility of entire refereed and published catalogs, since in this case expert astronomers have already made a selection or the entire catalog refers to the same measurement equipment, such that systematics tends to factor out among the results!

  • Last not least, my concern is to maintain a completely transparent documentation of what is displayed in Celestia. This becomes increasingly impossible if we start flipping in (officially) individual stars (or worse, partial results) from many /different/ sources, each based on different equipment and thus different kinds of systematic uncertainties.

    A good documentation of used data is a basic pillar of scientific soundness!

Unfortunately, as far as I can tell, the significant discovery that the secondary star of gamma Cephei has been directly detected hasn't managed to make it into one of these catalogues.
Usually there are good reasons for such an apparently conservative attitude of official catalogs!

Just trust me that in decades of interacting with experimental physics I have seen many "raw" discoveries vanish again or being strongly revised. Scientists usually put their findings only into official data bases when they are really confident that their detectors and the ambiguities of their analysis have been fully understood. This always takes time.

Since I served over many years as a theoretical advisor in various experimental research boards of big international laboratories, I suppose I should have sufficient insight into these delicate matters...

If on the other hand my objective is to view an up-to-date version of the gamma Cephei system, then the situation is surely different? That's why I suggested the switching option. How would you resolve this situation?
Your concrete example is clearly an application that would concern only very few people and very few concrete events. Hot discoveries don't happen all that often ;-) . For such comparatively rare events, I don't see why people like yourself should be unable to exploit the overriding facilities in the 'extras' directory for displaying data that are still too "raw" for appearing in official catalogs.
t00fri wrote:So what are you studying, if I may ask ?
You mean you haven't checked the "About Us" page at Celestial Matters? ;)

;-)

Incidentally, some years ago, my daughter graduated in Cambridge with the Tripos part III (mathematical physics, theoretical cosmology)... She also has British nationality by birth ;-)

F.