celcheck - a tool to check the syntax of .ssc, .stc, .dsc, .cel, ASCII .cmod, .cms (and new: .xyz and .xyzv) files v1.4

Here you find pointers to utilities that help you create addons for Celestia.
Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

celcheck - a tool to check the syntax of .ssc, .stc, .dsc, .cel, ASCII .cmod, .cms (and new: .xyz and .xyzv) files v1.4

Post #1by EarthMoon » 11.10.2023, 12:47

I have created a new useful program: celcheck

It can check .ssc, .stc, .dsc, .cel, .cms, ASCII .cmod, .xyz and .xyzv files for their syntax (but not the values of each parameter). This can be useful if you have an ssc file which is very long (e.g. a long timeline with many parameters each) and you are unsure if you have closed all opened braces.

To use it:

Open a command line, run the file from there and pass the name to the file as an argument, like this:

Code: Select all

celcheck_x.y.exe myfile.ssc

Where x.y is the actual version (1.0, 1.1, etc.).

Note: the script allows you to put ssc, stc, dsc objects, cel scripts and .cms models together into one file, so if you have a large addon with two or more types of objects and scripts, you don't have to check each file separately but all in once. Cmod model must be separated into own files (one model per file) whose file names end with ".cmod"! Also, cmod models must be in ASCII encoding mode (you can check this by opening the model in a text editor and looking at the first few bytes: if it starts with #celmodel__ascii, it is the correct format, if it's #celmodel_binary followed by a bunch of unreadable characters, it's the wrong format (notice the double underscores in the ascii header and the single underscore in the binary header (to make both headers 16 bytes long))!

This script doesn't check the names and values of the parameters (for compatibility reasons, one Celestia version has other/new paramaters than an older one).

Examples for defintions celcheck will mark as valid:

Code: Select all

"My object" "Sol/Parent" {
   Class "moon"
}

Modify "My other object" "Sol/Parent" {}

"" "" {}

# This definition has wrong values but celcheck doesn't care about that:
"My fourth object" "Sol/Parent" {
   Class 123
   Radius "invalid radius, but valid file"
   Texture [1 2 3]
   EllipticalOrbit true
   SomeUnknownParameter [1 [1 2] {Key "value"} true 1 3 false]
}


Invalid definitions:

Code: Select all

# [] instead of {}
"My object" "Sol/Parent" [
   Class "moon"
]

# Forget to specify the parent object
Modify "My other object" {}

# Forget the {}
"" ""

# The 'UniformRotation' parameter doesn't have a value +
# there is an unclosed { in the SomeUnknownParameter (or if it is closed by the end }, the main body isn't closed)
"My fourth object" "Sol/Parent" {
   Class 123
   Radius "invalid radius, but valid file"
   UniformRotation
   Texture [1 2 3]
   EllipticalOrbit true
   SomeUnknownParameter [1 [1 2] {Key "value" true 1 3 false]
}


Note: there must be at least one object definition in the file to be valid.



The script has been written in Python and uses the Lark parser.

How the script works internally:

1. Try to read input file
If everything runs well:
_2. Build the parser
_3. Parse the file
_If there are no errors:
__4. Tell the user that the file is valid
_If there are errors:
__4. Print the errors
If none is specified or it could not be read:
_2. print an appropriate message
5. Exit

Files for download:

celcheck_x.y.exe - binary program
source_code_x.y.zip - contains "celcheck.py" (for anyone who wants to modify/fix bugs/extend the script, but you have to credit me if you publish that modifed version), also contains a "grammar.lark" file for versions 1.0 and 1.1

Changelog:
If you need an older version than the latest one, comment below or PM me.

1.1: Added support for .cel scripts!

1.2: Added support for .cmod (only ASCII) and .cms models! A cmod model's name MUST end with ".cmod" to be recognized by celcheck! A file with any other file extension must countain .cel, .ssc, .stc, .dsc or .cms data! Builds are now in --onedir mode and not in --onefile (PyInstaller) anymore! The .cms mode has been tested with Celestia 1.7.0 srgb's models/asteroid.cms model. There are no grammer files in the source_code_x.y.zip files anymore. Open the .py script to find the grammar data.
The syntax definition of .cmod files comes from the Wikibooks page about CMOD files. I am not responsible for any errors in the docs (but I have tested celcheck on a model). Additionally, I have added the pointsize option to <vertex_semantic> (for sprite models).

1.3: Strings can now be multi-line in all files currently supported (except .cmod files), like multi-line object name definitions:

Code: Select all

"my object:other name:third name:
fourth name" "Parent" { <parameters> }


1.4: Added support for .xyz and .xyzv files: these files must have their corresponding file extension (e.g. a file with the ending .abc won't get parsed as xyz(v) but as ssc/dsc/stc).
Attachments
source_code_1.4.zip
(1.35 KiB) Downloaded 31 times
celcheck_1.4.zip
(116.08 MiB) Downloaded 34 times
Last edited by EarthMoon on 17.12.2023, 10:59, edited 18 times in total.
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Update 1.1! Added support for .cel scripts

Post #2by EarthMoon » 22.10.2023, 17:31

See the first post.
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #3by gironde » 05.11.2023, 10:14

sorry but I can't get it to work.
Where should we put the downloaded files and the file to check?
should we launch the celcheck.py file or the celcheck_1.1.exe file?

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #4by EarthMoon » 09.11.2023, 14:31

The celcheck.py file is the source code (but you can also use it if you want). I recommend the .exe file. The location of the exe and check file does not matter, although you then need to specifiy the full relative or absolute to the check file (the ssc/stc/dsc/cel file). Or drop the file on the exe (but you will have to be able to read very quickly as the exe terminates right after the check).
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #5by gironde » 10.11.2023, 11:38

the program cannot read the ssc file that I provide it :cry:

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #6by EarthMoon » 10.11.2023, 16:45

What are the contents of the ssc?
And what encoding has your ssc file?
What error message do you get?
Did you make sure the file exists and you have permissions to at least read it?
What command did you use to run celcheck?
Where are the file and the program on your system?
What is the current working dir in your command line window?

(Sorry for so many questions but I need to know all that as I don't know your exact situation except that "it cannot read the file")
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #7by gironde » 19.11.2023, 12:03

in cmd, I write:
celcheck_1.1 asteroids.ssc

both are in the same folder
asteroids.ssc is an asteroid declaration file that works in celestia (281310 lines et 5350Ko)

the result is:
"Could not read file 'asteroids.ssc'!
Make sure the file exists and you have read permissions"

with a file of the first 79 lines of the original (4Ko), the program works and the response is:
Specified file is valid.

there is a limit on the size of the accepted ssc file. Can you define this limit?

:hi:

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #8by EarthMoon » 20.11.2023, 09:29

Hmmm, that is weird. I did not specifiy a file size limit in the program, meaning this is really a bug and not intended. Could you send me the file so I can test it?
I am currently trying to find out what causes this issue with a self-made file and then with yours if needed.
Until it is fixed I suggest you to split your ssc into multiple smaller files for check and then merging all again.

Added after 25 minutes 25 seconds:
So, I have tested it with a self-made file which has 12339 asteroids (a bit more than yours which should have 281310/23=12231 asteroids, each definition+extra line has 23 lines) and 285086 lines and I don't get any errors when running (it only takes very long but that's to be expected when dealing with big files). So, possible causes could be:

- The file isn't existing (e.g. a tiny typo in the name, e.g. the real name is astreoids.ssc and you specify asteroids.ssc). You could check that with this way: Open Python, paste the file names from the command and the real file as strings connected with == in the interpreter ("filename_on_filesystem.ssc" == "filename_in_command.ssc") and then check if it's True. Or paste both names in a text editor in two lines below each other and then look if there are differences.
- It has a bad encoding (mine has UTF-8 and the program has only been tested with UTF-8 .ssc's).
- The file has bad newline chars (mine uses CR LF (\r\n))

Added after 12 minutes 8 seconds:
Could you run the command again, but with this test version, and post everything from the cmd in your next post?
Attachments
celcheck_test1.exe
(113.5 MiB) Downloaded 50 times
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #9by gironde » 22.11.2023, 19:27

hello,

name of file is asteroids.ssc
encode UTF-8

---------------------- First run -----------------------
D:\Public\Celestia extra\Celestia add\tools\Outil celcheck>celcheck_test1.exe asteroids.ssc
Traceback (most recent call last):
File "celcheck_test1.py", line 67, in <module>
File "encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 819162: character maps to <undefined>
[10632] Failed to execute script 'celcheck_test1' due to unhandled exception!

correction : removed a comment written in Russian
---------------------- 2e run -----------------------
D:\Public\Celestia extra\Celestia add\tools\Outil celcheck>celcheck_test1.exe asteroids.ssc
Specified file is invalid:
UnexpectedCharacters: No terminal matches '-' in the current parser context, at line 562 col 12

Texture ""-- "darkasteroid.dds"
^
correction : Texture "darkasteroid.dds"
Expected one of:
* RBRACE
* __ANON_2

---------------------- 3e run -----------------------
D:\Public\Celestia extra\Celestia add\tools\Outil celcheck>celcheck_test1.exe asteroids.ssc
Specified file is invalid:
UnexpectedCharacters: No terminal matches '.' in the current parser context, at line 31197 col 16

Eccentricity .187453063150358 -- normally accepted by celestia
^
correction : Eccentricity 0.187453063150358
Expected one of:
* LBRACE
* TRUE
* __ANON_1
* ESCAPED_STRING
* LSQB
* FALSE
* __ANON_0

---------------------- 4e run -----------------------
D:\Public\Celestia extra\Celestia add\tools\Outil celcheck>celcheck_test1.exe asteroids.ssc
Specified file is invalid:
UnexpectedCharacters: No terminal matches '-' in the current parser context, at line 131286 col 73

1973 UF1:1976 GC7:1976 JH:1986 UK" "Sol" -- AdeonaFamily (MBA-MBA)
^
correction : "Walford:4629 Walford:1986 TD7:1973 UF1:1976 GC7:1976 JH:1986 UK" "Sol" # AdeonaFamily (MBA-MBA)
Expected one of:
* LBRACE

---------------------- 5e run -----------------------
D:\Public\Celestia extra\Celestia add\tools\Outil celcheck>celcheck_test1.exe asteroids.ssc
Specified file is invalid:
UnexpectedCharacters: No terminal matches '"' in the current parser context, at line 179785 col 31

"Beslan:6374 Beslan:1986 PY4" "Sol" # (MBA-MBA)
^
correction : added the missing lines added the missing lines found in the original (asteroid 6369 to 6373)
Expected one of:
* RBRACE
* __ANON_2

---------------------- 6e run -----------------------
D:\Public\Celestia extra\Celestia add\tools\Outil celcheck>celcheck_test1.exe asteroids.ssc
Specified file is invalid:
UnexpectedCharacters: No terminal matches 'C' in the current parser context, at line 194551 col 2

Class "asteroid" -- no find error ??????
^
correction : no find error ??????
Expected one of:
* LBRACE

-----------------------------------------------------------------------------------------------------


-- after, more error of type '6e run'

the main blockage was the commentary in Russian

:clap: :clap: :clap:

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #10by EarthMoon » 23.11.2023, 08:15

1st run: I won't allow comments with non-ASCII caracters to make SSC files' comments readable for everyone and to prevent issues with Celestia (e.g. the right-to-left direction of the Arabian language).
3rd run: I forgot about this format of numbers. I will add that in a new version soon!
6th run: As the error tells, Celcheck expects a LBRACE (left brace, {) in that place, maybe you forgot (to open/close) a parameters block in one of the preceeding asteroids? Or you forgot to open the parameters block on this asteroid, like:

Code: Select all

"Name of Asteroid" "Sol"
    Class "asteroid"


If you cannot fix the issue, as I told in the last post, could you send me the file (e.g. as attachment or PM) so I can inspect what's wrong?
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Update 1.2! Added support for ASCII .cmod and .cms models

Post #11by EarthMoon » 08.12.2023, 10:37

See the first post.
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #12by gironde » 09.12.2023, 09:58

I continue to test your latest version
my test file is a file that I shared on the forum under asteroids from 0 to 9999
Your program does not recognize certain characters in the file name, for example "°", although celesia accepts them.

So my test is done with the file asteroids00001-09999_N=10000.ssc which I attach here

1st test with error
Specified file is invalid:
UnexpectedCharacters: No terminal matches '"' in the current parser context, at line 58215 col 1

"McCuskey:2007 McCuskey:1963 SQ:1970 QD1
^
Expected one of:
*ESCAPED_STRING
*LBRACE
* __ANON_0
* __ANON_2

the original lines are:
"McCuskey:2007 McCuskey:1963 SQ:1970 QD1:1969 EC1:1966 OB:1965 CC:1965 CA:1963 VC:
1963 TL:1958 GB:1952 UR1:1952 SB1:1951 LV:1947 GE:1943 EL:1941 SW1:1936 HP:A921 EK" "Sol" # PolanaFamily (MBA-MBA)
{

Apparently writing on 2 lines is not recognized by your program even though it is very well accepted by Celestia

asteroids00001-09999_N=10000.zip
(834.44 KiB) Downloaded 38 times


Added after 14 minutes 36 seconds:
2e test
Specified file is invalid:
UnexpectedCharacters: No terminal matches 'C' in the current parser context, at line 194551 col 2

Class "asteroid"
^
Expected one of:
* LBRACE

the original lines are:
"Saint-Marys:6898 Saint-Marys:1988 LE" "Sol" # EunomiaFamily (MBA-MBA)
Class "asteroid"
Mesh "asteroid-large.cmod"
Texture "asteroid.dds"

forget to {
correction:
"Saint-Marys:6898 Saint-Marys:1988 LE" "Sol" # EunomiaFamily (MBA-MBA)
{
Class "asteroid"
Mesh "asteroid-large.cmod"
Texture "asteroid.dds"

Added after 34 seconds:
after no error :smile:

Added after 43 minutes 36 seconds:
for manipulation of celcheck, I have created an batch file named celcheck.bat :

Code: Select all

celcheck_test1.exe asteroids030000-039999_N=10000.ssc
cmd/k


asteroids030000-039999_N=10000.ssc is the name of the file to study (without space and without special characters)
this file is in the same folder that .exe

cmd/k is a command that prevents cmd from closing at the end

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #13by EarthMoon » 09.12.2023, 13:49

gironde wrote:Your program does not recognize certain characters in the file name, for example "°", although celesia accepts them.
"Celesia accepts them" yes. But what about the file system the addons gets installed on?
Not all file systems support all characters, so I will let ° disallowed, and it isn't a big pain to rename one single file.

gironde wrote:after no error :smile:
Good to know that.

gironde wrote:for manipulation of celcheck, I have created an batch file named celcheck.bat:
This can be useful for debugging: You don't have to write/copy+paste the same command over and over again but just have to double click a file.

gironde wrote:Apparently writing on 2 lines is not recognized by your program even though it is very well accepted by Celestia
I have fixed that in 1.3 (see first post).
Last edited by EarthMoon on 09.12.2023, 14:23, edited 1 time in total.
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Update 1.3: Bugfix

Post #14by EarthMoon » 09.12.2023, 14:22

See first post.
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #15by gironde » 09.12.2023, 18:43

celcheck_1.3 does not work on large files
I'm trying to find up to how many lines it works

celcheck_test1 works very well with large files: 280132 lines

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #16by gironde » 14.12.2023, 09:18

cellcheck_1.3 is effective with an ssc file containing around thirty objects but beyond that, checking the parameters programmed on 2 or more lines takes a lot of time without us knowing if the program is stopped or running. With files of 10,000 asteroids, the program manages to empty the central memory (12 Gb) and even the virtual memory.
For this type of file, I use cellcheck_test1, which does not perform this check but triggers an error that is easy to correct temporarily so that the program can go to the end of the ssc file.
You could make a version with double line control and another without!

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #17by EarthMoon » 14.12.2023, 19:14

gironde wrote:You could make a version with double line control and another without!
The new 1.3 version already has this double line thing (strings [data enclosed in " "] can be even multi-line (2, 3, 4, ...), not only double line; parameters can also be multi-line: Name on one line, value on a later one). 1.2 is basically the same but only supports one line for a string.
And it's normal that it takes a lot of time (like Celestia takes long time to load like lots of GB of addons, for me at about 45 secs with 5.38 GB in the /extras/ directory). The only two solutions are: A better PC/RAM/CPU or (the easier way) using patience.

E.g.:

Valid in 1.3:

Code: Select all

"Object:
Other name" "Sol/Parent" {
   Name
   "value"
   Param1 "value1" Param2
   "value2" Param3
   "value3"
}


Valid in 1.2:

Code: Select all

"Object:Other name" "Sol/Parent" {
   Name
   "value"
   Param1 "value1" Param2
   "value2" Param3
   "value3"
}
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
gironde M
Posts: 818
Joined: 16.12.2016
Age: 71
With us: 7 years 4 months
Location: Montigny-Les-Metz, France

Post #18by gironde » 16.12.2023, 10:14

ok but that doesn't help those who provide large files: asteroids, stars, nebulas, galaxies, pulsars, spacecraft fleets, ...
multilines exist but are few in number (for my part, only 1 out of 600,000 asteroids). Once this error is resolved, even temporarily,
the celcheck program runs much faster.
I tried on a large .stc or .dsc file and celcheck_1.3 stopped due to a memory overflow. (12 Gb)

ps : 1.2 version is not published here !

:hi:

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Post #19by EarthMoon » 16.12.2023, 17:19

gironde wrote:ps : 1.2 version is not published here !

As the first post says: "If you need an older version than the latest one, comment below or PM me.".
I only upload the latest version and remove all older ones (except the celcheck_test1.exe because of your ongoing problems/errors, I will delete that too as soon as everything is fixed or solved) in the main post to clean up attachments/resources on this forum.

gironde wrote:Once this error is resolved, even temporarily,
the celcheck program runs much faster.
You mean that the program runs faster when there are no multi-line object definitions/strings?
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)

Avatar
Topic author
EarthMoon
Posts: 200
Joined: 02.07.2022
With us: 1 year 10 months
Location: Germany

Update 1.4: Added support for .xyz and .xyzv files!

Post #20by EarthMoon » 17.12.2023, 10:56

See the first post.
Crew: "We are orbiting a black hole."
Control Center: "Do not fly too close to the black hole!"
Crew: "OH OOPS..."

Celestia versions:
1.6.3, 1.7.0 sRGB, 1.6.1 ED (plain)

Current projects:
Celestial (a Celestia-like program written in Python)


Return to “Utilities”