celestia networking-patch (http/ftp)

[phoenix]

hi all,

I have made a patch for celestia that allows the program to download and update remote sources like scripts, models, textures etc. automatically.
this patch also includes the modified gzip-patch.

client-section:

So what does it change?

+ adds 4 new options for celestia.cfg:

EnableNetworking (can be either 0 to disable or 1 to enable the networking code)
BindIP (can be an interface name, an IP address or a host name of the outgoing network interface celestia will be using)
DownloadFileTypes (a list of allowed filetype-extensions celestia will download, anything else will be ignored)
RemoteURL (a list of remote URLs to file-hash-lists - see server-section below for more information)

+ modified the splash-screen to display download speed and status
+ adds gzip support for stars.dat, cmod-files, dds-files and 3ds-files (extension .gz)
+ new win32 dlls libpng (1.2.8) and zlib (1.2.3)

What does it do if I run the exec?

If you don't change anything in the provided example celestia.cfg it will contact my test-server containing ~45 MB of data
including all necessary celestia-data, models and textures plus some randomly chosen addons.
People with slow connections (e.g. analog-modems) really should edit the celestia.cfg and change RemoteURL to another server containing less data or disable networking completely!

If you run the program a second time it will only check for changes between the local and remote files using md5-hashes.
Changed files will be downloaded and replaced automatically meaning any local changes will be overwritten too.

server-section:

The RemoteURL points to a server-stored text-file containing md5-hashes and filenames.
for example:

http://www.purebytes.de/stuff/celestia/ ... lestia.md5

The "*" sign preceding all filenames describes the file-type where * stands for binary.
To successfully create and compare cross-platform md5-hashes we need to treat all files as binary.

As you can see here the server stores all files in the same way celestia stores its data (e.g. data, models, scripts, extras).
This is essential because celestia won't create these directorys for you except for everything under the "extras" directory!

The script to create this "celestia.md5" can be found here (also see makemd5).
You don't have to call it "celestia.md5", call it whatever you want as long as you update your celestia.cfg to the exact filename.

Limitations

Currently you cannot abort the loading-process, so if have problems with your connection to the server or the download-stops
you either have to wait for the 30 sec timeout (per file!) or kill the application.
Also linux-users must have write access to the directory where celestia stores its data (e.g. /usr/local/share/celestia) otherwise you may get unexpected errors.

Bugs

None that i know of but since this hasn't been tested by many people there could still be some serious bugs hiding in the code.
Use this at your own risk! ;)


------------------------------------------------------------------
Current version is alpha7 - grab it here (11/09/2006)

this includes:
- a celestia-win32-binary from latest CVS
- a basic directory-structure
- a sample celestia.cfg
- the makemd5-script to create the server-side hashlist
- all necessary win32-libs to run the program.

download the sources here.


have fun testing
phoenix

[phoenix]

well i was going to do this anyway so here is my first stage:

http://www.purebytes.de/stuff/celestia/ ... _alpha.zip

this package includes the modified source-code, 2 new dlls (libcurl and zlib1) and a win32-exec from latest CVS including my modifications.
other than win32 you can get tons of precompiled libcurl-builds here

it adds a new config-option in celestia.cfg "RemoteURL" which takes a remote http-directory and then checks for .md5 files
if the md5-checksum differs from the local file it will simply replace it.
it will do the same if the local file is missing.

the server has to store data equal to celestias directory-tree (e.g. data/*).
i've included a small bash-script which can be executed server-side to create corresponding .md5 files in a directory.

any feedback is welcome

phoenix

[phoenix]

hmm thats odd, 100 views of this thread and noone is interested or has even checked this out?
there are tons of possibilities and directions this could go to particularly with regard to addons and translations.

[selden]

You just need to be patient:

More sophisticated URLs are a capability that Addon developers have to consider using and software developers have to consider supporting.

Most people who visit the forum (and thus most who read the postings) are users of Celestia, not developers for it.

[symaski62]

well i was going to do this anyway so here is my first stage:

http://www.purebytes.de/stuff/celestia/ ... alpha2.zip

this package includes the modified source-code, 2 new dlls (libcurl and zlib1) and a win32-exec from latest CVS including my modifications.
other than win32 you can get tons of precompiled libcurl-builds here

it adds a new config-option in celestia.cfg "RemoteURL" which takes a remote http-directory and then checks for .md5 files
if the md5-checksum differs from the local file it will simply replace it.
it will do the same if the local file is missing.

the server has to store data equal to celestias directory-tree (e.g. data/*).
i've included a small bash-script which can be executed server-side to create corresponding .md5 files in a directory.

any feedback is welcome

phoenix

NOTE: this package is build upon the latest CVS but does NOT include any additions made since 1.4.1 (e.g. new shaders)
so you need a working cvs-environment to build or at least the latest cvs-checkout to run the program.

aaahhh celestia.exe "clic" => CVS t?©l?©charger auto

THX !

[Malenfant]

I'm just wondering what an "http-patch" is in the first place... what capabilities would it add to Celestia?

[phoenix]

I'm just wondering what an "http-patch" is in the first place... what capabilities would it add to Celestia?

ok i probably should rename this to networking-patch
it basically adds http and ftp support to celestia.

here is an updated version adding ftp & http-auth support and a new config-option "BindIP" which forces celestia to bind to a specific IP, hostname or network-interface.
look at my example "celestia.cfg" for the 2 new config-options...

http://www.purebytes.de/stuff/celestia/ ... _alpha.zip

if you experience any errors let me know.
if nothing happens or celestia hangs during startup have a look at the "curl.log" (or /tmp/celestia-curl.log for linux) if there are any errors..

greets
phoenix

[Malenfant]

I'm just wondering what an "http-patch" is in the first place... what capabilities would it add to Celestia?

ok i probably should rename this to networking-patch
it basically adds http and ftp support to celestia.

So what would that mean in practice? I thought we could already go to URLs in Celestia with the info-URL capability? And would FTP support add FTP links to Celestia? How would those be useful?

[t00fri]

hmm thats odd, 100 views of this thread and noone is interested or has even checked this out?
there are tons of possibilities and directions this could go to particularly with regard to addons and translations.

phoenix,

you should not be surprised about low reaction from the developers if you don't set out and carefully motivate the usefulness of your http-patch in detail.

Since the core development of celestia is entirely uninterested in game aspects, it seems to be worth asking what it's all supposed to be good for.

Bye Fridger

[phoenix]

alright I should have explained this in more detail. so here we go

the goal of this patch is to optional seperate the celestia-core from its data (stardb / textures / models / addons etc.)

for example:

someone creates a cool new addon with tons of new models, textures and scripts.
if finished, he builds a big package and distributes it on his webserver.
the user now has to download and install the package which often leads to confusion and errors (yes i know it's simple to install addons )

now my patch could simplify this.
the user just has to add 1 line in his celestia.cfg and it will automatically check and download the addon on startup.
furthermore everytime the author changes a script, texture or model on his server celestia will automatically download the new stuff.

and this is not limited to addons.
every part of celestia that is not part of its program-core could thereby be managed on a central server.

this may even be possible for translations (if not correct me please)

i hope this clarify things and the purpose of my patch

thanks
phoenix

[Paolo]

This is a simplified an add-on manager.
This issue was discussed may many times in the past and is really welcome.

But there is a lot to discuss about and it is not so simple like it seems. This kind of project has failed at least one time in the past despite the great effort of some developers.

As far I can know Chris don't like to add more external libraries.

Can you give us more details?

Kind regards

[phoenix]

This is a simplified an add-on manager.
This issue was discussed may many times in the past and is really welcome.

well... yes and no.
I wasn't going for an addon-manager just another option for celestia to fetch it's data-files.
the clou about this patch is that it keeps the current way celestia handles files and addons untouched.
it just adds more ways of retreiving those files.

But there is a lot to discuss about and it is not so simple like it seems.

yes and a lot todo like:
- proxy-support
- local backups
- more config options
- SSL support
- zipped-archives support
- md5sum-lists instead of single .md5-files for every file

As far I can know Chris don't like to add more external libraries.

hmm some feedback if this has the chance to be implemented in the official-release would be great

Can you give us more details?

details about the code or what it can do or will do when it's finished?

bye
phoenix

[symaski62]

FILE ERROR opening data/gliesexindex.dat
MD5 LOCAL: ERROR
CURL LOADING URL: http://test:test123@www.purebytes.de/st ... ex.dat.md5
CURL ERROR: no error
CURL RESPONSE: 404

ERROR 404

[phoenix]

http_error 404 is just a missing file (or page) ...
in that case "gliesexindex.dat" is missing and so is the corresponding md5-file

I don't know why celestia tries to read that file as it's not yet included in current cvs
but that's really nothing to worry about

[phoenix]

new testing-release alpha4 fixed some bugs and added gzip-support.
I will now edit my original post to include every new release

[Joe]

phoenix

Your patch is very useful for data sharing and some degree of collaborative work that helps some projects work for my students. Just want to say "great work" and thanks.

[phoenix]

thanks!
i've just completed another release improving file-handling and downloads.
see my original post for more details.

I really should think about a readme or turorial by now

[phoenix]

next version is finished.
again more speed improvements

[Joe]

It is getting better and better!!! . Definitely a useful tool for real use in group project. My short experience in using this tool convinced me it helps users to share resources and data files effortlessly easy and fast and I love it. Guys, this is really worthy of considering further developing and hope it can become part of Celestia.

[phoenix]

I made a new update which now makes "real" addon support possible.
meaning it can now create directorys under the "extras" folder and supports multiple URLs provided by the celestia.cfg

I also updated my original post with a detailed description of this patch.