Adding Lua Hooks to Celestia

[hank]

I've been experimenting with a scheme for hooking Lua code into Celestia at strategic points. The goal is a mechanism that would allow experimenters and add-on developers to add new features to Celestia using Lua without having to modify the source code and recompile the application.

Here's a brief explanation of what I've done: (Some familiarity with Celestia internals and Lua is assumed.)

I added a new (optional) entry called LuaHook to the Celestia configuration file. It specifies the path to a file containing a chunk of Lua code that is loaded and run when Celestia starts up. This Lua code calls a new method called celestia:setluahook which I added ito celx. The argument to this method is a Lua object (a table of method functions) which will have its methods called by Celestia at strategic points.

I added a LuaState* variable called luaHook to CelestiaCore which is used to call Lua. (This is separate from the LuaState variable celxScript used for end-user scripting.) The luaHook variable is initially nil. If a LuaHook startup file is specified in the Celestia configuration file, then a LuaState is created and assigned to luaHook, and the specified file is loaded and run there. This is done in a new method added to CelestiaCore called initLuaHook.

After luaHook is initialized, the methods of the Lua object which was set in the LuaHook startup file using celestia:setluahook() can be called from CelestiaCore using a new method added to LuaState called callLuaHook(). Obviously calls to this method are made only if luaHook is not nil.

For example, to experiment with custom info overlays, I added this call to the draw method of CelestiaCore:

Code: Select all

   if (luaHook) then luaHook->callLuaHook(this,"luaoverlay");


Similar calls are easily added at other points in the Celestia code. For example, to allow the Lua object to intercept keyboard input I added this call to the charentered method of CelestiaCore:

Code: Select all

   if (luaHook && luaHook->callLuaHook(this,"luacharentered", c))
         return;


Note that the callLuaHook method is overloaded to accomodate passing arguments to the Lua method. Also note that a reference to the CelestiaCore object is included in the call, so that Lua hooks can be associated with Celestia objects other than the CeletiaCore.

That's the basic hookup mechanism. I've also made some additions to celx and enabled loadlib for dynamic libraries. More details will follow.

- Hank

[hank]

Once you've built a version of Celestia with the Lua hook mechanism described in my previous post, you can create a customized version of Celestia using Lua.

First, you would add the following line to your celestia.cfg file:

Code: Select all

   LuaHook "luahooktest.lua"


Then you would create the file luahooktest.lua in your Celestia resources directory. It could look like this:

Code: Select all

hooktest = {};

hooktest.luaoverlay =
   function(self)
      local time = celestia:gettime();
      celestia:flash("Time "..time,1);
   end;

celestia:setluahook(hooktest);


In this very simple example, a Lua object (table) called hooktest is created and a method luaoverylay is defined for it. Then the object is attached to Celestia with a call to celestia:setluahook. The luaoverlay method of the hooktest object will be called from the CelestiaCore draw method each time Celestia redraws the display. The luaoverlay method gets the current simulation time and flashes it on the screen (nominally for one second, but actually the message is updated with a new time almost instantly by the next call to luaoverlay).

Keep in mind that this is a very simple example. The luaoverlay method is not limited to using celestia:flash, but can in principle draw any graphics that are possible with OpenGL, and can be defined in any way that can be programmed using Lua. As one example, the luaoverlay method could determine the current orientation of the Celestia observer, and display a graphical compass showing the current compass heading. The possibilities are endless.

- Hank

[hank]

I previously mentioned the celestia:setluahook method I added to celx.cpp that is used to install the Lua object that handles hook calls. Here's what it looks like:

Code: Select all

static int celestia_setluahook(lua_State* l)
{
    checkArgs(l, 2, 2, "One argument required for celestia:setluahook");
    CelestiaCore* appCore = this_celestia(l);
       
    if (!lua_istable(l, 2) && !lua_isnil(l, 2))
    {
        doError(l, "Argument for celestia:setluahook must be a table or nil");
    }
   
    lua_pushlightuserdata(l, appCore);
    lua_pushvalue(l, -2);
    lua_settable(l, LUA_REGISTRYINDEX);
    return 0;
}


This method stores the hook handler object in the Lua registry, indexed by a lightuserdata which is a pointer to the CelestiaCore object. This pointer is subsequently used to retrieve the hook handler object in order to call one of its methods. That is done in the callLuaHook method added to the LuaState class (also in celx.cpp), which looks like this:

Code: Select all

bool LuaState::callLuaHook(const void* obj, const char* method)
{
    lua_pushlightuserdata(costate, obj);
   lua_gettable(costate, LUA_REGISTRYINDEX);
    if (!lua_istable(costate, -1))
    {
        cerr << "Missing Lua hook object";
        lua_pop(costate, 1);
        return false;
    }
    bool handled = false;   

    lua_pushstring(costate, method);
   lua_gettable(costate, -2);
    if (lua_isfunction(costate, -1))
    {
        lua_pushvalue(costate, -2);          // push the Lua object the stack
        lua_remove(costate, -3);        // remove the Lua object from the stack
               
        timeout = getTime() + 1.0;
        if (lua_pcall(costate, 1, 1, 0) != 0)
        {
            cerr << "Error while executing Lua Hook: " << lua_tostring(costate, -1) << "\n";
        }
        else
        {
           handled = lua_toboolean(costate, -1) == 1 ? true : false;
        }
        lua_pop(costate, 1);             // pop the return value
    }
    else
    {
        lua_pop(costate, 2);
    }
   
    return handled;
}


This method retrieves the hook handler object from the Lua registry, using a pointer to the CelestiaCore object as a lightuserdata for its index, and then calls the handler object method with the indicated name. This can be used to call any hook method which has no arguments. The overloaded versions that are used for hook methods with arguments are similar, but include code to move the arguments onto the Lua stack before the method is called. Here's an example for a hook that has a single char argument (e.g. luacharentered):

Code: Select all

bool LuaState::callLuaHook(const void* obj, const char* method, const char ch)
{
    lua_pushlightuserdata(costate, obj);
   lua_gettable(costate, LUA_REGISTRYINDEX);
    if (!lua_istable(costate, -1))
    {
        cerr << "Missing Lua hook object";
        lua_pop(costate, 1);
        return false;
    }
    bool handled = false;   

    lua_pushstring(costate, method);
   lua_gettable(costate, -2);
    if (lua_isfunction(costate, -1))
    {
        lua_pushvalue(costate, -2);          // push the Lua object onto the stack
        lua_remove(costate, -3);        // remove the Lua object from the stack

        lua_pushlstring(costate, &ch, 1);          // push the char onto the stack
               
        timeout = getTime() + 1.0;
        if (lua_pcall(costate, 2, 1, 0) != 0)
        {
            cerr << "Error while executing Lua Hook: " << lua_tostring(costate, -1) << "\n";
        }
        else
        {
           handled = lua_toboolean(costate, -1) == 1 ? true : false;
        }
        lua_pop(costate, 1);             // pop the return value
    }
    else
    {
        lua_pop(costate, 2);
    }
   
    return handled;
}


There's really not a lot to this. It's similar to how the new celx script event handlers work, except that the method table is indexed by the CelestiaCore object pointer and the arguments are passed directly rather than in a table. Also, the hook methods are defined as part of an object, rather than being set with individual calls. This approach leverages Lua's object-oriented programming capabilities.

- Hank

[chris]

Hank,

I really like the overlay hook, but the charentered hook seems like it just duplicates functionality of the event handlers. Is there a good reason to have both?

The overlay hook needs a good set of drawing commands--something much better than flash() for text. Some basic commands to draw text, lines, filled polygons, and textured rectangles would suffice as a start.

--Chris

[hank]

I really like the overlay hook, but the charentered hook seems like it just duplicates functionality of the event handlers. Is there a good reason to have both?

The event handlers work with end user scripts, which run in the foreground. The Lua hooks described here would run in the background. They are intended as a custom application creation tool, rather than an end user scripting facility. The two are similar in that they use the same LuaState and celx infrastructure, but they run in different LuaState objects. End user scripts are terminated by the escape key and when a new script is started, whereas the Lua hook state is persistent.

The overlay hook needs a good set of drawing commands--something much better than flash() for text. Some basic commands to draw text, lines, filled polygons, and textured rectangles would suffice as a start.

I used flash() just to keep the example simple and focused on the hook mechanism itself. Obviously a more capable graphics api is needed. A simple starter api won't be difficult to write. But the key point is that the high-level apis can be written in Lua. They doesn't have to be hard-coded in Celestia. There can be more than one graphics api, with different trade-offs between simplicity and functionality, all the way to a full-blown gui. Various possible designs can be prototyped, and users aren't locked-in to one design. I'll say more about that in a later post. But I'll just mention that my preliminary custom overlay experiments have included draggable windows with colored borders and background fills, background images, formatted text, multiple subwindows, etc. So I think some pretty flashy stuff will be possible.

- Hank

[hank]

I've described celestia:setluahook() and LuaState::callLuaHook() which are the main additions to celx.cpp needed to implement the basic hook mechanism. The other main addition is CelestiaCore::initLuaHook() in celetiacore.cpp which handles the initialization of the hook mechanism. It looks like this:

Code: Select all

bool CelestiaCore::initLuaHook()
{
    if (config->luaHook == "")
       return false;
    else
    {
        string filename = config->luaHook;
        ifstream scriptfile(filename.c_str());
        if (!scriptfile.good())
        {
           char errMsg[1024];
           sprintf(errMsg, "Error opening LuaHook '%s'",  filename.c_str());
         if (alerter != NULL)
                alerter->fatalError(errMsg);
           else
                flash(errMsg);
        }
   
        luaHook = new LuaState();
        luaHook->init(this);

   string LuaPath = "?.lua;lua/?.lua;";
   
    // Find the path for lua files in the extras directories
    {
        for (vector<string>::const_iterator iter = config->extrasDirs.begin();
             iter != config->extrasDirs.end(); iter++)
        {
            if (*iter != "")
            {
                Directory* dir = OpenDirectory(*iter);

                LuaPathFinder loader("");
                loader.pushDir(*iter);
                dir->enumFiles(loader, true);
            LuaPath += loader.luaPath;      

                delete dir;
            }
        }
    }      
   luaHook->setLuaPath( LuaPath );

        int status = luaHook->loadScript(scriptfile, filename);
        if (status != 0)
        {
         cout << "lua hook load failed\n";
            string errMsg = luaHook->getErrorMessage();
            if (errMsg.empty())
                errMsg = "Unknown error loading hook script";
            if (alerter != NULL)
                alerter->fatalError(errMsg);
            else
                flash(errMsg);
            delete luaHook;
            luaHook = NULL;
        }
        else
        {
            // Coroutine execution; control may be transferred between the
            // script and Celestia's event loop
            if (!luaHook->createThread())
            {
                const char* errMsg = "Script coroutine initialization failed";
         cout << "hook thread failed\n";
                if (alerter != NULL)
                    alerter->fatalError(errMsg);
                else
                    flash(errMsg);
                delete luaHook;
                luaHook = NULL;
            }
         if (luaHook)
         {
         while ( !luaHook->tick(0.1) ) ;
         }
        }
    }
   return true;
}

This method gets the path to the Lua initialization file from the configuration info, creates and initializes the luaHook LuaState object, and loads and runs the initialization file.

I included code here to set the places where Lua searches for modules to load when the Lua "require" function is used, to include any subdirectories of the Extras directories listed in the configuration file that contain any .lua files. (This should probably be done for celx scripts also.) A special subclass of EnumFilesHandler was needed for the directory search. It looks like this:

Code: Select all

class LuaPathFinder : public EnumFilesHandler
{
 public:
   string luaPath;
    LuaPathFinder(string s) : luaPath(s) {};
   string lastPath;

    bool process(const string& filename)
    {
   if ( getPath() != lastPath )
   {
   lastPath = getPath();
    int extPos = filename.rfind('.');
    if (extPos != (int)string::npos)
   {
      string ext = string(filename, extPos, filename.length() - extPos + 1);
      if (ext == ".lua")
      {
            string newPatt = getPath()+"/?.lua;";
             extPos = luaPath.rfind(newPatt);
            if (extPos < 0)
            {            
               luaPath = luaPath + newPatt;
            }
         }
      }
   }
        return true;
    };
};


This also makes use of a new LuaState method setLuaPath, which looks like this:

Code: Select all

void LuaState::setLuaPath(const string& s)
{
    lua_pushstring(state, "LUA_PATH");
    lua_pushstring(state, s.c_str());
    lua_settable(state, LUA_GLOBALSINDEX);
}


I'm currently calling initLuaHook() at the end of initRenderer(), but it probably should be done just after the configuration file is read, to allow hooks during the Celestia setup process.

That pretty much covers the code for the Lua hookup mechanism. (I didn't include the addition of the LuaHook entry in the configuration file, but that addition is completely conventional.)

Later I'll describe some other additions I made to celx.cpp which aren't part of the hook mechanism per se.

- Hank

[phoenix]

hank, could you make a lua-patch available so that I can start experimenting with the code and get familiar with lua

[hank]

hank, could you make a lua-patch available so that I can start experimenting with the code and get familiar with lua 8)

I don't have a patch ready yet, but you can start getting familiar with Lua using .celx scripts.

- Hank

[Vincent]

Hank,

When do you think your patch will be available, so as we can start playing a little bit with our new toy ?

[Malenfant]

Do you have a plain english explanation and examples for what this would allow people to do, Hank? All that code is meaningless to me.

[Cham]

I really don't see the usefullness of that thing, from an average user point of view. To be honest, I'm very sceptic about all this.

[phoenix]

I really don't see the usefullness of that thing, from an average user point of view. To be honest, I'm very sceptic about all this.

because this is for developers only to be able to create addons to celestia without having to recode the main core.

[Cham]

because this is for developers only to be able to create addons to celestia without having to recode the main core.

Then my personal opinion is this : Don't add it to the default Celestia package. Make an hack or a patch if you wish, but keep the official Celestia PURE. I will vote NO for this.

[selden]

The descriptions and coding examples that Hank is providing are for people who will be developing Lua extensions to Celestia. Those developers need to already understand the paradigm used by object oriented languages like C++, Java, C# and Lua. This isn't your father's programming language. It is nothing like Fortran or Basic.

Most people, including Addon developers, will use the results of those Lua extensions. The interfaces to those extensions will be (had better be!) much simpler than what Hank needs to describe.

[phoenix]

because this is for developers only to be able to create addons to celestia without having to recode the main core.

Then my personal opinion is this : Don't add it to the default Celestia package. Make an hack or a patch if you wish, but keep the official Celestia PURE. I will vote NO for this.

I think you misunderstood something here.
this will make it possible for all special versions of celestia to merge with the current version.

for example the sound-patch, realtime-interface or even my http-patch could then be implementet as an "addon" by loading and controlling it via lua.

currently there are a lot of enhanced celestia-versions out there that will never make it into the core celestia.

[ElChristou]

For the average user it's not the time to interfere here; let's the dev discuss the topic and wait for some more tangible application...

[Cham]

If those patched versions didn't made it into the official version, it's for a good reason. It's all about HACKS. It's not the right way to do it. Again, I'm completelly against this thing. It will make Celestia become a mess !

[phoenix]

It's all about HACKS.

sorry, i really don't know what you mean.
there are a lot of nice improvements for celestia in form of a patched version of it.

so if someone would create a patched-celestia which includes at least half of the improvements of your wishlist would you also see this as a hack and refuse to use it?

and exactly what is "the right way to do it" ?

[hank]

Hank,

When do you think your patch will be available, so as we can start playing a little bit with our new toy ? :wink:

Soon. (Today or tomorrow, I hope.)

Do you have a plain english explanation and examples for what this would allow people to do, Hank? All that code is meaningless to me.

Don't worry about that for now. This discussion is intended for developers and experimenters who can work with Lua, C++ and Celestia internals.

I really don't see the usefullness of that thing, from an average user point of view. To be honest, I'm very sceptic about all this.

Please don't judge prematurely. If it works out, I think you'll recognize the benefits.

because this is for developers only to be able to create addons to celestia without having to recode the main core.

Exactly right. I think most experienced programmers will get this.

Then my personal opinion is this : Don't add it to the default Celestia package. Make an hack or a patch if you wish, but keep the official Celestia PURE. I will vote NO for this. :evil:

If there's no LuaHook entry in the celestia.cfg file (and there won't be one, by default) then Celestia will run essentially unchanged.

The interfaces to those extensions will be (had better be!) much simpler than what Hank needs to describe.

If they aren't simple to use, they simply won't be used, at least by most end users. It'll be up to the addon developers to provide a suitably simple external interface.

Normal end users shouldn't really need to know anything about what I've described so far. This is very low-level stuff. Even many Lua addon developers wouldn't need to know a lot of these details. At this point what I'm looking for is technical feedback on the design and code presented here.

- Hank

[t00fri]

Just go ahead, Hank .

It's quite interesting stuff! And indeed, this is the Development board and not the Educational one ...

Bye Fridger