PLua

Interfacing between Perl and Lua as seamlessly and efficiently as unreasonably possible!

Introduction

Lua (and specifically luajit) is really rather efficient. And it's a safe language. A "scripting" language. And it's not too painful to deal with from C. Consequently, the idea behind PLua is to allow Perl to use tightly-coupled blocks of embedded Lua code to speed up hot code paths.

In other words, you can do this:

use PLua;
bla bla bla Perl code bla bla
my $foo = 1;
lua {
  local bar = $foo.int * 42
  $foo = some_lua_function(bar)
}
bla more Perl using modified $foo here

The details are (likely unsurprisingly) a bit involved.

Since this is an experiment and proof-of-principle quality code base this README is initially (hopefully) going to be mostly a guide to getting started with hacking on PLua itself. That information is likely to move on to a different location in a later stage.

Building

Make sure to have a copy of Lua (likely luajit) compiled and available from a standard include/linking location. Possibly edit Makefile.PL to point at the directory of the location if you use a custom one. Then, it's just the usual:

perl Makefile.PL
make
make test (some tests fail!)

You can enable debug mode with a number of assertions like this:

DEBUG=1 perl Makefile.PL
make
make test

Since I dislike fuzzing with build systems and I am lazy (and the DEBUG=1 is a horrible hack), I usually work like this:

hack hack hack
make clean; DEBUG=1 perl Makefile.PL && make test
rinse, repeat

Which is still fast enough with this code base. Patches welcome (really).

UNDERSTANDING THE IMPLEMENTATION

You'll need at least some basic knowledge of the Perl API in order to make sense of this, I suppose.

PLua uses Perl's keyword plugin mechanism to generate a custom OP (OPs being the basic building blocks of Perl's ugly step-sister of an excuse for an AST).

What happens when you load "Plua"?

• We use an XS BOOT (see perlxs) section to call the bootstrapping function plu_init_global_state (see plu_global_state.h and plu_global_state.c). This in turn will perform the following steps:

• Create a global (I know...) lua_State object / Lua runtime for use by PLua. This lives in PLU_lua_int right now. Will eventually move elsewhere. (threads, reuse, coroutines, etc.)

• Create and register the description of our custom Lua OP (PLU_xop). Also register a hook for freeing OPs of our custom type since they have additional data attached to each instance.

• Register our keyword plugin (see function plu_my_keyword_plugin in plu_parse_kw.h).

• Register global destruction cleanup hook (Perl_call_atexit calling plu_global_state_final_cleanup) for the Lua interpreter and other global state.

What happens at compile time when you do "lua {...}"?

• Perl's tokenizer finds the unknown "lua" keyword. It invokes the keyword plugin mechanism which eventually finds our particular keyword plugin (see above) and calls it. We detect that it's "our" keyword and...

• HACK ALERT! This must change in the future.

  ... use the Perl lexer API to determine the Lua delimiter (one or many {) and then use said API to find the corresponding number of successive (without whitespace) closing braces to end the Lua block. This should be replaced by a proper (extended) Lua parser later.

• MORE HACKS!

  The scanner code is modified by plu_munge_lua_code in plu_lua_syntax_ext.c. This is currently deals with syntax like "$x.int" or "$x = ...". It works by scanning the code with regexes (in Perl, see lib/PLua.pm), then doing all lexical PAD offset lookups for the scalars (back in C using variations on pad_findmy / pad_findmy_pvn), then doing search/replace of the above snippets with things like "perl.val_to_$type($padoffset)".

  In order to close over Perl lexicals properly, this will most certainly have to be amended. Or Lua functions (NOT Lua blocks) closing over lexicals will have to be abandoned in favor of exposing functions directly.

• The modified code is passed to the Lua compiler, which puts a Lua function on the Lua stack (if all goes well), see plu_compile_lua_block_or_croak in plu_lua.c.

• Since it's not legal Lua API to pop functions off the stack for outside-Lua-use and then putting them back on, we keep them within Lua and use the ref mechanism to store it within the lua_State (see Lua's manual on luaL_ref). Then we construct a custom OP to hold all the necessary data for runtime, see plu_prepare_custom_op in plu_op.c

What happens when Perl is about to execute a Lua block?

TODO write about run-time behaviour

How does the Lua table wrapping work?

Lua tables are a kind of bastard child of arrays and dictionaries. Even if I'll be tried for heresy by saying this: Kind of like PHP "arrays", except more powerful (keys can be anything). This does not map perfectly to Perl arrays or hashes. Complete automatic mashalling just isn't going to DWYM entirely, so the design of PLua is to make it easy to get what you want, but not automatic. For this purpose, PLua will convert a table to an instance of the PLua::Table class instead of to a hash or array. This class (implemented in xs/Table.xs and plu_table.{c,h}) allows the user to explicitly convert the table to an array, a hash, or inspect and manipulate it directly.

Due to Lua's object ownership, the implementation of that class is a little interesting. It is not (safely) possible to get the actual table out of the lua_State and the only table inspection/manipulation API from C is to have the table object on the Lua stack and then calling stack-based API functions. If we want to be able to pass around references to such a table in Perl space, then we have to somehow reference it at a distance without having a straight C pointer. When creating the PLua::Table, we use luaL_ref to store a reference to the table in the Lua registry (read up on this in the Lua manual now if that's not familiar to you). To this effect, a PLua::Table is a struct that holds a pointer to the lua_State that the table belongs to, plus an integer indicating the registry index of our stashed reference. Whenever we want to do any operation on the table, we have to first fetch the actual table reference from the Lua registry and put it on the Lua stack. This is done using the PLU_TABLE_PUSH_TO_STACK macro (plu_table.h).

In the destructor of the PLua::Table object, we use luaL_unref to release our reference to the handle, allowing Lua to GC it.

Of course, this all means that PLua::Tables will never travel between lua_States and this could be "interesting" once having multiple Lua interpreters is actually possible.

How does the Lua function wrapping work?

Fundamentally, wrapping a Lua function for Perl works with the same Lua-side tricks as the table wrapping. The code lives in plu_lua_function.{c,h}. On top of that, function wrappers require a number of additional tricks to work:

These function wrappers are actual Perl subroutine references that just so happen to be blessed into the PLua::Function package. This is purely so that the destructor can release the reference to the Lua function that we've stashed (see above). But it gets even more weird!

A very simple way to implement this would be to have a generic "call a Lua function by name or registry index (see above)" function and then to attach the registry index (or conceivably, name) in Perl space using a closure:

# THIS IS NOT HOW IT REALLY WORKS!
sub make_lua_wrapper {
  my ($registry_index, $lua_state) = @_;
  return sub {
    return _invoke_lua_function($lua_state, $registry_index, \@_);
  }
}

This would just create a Perl closure over the necessary data and store & encapsulate it that way. Perl closures are fantastic devices. Alas, this scheme requires three function calls, one being the Perl closure, one being the _call_into_lua XSUB, one being the actual Lua function. PLua is all about performance (and convenience), so that doesn't fly.

Instead, we use a trick inspired by the Class::XSAccessor CPAN module. There is a detailed explanation of the technique in the source comments of that code base. In a nutshell, a Perl CV* (the Perl struct typedef for "Code Values", subroutines) contains a union (cf. XSANY) that is used to implement the XS ALIAS (see perlxs) feature. It can store an integer indicating the XSUB alias to call. It can also store a pointer. The trick is to have one invoker XSUB like the above _invoke_lua_function (in xs/Function.xs) and then we create a new Perl-side subroutine reference of that same XSUB over and over again using newXS. Yes, this means that all Lua function wrappers are really Perl-side references to the same C function. But the Perl-side references are curried by storing a pointer to a plu_function_t struct in the XSANY.any_ptr which holds the information about the Lua function to call. Et voila, we've done away with that pesky slow Perl closure. Queue maniacal laughter.

The rest is just marshalling data back and forth.