Add support for incorporating (some) Lua Standard libraries as builtins

[tcoram]

There are some standard libraries I miss when programming in Pallene. An example would be string.byte(). Other standard Lua functions, in os, io, math (sin/cos/etc) and string, would also be useful for system programming. While full support for purely variadic functions may be problematic (and defeat the type optimization of Pallene), support for non-variadic forms of the standard functions (ideally as first class functions) in Pallene would be extremely useful.

[srijan-paul]

I'd love this!
I remember wanting to integrate pallene into my C++ game engine and being deterred by this, and the inability to interop with other Lua libraries (at the time).

I wonder if there's a proposal to move ahead in this direction.

[tcoram]

An approach to doing this was outlined in #337

TBH, for my needs, I don't really need to manipulate the builtins as higher order functions. It's tempting, but upon further thought I am struggling with what I need from the standard libraries that have a fixed arity (or is usable with just a fixed arity subset).

There are a handful of select standard library functions (e.g. string.byte() and other string functions, maybe some io) that I want to use inside of Pallene functions.

Right now, I carefully hand add them (to the Pallene source code), but that now means I've forked Pallene.

Alternately, I can do a lot of boiler plate code (that, aside from being laboriously written boiler plate) and be less efficient (10x in this case), such as (this contrived example):
As a builtin:

function u.is_binary_builtin (s: string) : boolean
   for i=1,#s do
      local byte = string.byte(s,i)
      if byte ~= 48 and byte ~= 49 then return false end
   end
   return true
end

As string.byte passed in (via u.register_string_bye) from Lua:

typealias String_Byte_Func = (string, integer) -> integer
local string_byte : String_Byte_Func = function (s,i) return 0 end  -- place holder dummy function

function u.register_string_byte(f : String_Byte_Func)
   string_byte = f
end

function u.is_binary_str (s : string) : boolean
   for i=1,#s do
      local byte = string_byte(s,i)
      if byte ~= 48 and byte ~= 49 then return false end
   end
   return true
end

Or, is there a cleaner way to pass in Lua functions and store them in a variable?

[srijan-paul]

Ah yes, I remember #337.
It seems possible to implement (using LClosures when passing builtins as HOFs).
Perhaps you can hash this out with @hugomg, and get it merged into the pallene repo.

[hugomg]

Hi Todd, that idea with the "register string byte" is close to what I had in mind. The difference is that instead of writing high-level Pallene code, we could do everything under the hood, inside the Pallene compiler.

In the step where we create the intermediate representation, we could insert one local variable for every builtin. Then, during module initialization, we initialize that variable with the closure object for that function, which we require from Lua.

Because we have the closures, we would get "higher order builtins" for free. The reason current builtins cannot be used higher order is because we never create a function for them. They are always "inlined".

[tcoram]

Ah, I get it. I am getting a better idea of how to accomplish such a feature...

But, I have ask: Would we get better performance than what I did in the code snippet above? Or about the same.

Just for reference, 100,000,000 iterations of calling the above code (with an argument of "001100111000") yields:

Plain lua w/ std Lua string.byte:	395.573 seconds
Pallene w/ registered string_byte:	264.425 seconds
Pallene w/   inline string.byte:	 26.552 seconds

Once of the places I want to use Pallene is to encode/decode (network) protocol packets as fast as possible. Higher order builtins aren't as important as raw speed in this case. While I'd love to have more Lua standard library facilities at my disposal, there are still a few things (e.g. low level string stuff like string.char & string.byte, io write/read, math, etc) that as "inline" functions make a compelling reason to use Pallene in place of C.

[hugomg]

For something particularly simple such as string.byte, it might make sense to code it as an inlined builtin, the same way that we code the current builtins.

I just think we need to be judicious. For example, take a look at gen_cmd["Builtin and the helper functions in pallenelib.lua. Some of them, like the math operations, are pretty simple. But some of them, like tostring, are probably too complicated to be worth maintaining their reimplementation in Pallene. I expect that in the latter case it would make more sense to grab the function from the Lua library.

[tcoram]

Agreed!

But what about functions with variable arity? I'm struggling a bit with string.byte() as a "builtin". I chose the most flexible/efficient /useful-for-me format (2 parameters ): string.byte(s,idx). If a standard library function is "grabbed", then a fixed arity form will need to specified, right? Will the user be able to choose (via a typealias maybe), or will Pallene need to make that decision?

Also (an aside): as I write tests for string.byte(s,idx), I realize that the Lua standard lib version will return nil (not raise an error) on bad indexing or empty strings! This is odd to me, but makes me think that maybe other standard functions can return nil as a result instead of string/number/table??? Or maybe string.byte() is unusual, for a standard library function...

[hugomg]

We would like to support optional parameters / overloading in the future. In the meantime, we have to pick a single arity to support. For math.log, we created a temporary math.ln alternative to stand for the 1-argument variant.

Nil is fact of life in Pallene :) Pallene must to do run-time checks to deal with those. For example, see how we deal with array indexing arr[i], which also gives nil for out of bounds.