Elvish as an embedded scripting language?

[capnspacehook]

I am considering using Elvish as an embedded scripting language in a project, and just wondered if you guys had any tips or thoughts on how you would do that, if it's feasible at all, etc.

I was originally going to use Bash, but due to the fact the internal modules I will be calling will return objects, not text, Bash will not work.

Then I thought about using Elvish. It has support for lists and maps, which I figure I can use to create arbitrary 'objects', and is a shell scripting language.

My questions/concerns are:

• can I add user defined types/functions such that the Elvish interpreter will recognize them, much like you can in otto and gopher-lua?
• what should I remove that is unneeded in a embedded scripting scenario? Things like the file browser, package manager don't seem useful in my use case and would just bloat the compiled binary.
• can I parse input scripts/code server side, generate and AST and serialize and send it over the wire to a client which would deserialize the AST and execute it with the Elvish interpreter?

[xiaq]

In theory, Elvish is already embeddable by the virtue of having a modular design; in your Go code you can use github.com/elves/elvish/parse to parse and github.com/elves/elvish/eval to evaluate Elvish code.

Elvish functions and elvish values are Go functions and values of particular shapes. In fact, the line editor component (github.com/elves/elvish/edit, although a better, but incomplete version is github.com/elves/elvish/newedit) is written exactly as some kind of addon to the core.

The AST is probably serializable, but I have not checked.

In principle, you should be able to just refer to the godoc for the parse and eval packages (https://godoc.elv.sh/parse, https://godoc.elv.sh/eval) as well as the source code of the newedit package to see how Elvish can be embedded. However, the APIs are not very well documented, neither are they stable. Neither is the Elvish language stable.

Nonetheless, your request is a very reasonable one, and ensuring that Elvish is embeddable is beneficial for making sure that the internal architecture of Elvish is modular enough, with good internal APIs. Still, since the stability of internal API as well as the language will only become a goal after the 1.0 release, I encourage you to give it a try, but beware of all the caveats mentioned above.

[capnspacehook]

Perfect, from what I've read of the API from the godoc your answer is what I expected. One additional requirement I forgot to mention is that I would like to parse and compile the input code/script before sending and executing it on the client, but as you said that shouldn't be a problem.

I totally understand that Elvish isn't stable, and can accept that for this project. Just out of curiosity though, do you have any idea/timeline when Elvish will hit v1.0?

[xiaq]

I have written my thoughts in #815.

[xiaq]

@capnspacehook Compiling the Elvish code and sending the compiled code over wire is much harder than sending the AST, as Elvish's compiler actually compiles the AST into Go closures, which are inherently unserializable. You can only use the compiler to do the checking, and re-compile it on the client though.

[capnspacehook]

Hmm ok, that's really good to know. I will have to that then, compile to check on the server, send, recompile and execute on the client.

[capnspacehook]

So it seems the result of parse.AsChunk, parse.Chunk is not serializable, as it has unexported fields that also contain unexported fields etc...

[capnspacehook]

So after taking a deep dive into the source of the parse and eval packages, I've been able to understand and tweak what I needed to to make Elvish work as I need it to. There are some things I changed in my private fork that are specific to my use case, but there are other changes I had to make that are more general I think. So I will keep this issue open to discuss issues I come across when embedding elvish that may affect other users.

That being said, I can't express enough just how well written and designed Elvish is, it's a pleasure to work with, and it's syntax is very intuitive and simple. Thanks again for such a great project.

[xiaq]

Hi - really glad that you like the design!

I would love to learn from your changes. - I try to write the code in a general way, but since all of the packages are really only used in the Elvish program, it's almost unavoidable that some parts get unnecessarily specific. In general, I feel understanding more use cases can be quite helpful for improving the internal design.

[capnspacehook]

No problem, yeah the intuitive syntax and the ability to use lists and maps in a shell scripting context is great.

Yeah there's a few things I know looking back that I changed that could benefit others:

• I created an exported wrapper for eval.addBuiltinFns(), if people are embedding Elvish, a common use case is to add custom builtins/functions to the interpreter.
• I created an exported wrapper for parse.node.setSourceText(), when serializing an Elvish parse tree (parse.Chunk) the sourceText field is needed for tilde nodes only I believe.

There are more changes I made, but they were specific to my use case and would not be good changes to make to Elvish generally.