Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[udf] Unlock Lua for user-defined functions #669

Open
wants to merge 3 commits into
base: amo/javascript
Choose a base branch
from
Open

Conversation

amotl
Copy link
Member

@amotl amotl commented May 13, 2023

Hi again,

similar to what Webscript has been doing for HTTP, and @progrium explored on behalf of Airscript the other day, this patch explores the feasibility of "using Lua for MQTT scripting".

Thanks to the excellent lupa package by @scoder 💯, the patch is ridiculously small again, similar to the other one GH-667, which brings in JavaScript for the same reasons. Both patches are making mqttwarn a host for other languages, effectively bringing together the MQTT protocol, mqttwarn's transformation engine, and now, more languages than just Python, into a small-scale framework it is so much fun to fiddle with.

I think in the current era of serverless computing and FaaS systems, often operated by large cloud service vendors, it is so important to have open source alternatives around, which, more often than not, can do things more efficiently and conveniently in terms of development efforts, and also don't have several layers of indirection between caller and callee, omitting all the dynamic scaling and throttling aspects usually present in large application container / language host infrastructures, which I found to have a significant cost, both mentally and operationally.

Maybe it works well when it's deployed well, but well, long story short, I prefer personal computing over needing to wrap my head around how to invoke services/programs/functions on 3rd-party infrastructure, which is mostly not completely transparent, or impossible to run on a personal computer at all.

My motivation is mainly but not exclusively driven by my bad experiences in the past when I have been sucked into analyzing subtle errors on shared cloud infrastructure. It's ridiculous, and I am so happy to have learned how to operate my own machines for hosting and publishing digital goods. Enjoy reading! 1

The documentation of the new feature reflects my experiences, and its preview is available here:

With kind regards,
Andreas.

Footnotes

  1. And no offense to the Azure and Kafka authors, engineers and users, I know I am comparing apples with oranges here, you are doing an excellent job, and nobody would want to professionally run Kafka on personal computers anyway ;]. It's all just my personal idea of a small-scale FaaS framework with an adapter to a message bus system - here, MQTT.

@scoder
Copy link

scoder commented May 14, 2023

Might be worth mentioning that Lupa 2.0 is in the making (no release date yet), which provides different Lua implementations in separate modules, i.e. lupa.lua53, lupa.lua54, lupa.luajit21, etc., all having the same API as before. You could then be specific about the Lua version that you want to provide, just with an import. Or, you could let users decide, e.g. by looking at the shebang line of a script. That's probably out of scope for this PR, but worth keeping an eye on for later.

@amotl
Copy link
Member Author

amotl commented May 15, 2023

Hi Stefan,

thanks for the heads up. I had the impression that I was using Lupa 2.0 already. It works very well, thank you so much!

$ pip install lupa
Collecting lupa
  Using cached lupa-2.0.tar.gz (6.3 MB)
$ python -c 'import lupa; print(lupa.__version__)'
2.0

I recognized that possibility to invoke different interpreters already -- very impressive! -- but you are right, I did not look into making this configurable yet, and was happy that Lupa just works out of the box. But, very true, unlocking LuaJIT with a subsequent iteration very much resonates with me for this use case.

With kind regards,
Andreas.

#### Lua

For running user-defined functions code written in Lua, mqttwarn uses the excellent
[lupa] package. For adding JavaScript support to mqttwarn, install it using pip like
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy/paste error. Apologies.

Suggested change
[lupa] package. For adding JavaScript support to mqttwarn, install it using pip like
[lupa] package. For adding Lua support to mqttwarn, install it using pip like

Comment on lines +294 to +337
class LuaJsonAdapter:
"""
Support Lua as if it had its `json` module.

Wasn't able to make Lua's `json` module work, so this provides minimal functionality
instead. It will be injected into the Lua context's global `json` symbol.
"""

@staticmethod
def decode(data):
if data is None:
return None
return json.loads(data)
Copy link
Member Author

@amotl amotl May 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite the same, but I've discovered another monkey-patching around the JSON library at JSPyBridge/Pythonia last night. JFYI.

-- https://github.com/extremeheat/JSPyBridge/blob/1.0.1/src/pythonia/json_patch.py

@@ -200,6 +203,9 @@
"Operating System :: MacOS",
"Operating System :: Microsoft :: Windows",
"Programming Language :: JavaScript",
"Programming Language :: Lua",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is connected to a corresponding request to add it as a new Trove classifier.

@scoder
Copy link

scoder commented May 15, 2023 via email

@amotl amotl force-pushed the amo/javascript branch 2 times, most recently from 3b47f82 to 8cd99f7 Compare January 5, 2024 00:22
Copy link

codecov bot commented Jan 5, 2024

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (8cd99f7) 49.96% compared to head (7d2fc46) 50.12%.

Files Patch % Lines
mqttwarn/util.py 80.95% 4 Missing ⚠️
Additional details and impacted files
@@                Coverage Diff                 @@
##           amo/javascript     #669      +/-   ##
==================================================
+ Coverage           49.96%   50.12%   +0.15%     
==================================================
  Files                  81       81              
  Lines                4051     4072      +21     
==================================================
+ Hits                 2024     2041      +17     
- Misses               2027     2031       +4     
Flag Coverage Δ
unittests 50.12% <80.95%> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants