[meta] HXLm.lisp and/or related strategies for portable 'Turing complete' HDP custom functions

[fititnt]

Related topics:

• [meta] HDP Declarative Programming (working draft) [meta] HDP Declarative Programming (working draft) #16
  • The HDP YAML (+JSON) syntax have potential to, at minimum, be an way to humans document (without leak access keys and passwords, yet with minimum viability to allow be parsed by programs) how to access datasets
    • The urnresolver: Uniform Resource Names - URN Resolver urnresolver: Uniform Resource Names - URN Resolver #13 could aid the part of how to point to the resources themselves
    • The way to express Acceptable Use Policies (ones that could be enforced by tools) still not defined, because this need be good enough to allow localization in natural languages supported by the drafted HDP
• hxl-yml-spec-to-hxl-json-spec: HXL Data processing specs exporter hxl-yml-spec-to-hxl-json-spec: HXL Data processing specs exporter #14
  • The HXL Data processing specs is known to work in production for years. The official wiki says the underlining usage is focused for coders, but the HXL-Proxy already provides a front-end for end users.
    • At bare minimum any syntax should be able to also be transpiled to an to JSON processing specs for HXL data so it could run on existing HXL-proxy (and command line tools) with the same stability of tools already working in the past.
      • Note: Lisp-like syntax is even less user-friendly than YAML syntactic sugar to JSON processing specs for HXL data, so this approach is not expected to replace the possibility of users doing without Lisp-like syntax.
  • Note: as this comment hxl-yml-spec-to-hxl-json-spec: HXL Data processing specs exporter #14 (comment) an early version, not integrated on HXLm.HDP, but accessible via hdpcli, already transpile YAML structure to HXL Data processing specs, supported by libhxl cli tools and the HXL-proxy. The notable exception is the inline to use as input (so instead of giving an external URL, the processing spec would already have the input data) and the inline output (that could be used to test if the filter was validated).
    • But if we manage to let users, before pass data to libhxl-python cli tool or the HXL-proxy to save the input, like on a local file or implement the HXLm.core.io to allow even write google spreadsheets, as hard as this extra step may be, it would allow validation of entire HXL-like systems, like if it was with more close to real data and meet requirements of an 'design-by-contract programming' testing the full chain of components, not just the internals of the HXL data processing implementation
• [meta] Internationalization and localization (i18n, l10n) and internal working vocabulary [meta] Internationalization and localization (i18n, l10n) and internal working vocabulary #15
  • An proof of concept the HDP already support 6 UN working languages plus Latin and Portuguese.
    • The early prototypes actually were faster to make than core refactoring to an more OOP approach with strong focus on automated testing for internals. But the point here is that the HDP yaml files already support several natural languages and to add new ones , do not need to know internals, just change some files on hxlm/ontologia
  • While the same supported languages do not need to be both for the high level HDP yaml keywords (ideally, since is less terms, that should be done first!) the pertinence here is, whatever would be an language syntax for create HDP macro/extensions/plugins, this must be done already optimizing to to provide an Source-to-source compiler
    • In other words: even if we decide to implement keywords using English/Latin, as soon as do exist help to add new natural languages to knowledge graph, this should allow source to source compiling. This means that the syntax should be "L10N friendly" from start

Related concepts:

• Concept: Spreadsheets functions (both the part that is viable localize the key terms and also the need to be able to have some sort of Turing completeness to make them, even if direct access to file system or network we do not allow)
  • Google Sheets, several languages options: https://support.google.com/docs/answer/58515#zippy=%2Cchange-the-language-for-functions
  • Microsoft Excel, several language options: https://support.microsoft.com/en-us/office/supported-languages-79be5ebe-9130-455e-b019-e3ce99367bb5
• Concept: design-by-contract programming / Design by contract (DbC) https://en.wikipedia.org/wiki/Design_by_contract
• Concept: International programming languages
  • https://en.wikipedia.org/wiki/Non-English-based_programming_languages#International_programming_languages

---

About this topic

This topic is a draft. It will be referenced on specific commits and other discussions.

But to say upfront that as much as possible, the idea here is keep as much as possible documents that could be used by decision makers to authorize usage and/or for people who document that datasets do exist (even if they do not say on the document how to find them) and, for what is not feasible already have via the underlining python implementation, allow customization.

Note that these customization, while not explicitly sandboxes (but could be) do not need to be allowed to have direct disk or network access. This approach is not just more safe, also open room for they be more reusable and (this is very important!) simplify documentation on how to use, even by individuals who would not speak the same language.

[fititnt]

I believe that, at least at this moment for the JavaScript part of HDP, we use prefix HDPb instead of HDP

Maybe this never get implemented, but Lisp dialects have the tradition to be the easiest ones to allow Self-hosting (compilers). I'm not saying that the entire HDP implementaton in python should be written in an lisp dialect that is not even formalized, but at least the javascript part could have this b as a hint for Bootstrapping (compilers)

[fititnt]

Ok. While JavaScript since the beginning support UTF-16, often we will deal with more than 2 bytes per char. So it's a bit different the loops.

[fititnt]

In the moment, we already drafted a very primitive HDPLisp with JavaScript on hxlm-js/bootstrapper/lisp. It's does not have automated translation like [meta] HDP Declarative Programming (working draft) #16 but is better than the drafted python version.

But then, there is a new underlining host language

Prototype of HDPLisp using Racket

I believe we could take at least two weeks just to get some early feedback on the language syntax. (see https://racket-lang.org/)

So, why do another host language (beyond Python and JavaScript, the ones intended to be used in production for typical end users)? For two reasons:

1. Some old friends at my university are granted to learn Racket and even know a bit of DrRacket platform.
   • Most of them either have strong computer science or computer engineering related background, so they at least can give quick feedback.
   • Some of them actually know at least one natural language in non-Latin script. Better than me.
2. Racket does have a good platform to test new languages and do it quickly.
   • Deployment to production is not that friendly
     • That's why even if we bootstrap another place to run HDPLisp, is more because early users Feedback
     • The default way to create users interfaces with Racket is to enforce use of GPL license. This is also one why, while I'm not against people releasing quick GUIs, would be preferable have the production ready JavaScript

About keep updated the Racket version

Even if we literally use the HDPLisp on Racket to do advanced tests on the JavaScript (and the on the Python versions) I'm not sure if I myself will keep the Racket version updated and definely some parts (like care about isolation of unsafe code) may not implemented even on Lisp version.

The idea of HDPLisp also I'd about be translatable in several natural languages and provide advanced help for who uses IDEs (like code assistance on some browser editing or my preferred VSCode). So somewhat both versions in JavaScript and Python would use ontologies to understand how each code translates to another.

This approach may actually not be necessary on Racket because if people really care about having code aid, it could be simpler to just add a single file manually crafted for each language. Also, even if later someone using Racket want to import HDPLisp files, the person very likely would want to use HDPLisp with other Lisps, so trying to give time to "isolate code" makes less sense.

Note that both JavaScript and Python versions tend to be used as part of bigger applications and with users that will trust the software to check integrity of files. They're more likely to be a script import or an extra command like your jupyter Notebook importing pandas or numpy (but in general HDP, the initial target use for HDPLisp, requires some way to validate authenticity as core functionality).

Also, another strong reason to potentially not suggest usage of DrRacket (the IDE, not the language) is (on my tests) the lack of support for Right-to-Left scrips. So, even if the platform is very friendly for usage, since both HDP and HDPLisp are very strong on the path of internationalization/localization, recommend the usage of an interface that do not even allow change the text direction of a file would be against this.