Embedded support

[kddnewton]

Following up on @hasumikin's #2171 PR, this discussion is around how we want to support embedded devices, particularly ones without a file system.

If I understand correctly, the only real changes we would need to support are in pm_string.{c,h}, which performs all of the IO to set up the string so that the parser can call pm_string_source and pm_string_length. Everything else depends only on standard C99 libraries with no IO.

@hasumikin what else can we do to support this work?

[hasumikin]

Thank you for opening this discussion.

Regarding the #2171 (comment),
If you prefer automatic detection, the _POSIX_MAPPED_FILES macro may be more meaningful.

I made a new branch using _POSIX_MAPPED_FILES: hasumikin/prism@main...prism_has_mmap

If you're happy with this, I'll send you a new PR :)

@hasumikin what else can we do to support this work?

In addition to the MMAP, I want Prism to be able to replace functions relating to heap like malloc() with user-specific ones.
Generally, standard libraries for embedded systems also have malloc().
However, in some cases including PicoRuby, embedded runtimes have special-made memory management functions.

My first idea to replace malloc() looks like:

/* defines.h */
#ifdef USE_CUSTOM_ALLOC
#include "custom_alloc.h"
#else
#    define MALLOC malloc
#    define CALLOC calloc
#    define REALLOC realloc
#    define FREE free
#endif

And replace {m|c|re}alloc in src/**/*.c with {M|C|RE}ALLOC (and free, too)

Then the user can write their own custom_alloc.h like:

/* custom_alloc.h */
#define MALLOC my_malloc
#define CALLOC my_calloc
#define REALLOC my_realloc
#define FREE my_free

Although I am OK with this simple solution, mruby has a more flexible way that malloc() is replaceable in runtime: https://github.com/mruby/mruby/blob/3.2.0/include/mruby.h#L259

What do you think?

[kddnewton]

Hey @hasumikin —

The _POSIX_MAPPED_FILES macro looks great. Please open that PR, I would be happy to merge it.

As for the memory functions, I think that could work in the short-term, but in the long-term I'd prefer that we allow users to define memory functions on the parser itself, much like we have for mruby. But until that time, I'd be okay with a workaround like the one you suggest. The only thing I would do is prefix it with PRISM, as in PRISM_CUSTOM_MEMORY_ALLOCATORS or something.

[hasumikin]

(Created a PR but CI fails. Trying to figure it out...)
#2420

[hasumikin]

@kddnewton
Regarding the memory functions, as a starting point, I wrote a simple version of replacing the alloc interface.
hasumikin/prism@main...hasumikin:prism:custom_allocation_interface

I think this works enough.
If you want the interface to be replaceable runtime, we would do:

• Add function pointer members to struct pm_parser

  /* include/prism/parser.h */
  struct pm_parser {
      (...)
      /**
       * Alloc functions
       */
      struct {
          void *(*malloc)(size_t size);
          void *(*calloc)(size_t number, size_t size);
          void *(*realloc)(void *ptr, size_t newsiz);
          void (*free)(void *ptr);
      } alloc_funcs;
  };

• Add a param pm_parser_t *parser to all the functions that handle the heap

• In defines.h (or somewhere), define macros like #define xmalloc parser->alloc_funcs.malloc

What do you think?

[kddnewton]

I think your branch is good, I don't think we need to be able to do it at runtime. The only change I would make to your branch would be to call it prism_xallocator.h and PRISM_XALLOCATOR.

[kddnewton]

Thank you for your work on this!

[hasumikin]

After merging the PR #2542, I think we may better have a static check of detecting usage of malloc/calloc/realloc/free in src/*c and templates/src/*.c.erb to promote x-prefixed ones

[kddnewton]

That would be a very good idea

[hasumikin]

I want to discuss restricting encodings in terms of reducing the code (ROM) size.
mruby assumes only UTF-8 as its encoding (I talked to Matz and he agrees).
PicoRuby currently accepts only ASCII8BIT while we plan to support UTF-8 in the future.
So, I'd be happy to have macros that work like this:

#if defined(PRISM_ONLY_UTF8)
  // for mruby (and PicoRuby in the future)
  // exclude encoding resources other than UTF-8
#elif defined(PRISM_ONLY_ASCII8BIT)
  // for PicoRuby. mruby may also be here depending on the app
  // exclude even UTF-8
#else
  // for CRuby, JRuby, TruffleRuby, etc.
  // include all
#endif

(Naming PRISM_ONLY_xxx is just a casual idea)

[kddnewton]

Yeah I would be fine with that! I think for naming I would like:

PRISM_ENCODING_UTF_8_ONLY
PRISM_ENCODING_ASCII_8BIT_ONLY

[hasumikin]

They look good 👍
Shall I do the task of introducing conditional build? Or can you work on it when you have time? (I'm myself ready to do it)

[kddnewton]

I'm happy to take a look at it next week, but you're welcome to if you have time before then.

Just for my own education — how are you measuring the ROM size? I would like to know how much this reduces it by, but I don't know how to measure it.

[hasumikin]

I'm happy to take a look at it next week,

Yeah, please do it. I'm focusing on my code generator for now🕵️‍♂️

how are you measuring the ROM size?

size build/libprism.a gives you a table like this:

If text and data are reduced, you successfully made it 👍
The figures themselves are just a reference but I believe they will similarly affect the size of the executable on any build

[hasumikin]

As in, I will possibly want to consider restricting source objects in the build
eg: excluding prettyprint in the production build for microcontroller while including it in the debug build

[kddnewton]

Going to close for now. Please open if there are any other ideas for reducing the size!