[WIP] OCaml 5.0 support #122

kit-ty-kate · 2022-10-13T11:38:38Z

~~For now this branch will only work with ocaml-src pointing to https://github.com/kit-ty-kate/ocaml/tree/solo5-500 (all changes are waiting to be backported to the 5.0 branch)~~ EDIT: ocaml-src from opam-repository works fine now.

[copying my first comment for a more visible disclaimer]
For the ocaml-solo5 devs: Feel free to ask questions, rebase the branch, add changes to this branch (you should have the rights to, though don't forget to use git push --force-with-lease if you force-push in case i also add some changes at the same time). I might have not time to finish the work so please do continue it.

joint work with @TheLortex at the MirageOS retreat 2022 with some help of all the people present at the event.

kit-ty-kate · 2022-10-13T11:51:48Z

For the ocaml-solo5 devs: Feel free to ask questions, rebase the branch, add changes to this branch (you should have the rights to, though don't forget to use git push --force-with-lease if you force-push in case i also add some changes at the same time)

haesbaert

I notice some minor things and had some questions, also wrote a mini strerror_r.

ocaml-solo5.opam

nolibc/include/stdio.h

haesbaert · 2022-10-13T12:08:27Z

nolibc/include/stdatomic.h

+     *OBJ = *OBJ | ARG; \
+     tmp; \
+  })
+


Could be easier to just call into the compiler atomics (__atomic_bla) no ?

nolibc/include/pthread.h

haesbaert · 2022-10-13T12:24:01Z

nolibc/stubs.c

+STUB_ABORT(pthread_cond_broadcast);
+STUB_ABORT(pthread_self);
+STUB_ABORT(pthread_detach);
+STUB_ABORT(sigfillset);


This could probably be STUB_IGNORE, we'll never have signals.

Indeed it can be STUB_IGNORE, during the small experiences I've done so far it doesn't seems to be an issue as I never saw any abort exception raised. Could it worth to keep it as STUB_ABORT and wait for the time an unikernel tells us when we need to add signal to solo5? :)

haesbaert · 2022-10-13T12:30:09Z

nolibc/stubs.c

+STUB_ABORT(pthread_self);
+STUB_ABORT(pthread_detach);
+STUB_ABORT(sigfillset);
+STUB_ABORT(usleep);


This would be something like calling solo5_yield(deadline, emptyset)

I agree, we currently can sleep with Solo5 since we have the monotonic clock (it's just matter that we will block the only CPU we have).

haesbaert · 2022-10-13T13:04:41Z

nolibc/stubs.c

+STUB_ABORT(pthread_detach);
+STUB_ABORT(sigfillset);
+STUB_ABORT(usleep);
+STUB_ABORT(strerror_r);


Here is one:

#include <assert.h> #include <errno.h> #include <stdio.h> #include <string.h> int mystrerror_r(int num, char *buf, size_t buflen) { static const char *errlist[] = { "Undefined error: 0", /* 0 - ENOERROR */ "Bad file descriptor", /* 1 - EBADF */ "Result too large", /* 2 - ERANGE */ "Function not implemented", /* 3 - ENOSYS */ "Value too large to be stored in data type", /* 4 - OVERFLOW */ "No such file or directory", /* 5 - ENOENT */ "Invalid argument", /* 6 - EINVAL */ "Cannot allocate memory", /* 7 - ENOMEM */ "Too many open files", /* 8 - EMFILE */ "Device busy", /* 9 - EBUSY */ }; int nerr; nerr = sizeof(errlist) / sizeof(errlist[0]); if (num >= nerr || num < 0) { errno = EINVAL; return (EINVAL); } if (snprintf(buf, buflen, "%s", errlist[num]) >= buflen) { errno = ERANGE; return (ERANGE); } return (0); } int main(void) { char buf[256]; int i; assert(mystrerror_r(0, buf, strlen("Undefined error: 0")) == ERANGE); assert(mystrerror_r(-1, buf, sizeof(buf)) == EINVAL); assert(mystrerror_r(10, buf, sizeof(buf)) == EINVAL); for (i = 0; i < 10; i++) { assert (mystrerror_r(i, buf, sizeof(buf)) == 0); printf("strerror_r[%d] = %s\n", i, buf); } return (0); }

sam:tmp: cc -o strerror strerror.c -Wall && ./strerror strerror_r[0] = Undefined error: 0 strerror_r[1] = Bad file descriptor strerror_r[2] = Result too large strerror_r[3] = Function not implemented strerror_r[4] = Value too large to be stored in data type strerror_r[5] = No such file or directory strerror_r[6] = Invalid argument strerror_r[7] = Cannot allocate memory strerror_r[8] = Too many open files strerror_r[9] = Device busy

oh forgot to store errno on the error cases, fixed.

Thank you! I agree it should be added, but as for sigfillset, if no exception raised, maybe it's not needed?

haesbaert · 2022-12-17T15:06:09Z

I think the debugging is imprecise, this is pretty much just after the first deference of s: s->unique_id = fresh_domain_unique_id(); s is probably screwed up, which means &d->interruptor is bad. Don't be tricked by the CAMLassert before, I think that is not compiled in.

I can't really see how &d->interruptor would be bad, but I'd try to dereference s before your call just to make sure the problem is that.

palainp · 2022-12-18T08:39:19Z

Here are some printing (sorry, this will be a bit long :/), we probably segfault before accessing s and gdb seems to misalign the code and the assembly:

$ gdb -q --args solo5-spt dist/hello.spt
Reading symbols from solo5-spt...
(gdb) add-symbol-file dist/hello.spt 0x100000
add symbol table from file "dist/hello.spt" at
	.text_addr = 0x100000
(y or n) y
Reading symbols from dist/hello.spt...
(gdb) r
Starting program: /home/user/.opam/5.0.0/bin/solo5-spt dist/hello.spt
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000

Program received signal SIGSEGV, Segmentation fault.
0x00000000001e8f89 in fresh_domain_unique_id () at runtime/domain.c:517
warning: Source file is more recent than executable.
517	    uintnat next = next_domain_unique_id++;
Missing separate debuginfos, use: dnf debuginfo-install libseccomp-2.5.3-1.fc34.x86_64
(gdb) up
#1  domain_create (initial_minor_heap_wsize=262144) at runtime/domain.c:583
583	  s->unique_id = fresh_domain_unique_id();
(gdb) p *d
$1 = {id = 0, state = 0x0, interruptor = {interrupt_word = 0x0, lock = 0, 
    cond = {cond = 0, mutex = 0x2c7598 <all_domains+24>}, running = 0, 
    terminating = 0, unique_id = 0, interrupt_pending = 0}, 
  backup_thread_running = 0, backup_thread = 0, backup_thread_msg = 3, 
  domain_lock = 0, domain_cond = {cond = 0, 
    mutex = 0x2c75e0 <all_domains+96>}, minor_heap_area_start = 3223552, 
  minor_heap_area_end = 5320704}
(gdb) p *s
$2 = {interrupt_word = 0x0, lock = 0, cond = {cond = 0, 
    mutex = 0x2c7598 <all_domains+24>}, running = 0, terminating = 0, 
  unique_id = 0, interrupt_pending = 0}
(gdb) p s
$3 = (struct interruptor *) 0x2c7590 <all_domains+16>
(gdb) p &d->interruptor
$4 = (struct interruptor *) 0x2c7590 <all_domains+16>
(gdb) p domain_state
$5 = (caml_domain_state *) 0x80
(gdb) p $fs
$6 = 0

EDIT: So far I'm confused why we have d->state==NULL and why domain_state==0x80 as the following block code should update the d->state field and provides a legit address for domain_state with the calloc call:

  if (d->state == NULL) {
    /* FIXME: Never freed. Not clear when to. */
    domain_state = (caml_domain_state*)
      caml_stat_calloc_noexc(1, sizeof(caml_domain_state));
    if (domain_state == NULL)
      goto domain_init_complete;
    d->state = domain_state;
  } else {
    domain_state = d->state;
  }

Here is how I read the assembly for domain_create:

00000000001e8f20 <domain_create>:
  1e8f20:       41 56                   push   %r14
  1e8f22:       41 55                   push   %r13
  1e8f24:       41 54                   push   %r12
  1e8f26:       49 89 fc                mov    %rdi,%r12
  1e8f29:       55                      push   %rbp
  1e8f2a:       53                      push   %rbx

;   uintnat stack_wsize = caml_get_init_stack_wsize();
  1e8f2b:       e8 00 44 00 00          callq  1ed330 <caml_get_init_stack_wsize>
  1e8f30:       bf 88 b9 2c 00          mov    $0x2cb988,%edi
  1e8f35:       49 89 c5                mov    %rax,%r13

;   
  1e8f38:       e8 73 ff f8 ff          callq  178eb0 <pthread_mutex_lock>
  1e8f3d:       85 c0                   test   %eax,%eax
  1e8f3f:       74 19                   je     1e8f5a <domain_create+0x3a>
  1e8f41:       e9 7c 03 00 00          jmpq   1e92c2 <domain_create+0x3a2>

;  /* Wait until any in-progress STW sections end. */
;  while (atomic_load_acq(&stw_leader)) {
;    /* [caml_plat_wait] releases [all_domains_lock] until the current
;       STW section ends, and then takes the lock again. */
;    caml_plat_wait(&all_domains_cond);
;  }
  1e8f46:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  1e8f4d:       00 00 00 
  1e8f50:       bf 90 6a 2c 00          mov    $0x2c6a90,%edi
  1e8f55:       e8 d6 3f 01 00          callq  1fcf30 <caml_plat_wait>
  1e8f5a:       48 8b 05 1f 2a 0e 00    mov    0xe2a1f(%rip),%rax        # 2cb980 <stw_leader>
  1e8f61:       48 85 c0                test   %rax,%rax
  1e8f64:       75 ea                   jne    1e8f50 <domain_create+0x30>

;   d = next_free_domain(); (inlined? Max_domains==128?)
;   if (d == NULL)
;     goto domain_init_complete;
  1e8f66:       48 63 05 f3 e1 0d 00    movslq 0xde1f3(%rip),%rax        # 2c7160 <stw_domains>
  1e8f6d:       3d 80 00 00 00          cmp    $0x80,%eax
  1e8f72:       0f 84 e9 02 00 00       je     1e9261 <domain_create+0x341>  ;  this is a goto end
  1e8f78:       48 8b 1c c5 68 71 2c    mov    0x2c7168(,%rax,8),%rbx
  1e8f7f:       00 
  1e8f80:       48 85 db                test   %rbx,%rbx
  1e8f83:       0f 84 d8 02 00 00       je     1e9261 <domain_create+0x341>  ;  this is a goto end

;  s = &d->interruptor;
;  CAMLassert(!s->running);
;  CAMLassert(!s->interrupt_pending);
;
;  domain_self = d;

; here is the segfault line, so far %fs==0
  1e8f89:       64 48 89 1c 25 e8 ff    mov    %rbx,%fs:0xffffffffffffffe8
  1e8f90:       ff ff 
  1e8f92:       48 8b 6b 08             mov    0x8(%rbx),%rbp
  1e8f96:       48 85 ed                test   %rbp,%rbp
  1e8f99:       0f 84 e9 02 00 00       je     1e9288 <domain_create+0x368>  ;  this is the test   if (d->state == NULL) {

; this is the return from the calloc_noexec call, so this is probably   d->state = domain_state; ?
  1e8f9f:       64 48 89 2c 25 f0 ff    mov    %rbp,%fs:0xfffffffffffffff0
  1e8fa6:       ff ff 

; here are the computation around inligned static uintnat fresh_domain_unique_id(void) ?
  1e8fa8:       48 8b 05 99 e1 0d 00    mov    0xde199(%rip),%rax        # 2c7148 <next_domain_unique_id>
  1e8faf:       b9 01 00 00 00          mov    $0x1,%ecx
  1e8fb4:       48 89 6b 10             mov    %rbp,0x10(%rbx)
  1e8fb8:       48 89 c2                mov    %rax,%rdx
  1e8fbb:       48 89 43 38             mov    %rax,0x38(%rbx)
  1e8fbf:       48 83 c2 01             add    $0x1,%rdx
  1e8fc3:       c7 43 30 01 00 00 00    movl   $0x1,0x30(%rbx)
  1e8fca:       48 0f 44 d1             cmove  %rcx,%rdx
  1e8fce:       48 89 15 73 e1 0d 00    mov    %rdx,0xde173(%rip)        # 2c7148 <next_domain_unique_id>
  1e8fd5:       48 8b 05 64 e1 0d 00    mov    0xde164(%rip),%rax        # 2c7140 <caml_num_domains_running>
  1e8fdc:       48 83 c0 01             add    $0x1,%rax
  1e8fe0:       48 87 05 59 e1 0d 00    xchg   %rax,0xde159(%rip)        # 2c7140 <caml_num_domains_running>
  1e8fe7:       48 8d 7b 60             lea    0x60(%rbx),%rdi
  1e8feb:       e8 c0 fe f8 ff          callq  178eb0 <pthread_mutex_lock>
  1e8ff0:       85 c0                   test   %eax,%eax
  1e8ff2:       0f 85 ca 02 00 00       jne    1e92c2 <domain_create+0x3a2>
...
; this is the end of the function
; domain_init_complete:
;   caml_gc_log("domain init complete");
;   caml_plat_unlock(&all_domains_lock);
  1e9261:       bf 72 57 21 00          mov    $0x215772,%edi
  1e9266:       31 c0                   xor    %eax,%eax
  1e9268:       e8 63 2f 01 00          callq  1fc1d0 <caml_gc_log>
  1e926d:       bf 88 b9 2c 00          mov    $0x2cb988,%edi
  1e9272:       e8 59 fc f8 ff          callq  178ed0 <pthread_mutex_unlock>
  1e9277:       85 c0                   test   %eax,%eax
  1e9279:       75 53                   jne    1e92ce <domain_create+0x3ae>
  1e927b:       5b                      pop    %rbx
  1e927c:       5d                      pop    %rbp
  1e927d:       41 5c                   pop    %r12
  1e927f:       41 5d                   pop    %r13
  1e9281:       41 5e                   pop    %r14
  1e9283:       c3                      retq   

;    /* FIXME: Never freed. Not clear when to. */
;    domain_state = (caml_domain_state*)
;      caml_stat_calloc_noexc(1, sizeof(caml_domain_state)); // caml_stat_calloc_noexc is probably inligned as not called later
;    if (domain_state == NULL)
;      goto domain_init_complete;
  1e9284:       0f 1f 40 00             nopl   0x0(%rax)
  1e9288:       be d0 03 00 00          mov    $0x3d0,%esi
  1e928d:       bf 01 00 00 00          mov    $0x1,%edi
  1e9292:       e8 29 16 01 00          callq  1fa8c0 <caml_stat_calloc_noexc>
  1e9297:       48 89 c5                mov    %rax,%rbp
  1e929a:       48 85 c0                test   %rax,%rax
  1e929d:       74 c2                   je     1e9261 <domain_create+0x341>    ; this is the goto end
  1e929f:       48 89 43 08             mov    %rax,0x8(%rbx)
  1e92a3:       e9 f7 fc ff ff          jmpq   1e8f9f <domain_create+0x7f>

; this is a CAMLAssert failures for the function
  1e92a8:       bf 01 00 00 00          mov    $0x1,%edi
  1e92ad:       e8 7e a7 ff ff          callq  1e3a30 <caml_record_backtraces>
  1e92b2:       e9 0f ff ff ff          jmpq   1e91c6 <domain_create+0x2a6>
  1e92b7:       4c 89 c1                mov    %r8,%rcx
  1e92ba:       4c 89 ca                mov    %r9,%rdx
  1e92bd:       e9 49 ff ff ff          jmpq   1e920b <domain_create+0x2eb>
  1e92c2:       89 c6                   mov    %eax,%esi
  1e92c4:       bf a0 6d 21 00          mov    $0x216da0,%edi
  1e92c9:       e8 f2 3a 01 00          callq  1fcdc0 <caml_plat_fatal_error>
...

palainp · 2022-12-18T15:45:59Z

I edited the asm output, to my understanding, we fail before the test d->state==NULL at line 570.

As s seems to be properly affected to &d->interruptor in the debugger console, maybe we fail on the next line domain_self = d; which makes sense as fs seems to be used for thread stuff and domain_self is thread related. I'm not sure how fs is supposed to be initialised, I'll continue on that track.

nolibc/stubs.c

palainp · 2022-12-20T09:16:17Z

I'm now sure that the failing line is domain_self = d (with domain_self defined as a thread static variable) and this comment may be helpful: https://github.com/Solo5/solo5/blob/85ce3239ce87a6e49e550a7a0756b072a8c9fa6c/bindings/crt_init.h#L47 . fs register set to zero_page and the need of threads in Ocaml 5 can explain our troubles right?

haesbaert · 2022-12-20T12:08:50Z

I'm now sure that the failing line is domain_self = d (with domain_self defined as a thread static variable) and this comment may be helpful: https://github.com/Solo5/solo5/blob/85ce3239ce87a6e49e550a7a0756b072a8c9fa6c/bindings/crt_init.h#L47 . fs register set to zero_page and the need of threads in Ocaml 5 can explain our troubles right?

Nice find ! For sure that is the problem. Interesting that the TLS is kept in fs, I was thinking it would be gs, but now I read it's more common to use fs in userland.

For now you can cheat, just change the runtime for domain_self to be a common pointer, as long as there is no actual pthread_create taking place, a global pointer will work

palainp · 2022-12-20T14:17:50Z

Ok, I tried and had to fight a bit with the caml_state teammate too, contrary to domain_self it's not static and used everywhere, so I decided to remove domains.c:581 caml_state = domain_state;.
It continues well and now fails in 00000000001fe090 <caml_init_shared_heap>: on a line with talking about fs (1fe11c: 64 48 8b 00 mov %fs:(%rax),%rax).
I guess the only way out from here is to follow the hard path of adding threads in solo5.

haesbaert · 2022-12-20T14:48:04Z

Ok, I tried and had to fight a bit with the caml_state teammate too, contrary to domain_self it's not static and used everywhere, so I decided to remove domains.c:581 caml_state = domain_state;. It continues well and now fails in 00000000001fe090 <caml_init_shared_heap>: on a line with talking about fs (1fe11c: 64 48 8b 00 mov %fs:(%rax),%rax). I guess the only way out from here is to follow the hard path of adding threads in solo5.

I guess there are moar cases:

balin:ocaml: find . -name '*.c' -exec grep __thread {} \; | wc -l
9

Writing a userland pthread scheduler is not an insane amount of work, especially because we need a limited subset of it.
When I was planning to do it I checked how some other implementations were doing it, the trick part is allocating and switching between two stacks, I can do it for x86 but would have to study for anything different. Ofc it would have to be non-preemptive as we don't have an interrupting source on on solo5, but it's enough to get the ball rolling for single domain.

At any rate, if you're up to it we can do it at some point, I'm on holidays now contemplating the end of the universe and playing dwarf fortress, so can't really commit to anything.

palainp · 2022-12-22T08:47:32Z

Update: I tried the dumb approach by allocating some memory for TLS and running the thing:

$ git diff
diff --git a/nolibc/mmap.c b/nolibc/mmap.c
index 8a10fcc..330b0e0 100644
--- a/nolibc/mmap.c
+++ b/nolibc/mmap.c
@@ -5,7 +5,8 @@
 
 void *mmap(void *addr, size_t len, int prot, int flags, int fildes, off_t off) {
   if (addr != NULL) {
-    printf("mmap: non-null addr is unsupported.\n");
+    printf("mmap: non-null addr (=%p) is unsupported.\n", addr);
+    printf("Return addr (in caller function): %p\n", __builtin_return_address(0));
     abort();
   }
   if (fildes != -1) {
diff --git a/nolibc/sysdeps_solo5.c b/nolibc/sysdeps_solo5.c
index dacd021..b6dbc7e 100644
--- a/nolibc/sysdeps_solo5.c
+++ b/nolibc/sysdeps_solo5.c
@@ -96,6 +96,10 @@ void _nolibc_init(uintptr_t heap_start, size_t heap_size)
 
     sbrk_start = sbrk_cur = heap_start;
     sbrk_end = heap_start + heap_size;
+
+    size_t tdata = 4096; // FIXME: hard coded
+    uintptr_t tls = (uintptr_t)malloc(tdata);
+    solo5_set_tls_base(tls);
 }

And now I have the following:

            |      ___|                  
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000
mmap: non-null addr (=0x314000) is unsupported.
Return addr (in caller function): 0x1fd0f9
Aborted
Solo5: solo5_abort() called

The caller for mmap is caml_mem_commit (defined here https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/platform.c#L191 and seems to be platform dependant. The unix version uses mprotect, not sure why I have a mmap call here).

I'll be afk for the next week, but I think we're on good track for resolving that PR. Enjoy the end of the world and be safe ;)

palainp · 2022-12-22T15:55:40Z

Update: The mmap's manpage leaves the door open to disregard the addr argument, so with spt or qubes as target (with 1 domain limitation and 512MB of ram) and the PR on @kit-ty-kate repo kit-ty-kate#1:

[2022-12-22 16:44:18] Solo5: Xen console: port 0x2, ring @0x00000000FEFFF000
[2022-12-22 16:44:18]             |      ___|
[2022-12-22 16:44:18]   __|  _ \  |  _ \ __ \
[2022-12-22 16:44:18] \__ \ (   | | (   |  ) |
[2022-12-22 16:44:18] ____/\___/ _|\___/____/
[2022-12-22 16:44:18] Solo5: Bindings version v0.7.5
[2022-12-22 16:44:18] Solo5: Memory map: 512 MB addressable:
[2022-12-22 16:44:18] Solo5:   reserved @ (0x0 - 0xfffff)
[2022-12-22 16:44:18] Solo5:       text @ (0x100000 - 0x22afff)
[2022-12-22 16:44:18] Solo5:     rodata @ (0x22b000 - 0x248fff)
[2022-12-22 16:44:18] Solo5:       data @ (0x249000 - 0x31dfff)
[2022-12-22 16:44:18] Solo5:       heap >= 0x31e000 < stack < 0x20000000
[2022-12-22 16:44:18] 2022-12-22 15:44:18 -00:00: INF [application] hello
[2022-12-22 16:44:19] 2022-12-22 15:44:19 -00:00: INF [application] hello
[2022-12-22 16:44:20] 2022-12-22 15:44:20 -00:00: INF [application] hello
[2022-12-22 16:44:21] 2022-12-22 15:44:21 -00:00: INF [application] hello
[2022-12-22 16:44:22] Solo5: solo5_exit(0) called

The next step is to have a proper threads implementation and eventually checks why I can have with not enough memory (128MB is not sufficient :/):

[2022-12-22 16:43:35] Fatal error: Not enough heap memory to reserve minor heaps

haesbaert · 2022-12-22T17:48:20Z

Update: I tried the dumb approach by allocating some memory for TLS and running the thing:

$ git diff
diff --git a/nolibc/mmap.c b/nolibc/mmap.c
index 8a10fcc..330b0e0 100644
--- a/nolibc/mmap.c
+++ b/nolibc/mmap.c
@@ -5,7 +5,8 @@
 
 void *mmap(void *addr, size_t len, int prot, int flags, int fildes, off_t off) {
   if (addr != NULL) {
-    printf("mmap: non-null addr is unsupported.\n");
+    printf("mmap: non-null addr (=%p) is unsupported.\n", addr);
+    printf("Return addr (in caller function): %p\n", __builtin_return_address(0));
     abort();
   }
   if (fildes != -1) {
diff --git a/nolibc/sysdeps_solo5.c b/nolibc/sysdeps_solo5.c
index dacd021..b6dbc7e 100644
--- a/nolibc/sysdeps_solo5.c
+++ b/nolibc/sysdeps_solo5.c
@@ -96,6 +96,10 @@ void _nolibc_init(uintptr_t heap_start, size_t heap_size)
 
     sbrk_start = sbrk_cur = heap_start;
     sbrk_end = heap_start + heap_size;
+
+    size_t tdata = 4096; // FIXME: hard coded
+    uintptr_t tls = (uintptr_t)malloc(tdata);
+    solo5_set_tls_base(tls);
 }

And now I have the following:

            |      ___|                  
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000
mmap: non-null addr (=0x314000) is unsupported.
Return addr (in caller function): 0x1fd0f9
Aborted
Solo5: solo5_abort() called

The caller for mmap is caml_mem_commit (defined here https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/platform.c#L191 and seems to be platform dependant. The unix version uses mprotect, not sure why I have a mmap call here).

I'll be afk for the next week, but I think we're on good track for resolving that PR. Enjoy the end of the world and be safe ;)

Can't you just point to a static buffer though ? no need to malloc, but anyway, good find !

palainp · 2022-12-31T09:01:10Z

Can't you just point to a static buffer though ?

Yes, done in an update branch, I'll PR it at some point.

And in the meantime, I discovered why the unikernel now needs a lot more memory. This is due to https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/domain.c#L218 and https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/domain.c#L724. The heap reservation is done once for the upper limit of possible domains (here Max_domains==128 and the heap per domain seems to be 2MB). As Max_domains is defined as a macro (https://github.com/ocaml/ocaml/blob/021619f25d84dc0161537366f01c92f6a3980851/runtime/caml/domain.h#L32), I'm not sure if the code can be soon (regarding the comment This hard limit may go away in the future.) adapted to set it dynamically and this will probably cause issues for mirage with a 256MiB entry ticket for unikernels (I'm thinking in particular of qubes-mirage-firewall which aims at low memory consumption).

I got the traces:

$ solo5-spt dist/hello.spt
            |      ___|                  
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x207fff)
Solo5:     rodata @ (0x208000 - 0x224fff)
Solo5:       data @ (0x225000 - 0x2d2fff)
Solo5:       heap >= 0x2d3000 < stack < 0x20000000
malloc: 176 caller 0x1fa92d     ; this is in caml_stat_alloc
malloc: 176 caller 0x1fa92d
malloc: 176 caller 0x1fa92d
malloc: 176 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
... ; lots of similar lines
malloc: 16 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
malloc: 32 caller 0x1fa92d
malloc: 262144 caller 0x1fa92d
malloc: 268439552 caller 0x1fd266  ; this is in caml_mem_map, this is our 256MiB allocation
malloc: 976 caller 0x1fab50   ; this is in caml_stat_calloc_noexc
malloc: 168 caller 0x1fb75c  ; caml_alloc_minor_tables
malloc: 1120 caller 0x1fe36c ; caml_init_shared_heap
malloc: 24 caller 0x1f925d  ; caml_init_major_gc
malloc: 65536 caller 0x1f927e ; caml_init_major_gc
malloc: 112 caller 0x1ee92c ; caml_alloc_final_info
malloc: 40 caller 0x20370c
malloc: 32 caller 0x1f9333
malloc: 2097152 caller 0x1fd2f9 ; caml_mem_commit
malloc: 40 caller 0x1ed5de
malloc: 32864 caller 0x1ed510
malloc: 557056 caller 0x1fd266
malloc: 24 caller 0x1fa92d
malloc: 40 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 40 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 64 caller 0x1fa92d
malloc: 7 caller 0x1fac38
free: 80 caller 0x202b8d
malloc: 24 caller 0x1fa92d
malloc: 232 caller 0x1fa92d
malloc: 52 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 264192 caller 0x1fb062
malloc: 65616 caller 0x1fa92d
malloc: 792576 caller 0x1fb062
malloc: 65616 caller 0x1fa92d
malloc: 65616 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
free: 48 caller 0x1fff94
free: 48 caller 0x1fff94
malloc: 45 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 40 caller 0x1fa92d
malloc: 58 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 14 caller 0x1fac38
free: 32 caller 0x202041
malloc: 13 caller 0x1fac38
free: 32 caller 0x202041
malloc: 7 caller 0x1fac38
free: 32 caller 0x202041
malloc: 5 caller 0x1fac38
free: 32 caller 0x202041
malloc: 12 caller 0x1fac38
free: 32 caller 0x202041
malloc: 32 caller 0x1fa92d
malloc: 8192 caller 0x204162
2022-12-31 08:34:46 -00:00: INF [application] hello
free: 48 caller 0x1fff0e
malloc: 24 caller 0x1fa92d
2022-12-31 08:34:47 -00:00: INF [application] hello
2022-12-31 08:34:48 -00:00: INF [application] hello
2022-12-31 08:34:49 -00:00: INF [application] hello
malloc: 16 caller 0x1f5f4b
malloc: 16 caller 0x1f5f4b
free: 32 caller 0x1f5fea
malloc: 24 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
free: 48 caller 0x1fff94
free: 64 caller 0x1fff94
malloc: 557056 caller 0x1fd266
free: 32 caller 0x1f5fea
Solo5: solo5_exit(0) called

haesbaert · 2022-12-31T10:09:19Z

At this stage it's probably easier to make a compiler patch like '--with-max-domains=1'.
I haven't looked on how memory management is done in solo5, I'm assuming none, which means all mapped pages are wired in ? It should be fine to map 256MB (or even like a brazillion GB) if the pages are never wired.
At any rate, in the future we need a better malloc+mmap, one that is multicore aware and that is mmap based, not just sbrking left and right.

dinosaure · 2022-12-31T14:24:19Z

At this stage it's probably easier to make a compiler patch like '--with-max-domains=1'.

It's probably a task that @shindere should be aware 👍. Any requests like this one deserves special attention in order to improve the toolchain compiling OCaml for our specific use. However, I'm not sure how to organize that in the middle/long term - and we still are on the experiment stage.

palainp · 2023-01-05T13:51:05Z

Makefile

+	mv ocaml/Makefile.sed ocaml/Makefile
+	echo -e "\$$(ocamlyacc_PROGRAM)\$$(EXE):\n\tcp $(shell which ocamlyacc) yacc/\n" >> ocaml/Makefile
+# patch ocaml 5.0.0 runtime for single domain/thread solo5
+	sed -e 's/#define Max_domains 128/#define Max_domains 1/' ocaml/runtime/caml/domain.h > ocaml/runtime/caml/domain.h.sed && \


Maybe it can be useful to be more robust here and use 's/#define Max_domains .+$/#define Max_domains 1/'? It will modify both paths for Max_domains definition (https://github.com/ocaml/ocaml/blob/021619f25d84dc0161537366f01c92f6a3980851/runtime/caml/domain.h#L34) and be resilient if in the future Max_domains is set to anything else than 128.

Or, just remove the original domain.h after moving to .sed, if sed -e fails, then domain.h will never be produced and compilation will fail. This also makes sure all the steps in && will have to succeed.

shindere · 2023-01-16T15:18:38Z

Just commenting on the `Max_domain` aspect here but let me know whether it was a wider comment which was expected. I quickly went through the code and read a comment in `runtime/caml/comain.h` saying that the hard limit on the maximum number of domains may go away in the future. This makes me unsure that upstream will be willing to introduce an interface to specify it, because it means that one will have to handle its deprecation if one day the hard limit indeed goes away. But that's just my uninformed opinion, so if being able to specify a max doamin is important to you it may be worth discussiing with developers more involved than me in this particular area of domains.

palainp · 2023-01-17T08:20:24Z

This makes me unsure that upstream will be willing to introduce an interface to specify it, because it means that one will have to handle its deprecation if one day the hard limit indeed goes away.

Hi @shindere, thanks for your feedback! I think you're right here :)

But that's just my uninformed opinion, so if being able to specify a max doamin is important to you it may be worth discussiing with developers more involved than me in this particular area of domains.

As @haesbaert said in #122 (comment) it's probably best at some point to change the allocation system in solo5 and to allow some memory to be requested and only reserve it when it's really used (the hint should be MAP_NONE in the flags argument to mmap), will have to read how to do that :). For the time being I think we can stay with this patch in the Makefile as solo5 is single core.

hannesm · 2023-05-19T09:57:49Z

I understand #124 superseeds this PR to some degree, but in this PR there are still some review comments left as far as I can tell.

Someone wants to drive the comments from here over to the other PR, so we can close this one and re-review the other one?

dinosaure · 2023-05-19T10:11:06Z

I think, most of what we said on this PR was integrated into #124. The MacOS compilation is missing I think - but Solo5 already has an issue on such platform. I think we can safely close this PR.

hannesm · 2023-05-19T10:17:24Z

What about e.g. #122 (comment) ?

hannesm · 2024-05-06T09:34:17Z

Given that #134 moved forward, I'll close this. If this is errorneous, please let me know. As I understand @shym, the review comments from this PR have been addressed in #134.

kit-ty-kate added 27 commits October 5, 2022 18:56

tmp

de0299f

tmp

2b31cbb

fix test-include weird Makefile issues

cad5c3b

tmp

3f473c7

test

563fd92

tmp

665e955

test

fa430fb

tmp

82205a7

bump

f147b12

bump

92fc584

fix

d6b9bcb

fix linking

6b06b3f

fix

1d5f77c

fix 10_000 feet above the floor

97671f2

more stubs (actually needed)

aa1ac8f

atomic_thread_fence

ad01905

pthread_mutexattr_*

2b65ad2

atomic_fetch_add + pthread_sigmask + pthread_equal

5b378ed

fixup

f91352a

more atomics

8017dfd

fixup

c235799

fixup

401371d

atomic_fetch_or

0c2ec55

pthread_cond_*

59621fd

more

f154280

end?

5dc13d7

cleanup

840db53

kit-ty-kate marked this pull request as draft October 13, 2022 11:41

haesbaert reviewed Oct 13, 2022

View reviewed changes

palainp reviewed Dec 18, 2022

View reviewed changes

nolibc/stubs.c Show resolved Hide resolved

dumb thread local storage

72eb013

palainp added 3 commits January 4, 2023 14:21

tls as static

8a23cdf

patch ocaml runtime for 1 domain

d3e6e33

remove -i argument which is not posix

f118992

palainp reviewed Jan 5, 2023

View reviewed changes

This was referenced Jan 9, 2023

Support ocaml 5.0.0 dinosaure/esperanto#31

Open

Support of OCaml 5.0 (cleaned version) #124

Closed

palainp mentioned this pull request Jan 30, 2023

fix linker scripts with thread section Solo5/solo5#542

Merged

shym mentioned this pull request Apr 18, 2024

Support OCaml 5.2 #134

Open

hannesm closed this May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] OCaml 5.0 support #122

[WIP] OCaml 5.0 support #122

kit-ty-kate commented Oct 13, 2022 •

edited

Loading

kit-ty-kate commented Oct 13, 2022

haesbaert left a comment

haesbaert Oct 13, 2022

haesbaert Oct 13, 2022

palainp Mar 1, 2024

haesbaert Oct 13, 2022

dinosaure Oct 13, 2022

haesbaert Oct 13, 2022 •

edited

Loading

haesbaert Oct 13, 2022 •

edited

Loading

palainp Mar 1, 2024

haesbaert commented Dec 17, 2022

palainp commented Dec 18, 2022 •

edited

Loading

palainp commented Dec 18, 2022 •

edited

Loading

palainp commented Dec 20, 2022

haesbaert commented Dec 20, 2022

palainp commented Dec 20, 2022

haesbaert commented Dec 20, 2022 •

edited

Loading

palainp commented Dec 22, 2022

palainp commented Dec 22, 2022

haesbaert commented Dec 22, 2022

palainp commented Dec 31, 2022

haesbaert commented Dec 31, 2022

dinosaure commented Dec 31, 2022

palainp Jan 5, 2023

haesbaert Jan 5, 2023

shindere commented Jan 16, 2023 via email

palainp commented Jan 17, 2023

hannesm commented May 19, 2023

dinosaure commented May 19, 2023

hannesm commented May 19, 2023

hannesm commented May 6, 2024

[WIP] OCaml 5.0 support #122

[WIP] OCaml 5.0 support #122

Conversation

kit-ty-kate commented Oct 13, 2022 • edited Loading

kit-ty-kate commented Oct 13, 2022

haesbaert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haesbaert Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

haesbaert Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haesbaert commented Dec 17, 2022

palainp commented Dec 18, 2022 • edited Loading

palainp commented Dec 18, 2022 • edited Loading

palainp commented Dec 20, 2022

haesbaert commented Dec 20, 2022

palainp commented Dec 20, 2022

haesbaert commented Dec 20, 2022 • edited Loading

palainp commented Dec 22, 2022

palainp commented Dec 22, 2022

haesbaert commented Dec 22, 2022

palainp commented Dec 31, 2022

haesbaert commented Dec 31, 2022

dinosaure commented Dec 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shindere commented Jan 16, 2023 via email

palainp commented Jan 17, 2023

hannesm commented May 19, 2023

dinosaure commented May 19, 2023

hannesm commented May 19, 2023

hannesm commented May 6, 2024

kit-ty-kate commented Oct 13, 2022 •

edited

Loading

haesbaert Oct 13, 2022 •

edited

Loading

haesbaert Oct 13, 2022 •

edited

Loading

palainp commented Dec 18, 2022 •

edited

Loading

palainp commented Dec 18, 2022 •

edited

Loading

haesbaert commented Dec 20, 2022 •

edited

Loading