Host process becomes a zombie when calling execve from another than the initial thread

[thomasten]

Description of the problem

I have the following program:

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

void* f(void* p) {
    char* argv[] = {p, "", NULL};
    execve(p, argv, NULL);
    abort();
}

int main(int argc, char** argv) {
    if (argc == 1) {
        pthread_t t;
        pthread_create(&t, NULL, f, argv[0]);
        pthread_join(t, NULL);
        abort();
    }

    for (;;) {
        // fputs("a\n", stderr);
        sleep(1);
    }
}

When I run this with Gramine, the host process becomes a zombie:

$ gramine-direct helloworld &
[1] 1457
$ ps
    PID TTY          TIME CMD
   1436 pts/0    00:00:00 bash
   1457 pts/0    00:00:00 exe <defunct>
   1461 pts/0    00:00:00 ps

I think this is because the execve implementation kills all other threads and the initial thread is gone.

Steps to reproduce

No response

Expected results

No response

Actual results

No response

Gramine commit hash

v1.4

[dimakuv]

Do I understand correctly that the problem is that the execve'd process doesn't respond to any signals anymore?

This is probably because Gramine forgets to "keep" some important metadata that was somehow stored in the main thread but not in any child thread. Looks like some bug in signal dispositions. I'll take a look at it.

P.S. This is the most contrived program I've seen in my life :)

[thomasten]

It responds to signals if I use kill. But when I run this in a container and use docker kill, it doesn't work. I believe this is because containerd can't find the child process anymore.

I'm not so familiar with Gramine's codebase. My assumption was that from the host's perspective, something like this happens:

void* f(void* p) {
    for (;;) {
        sleep(1);
    }
}

int main() {
    pthread_t t;
    pthread_create(&t, NULL, f, NULL);
    pthread_exit(0);
}

This causes the process to be marked as defunct. (Running the execve example without Gramine doesn't have this behavior.)

[dimakuv]

@thomasten Thanks for the hints! I see the problem now.

The issue is that from the Linux host perspective, the main thread (with its pid) is terminated, and there are only child threads executing in the original process -- which makes the whole process defunct (it is not visible in a list of alive processes, aka it's a zombie).

This happens because Gramine doesn't "rewire" the execve() logic to be executed on the main thread, but just continues executing this child thread, killing all other threads (including the main thread) in the process.

So I see two solutions:

1. Workaround -- keep the main thread alive and just waiting forever (sounds a bit brittle), or
2. When Gramine notices on execve() that the thread is not the main thread, then force the main thread to continue with execve() logic.

[thomasten]

Is it a requirement that the main thread of the enclave process is executed by the main thread of the host process? If not, a third option could be to use the host process's main thread for something else than enclave execution (not sure if Gramine has any mandatory helper threads) or let it wait forever.

[dimakuv]

I found the related Linux code: https://elixir.bootlin.com/linux/latest/source/fs/exec.c#L1077 (until line 1155)

I don't think that this can map in any way to Gramine, because there is no such syscall as to "rewire" a non-main thread to assume the identity of the main thread (mainly, PID and some other IDs).

[mkow]

Ok, after some tests and talking with @boryspoplawski I think I understand the issue.

Regarding the solutions:

• Whichever we implement it will require extensive explanation in a comment how this part of Linux works and why on Gramine it's requires some tricks.
• 2. will be rather hard, we'd need to somehow stop the first thread and reroute it, while it may be busy with something else. This is a similar issue to multithreaded fork ([LibOS] fork in multi-threaded app may fail #1156).
• 1. is a bit uglier, but it's still solid and should be trivial to implement, so I guess we should go with it?

[dimakuv]

@thomasten Could you check #1268?

[thomasten]

I tested the fix with my actual use case. Works fine.
(Code also LGTM, but I don't know the codebase well enough of course.)