Issues preventing running Go applications in Gramine

[meithecatte]

Go (also known as Golang) is one of the languages currently popular for writing applications in. Ideally it would be possible to run those within Gramine. However, some issues within the Go runtime make that unreliable and slow.

Syscall interface

Gramine needs to be in charge of handling all syscalls made by the application. The recommended way to achieve this is to use a specific call instruction instead of syscall. We carry patches for glibc and musl to handle this for most applications. However, because of an interesting decision made by the Go team, Go doesn't use the libc to issue syscalls. This is not the first time this has caused issues.

For Gramine, the impact is that the fallback that traps the syscall instruction will be used. This causes a significant performance degradation, as many more context switches are required for each syscall. This is signalled by the following message printed during startup:

Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.

As Go binaries are fully statically linked, the only solution seems to be patching the Go toolchain to create binaries with the Gramine-specific syscall ABI. Apart from the overhead of maintaining the patches, this would be somewhat annoying for users, as for most other stacks, the build system doesn't need to know about Gramine, and the binaries produced can be run on Linux directly for testing.

Go runtime thread count

Due to SGX limitations, we need to specify the total number of threads each process in the enclave might use (this is the sgx.thread_num manifest key). However, the number of threads the Go runtime may use is unbounded. The closest we have to limiting this is the GOMAXPROCS environment variable, however this doesn't count the threads that are blocked on a syscall, so it doesn't solve the problem.

Possible solutions:

• patch Go to introduce a proper thread count limit
• as above, but campaign to have it included upstream to ease the maintenance burden
• set sgx.thread_num to a somewhat high number, cross your fingers and hope for the best (this is what's happening in practice right now)
• wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)

[lejunzhu]

Just my two cents, for the syscall interface, there is another possibility: to patch the ELF file after the Go toolchain produces it. And this could be more convenient if the user only has the binary.
For example, a tool could scan the whole ELF and find the pattern "mov ...; syscall", and rewrite it as a long jump to a new piece of code, to call Gramine then jump back.

[mkow]

@lejunzhu: This method is super unreliable and hard to maintain, we don't plan to go in this direction.

[StanPlatinum]

• wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)

Are there any updates on this "waiting for EDMM/SGXv2 solution"?

And AFAIK, SGX2 only allows you to manage memory dynamically. I am not sure it can allow dynamically allocating more threads.

[kailun-qin]

• wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)

Are there any updates on this "waiting for EDMM/SGXv2 solution"?

Yes, it's been supported since Gramine v1.6. Pls see the related manifest option, issue #1223 and PR #1451 for details.

And AFAIK, SGX2 only allows you to manage memory dynamically. I am not sure it can allow dynamically allocating more threads.

Dynamic threading is achieved based on SGX2 that allows TCS to be added at runtime (specifically, by changing regular enclave pages into TCS pages). Pls see above and Section 3.4 of this EDMM paper.

[dimakuv]

@mkow @kailun-qin Should this be moved to Discussions?

[kailun-qin]

Should this be moved to Discussions?

Yes, make sense to me.

[tiagorvmartins]

Hey Gramine team!
I am currently trying to improve the performance of a few golang applications running on Gramine.

Trying to reach a point where the following warning dissapears:
"Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API."

And after discussing previously with gramine team, go build (with compiler -gccgo) is not an option for now, because its quite back in compatibility with golang version, current gccgo supports only go 1.18. And currently we need at least support for Go 1.21.

So having this said, I have been trying to build the golang application using the gramine binary using gramine glibc instead, but not sure if I am following the correct paths, or if this makes sense at all.

Here is a small sample that I am currently testing to build the application inside docker:

...more things not relevant for the use case...

COPY --from=gramineproject/gramine --chown=root /usr/lib/x86_64-linux-gnu/gramine/runtime/glibc/ /gramine-glibc/
COPY --from=gramineproject/gramine --chown=root /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /gramine-linker/ld-linux-x86-64.so.2

RUN export CGO_LDFLAGS="-Xlinker -rpath=/gramine-glibc/" && CGO_LDFLAGS="$CGO_LDFLAGS -Xlinker --dynamic-linker='/gramine-linker/ld-linux-x86-64.so.2'" CGO_ENABLED=${CGO_ENABLED} GOOS=${GOOS} GOARCH=${GOARCH} GOBIN=${GOBIN} go build -ldflags "-linkmode=external" -ldflags "all=${LDFLAGS}" -o ${GOBIN}/${GOBINARY} ${GOCMD}

...more things not relevant for the use case...

However it fails on boot:
[P1:T1:my-exec-name] error: libos_init() failed in init_elf_objects: No such file or directory (ENOENT)

Am I following the correct path here? Makes sense?
Would be grateful for any feedback or comments on this,
Thank you!