Host injected malicious signal / memory fault when running java app

[paul-jubo]

I am trying to run a java helloworld application on an SGX worker node in a kubernetes cluster.
The container image extends the prebuilt image docker.io/gramineproject/gramine.
When running the app with "gramine-sgx", I get the following error:

(..)
(libos_rtld.c:1045:register_library) [P1:T2:java] debug: glibc register library /opt/java/openjdk/lib/libjimage.so loaded at 0x58771000
(libos_rtld.c:1045:register_library) [P1:T2:java] debug: glibc register library /opt/java/openjdk/lib/libjava.so loaded at 0x5873c000
(pal_exception.c:289:_PalExceptionHandler) error: Host injected malicious signal 2
(pal_process.c:248:_PalProcessExit) debug: PalProcessExit: Returning exit code 1

When I run gramine-direct instead of gramine-sgx, a memory fault is stated in the output, exactly where gramine-sgx had failed, but then the app keeps running and successfully prints the helloworld message:

(..)
(libos_rtld.c:1045:register_library) [P1:T2:java] debug: glibc register library /opt/java/openjdk/lib/libjimage.so loaded at 0x39917bc16000
(libos_rtld.c:1045:register_library) [P1:T2:java] debug: glibc register library /opt/java/openjdk/lib/libjava.so loaded at 0x39917bbe1000
(libos_signal.c:351:memfault_upcall) [P1:T2:java] debug: memory fault at 0x00000000 (IP = 0x39916498062d)
(libos_context.c:279:prepare_sigframe) [P1:T2:java] debug: Created sigframe for sig: 11 at 0x39917bd33490 (handler: 0x39917cc9d3a0, restorer: 0x39917d29e6f0)
(..)
Hello world!
(..)

I would deduce from that, that the memory fault is also happening in the case of gramine-sgx and that this is the root cause for the error. I double-checked that I can successfully execute the java app without gramine. I also made sure that the java compilation as well as the gramine signing is executed on the target worker node in order to exclude any possible disturbances in terms of CPU architecture.
Does anyone have an idea why this memory fault is happening? I appended the manifest file for reference. I am aware that the enclave size of 1GB might be a bit small but I have varied it and went up until 4GB and the error persists. Any thoughts highly appreciated!

manifest.txt

[kailun-qin]

Hi @paul-jubo, thanks for the question!

OpenJDK requires memfault handling, so you'll need to set sgx.use_exinfo = true in the Gramine manifest for it to work (pls see https://gramine.readthedocs.io/en/stable/manifest-syntax.html#sgx-exinfo and https://github.com/gramineproject/examples/blob/da07254dc93264e9399c1da44446e822f98a2079/openjdk/java.manifest.template#L26-L27 for details). Note that this will need hardware feature EXINFO support, pls first check w/ the is-sgx-available tool.

(pal_exception.c:289:_PalExceptionHandler) error: Host injected malicious signal 2

The shown log was due to that when EXINFO feature was disabled, the #PF hardware exception was considered invalid/unsupported (in SSA EXINFO) while an unexpected SW signal was actually generated (pls see below for details:

gramine/pal/src/host/linux-sgx/pal_exception.c

Lines 245 to 292 in 5d38bb1

	/*
	* Intel SGX hardware exposes information on a HW exception in the EXITINFO struct.
	* Host OS + Gramine's untrusted part of PAL deliver a SW signal. The SW signal can be a
	* reaction to HW exception (synchronous signal) or a reaction to software events (asynchronous
	* signal). For security, it is important to cross-check HW exception state vs SW signal state.
	*
	* The below table shows the cross checks. "yes" means allowed combination, "no" means
	* prohibited combination (Gramine terminates). "yes*" means a special case of #PF, see comments
	* below on #PF handling.
	*
	* +-----------------------------+-----+-----+-----+-----+------------------+------------+
	* | HW exceptions (trusted) -> | | #DE | | | | |
	* | --------------------------- | | #MF | | #GP | others | none |
	* | SW signals (untrusted) | | #UD | #XM | #PF | #AC | (#BR,#DB,#BP,#CP)| (valid=0) |
	* | v | | | | | | |
	* --+-----------------------------+-----+-----+-----+-----+------------------+------------+
	* s | | | | | | | |
	* y | PAL_EVENT_ILLEGAL | yes | no | no | no | | |
	* n | | | | | | | |
	* c +-----------------------------+-----+-----+-----+-----+ no | no |
	* h | | | | | | (exceptions | (malicious |
	* r | PAL_EVENT_ARITHMETIC_ERROR | no | yes | no | no | unsupported | host |
	* o | | | | | | by Gramine) | injected |
	* n +-----------------------------+-----+-----+-----+-----+ | SW signal)|
	* o | | | | | | | |
	* u | PAL_EVENT_MEMFAULT | no | no |yes* | yes | | |
	* s | | | | | | | |
	* --+-----------------------------+-----+-----+-----+-----+------------------+------------+
	* | | | |
	* a | PAL_EVENT_QUIT | | yes |
	* s | | no, except #PF case* | |
	* y +-----------------------------+ (malicious host ignored HW exception) +------------+
	* n | | | |
	* c | PAL_EVENT_INTERRUPTED | | yes |
	* | | | |
	* --+-----------------------------+------------------------------------------+------------+
	*/
	uint32_t event_num = 0; /* illegal event */
	if (!trusted_exit_info.valid) {
	/* corresponds to last column in the table above */
	if (untrusted_external_event != PAL_EVENT_QUIT
	&& untrusted_external_event != PAL_EVENT_INTERRUPTED) {
	log_error("Host injected malicious signal %u", untrusted_external_event);
	_PalProcessExit(1);
	}
	event_num = untrusted_external_event;

).

[paul-jubo]

@kailun-qin thank you so much for your fast reply! Currently, the is-sgx-available tool shows that EXINFO is not supported. Therefore, when I activate it in the gramine manifest, the enclave cannot be initialised. Just for completeness, here is the output I am receiving now:

(host_framework.c:562:init_enclave) error: Enclave initialization IOCTL failed: Invalid argument (EINVAL)
(host_main.c:567:initialize_enclave) error: Initializing enclave failed: Invalid argument (EINVAL)
(host_main.c:1250:main) error: load_enclave() failed with error: Invalid argument (EINVAL)

I will try to activate the feature at the hardware level and let you know if it worked. Thanks a lot already. :-)

[dimakuv]

@paul-jubo Yes, as @kailun-qin explained. I just wanted to add that this is a breaking change in the new release of Gramine v1.6. See here: https://github.com/gramineproject/gramine/releases/tag/v1.6.

We had to do this change because there was a potential vulnerability otherwise, and old SGX hardware (without the EXINFO feature) simply cannot cope with that potential vulnerability.

[paul-jubo]

@dimakuv thanks a lot for the additional info.

Does anyone know whether I am correct to assume that the EXINFO support would usually be something that can be activated in the BIOS? Or is it definitely not supported by the hardware when is-sgx-available says that it's not there?

[paul-jubo]

Very useful information, thank you.

I'm using an Intel® Xeon® Gold 6416H processor, according to this spec overview it was launched in Q1'23.

Unfortunately, I could not find any related BIOS settings, nor did I find specific information on the linked github site @dimakuv .
Neither could I find other information on which CPUs provide this feature. However, the SGX Exception Handling Guide kind of suggests that this feature is only part of SGX2:

In SGX2, Page faults (#PF) and general-protection faults (#GP) may also be reported to an enclave by configuring MISCSELECT.

That would mean that I could not execute any java app on my processor. For reference, here is a excerpt form my is-sgx-available output:

$ is-sgx-available
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
...
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: false
#CP information in EXINFO in MISC region of SSA supported: false

Maybe there is a workaround to start a java app without this type of memory fault exception handling? Or do you think that any java app executed with Gramine on SGX definitely depends on the EXINFO feature?

[dimakuv]

I'm using an Intel® Xeon® Gold 6416H processor, according to this spec overview it was launched in Q1'23.

Such CPU must definitely support EXINFO. Please enable SGX2 in your BIOS settings (it should be possible). Then you'll see that EXINFO lines will be set to true in the tool output.

[dimakuv]

Maybe there is a workaround to start a java app without this type of memory fault exception handling? Or do you think that any java app executed with Gramine on SGX definitely depends on the EXINFO feature?

There is no workaround. It is known that Java requires memory fault exception handling unconditionally.

[paul-jubo]

Okay, thanks again. Until now I couldn't find the options in the BIOS but I'm working on it. Since I'm operating in a rather complex cloud setup it's not simple to figure out why the SGX2 feature is not working currently, but if I find I solution I will update you. You already helped me a lot with your explanations, really appreciate it!

[mkow]

Hmm, maybe it's just an old BIOS and updating it will help?

[paul-jubo]

Still working on it, it might be associated to an old kernel version of the host system.

From my point of view, this discussion helped a lot and resolved my original problem. It was definitely the case that SGX2 was not available to the application, albeit the hardware supports it. We can either close this discussion or I can give you an update on why SGX2 capabilities were not passed on to the application (as soon as I find out).

[dimakuv]

We can either close this discussion or I can give you an update on why SGX2 capabilities were not passed on to the application (as soon as I find out).

Yeah, I'd be happy to learn what happens in your particular case. Let's keep this discussion open for now.

[pravinrajr9]

@paul-jubo is this issue resolved? if so how?

[paul-jubo]

The bottleneck in my case was a virtualisation software from VMWare which did not pass on the full hardware capabilities to the virtual host. Unfortunately, I don't have the specific versions but after an update of this virtualisation software, the SGX2 capabilities were finally available in the application.