When exactly files are loaded inside the enclave? and what happens during enclave initialization?

[jogi343]

Hi All,

I think I am a bit confused about when exactly the files mentioned in the manifest.template file are loaded (or copied) inside an enclave (i.e. inside the EPC memory).

My questions are:

1. When exactly all the files mentioned in sgx.trusted_files TOML array, fs.mounts TOML array and libos.entrypoint are loaded / copied inside the enclave? Are they copied during enclave initialization? or are they copied when they are actually needed during execution?

2. If they are copied inside the enclave when they are actually needed, then what things happen during enclave initialization? (as enclave initialization takes a relatively long time to finish), also what things are present inside a freshly initialized enclave?

Thank you in advance.

[dimakuv]

Are they copied during enclave initialization? or are they copied when they are actually needed during execution?

The latter. The files are copied inside the enclave only on demand.

If they are copied inside the enclave when they are actually needed, then what things happen during enclave initialization? (as enclave initialization takes a relatively long time to finish)...

Enclave initialization takes a long time to finish for a single reason -- in SGX v1 (without the EDMM feature), the whole enclave memory is mapped at startup. So, if you have a sgx.enclave_size = "16G" enclave, then the whole 16GB of enclave pages will be loaded at startup (more specifically, these pages will be EADDed). Which is a slow operation, especially if the EPC (physical amount of enclave pages memory) is low, which happens on older CPUs.

If you have a modern big CPU (e.g., Intel Icelake Server, aka ICX) and you enable the EDMM feature (will be available in Gramine v1.4 under sgx.edmm_enable = true), you will notice that even 16GB enclaves are started in less than 1 second.

...also what things are present inside a freshly initialized enclave?

Very good question. Basically, three things:

1. The initial in-enclave Gramine binary (so-called libpal.so)
2. The manifest.sgx file contents
3. Helper data like TCS, TLS, SSA, stack pages

After this fresh enclave is created, Gramine jumped into its entrypoint, and then the rest of the initialization happens (now inside the SGX enclave itself): loading the LibOS component of Gramine, loading the interpreter (ld.so), loading the application binary, loading the shared libs, initializing the heap area, etc.

For the sources, you can look at this:

gramine/python/graminelibos/sgx_sign.py

Line 205 in 5fb3154

def gen_area_content(attr, areas, enclave_base, enclave_heap_min):

[jogi343]

Thank you for your help @dimakuv !!