Error running the ra-tls-mbedtls example

[tanishq-akula]

While running the command 'ARCH_LIBDIR=/lib64 make app', I am getting following error:-
(Server:- RHEL 8.6, Kernel Version:- 4.18.0-348.20.1.el8_5.x86_64)

gramine-manifest \
        -Dlog_level=error \
        -Darch_libdir=/lib/x86_64-redhat-linux \
        -Dra_type=none \
        -Dra_client_spid= \
        -Dra_client_linkable=0 \
        server.manifest.template > server.manifest
cc src/server.c -O2 -fPIE -I./mbedtls/include -pie -ldl -Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64 -lmbedcrypto_gramine -lmbedtls_gramine -lmbedx509_gramine  -o server
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `mbedtls_ssl_conf_dh_param_ctx':
(.text+0x2055): undefined reference to `mbedtls_dhm_get_value'
(.text+0x2069): undefined reference to `mbedtls_dhm_get_value'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `mbedtls_ssl_handshake_step':
(.text+0x280c): undefined reference to `mbedtls_ssl_states_str'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `mbedtls_ssl_handshake_free':
(.text+0x29c8): undefined reference to `mbedtls_dhm_free'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `ssl_handshake_init':
(.text+0x3f43): undefined reference to `mbedtls_dhm_init'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `mbedtls_ssl_parse_sig_alg_ext':
(.text+0x4ad0): undefined reference to `mbedtls_ssl_sig_alg_to_str'
(.text+0x4bca): undefined reference to `mbedtls_ssl_sig_alg_to_str'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `mbedtls_ssl_psk_derive_premaster':
(.text+0x5226): undefined reference to `mbedtls_dhm_calc_secret'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls.o): In function `mbedtls_ssl_write_sig_alg_ext':
(.text+0x6b42): undefined reference to `mbedtls_ssl_sig_alg_to_str'
(.text+0x6bc9): undefined reference to `mbedtls_ssl_sig_alg_to_str'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls12_server.o): In function `ssl_parse_client_dh_public':
(.text+0x43): undefined reference to `mbedtls_dhm_read_public'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls12_server.o): In function `mbedtls_ssl_handshake_server_step':
(.text+0x323f): undefined reference to `mbedtls_dhm_set_group'
(.text+0x3274): undefined reference to `mbedtls_dhm_get_len'
(.text+0x3292): undefined reference to `mbedtls_dhm_make_params'
(.text+0x3964): undefined reference to `mbedtls_dhm_calc_secret'
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libmbedtls_gramine.a(ssl_tls12_client.o): In function `mbedtls_ssl_handshake_client_step':
(.text+0x1469): undefined reference to `mbedtls_dhm_get_len'
(.text+0x14b4): undefined reference to `mbedtls_dhm_get_len'
(.text+0x14cf): undefined reference to `mbedtls_dhm_make_public'
(.text+0x1559): undefined reference to `mbedtls_dhm_calc_secret'
(.text+0x2833): undefined reference to `mbedtls_dhm_read_params'
(.text+0x284b): undefined reference to `mbedtls_dhm_get_bitlen'
(.text+0x2df3): undefined reference to `mbedtls_dhm_get_len'
(.text+0x2e57): undefined reference to `mbedtls_dhm_get_len'
(.text+0x2e73): undefined reference to `mbedtls_dhm_make_public'
collect2: error: ld returned 1 exit status
make: *** [Makefile:80: server] Error 1

Thanks

[dimakuv]

GCC cannot find mbedtls_gramine shared library.

How did you install Gramine? Did you git-clone, build and install? If yes, under which prefix? And did you modify environment variables if the prefix was specified by you? (See the table at https://gramine.readthedocs.io/en/latest/devel/building.html#additional-build-options).

[tanishq-akula]

We have directly installed Gramine with dnf command on RHEL 8.6 host as per steps specified in the documentation.

https://gramine.readthedocs.io/en/stable/quickstart.html

To get other DCAP libraries we have built Gramine with the meson command as per building instructions you have specified above.

mbedtls_gramine.so is already available in /usr/lib64.

[dimakuv]

I think you're right and we have a bug in Gramine.

Could you do two things:

1. Show us the output of pkg-config --libs mbedtls_gramine.
2. Replace this line:

   gramine/CI-Examples/ra-tls-mbedtls/Makefile

   Line 77 in 670cc12

   LDFLAGS += -ldl -Wl,--enable-new-dtags $(shell pkg-config --libs mbedtls_gramine)

with this:

LDFLAGS += -ldl -Wl,--enable-new-dtags -Wl,--start-group $(shell pkg-config --libs mbedtls_gramine) -Wl,--end-group

---

Explanation: the ra-tls-mbedtls example links against static libraries libmbedcrypto_gramine, libmbedtls_gramine, libmbedx509_gramine. The order of linking in case of static libraries is very important, see e.g. https://stackoverflow.com/questions/45135/why-does-the-order-in-which-libraries-are-linked-sometimes-cause-errors-in-gcc

In the case of mbedTLS, the dependencies between the three static libraries are like this:

• libmbedtls_gramine -> libmbedcrypto_gramine
• libmbedx509_gramine -> libmbedcrypto_gramine

So the correct order should be: -lmbedtls_gramine -lmbedx509_gramine -lmbedcrypto_gramine.

On my Ubuntu machine, I see the wrong order:

$ pkg-config --libs mbedtls_gramine
-L/home/dimakuv/gramineproject/gramine/built-debug/lib/x86_64-linux-gnu -Wl,-rpath,/home/dimakuv/gramineproject/gramine/built-debug/lib/x86_64-linux-gnu -lmbedcrypto_gramine -lmbedtls_gramine -lmbedx509_gramine

I'm unsure why on my Ubuntu, everything works and builds correctly. Maybe there is some difference in default GCC behavior...

@mkow @boryspoplawski @woju Did I root cause this correctly? Should we modify our pkg-config of Gramine?

[tanishq-akula]

1. pkg-config --libs mbedtls_gramine

-Wl,-rpath,/usr/lib64 -lmbedcrypto_gramine -lmbedtls_gramine -lmbedx509_gramine

2. I made the suggested changes in Makefile and now it's working.

Appreciate your quick support !!!

Do you want me to raise a bug or will it be raised internally ?

[dimakuv]

Do you want me to raise a bug or will it be raised internally ?

Yes, please, raise a bug. Thanks!

[tanishq-akula]

I have raised a bug for this issue.
#1046