Delta Table Loading Issue in Azure Function App

[kimzed]

We're experiencing an issue loading Delta tables in an Azure Function App using delta-dotnet. Key details:
Working scenario:

Works locally on Windows
Using path format:

filePath = $"abfss://{containerName}@{storageAccountName}.dfs.core.windows.net/{path-to-delta-table}"

Using access_key for storage authentication

var options = new TableOptions
{
};
options.StorageOptions["access_key"] = [access key];

Issue:

Same code fails when deployed to Azure Function App

Error: DeltaLake.Errors.DeltaRuntimeException: Failed to read delta log object: Generic HTTP client error: builder error
Error occurs during table loading operation

For this line:

DeltaRuntime runtime = new DeltaRuntime(RuntimeOptions.Default);
_table = DeltaTable.LoadAsync(runtime,
    filePath,
    tableOptions,
    CancellationToken.None).Result;

Has anyone encountered similar issues or have suggestions ?

[kimzed]

we are using 0.1.0 for deltalake.dotnet

failing method is

    internal unsafe async Task<Table> LoadTableAsync(Memory<byte> tableUri, DeltaLake.Table.TableOptions options, System.Threading.CancellationToken cancellationToken)
    {
        TaskCompletionSource<Table> tsc = new TaskCompletionSource<Table>();
        using Scope scope = new Scope();
        DeltaLake.Bridge.Interop.TableOptions tableOptions = default(DeltaLake.Bridge.Interop.TableOptions);
        tableOptions.version = (options.Version.HasValue ? ((long)options.Version.Value) : (-1L));
        tableOptions.without_files = (options.WithoutFiles ? ((byte)1) : ((byte)0));
        tableOptions.log_buffer_size = options.LogBufferSize ?? 0;
        tableOptions.storage_options = ((options.StorageOptions != null) ? scope.Dictionary(this, options.StorageOptions) : null);
        DeltaLake.Bridge.Interop.TableOptions value = tableOptions;
        Methods.table_new(Ptr, scope.Pointer(scope.ByteArray(tableUri)), scope.Pointer(value), scope.CancellationToken(cancellationToken), scope.FunctionPointer<TableNewCallback>(delegate (RawDeltaTable* success, DeltaTableError* fail)
        {
            if (cancellationToken.IsCancellationRequested)
            {
                Task.Run(() => tsc.TrySetCanceled(cancellationToken));
            }
            else if (fail != null)
            {
                Task.Run(() => tsc.TrySetException(DeltaRuntimeException.FromDeltaTableError(Ptr, fail)));
            }
            else
            {
                Task.Run(() => tsc.TrySetResult(new Table(this, success)));
            }
        }));
        return await tsc.Task.ConfigureAwait(continueOnCapturedContext: false);
    }

[mightyshazam]

@kimzed Check your environment variables in both environments. It is possible something about the function environment is triggering different behavior. The process appears to be failing before ever attempts a request because it can't build an http client.

[kimzed]

What environment variables are you talking about exactly?
for credentials we checked and everything is correct.
for other environment variables we checked in settings but nothing related pops out. I looked for proxy or http related variable

We tried to lower security rules in the azure function app settings to try to isolate the error more but we do not get a different result so far.

im wondering as well when the package is going to get a new release in nuget? what we have seems quite outdated

[mightyshazam]

I don't know which environment variable would cause the build statement to fail. I was just suggesting looking at differences. Are the OS and .net version the same in the function as the local environment?
Something is different, but without knowing what, it's hard to figure out.
We were waiting for the kernel to gain write support and/or come out of the incubator. Now that it is out and the default engine has write support, we can move forward. I think it should be possible to get a new release out next month.

[kimzed]

thanks for your answer

the function app works on ".NET 8 (LTS), isolated worker model" on a linux environment

we work on a windows 10

I could try to replicate that on a linux OS

[kimzed]

I also tried on a linux machine but did not get any issue

[kimzed]

does someone know on which version the used rust packages are?

[kimzed]

so we tried a couple of things, identified the differences in env variables
I tried to remove all managed identities related variables on the function app to try to force it to use the access key
also tried additional managed identities options in table options but not working either

so we are thinking of switching to another tech stack to solve this issue

[mightyshazam]

@kimzed Since this is still experimental, the CI didn't establish any tags for releases. However, I pushed a tag so that people can see the current nuget version. The current 0.1.0 version references delta-rs rust-v0.20.0.

[mdrakiburrahman]

@kimzed - just curious, can you repro this (on the old delta-dotnet Nuget you originally tried in Cloud) using Azure Function Emulator locally?

https://learn.microsoft.com/en-us/azure/azure-functions/functions-develop-local
https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=windows%2Cisolated-process%2Cnode-v4%2Cpython-v2%2Chttp-trigger%2Ccontainer-apps&pivots=programming-language-csharp

Or does it only repro in their Cloud runtime?

[mdrakiburrahman]

Also, if it's only in Cloud, could you paste a JSON representation of the Azure Function? (i.e. what the ARM props say):

[dmunch]

I'm planning to use this lib in the mid-term on Azure Functions as well, so got interested in this and had a quick look.

It's a long shot, but since delta-dotnet links to native code, I followed the dependency chain and found that the http client library reqwest used by the rust implementation depends in turn on native-tls, which on Linux depends on a shared library to be installed on the system. Since it's notoriously tricky to get the right version of OpenSSL on a Linux system (at least in my experience), this might be the cause of the issue: The Azure Function host on Linux might no have the right version installed.

As a solution it seems to be possible to statically compile OpenSSL by setting the vendored feature of native-tls (https://docs.rs/native-tls/latest/native_tls/#cargo-features) - I've got no experience in Rust, but maybe it's as simple as adding the following line (or something similar) to this projects Cargo.toml ?

[dependencies]
native-tls = { version = "0.2", features = ["vendored"] }

[dmunch]

It seems that most recent Microsoft Docker Images (and AFAIR Azure Functions run in a Docker Container) have removed OpenSSL 1.X (dotnet/dotnet-docker#4740) due to it being EOL. It's not entirely sure to me though if reqwest supports OpenSSL 3? seanmonstar/reqwest#1978

[mightyshazam]

@dmunch It is true that the openssl dependency is usually a problem. However, the rust dependencies all use rustls to avoid that pitfall. The latest version includes delta-kernel-rs, and that use reqwest with default features, so that may be an issue.

[dmunch]

I suspected arrow-rs having the dependency on reqwest due to your earlier post here, so it might not only be delta-kernel-rs?

The process appears to be failing before ever attempts a request because it can't build an http client.

I'd love to reproduce but am currently lacking a bit of time - However I was able to login to our function host via SSH and was able to confirm that it's running on "Debian GNU/Linux 12 (bookworm)"

root@bf96500ecbcb:/# ls / 
appservice  appsvctmp  azure-functions-host  bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@bf96500ecbcb:/# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@bf96500ecbcb:/# dpkg -L libssl3
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/engines-3
/usr/lib/x86_64-linux-gnu/engines-3/afalg.so
/usr/lib/x86_64-linux-gnu/engines-3/loader_attic.so
/usr/lib/x86_64-linux-gnu/engines-3/padlock.so
/usr/lib/x86_64-linux-gnu/libcrypto.so.3
/usr/lib/x86_64-linux-gnu/libssl.so.3
/usr/lib/x86_64-linux-gnu/ossl-modules
/usr/lib/x86_64-linux-gnu/ossl-modules/legacy.so
/usr/share
/usr/share/doc
/usr/share/doc/libssl3
/usr/share/doc/libssl3/changelog.Debian.gz
/usr/share/doc/libssl3/changelog.gz
/usr/share/doc/libssl3/copyright

So one way to reproduce locally would be to run the console example in the MS provided container locally, i.e. mcr.microsoft.com/dotnet/runtime:8.0-bookworm-slim

[dmunch]

I was now able to reproduce the issue, running the simple example I used in #106 inside mcr.microsoft.com/dotnet/runtime:8.0-bookworm-slim.

Running the app gives the following output:

root@8121720c9c9d:/app/bin/Debug/net8.0# dotnet cli.dll t 30
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'delta_rs_bridge' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 
/app/bin/Debug/net8.0/runtimes/linux-arm64/native/delta_rs_bridge.so: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/delta_rs_bridge.so: cannot open shared object file: No such file or directory
/app/bin/Debug/net8.0/delta_rs_bridge.so: cannot open shared object file: No such file or directory
libssl.so.1.0.0: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/libdelta_rs_bridge.so: cannot open shared object file: No such file or directory
/app/bin/Debug/net8.0/libdelta_rs_bridge.so: cannot open shared object file: No such file or directory
/app/bin/Debug/net8.0/runtimes/linux-arm64/native/delta_rs_bridge: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/delta_rs_bridge: cannot open shared object file: No such file or directory
/app/bin/Debug/net8.0/delta_rs_bridge: cannot open shared object file: No such file or directory
/app/bin/Debug/net8.0/runtimes/linux-arm64/native/libdelta_rs_bridge: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/libdelta_rs_bridge: cannot open shared object file: No such file or directory
/app/bin/Debug/net8.0/libdelta_rs_bridge: cannot open shared object file: No such file or directory

   at DeltaLake.Bridge.Interop.Methods.runtime_new(RuntimeOptions* options)
   at DeltaLake.Bridge.Runtime..ctor(EngineOptions options)
   at DeltaLake.Kernel.Core.Runtime..ctor(EngineOptions options)
   at DeltaLake.Table.DeltaEngine..ctor(EngineOptions options)
   at local.Program.Main(String[] args) in /Users/danielmunch/Projects/stripe-recon/dl.net/cli/Program.cs:line 51
   at local.Program.<Main>(String[] args)
Aborted

Inspecting the dependencies of libdelta_rs_bridge.so we can indeed see that it's failing to load OpenSSL 1.0 - Which as mentioned before is not part of bookworm.

root@8121720c9c9d:/app/bin/Debug/net8.0# ldd --verbose runtimes/linux-arm64/native/libdelta_rs_bridge.so 
        linux-vdso.so.1 (0x0000ffffa5df7000)
        libssl.so.1.0.0 => not found
        libcrypto.so.1.0.0 => not found
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffffa0c30000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffffa0c00000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffa0b60000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffffa0b30000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffa0980000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffffa5dba000)

        Version information:
        runtimes/linux-arm64/native/libdelta_rs_bridge.so:
                ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
                libdl.so.2 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libdl.so.2
                libc.so.6 (GLIBC_2.18) => /lib/aarch64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libc.so.6
                libssl.so.1.0.0 (OPENSSL_1.0.2) => not found
                libssl.so.1.0.0 (OPENSSL_1.0.0) => not found
                libcrypto.so.1.0.0 (OPENSSL_1.0.2) => not found
                libcrypto.so.1.0.0 (OPENSSL_1.0.0) => not found
                libgcc_s.so.1 (GCC_4.2.0) => /lib/aarch64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.3) => /lib/aarch64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.0) => /lib/aarch64-linux-gnu/libgcc_s.so.1
                libpthread.so.0 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libpthread.so.0
                libm.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libm.so.6
        /lib/aarch64-linux-gnu/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.35) => /lib/aarch64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/aarch64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libc.so.6
        /lib/aarch64-linux-gnu/libpthread.so.0:
                libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libc.so.6
        /lib/aarch64-linux-gnu/libm.so.6:
                ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
                libc.so.6 (GLIBC_PRIVATE) => /lib/aarch64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libc.so.6
        /lib/aarch64-linux-gnu/libdl.so.2:
                libc.so.6 (GLIBC_2.17) => /lib/aarch64-linux-gnu/libc.so.6
        /lib/aarch64-linux-gnu/libc.so.6:
                ld-linux-aarch64.so.1 (GLIBC_2.35) => /lib/ld-linux-aarch64.so.1
                ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
                ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1

I'm however failing to understand where and when libdelta_rs_bridge.so is linked against OpenSSL 1.0 - The Github Action Workflows run against ubuntu-latest where OpenSSL 3.0 should be the default... ?

[mightyshazam]

The Linux arm target is built using cross. It appears that it is getting libssl 1.0. The native tls dependency is a regression introduced into delta-rs by the kernel's reqwest dependency.
We discovered it yesterday when I had to update the build to install SSL, and some others mentioned it in the delta slack. I don't think there is an issue to link yet, but the fix is coming.
Are you able to reproduce the issue in a linux x64 environment?

[dmunch]

I managed to start a x64 container on my ARM Mac - The app seems to run (albeit very slowly, likely due to x64 on ARM), and the OpenSSL dependency seems to be correctly resolved:

root@e2101bfd7aa9:/app/cli/bin/Debug/net8.0# ldd --verbose runtimes/linux-x64/native/libdelta_rs_bridge.so 
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x0000004007489000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x0000004007532000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00000040079b8000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00000040079d8000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000004007ab7000)
        /lib64/ld-linux-x86-64.so.2 (0x0000004000000000)

        Version information:
        runtimes/linux-x64/native/libdelta_rs_bridge.so:
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                libgcc_s.so.1 (GCC_4.2.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.3) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.0) => /lib/x86_64-linux-gnu/libgcc_s.so.1
                libcrypto.so.3 (OPENSSL_3.0.0) => /lib/x86_64-linux-gnu/libcrypto.so.3
                libm.so.6 (GLIBC_2.35) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.27) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
                libm.so.6 (GLIBC_2.29) => /lib/x86_64-linux-gnu/libm.so.6
                libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.32) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.29) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.18) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.9) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.25) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.16) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.28) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libssl.so.3 (OPENSSL_3.0.0) => /lib/x86_64-linux-gnu/libssl.so.3
        /lib/x86_64-linux-gnu/libssl.so.3:
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libcrypto.so.3 (OPENSSL_3.0.3) => /lib/x86_64-linux-gnu/libcrypto.so.3
                libcrypto.so.3 (OPENSSL_3.0.0) => /lib/x86_64-linux-gnu/libcrypto.so.3
        /lib/x86_64-linux-gnu/libcrypto.so.3:
                libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.25) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.16) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.35) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.35) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
root@e2101bfd7aa9:/app/cli/bin/Debug/net8.0# 

I'll see if I can try to reproduce OPs issue on Azure Functions itself, now that apparently locally it does work. (Maybe yesterday bumps of delta-rs versions did solve the original issue?)

[mightyshazam]

The PR is pending for delta-kernel delta-io/delta-kernel-rs#572