SIGILL from InitializeOpenSSLShim on alpine arm64 during lldb debugging

[am11]

(ran into it while debugging an unrelated nuget issue dotnet/diagnostics#4753)

On linux-musl-arm64, apk add lldb openssl-dbg and dotnet tool install -g dotnet-sos && dotnet-sos install, running dotnet build under lldb reveals a (non-fatal) SIGILL:

/runtime # lldb -- .dotnet/dotnet build -p:OutputRid=linux-musl-arm64 /runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Ref.sfxproj

(lldb) target create ".dotnet/dotnet"
Current executable set to '/runtime/.dotnet/dotnet' (aarch64).
(lldb) settings set -- target.run-args  "build" "-p:OutputRid=linux-musl-arm64" "/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Ref.sfxproj"
(lldb) r
Process 38495 launched: '/runtime/.dotnet/dotnet' (aarch64)
Process 38495 stopped
* thread #1, name = 'dotnet', stop reason = signal SIGILL: illegal opcode
    frame #0: 0x0000ffbf55eddc20 libcrypto.so.3
->  0xffbf55eddc20: sm3partw1 v4.4s, v0.4s, v3.4s
    0xffbf55eddc24: ret    
    0xffbf55eddc28: nop    
    0xffbf55eddc2c: nop    
(lldb) target symbols add /usr/lib/debug/usr/lib/libcrypto.so.3.debug
symbol file '/usr/lib/debug/usr/lib/libcrypto.so.3.debug' has been added to '/lib/libcrypto.so.3'
(lldb) clrstack -f
        Child SP               IP Call Site
0000FFFFFFFFBEC0 0000FFBF55EDDC20 libcrypto.so.3!_armv8_sm3_probe at /home/buildozer/aports/main/openssl/src/openssl-3.3.1/crypto/arm64cpuid.S:111
0000FFFFFFFFBEC0 0000FFBF55DD102C libcrypto.so.3!arm_probe_for + 44 at /home/buildozer/aports/main/openssl/src/openssl-3.3.1/crypto/armcap.c:242
0000FFFFFFFFBEE0 0000FFBF55DD11EC libcrypto.so.3!OPENSSL_cpuid_setup + 428 at /home/buildozer/aports/main/openssl/src/openssl-3.3.1/crypto/armcap.c:387
0000FFFFFFFFC0D0 0000FFFFF7FA82FC ld-musl-aarch64.so.1!do_init_fini + 340 at /home/buildozer/aports/main/musl/src/musl-1.2.5/ldso/dynlink.c:1608
0000FFFFFFFFC250 0000FFFFF7FAA058 ld-musl-aarch64.so.1!dlopen + 196 at /home/buildozer/aports/main/musl/src/musl-1.2.5/ldso/dynlink.c:2222
0000FFFFFFFFC4D0 0000FFBF562DD428 libSystem.Security.Cryptography.Native.OpenSsl.so!OpenLibraryOnce + 184 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/opensslshim.c:58
0000FFFFFFFFC4D0 0000FFBF562DD418 libSystem.Security.Cryptography.Native.OpenSsl.so!OpenLibraryOnce + 168 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/opensslshim.c:58
0000FFFFFFFFC530 0000FFFFF7FA09F4 ld-musl-aarch64.so.1!__pthread_once_full + 200 at /home/buildozer/aports/main/musl/src/musl-1.2.5/src/thread/pthread_once.c:23
0000FFFFFFFFC580 0000FFFFF7FA0A3C ld-musl-aarch64.so.1!pthread_once + 16 at /home/buildozer/aports/main/musl/src/musl-1.2.5/arch/aarch64/atomic_arch.h:20
0000FFFFFFFFC580 0000FFBF562DD5AC libSystem.Security.Cryptography.Native.OpenSsl.so!InitializeOpenSSLShim + 48 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/opensslshim.c:165
0000FFFFFFFFC580 0000FFBF562DD594 libSystem.Security.Cryptography.Native.OpenSsl.so!InitializeOpenSSLShim + 24 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/opensslshim.c:163
0000FFFFFFFFC5D0 0000FFBF562D1898 libSystem.Security.Cryptography.Native.OpenSsl.so!EnsureOpenSslInitializedOnce + 20 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/openssl.c:1494
0000FFFFFFFFC5D0 0000FFBF562D1894 libSystem.Security.Cryptography.Native.OpenSsl.so!EnsureOpenSslInitializedOnce + 16 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/openssl.c:1492
0000FFFFFFFFC600 0000FFFFF7FA09F4 ld-musl-aarch64.so.1!__pthread_once_full + 200 at /home/buildozer/aports/main/musl/src/musl-1.2.5/src/thread/pthread_once.c:23
0000FFFFFFFFC580 0000FFFFF7FA0A3C ld-musl-aarch64.so.1!pthread_once + 16 at /home/buildozer/aports/main/musl/src/musl-1.2.5/arch/aarch64/atomic_arch.h:20
0000FFFFFFFFC650 0000FFBF562D1874 libSystem.Security.Cryptography.Native.OpenSsl.so!CryptoNative_EnsureOpenSslInitialized + 28 at /__w/1/s/src/native/libs/System.Security.Cryptography.Native/openssl.c:1528
0000FFFFFFFFC660 0000FFFFF1F6AD34 
0000FFFFFFFFC680                  [InlinedCallFrame: 0000ffffffffc680] 
0000FFFFFFFFC680                  [InlinedCallFrame: 0000ffffffffc680] 
0000FFFFFFFFC660 0000FFFFF1F6AD1C System.Security.Cryptography.dll!Interop+CryptoInitializer..cctor() + 60 [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs @ 34]
0000FFFFFFFFD308                  [DynamicHelperFrame: 0000ffffffffd308] 
0000FFFFFFFFD4D0 0000FFFFF1F406B0 System.Security.Cryptography.dll!Interop+Crypto..cctor() + 272 [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Initialization.cs @ 18]
0000FFFFFFFFE188                  [DynamicHelperFrame: 0000ffffffffe188] 
0000FFFFFFFFE350 0000FFFFF1F4DA74 System.Security.Cryptography.dll!Interop+Crypto.HashAlgorithmToEvp(System.String) + 36
0000FFFFFFFFE380 0000FFFFF1FE92B8 System.Security.Cryptography.dll!System.Security.Cryptography.HashProviderDispenser+OneShotHashProvider.HashData(System.String, System.ReadOnlySpan`1<Byte>, System.Span`1<Byte>) + 56 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/HashProviderDispenser.Unix.cs @ 58]
0000FFFFFFFFE4A0 0000FFFFF1FC7EFC System.Security.Cryptography.dll!System.Security.Cryptography.SHA256.TryHashData(System.ReadOnlySpan`1<Byte>, System.Span`1<Byte>, Int32 ByRef) + 76 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/SHA256.cs @ 111]
0000FFFFFFFFE4C0 0000FFFFF1FC7E30 System.Security.Cryptography.dll!System.Security.Cryptography.SHA256.HashData(System.ReadOnlySpan`1<Byte>, System.Span`1<Byte>) + 32 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/SHA256.cs @ 84]
0000FFFFFFFFE4F0 0000FFFFF1FC7DF4 System.Security.Cryptography.dll!System.Security.Cryptography.SHA256.HashData(System.ReadOnlySpan`1<Byte>) + 100 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/SHA256.cs @ 66]
0000FFFFFFFFE520 0000FFFFF1FC7D7C System.Security.Cryptography.dll!System.Security.Cryptography.SHA256.HashData(Byte[]) + 108 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/SHA256.cs @ 54]
0000FFFFFFFFE550 0000FFFFF09D41F8 Microsoft.DotNet.Cli.Utils.dll!Microsoft.DotNet.Cli.Utils.Sha256Hasher.Hash(System.String) + 72
0000FFFFFFFFE570 0000FFFFF09DA784 Microsoft.DotNet.Cli.Utils.dll!Microsoft.DotNet.Cli.Utils.ApplicationInsightsEntryFormat+<>c__DisplayClass10_0.<WithAppliedToPropertiesValue>b__1(System.Collections.Generic.KeyValuePair`2<System.String,System.String>) + 68
0000FFFFFFFFE5A0 0000FFFFF04B3248 System.Linq.dll!System.Linq.Enumerable.ToDictionary[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>>, System.Func`2<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>,System.__Canon>, System.Func`2<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>,System.__Canon>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>) + 1880 [/_/src/libraries/System.Linq/src/System/Linq/ToCollection.cs @ 235]
0000FFFFFFFFE7D0 0000FFFFF04B22C0 System.Linq.dll!System.Linq.Enumerable.ToDictionary[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>>, System.Func`2<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>,System.__Canon>, System.Func`2<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>,System.__Canon>) + 128 [/_/src/libraries/System.Linq/src/System/Linq/ToCollection.cs @ 194]
0000FFFFFFFFE820 0000FFFFF09D52D8 Microsoft.DotNet.Cli.Utils.dll!Microsoft.DotNet.Cli.Utils.ApplicationInsightsEntryFormat.WithAppliedToPropertiesValue(System.Func`2<System.String,System.String>) + 264
0000FFFFFFFFE860 0000FFFFF160218C System.Linq.dll!System.Linq.Enumerable+ListSelectIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Fill(System.ReadOnlySpan`1<System.__Canon>, System.Span`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>) + 92 [/_/src/libraries/System.Linq/src/System/Linq/Select.SpeedOpt.cs @ 386]
0000FFFFFFFFE8B0 0000FFFFF1602114 System.Linq.dll!System.Linq.Enumerable+ListSelectIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].ToList() + 388 [/_/src/libraries/System.Linq/src/System/Linq/Select.SpeedOpt.cs @ 379]
0000FFFFFFFFE900 0000FFFFF064EF20 dotnet.dll!Microsoft.DotNet.Cli.Telemetry.TelemetryFilter.Filter(System.Object) + 1248
0000FFFFFFFFE970 0000FFFFF09D4CC8 Microsoft.DotNet.Cli.Utils.dll!Microsoft.DotNet.Cli.Utils.TelemetryEventEntry.SendFiltered(System.Object) + 72
0000FFFFFFFFE9B0 0000FFFFF06455D8 dotnet.dll!Microsoft.DotNet.Cli.Program.ProcessArgs(System.String[], System.TimeSpan, Microsoft.DotNet.Cli.Telemetry.ITelemetry) + 2280
0000FFFFFFFFEAB0 0000FFFFF0644854 dotnet.dll!Microsoft.DotNet.Cli.Program.Main(System.String[]) + 692

continuing (with c command) a few times continues the debugging.

cc @wfurt, @janvorli might be that there is nothing actionable for us.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[vcsjones]

For OpenSSL this is "normal". Based on the stack, _armv8_sm3_probe it's probing for available CPU features. See https://stackoverflow.com/a/29481360/492405 for another example.

You can do export OPENSSL_armcap=0 to disable the probing for the purpose of debugging (and will disable all CPU feature use).

[vcsjones]

Here is where OpenSSL does that probing. It installs a SIGILL handler to test for available CPU features. https://github.com/openssl/openssl/blob/89c9c3b857b5d68d835c3c3d371dc74a26f568fd/crypto/armcap.c#L368

since you are running under a debugger, you are getting a "first chance" error for the SIGILL.

[am11]

You can do export OPENSSL_armcap=0 to disable the probing for the purpose of debugging (and will disable all CPU feature use).

Thanks @vcsjones! After the export, I don't see SIGILL anymore. 👍