Support running CCF on Azure Linux #6588

maxtropets · 2024-10-23T10:17:15Z

In this issue have to support building and testing CCF on Azure Linux.

The major obstacle so far - openssl.

On Azure Linux, we shall use OpenSSL of (major) version 3. It is to be determined in this ticket which exact version is supported by Azure Linux.

Ticket for reference: #5291

Subtasks to track the progress

Try fix ./crypto_test (patched, see discussions in comments)
What backend is used in Azure Linux? SymCrypt? Prove by tracking the lib we link against
breakdown initial issues

The text was updated successfully, but these errors were encountered:

achamayou · 2024-10-23T15:16:30Z

@maxtropets current is 3.3.2, so that one :)

maxtropets · 2024-10-25T14:26:07Z

Currently removed the SECP256K1 curve from crypto_test, fails on EVP_PKEY_CTX_set_ec_paramgen_curve_nid with err = 0.

Checked OpenSSL source, same NID for 3.1 and 3.3 impl, NID matches.

#define NID_secp256k1           714

Needs investigation.

maxtropets · 2024-10-25T14:30:24Z

Tried all other unit tests (crypto test patched with #6591)

 ./tests.sh -VV -L unit
...
The following tests FAILED:
	 11 - crypto_test (Failed)
	 13 - key_exchange_test (SEGFAULT)
	 18 - endorsements_test (Failed)
	 19 - historical_queries_test (SEGFAULT)
	 21 - snapshotter_test (SEGFAULT)
	 27 - channels_test (SEGFAULT)
	 28 - http_test (Failed)
	 37 - csr_test (Failed)

achamayou · 2024-10-28T11:37:40Z

256k1 is not a NIST standard, which is likely why SymCrypt rejects it - it would be good to confirm and document though

maxtropets · 2024-10-28T12:32:40Z

What backend is used in Azure Linux? SymCrypt? Prove by tracking the lib we link against

Figured out it's not explicitly linked, but rather being loaded during runtime

openat(AT_FDCWD, "/usr/lib/libsymcrypt.so.103", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=778048, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 557528, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f2af8423000
mmap(0x7f2af842f000, 380928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f2af842f000
mmap(0x7f2af848c000, 118784, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x69000) = 0x7f2af848c000
mmap(0x7f2af84a9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x86000) = 0x7f2af84a9000

Confirmed dependency chain ccf -> librypto -> libsymcrypt (monolithic lib):

root [ /workspace/build ]# ldd /usr/lib/libsymcrypt.so.103
	statically linked

maxtropets · 2024-10-28T12:35:14Z

Tried all other unit tests (crypto test patched with #6591)

 ./tests.sh -VV -L unit
...
The following tests FAILED:
	 11 - crypto_test (Failed)
	 13 - key_exchange_test (SEGFAULT)
	 18 - endorsements_test (Failed)
	 19 - historical_queries_test (SEGFAULT)
	 21 - snapshotter_test (SEGFAULT)
	 27 - channels_test (SEGFAULT)
	 28 - http_test (Failed)
	 37 - csr_test (Failed)

Split into smaller sub-issues except http_test, doesn't look openssl related, looking into it

UPD: Fixed in #6598

maxtropets · 2024-10-29T14:07:53Z

Got to the bottom of UVM endorsements test failure, filed a ticket (#6600)

So far, all the known issues are filed in sub-issues or solved.

Next steps

solve all sub-issues
rebase Enable unit tests for Azure Linux in CI #6591 on fixed main and rerun CI
file/solve more issues if any
extend CI with long tests
wrap up the Enable unit tests for Azure Linux in CI #6591 itself and merge

maxtropets · 2024-12-20T14:13:02Z

Dumping to not lose, how to run the container + tests

docker run -ti --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $(pwd):/workspace mcr.microsoft.com/azurelinux/base/core:3.0 /bin/bash

gpg --import /etc/pki/rpm-gpg/MICROSOFT-RPM-GPG-KEY && tdnf -y update && tdnf -y install ca-certificates git && tdnf -y install build-essential clang cmake ninja-build which && tdnf -y install openssl-devel libuv-devel perl && tdnf -y install libarrow-devel parquet-libs-devel && tdnf -y install lldb strace npm jq expect

git config --global --add safe.directory /workspace

CC=`which clang` CXX=`which clang++` cmake -GNinja -DCOMPILE_TARGET=virtual -DCMAKE_BUILD_TYPE=Debug -DLVI_MITIGATIONS=OFF -DUSE_LIBCXX=OFF ..

maxtropets · 2024-12-27T18:48:02Z

Currently running standard e2e tests set (no perf, no ASAN/TSAN, etc) is blocked by

./tests.sh -VV -R raft_scenario_test
- Node n[2] detail leadership_state is not as expected: Follower != Leader on line 46 <br>
- Node n[2] not at expected commit idx (13) on line 95 : 9 <br>
- Looks like a proper raft failure, needs investigation
./tests.sh -VV -R "programmability|modules_test"
- Only JWK API part is broken, pem missmatch for certain keys, needs investigation
./tests.sh -VV -R lts_compatibility
- no dpkg, needs other way of installing CCF releases
- possibly more problems can turn up, not clear at the moment

maxtropets · 2024-12-30T14:26:52Z

./tests.sh -VV -R "programmability|modules_test"
Only JWK API part is broken, pem missmatch for certain keys, needs investigation

Checked it in-depth, key things to mention

this's random failure
the part of a test that fails is in a nutshell
- create rsa key pair locally (on the python side)
- call python.jwk to construct a JWK for the private key
- call ccfcrypto.rsaPemToJwk and assert JWK is the same (always passes (!))
- pass that JWK to ccfcrypto.rsaJwkToPem and verify the obtain PEM against the created one
fails, because they sometimes (!) don't match
even though they don't match, they seem to represent the same private key (at least this online checker accepts both PEMs: https://www.sslchecker.com/matcher)

As soon as both PEMs represent the same private key, it's to be figured out what metadata encoded makes the difference here

maxtropets · 2024-12-30T14:33:36Z

Example

Priv key converted to JWK and back to PEM

The original one

Public key

maxtropets · 2024-12-30T16:05:37Z

Tried ASN1 decoder, surely there's a difference in metadata, trying to figure out what exactly is it

maxtropets · 2024-12-30T16:16:44Z

Converted the key obtained from CCF back to JWK with python.jwk, here's the the original pem->jwk on the left and new->jwk on the right

The d parameter doesn't match.

maxtropets · 2024-12-30T16:48:22Z

I've also tried a simple python test with sign/verify with both keys (different pems), worked out just well.

Learned some RSA docs, seems d is a defining parameter for a private key, so these are actually two different private keys. Verification with a public key works, because same n and e are applied.

Will dig up the implementation, but seems like either openssl version bump-up issue or a symcrypt problem.

maxtropets · 2024-12-31T11:45:10Z

Current summary

generating fresh key and getting it's PEM from python crypto
converting that PEM to JWK with both python.jwk and ccfcrypto - they match
converting JWK back to PEM* with python.jwk and ccfcrypto - they differ
converting PEM* back to JWK* with python.jwk and ccfcrypto - they match

Tried running local ccf crypto test as follows

generate new key
get its jwk and create a new key with it
match

That always passes, meaning there's something wrong with creating the key from parameters.

Open question - is it possible that PEM->JWK on one openssl version and then JWK->PEM on another openssl version aren't compatible?

maxtropets · 2024-12-31T11:46:55Z

Also, found this online jwk->pem->jwk thing: https://8gwifi.org/jwkconvertfunctions.jsp

JWK->PEM there matches the CCF version

maxtropets · 2024-12-31T12:10:22Z

Saving an interesting post here: https://crypto.stackexchange.com/questions/76253/requirements-for-rsa-key-parameters

maxtropets · 2024-12-31T12:21:05Z

Reproduced it in another way, getting a pem from a failed test, then unit-testing as follows

  const auto pem =
    "-----BEGIN PRIVATE KEY-----\n"
    "MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDMm8GpgeYyat7+\n"
    "K4FgIIuftNj2f8asetGaqcG3idkd11DMEXRs6CEpsiUEIN7JEOsqyhL7lSgblfia\n"
    "HFV6Cius5kJObINgDicDpITRjAn1XHOSGZiotMXiOe565/Zzo1gxg9Bu0TroG0H1\n"
    "pt4f9ekrhv7ZkVP6hxSCQ/j/dJyHS7xXvV5LIeERJTYssuyr0749InE1Sss/p77Y\n"
    "2rPJR8GF4S9JCTJEcE2NMNa8WxO4g6Gg+uS+P3TPRUlc9H2lvLf31QDbtGKhRG1F\n"
    "wYqHYRuk8EcB6bH6xiNQ0a5pwNVcw2bIGbwfJeQO9dnDjISWIt8sZQr7kLzB2OeP\n"
    "+BSdRA2hAgMBAAECggEABvQsNXSYEZKW7ZB7nya8AruEvCvvi/ZzSoxbwb8fmpIH\n"
    "dEgLlZVLZegLYK6Ko869ZKuQJZjAz4nd9G7ja3hjr5v2bkU7LyKL/5BcgI//2U+W\n"
    "ZfAUw3cRWp+ZaU1E5P03hsdHaDIYNxc7ZfvmhLOOLyUxYszmS8IXgliJDTWLXwBh\n"
    "C+OQACa2LC9x2CBJEWMaO0yAeFa601UeyNgAL4y8LAXED6Yk+PgM8cQ5QAJPKopC\n"
    "GQYVv1scxyimwBJar4CUtgCVLUsJ9xrZ1IaH9+GIZsy4eMu8TCJ9fchlCHSthObm\n"
    "2/IAGaDfHRBM6PBubtE2nUFqr5KgvKlcE/nnOcscWQKBgQDow0YXcg0vYsdSEc4S\n"
    "a0wMzE2XSGRT8mV43zCCd5azamq6LeO5nGwbcTxi19MmdlrWCAG1dlpTAtx7DkrI\n"
    "/187edqzWhfO6rgpTGwMc+309/WUkpW56Lpa9cY8gaLwvmz6XEcWhj7SUBoa2mP4\n"
    "dqnuvieukYXp9H1UEaaB2FOa+wKBgQDhCPDQkuNjRL8fia1Q+twF6R+CZqu3C6zX\n"
    "8poPWY3+T/UHwSqcPCpw4pUrTaH/PniQX4+paCGmP/JuwD7huJv8nduYJvDfK8Pd\n"
    "EDPzfircxa0pLkAmrDL6+Pq5TIbydYZ+e9CmQMAPJdTaRrcQJ8nXtDGRUvsUeUpT\n"
    "0ZxfivkXEwKBgHjrKPh5GXxqdl2ePk23S/TCFfTneKdbn3dhixPcCIAAYp2BxXHh\n"
    "XxAKjXJT6K9sgNf7pNeeOmIxdFTBIfy1HcYKEaFmxkOKEE30E5WMjCvcvMmINRw1\n"
    "WMp0L52zQvCKHJcbZRrAdtZPIqCcn56uFq2eP4MiukzF5G3OFZASz6wNAoGBAM3b\n"
    "j/A9byXwTaZcG+Bom74MdHal1Y+tboBdvfEsy2hJXK2rV390yp5ZqHvkmL0WONRZ\n"
    "luIyEBY1iWEnHF8AsjF1dtzJQq/lKVeg6uVkFXHAo5d4Z9b4z+omubWo/Anw2Qf7\n"
    "2/M6aHdy9DR5XB27SDm67V0zu9wO2J+0sIf2acdhAoGBALp+50gE+yOZqZLUucTG\n"
    "pw5RpAnBjgIlnOpfKXkgQdYsNyj2n3hGRPoh42Or9gL8PmEupDa2V3SiuKLCevFH\n"
    "DZT1zBqh3f+3q5Tko7nYL13+uWlwz4COa551LYeTjKDqSEq40eBQ7HFTWOW0+zqF\n"
    "frD1OBFK+2NytBQFs+EvbWUC\n"
    "-----END PRIVATE KEY-----\n";

  const auto key = ccf::crypto::make_rsa_key_pair(ccf::crypto::Pem(pem));

  const auto jwk = key->private_key_jwk_rsa();
  std::cout << "JWK: d old=" << jwk.d << std::endl;

  const auto key2 = ccf::crypto::make_rsa_key_pair(jwk);
  const auto jwk2 = key2->private_key_jwk_rsa();
  std::cout << "JWK: d new=" << jwk2.d << std::endl;

  REQUIRE(jwk.d == jwk2.d);

This passes on our current Ubuntu 20.04 with openssl 1.1.1, but fails on Mariner with openssl 3.3

maxtropets added the enhancement label Oct 23, 2024

maxtropets self-assigned this Oct 23, 2024

maxtropets changed the title ~~Upgrade OpenSSL to v.3+~~ Solve OpenSSL on Azure Linux Oct 25, 2024

maxtropets mentioned this issue Oct 25, 2024

Enable unit tests for Azure Linux in CI #6591

Merged

maxtropets mentioned this issue Oct 28, 2024

Drop SECP256K1 support #6592

Closed

maxtropets mentioned this issue Nov 8, 2024

Azure Linux: tweak dataclass for compatibility with python 3.12 #6620

Merged

achamayou removed the enhancement label Nov 12, 2024

maxtropets changed the title ~~Solve OpenSSL on Azure Linux~~ Support CCF testing&running on Azure Linux Nov 15, 2024

maxtropets changed the title ~~Support CCF testing&running on Azure Linux~~ Support Azure Linux in CCF Nov 15, 2024

maxtropets changed the title ~~Support Azure Linux in CCF~~ Support running CCF on Azure Linux Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support running CCF on Azure Linux #6588

Support running CCF on Azure Linux #6588

maxtropets commented Oct 23, 2024 •

edited

Loading

achamayou commented Oct 23, 2024 •

edited

Loading

maxtropets commented Oct 25, 2024

maxtropets commented Oct 25, 2024 •

edited

Loading

achamayou commented Oct 28, 2024

maxtropets commented Oct 28, 2024

maxtropets commented Oct 28, 2024 •

edited

Loading

maxtropets commented Oct 29, 2024

maxtropets commented Dec 20, 2024 •

edited

Loading

maxtropets commented Dec 27, 2024 •

edited

Loading

maxtropets commented Dec 30, 2024 •

edited

Loading

maxtropets commented Dec 30, 2024

maxtropets commented Dec 30, 2024 •

edited

Loading

maxtropets commented Dec 30, 2024

maxtropets commented Dec 30, 2024 •

edited

Loading

maxtropets commented Dec 31, 2024

maxtropets commented Dec 31, 2024

maxtropets commented Dec 31, 2024

maxtropets commented Dec 31, 2024

Support running CCF on Azure Linux #6588

Support running CCF on Azure Linux #6588

Comments

maxtropets commented Oct 23, 2024 • edited Loading

achamayou commented Oct 23, 2024 • edited Loading

maxtropets commented Oct 25, 2024

maxtropets commented Oct 25, 2024 • edited Loading

achamayou commented Oct 28, 2024

maxtropets commented Oct 28, 2024

maxtropets commented Oct 28, 2024 • edited Loading

maxtropets commented Oct 29, 2024

maxtropets commented Dec 20, 2024 • edited Loading

maxtropets commented Dec 27, 2024 • edited Loading

maxtropets commented Dec 30, 2024 • edited Loading

maxtropets commented Dec 30, 2024

maxtropets commented Dec 30, 2024 • edited Loading

maxtropets commented Dec 30, 2024

maxtropets commented Dec 30, 2024 • edited Loading

maxtropets commented Dec 31, 2024

maxtropets commented Dec 31, 2024

maxtropets commented Dec 31, 2024

maxtropets commented Dec 31, 2024

maxtropets commented Oct 23, 2024 •

edited

Loading

achamayou commented Oct 23, 2024 •

edited

Loading

maxtropets commented Oct 25, 2024 •

edited

Loading

maxtropets commented Oct 28, 2024 •

edited

Loading

maxtropets commented Dec 20, 2024 •

edited

Loading

maxtropets commented Dec 27, 2024 •

edited

Loading

maxtropets commented Dec 30, 2024 •

edited

Loading

maxtropets commented Dec 30, 2024 •

edited

Loading

maxtropets commented Dec 30, 2024 •

edited

Loading