FIPS enabled system unable to use TLS

[h0tw1r3]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The documentation at https://vector.dev/docs/reference/configuration/tls/#fips-provider-example suggests that openssl configured in "FIPS mode" should or was working at one time.

Any attempt to set up a TLS input, or connect to a sink TLS endpoint that uses openssl fails.

Configuration

---
api:
  enabled: true
  address: 0.0.0.0:8686

sources:
  syslog:
    type: socket
    address: 0.0.0.0:5000
    mode: tcp
    tls:
      ca_file: /etc/vector/ssl/ca.crt
      crt_file: /etc/vector/ssl/server.crt
      key_file: /etc/vector/ssl/server.key
      key_pass: password123
      enabled: true
      verify_certificate: true
      verify_hostname: false

sinks:
  testing:
    type: console
    inputs: [ syslog ]
    encoding:
      codec: json

  datadog:
    type: datadog_logs
    inputs: [ syslog ]
    tls:
      enabled: true
    site: ${DD_SITE-datadoghq.com}
    default_api_key: ${DD_API_KEY-testing}

Version

0.40.1

Debug Output

2024-09-09T03:40:55.312585Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-09-09T03:40:55.312628Z  INFO vector::app: Log level is enabled. level="trace"
2024-09-09T03:40:55.312681Z DEBUG vector::app: messaged="Building runtime." worker_threads=4
2024-09-09T03:40:55.313849Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-09-09T03:40:55.314984Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2024-09-09T03:40:55.315302Z DEBUG vector::topology::builder: Building new source. component=syslog
2024-09-09T03:40:55.316659Z DEBUG vector::topology::builder: Building new sink. component=datadog
2024-09-09T03:40:55.316906Z DEBUG vector::topology::builder: Building new sink. component=testing
2024-09-09T03:40:55.316996Z ERROR vector::topology::builder: Configuration error. error=Source "syslog": Could not parse private key in "/etc/vector/ssl/server.key": error:1E000066:DECODER routines:OSSL_DECODER_from_bio:reason(102):crypto/encode_decode/decoder_lib.c:60:No decoders were found. For standard decoders you need at least one of the default or base providers available. Did you forget to load them?, error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:355:Global default library context, Algorithm (AES-256-CBC : 9), Properties ()
2024-09-09T03:40:55.317011Z ERROR vector::topology::builder: Configuration error. error=Sink "datadog": Failed to build TLS connector: Could not build TLS connector: error:0A0000A1:SSL routines:SSL_CTX_new_ex:library has no ciphers:ssl/ssl_lib.c:3955:

Example Data

Test workflows:
FIPS (not working): https://github.com/h0tw1r3/vector-fips/actions/runs/10766127156/job/29851381228
Default (working): https://github.com/h0tw1r3/vector-fips/actions/runs/10766176636/job/29851509861

Additional Context

Originally discovered this issue while trying to run vector on Ubuntu 22.04 with Pro and FIPS enabled. Was able to create a alpine container configured with the fips provider to reproduce the issue.

Project and pre-built image here: https://github.com/h0tw1r3/alpine-fips/pkgs/container/alpine-fips

Created a project to easily track testing different versions of vector and alpine fips configurations here: https://github.com/h0tw1r3/vector-fips

Excerpt from an strace shows that the fips provider is never opened.

openat(AT_FDCWD, "/etc/vector/ssl/server.crt", O_RDONLY|O_CLOEXEC) = 9
statx(9, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=1675, ...}) = 0
openat(AT_FDCWD, "/etc/ssl/openssl.cnf", O_RDONLY) = 9
newfstatat(9, "", {st_mode=S_IFREG|0644, st_size=12815, ...}, AT_EMPTY_PATH) = 0
newfstatat(AT_FDCWD, "/etc/ssl/fipsmodule.cnf", {st_mode=S_IFREG|0644, st_size=351, ...}, 0) = 0
openat(AT_FDCWD, "/etc/ssl/fipsmodule.cnf", O_RDONLY) = 10
newfstatat(10, "", {st_mode=S_IFREG|0644, st_size=351, ...}, AT_EMPTY_PATH) = 0
openat(AT_FDCWD, "/etc/vector/ssl/server.key", O_RDONLY|O_CLOEXEC) = 9
statx(9, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0600, stx_size=3446, ...}) = 0

References

No response

[h0tw1r3]

First piece of the puzzle. Found that openssl-src does not build with DSO API support.
Opened PR alexcrichton/openssl-src-rs#251

[jszwedko]

Thanks for looking into this @h0tw1r3 ! I see the openssl-src PR was merged so we can include it in the next openssl-src version bump.

I have a vague memory that some people using the OpenSSL FIPS module with Vector have been rebuilding Vector with OpenSSL dynamically linked as opposed to using the distributed artifacts which have OpenSSL statically linked, but it seems like it may be possible to include the module dynamically? I haven't played with it myself.

@hhromic wrote the doc that you linked to. cc/ing him here for possible input.