Need help with upgrading rabbitmq to 3.13.2 from 3.11.24 and erlang to 26.2.3.

[devfinwiz]

• Before upgrade, everything seemed to be running fine as per expectations.
• However, after upgrading the version of rabbitmq and erlang, deployment of my project started failing with the following error:

• Need help identifying the root cause.

[devfinwiz]

Additional information:

• All my certificates are in place as mentioned in the rabbitmq.conf file.
• auth_http.user_path, auth_http.vhost_path, auth_http.resource_path , auth_http.topic_path are also valid and correct.

[mkuratczyk]

Read the release notes https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.13.0
Specifically: TLS Defaults paragraph.

For more details: https://www.erlang.org/blog/otp-26-highlights/#ssl-safer-defaults

[devfinwiz]

ssl_options.verify = verify_none
ssl_options.fail_if_no_peer_cert = false

If i keep these attributes like this in my conf file (to disable the peer verification), it should work then right? I am pretty new to this stuff, any amount of patience would be appreciated in helping me figure out what changes might be required... :)

[mkuratczyk]

there are multiple places where TLS is configured (different protocols, shovels, etc) but generally speaking yes - if you set certificate verification to verify_none, it should work as before (meaning: in a non-secure way).

[devfinwiz]

Experiencing this issue now:
Why is it still looking for peer certificates?

8babc93db283a7a1ac906687611c15cf366c24923880e2a5d2a37fc1861e0cec: "2024-06-17T18:54:44.211747081+10:00  ^[[38;5;160m2024-06-17 08:54:44.211569+00:00 [error] <0.56349.0> Error on AMQP connection <0.56349.0> ([fd00::1:8:59]:49020 -> [fd00::1:8:58]:5671, state: starting):^[[0m"
8babc93db283a7a1ac906687611c15cf366c24923880e2a5d2a37fc1861e0cec: "2024-06-17T18:54:44.211747081+10:00  ^[[38;5;160m2024-06-17 08:54:44.211569+00:00 [error] <0.56349.0> EXTERNAL login refused: connection peer presented no TLS (x.509) certificate^[[0m"
8babc93db283a7a1ac906687611c15cf366c24923880e2a5d2a37fc1861e0cec: "2024-06-17T18:54:44.212522417+10:00  2024-06-17 08:54:44.212226+00:00 [info] <0.56349.0> closing AMQP connection <0.56349.0> ([fd00::1:8:59]:49020 -> [fd00::1:8:58]:5671)^[[0m"
8babc93db283a7a1ac906687611c15cf366c24923880e2a5d2a37fc1861e0cec: "2024-06-17T18:54:44.212846059+10:00  ^[[38;5;160m2024-06-17 08:54:44.212596+00:00 [error] <0.56323.0> Error on AMQP connection <0.56323.0> ([fd00::1:8:5c]:52466 -> [fd00::1:8:58]:5671, state: starting):^[[0m"
8babc93db283a7a1ac906687611c15cf366c24923880e2a5d2a37fc1861e0cec: "2024-06-17T18:54:44.212846059+10:00  ^[[38;5;160m2024-06-17 08:54:44.212596+00:00 [error] <0.56323.0> PLAIN login refused: rabbit_auth_backend_http failed authenticating api_gateway.0: {failed_connect,^[[0m"

[devfinwiz]

Another alternative to this? If we do not want to disable peer verification?

[mkuratczyk]

Fix you certificates and/or configure your CAs so that RabbitMQ can verify the certs

[devfinwiz]

Last question:
What is it with the certificates presently? What is the correct format to have the certificates configured to avoid this issue?

[mkuratczyk]

https://www.rabbitmq.com/docs/ssl

[devfinwiz]

Thanks @mkuratczyk ! Really appreciate your help.

[devfinwiz]

ssl_options.cacertfile = /rabbitmq/keys/cacert.pem
ssl_options.certfile = /rabbitmq/keys/server.cert.pem
ssl_options.keyfile = /rabbitmq/keys/server.key.pem

In my whole deployment process, there is only 1 ca certifcate involved and there are no intermediate certificates (which means I should be good here in my conf file as I need not append anything else to cacert.pem). I am still not getting past the rabbit_auth_backend connection refused error. What am I missing here? @mkuratczyk

[mkuratczyk]

Sorry mate, but we explicitly exclude TLS debugging from our community support. It's just too time consuming and not really RabbitMQ specific (RabbitMQ doesn't implement TLS - Erlang/OpenSSL do).
https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md#exceptions-question-that-will-be-ignored

[lukebakken]

If you are getting CLIENT ALERT - Unknown CA errors in your RabbitMQ logs, it means your client application does not recognize the CA presented by your RabbitMQ server, and you must ensure that your client application has access to the cacert.pem file.

[devfinwiz]

No worries! Thanks for responding! Appreciate it :)

[devfinwiz]

I know you guys mentioned that this is the area you don't offer help in but I just wanted to know the following:

1. Are self-signed certificates causing the problem? I have my own application to generate these certificates.
2. If yes, is it possible that it worked earlier with rabbit 3.11 and erlang 25 but not with newer versions?

[devfinwiz]

I see many old threads where folks faced the same issue (unknown ca) with much older versions of rabbit and erlang which is why i was wondering that if self signed certificates were not an issue earlier (while having tls enabled), why are they now?

[lukebakken]

No, if the certs worked before they will work with Erlang 26.

I'm 99.9999% sure this issue is due to your environment / configuration and is not a bug, but I want to verify that. Please create a git repository (on GitHub, for instance) that contains the following:

• Your FULL RabbitMQ configuration file.
• A new set of self-signed certs that you generate the same way as your production certs. Please use CN=localhost in the client and server certificates. I need only the public part of the CA, of course, but I need both the public and private key for the client and server certs.
• A README.md file describing your HTTP auth backend - what are you using for this?

If you do not want this information to be public, make the repo private and add me as a collaborator.

If you provide all of the above, I will assist you.

[devfinwiz]

I am not sure but I guess my issue might be similar to - https://groups.google.com/g/rabbitmq-users/c/l9slEl1fq_s .
In my rabbitmq.conf file I do not have seperate ssl_options configured for inbound and outbound connections.

I tried doing this but these options actually do not exist:

auth_ssl_options.cacertfile             = /rabbitmq/keys/cacert.pem
auth_ssl_options.verify                 = verify_none
auth_ssl_options.fail_if_no_peer_cert   = false

My guess/explanation can be very vague since I am very new to this...

I see this response from on of the users -

陈世飞（china clever eyes）
unread,
Jan 10, 2024, 8:21:57 AM
to rabbitmq-users
ok, It works when I add advanced config:

[
    {rabbitmq_auth_backend_http, [
        {ssl_options, [
            {verify, verify_none},
            {fail_if_no_peer_cert, false}
        ]}
    ]},
	{rabbitmq_event_exchange, [
      {vhost, <<"mqtt">>}
	]}
].

I was wondering how I can have the same setting in my conf file (seperating ssl_options for inbound and outbound connections).

[michaelklishin]

TLS troubleshooting is explicitly limited to paying customers and regular contributors.

[devfinwiz]

Issue resolved!!!!!
Ignore my earlier query, needs no attention.

Thank you so much @mkuratczyk , @lukebakken , @michaelklishin , @potatosalad for bearing with me on this! :)