"ConnectionError: Couldn't log in: Unknown error 536871016"

[dfujiwara]

Since we upgraded our celery to version 3 and started using librabbitmq, we started to see this particular error quite frequently:

"ConnectionError: Couldn't log in: Unknown error 536871016"

I looked through the librabbitmq c code, but it seems like this is the server error. Given that, not sure if this is the right place to ask about this issue, but I was wondering if anyone out there has experienced a similar issue. Non-descriptive message certainly doesn't help and I don't really see any errors in the rabbitmq server log.

I have provided information below. If you have a good idea of how to investigate this issue further, I would much appreciate it!

Thanks in advance.

Here are what the versions of software we are using:
Django==1.4
celery==3.0.5
librabbitmq==0.9.9
billiard==2.7.3.12
kombu==2.3.2

And below is the stacktrace of what we see.

File "/opt/nextdoor-ve/lib/python2.6/site-packages/celery/app/task.py", line 343, in delay
return self.apply_async(args, kwargs)

File "/opt/nextdoor-ve/lib/python2.6/site-packages/celery/app/task.py", line 458, in apply_async
with app.producer_or_acquire(producer) as P:

File "/usr/lib/python2.6/contextlib.py", line 16, in enter
return self.gen.next()

File "/opt/nextdoor-ve/lib/python2.6/site-packages/celery/app/base.py", line 247, in producer_or_acquire
with self.amqp.producer_pool.acquire(block=True) as producer:

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/connection.py", line 705, in acquire
R = self.prepare(R)

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/pools.py", line 54, in prepare
p = p()

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/pools.py", line 45, in
return lambda: self.create_producer()

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/pools.py", line 42, in create_producer
return self.Producer(self._acquire_connection())

File "/opt/nextdoor-ve/lib/python2.6/site-packages/celery/app/amqp.py", line 160, in init
super(TaskProducer, self).init(channel, exchange, _args, *_kwargs)

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/messaging.py", line 83, in init
self.revive(self.channel)

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/messaging.py", line 174, in revive
channel = self.channel = maybe_channel(channel)

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/connection.py", line 879, in maybe_channel
return channel.default_channel

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/connection.py", line 617, in default_channel
self.connection

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/connection.py", line 610, in connection
self._connection = self._establish_connection()

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/connection.py", line 569, in _establish_connection
conn = self.transport.establish_connection()

File "/opt/nextdoor-ve/lib/python2.6/site-packages/kombu/transport/librabbitmq.py", line 108, in establish_connection
connect_timeout=conninfo.connect_timeout)

File "/opt/nextdoor-ve/lib/python2.6/site-packages/librabbitmq/init.py", line 168, in init
self.connect()

ConnectionError: Couldn't log in: Unknown error 536871016

[fetzig]

Couldn't log in

that should be enough evidence to try if that fixes it:

$ rabbitmqctl add_user joe joespassword
$ rabbitmqctl add_vhost something.local
$ rabbitmqctl set_permissions -p something.local joe ".*" ".*" ".*"

then make sure your broker url is correct by checking what those commands print out:

$ rabbitmqctl report
...
{ssl_listeners,[]},
...
{tcp_listeners,[{"127.0.0.1",5678}]},
...
$ rabbitmqctl list_vhosts
Listing vhosts ...
/
something.local
...done.
$ rabbitmqctl list_users
Listing users ...
joe []
guest   [administrator]
...done.

so the brocker url should look like this:

BROKER_URL = 'amqp://joe:[email protected]:5678/something.local'

[tommie]

I just had this same error. In my case it was due to librabbitmq connecting to a TLS-port on the server. librabbitmq doesn't support TLS, it seems, and so tries to send a plaintext header causing the server to close the connection immediately.

The error 536871016 (0x20000068) is a code created by librabbitmq-c, meaning ERROR_CATEGORY_CLIENT | ECONNRESET (on Linux.)

For completeness, the server logs said

error on AMQP connection <0.17345.1>: {ssl_upgrade_error,
                                       {tls_alert,"record overflow"}} (unknown POSIX error)