Connect using authentication - Cipher list not correct #104 [JIRA: CLIENTS-640]

[hytvi]

Hi,

How can I connect to a cluster using authentication?

Currently I am using

 var nodes = [];
 hosts.forEach(function (host) {
        var hostPort = host.split(':');

        nodes.push(new Riak.Node({
            remoteAddress: hostPort[0],
            remotePort: hostPort[1],
            auth: {
                user: "",
                password: ""
            },
            connectionTimeout: 5000,
            cork: true
        }));

    });

    client = new Riak.Client(new Riak.Cluster({nodes: nodes}));

But I cant seem to get a connection so far ( RiakConnection Timed out trying to connect)

I can't seem to find documentation for connecting using credentials either. Is there documentation avialable?

Looking forward to your reply

[lukebakken]

Do you have Riak Security enabled?

http://docs.basho.com/riak/latest/ops/running/authz/#Security-Checklist

[hytvi]

Hi Luke, Thanks for your reply. Yes, security is configured and enabled on the cluster. Currently I am using riak-pbc for connection to Riak, but I'd like to switch to the official client. In riak-pbc i am using the following code to connect (and its working).

 riak.createClient({
      nodes: {
         { host: "", port: 8087 },
         { host: "", port: 8087 },
         { host: "", port: 8087 }
    }
    auth: {
         user: "",
         password: ""
     }
 });

[lukebakken]

I don't think that security is enabled, since an empty user is not allowed.

Can you provide the output of these commands?

riak-admin security status
riak-admin security print-users
riak-admin security print-groups
riak-admin security print-sources

[hytvi]

I removed user credentials for security reasons, sorry for not marking it accordingly.

Below the output of your requested commands. (removed) means a removed user, password or host.

 $ sudo riak-admin security status
 Enabled

 $ sudo riak-admin security print-users
 +----------+---------------+----------------------------------------+------------------------------+
 | username |   member of   |                password                |           options            |
 +----------+---------------+----------------------------------------+------------------------------+
 |(removed) | applications  |(removed)|              []              |
 | (removed) |               |(removed)|              []              |
 |   (removed)   |               |(removed)|              []              |
 | (removed) |               |                                        |              []              |
 |(removed) | applications  |(removed)|              []              |
 |  (removed)   |     admin     |(removed)|              []              |
 |(removed) | applications  |(removed)|              []              |
 |   (removed)   |     admin     |(removed)|              []              |
 +----------+---------------+----------------------------------------+------------------------------+

 $ sudo riak-admin security print-groups
 +------------+---------------+------------------------------+
 |   group    |   member of   |           options            |
 +------------+---------------+------------------------------+
 |applications|               |              []              |
 |   admin    |               |              []              |
 +------------+---------------+------------------------------+


 $ sudo riak-admin security print-sources
 +--------------------+-----------------+----------+----------+
 |       users        |      cidr       |  source  | options  |
 +--------------------+-----------------+----------+----------+
 |       admin        |(removed)/32| password |    []    |
 |       admin        |(removed)/32 | password |    []    |
 |        all         |(removed)/32| password |    []    |
 |        all         |(removed)/32 | password |    []    |
 +--------------------+-----------------+----------+----------+

I am quite sure that the security setup is not the problem, as multiple applications (ruby, java, nodejs-pbc) are working with username and password.

[lukebakken]

Thanks for that information, your previous examples were confusing since they showed empty credentials.

Can you provide the /var/log/riak/error.log file from the Riak node to which you're connecting as well as the exact connection error text from your application?

Please also enable debug logging (example of how: https://github.com/basho/riak-nodejs-client/blob/master/test/debug-log.js) and re-try your connection. We may get more information in that output.

I am re-running the tests I have: https://github.com/basho/riak-nodejs-client/blob/master/test/security/security.js

I will add a test for connecting a Riak.Cluster object with security enabled.

[hytvi]

Yeah sorry for that, I should have pointed that out.

 $ tail -f /var/log/riak/error.log -n 0
 2015-10-21 15:55:13.204 [error] <0.16524.716> gen_fsm <0.16524.716> in state hello terminated with reason: no function clause matching tls_v1:enum_to_oid(28) line 404
 2015-10-21 15:55:13.204 [error] <0.16524.716> CRASH REPORT Process <0.16524.716> with 0 neighbours exited with reason: no function clause matching tls_v1:enum_to_oid(28) line 404 in gen_fsm:terminate/7 line 622
 2015-10-21 15:55:13.204 [error] <0.87.0> Supervisor ssl_connection_sup had child undefined started with {tls_connection,start_link,undefined} at <0.16524.716> exit with reason no function clause matching tls_v1:enum_to_oid(28) line 404 in context child_terminated
 2015-10-21 15:55:13.204 [error] <0.15593.716> gen_fsm <0.15593.716> in state wait_for_tls terminated with reason: {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...}
 2015-10-21 15:55:13.205 [error] <0.15593.716> CRASH REPORT Process <0.15593.716> with 0 neighbours exited with reason: {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...} in gen_fsm:terminate/7 line 622
 2015-10-21 15:55:13.205 [error] <0.16535.716> gen_fsm <0.16535.716> in state hello terminated with reason: no function clause matching tls_v1:enum_to_oid(28) line 404
 2015-10-21 15:55:13.205 [error] <0.16535.716> CRASH REPORT Process <0.16535.716> with 0 neighbours exited with reason: no function clause matching tls_v1:enum_to_oid(28) line 404 in gen_fsm:terminate/7 line 622
 2015-10-21 15:55:13.205 [error] <0.329.0> Supervisor riak_api_pb_sup had child undefined started with {riak_api_pb_server,start_link,undefined} at <0.15593.716> exit with reason {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...} in context child_terminated
 2015-10-21 15:55:13.205 [error] <0.87.0> Supervisor ssl_connection_sup had child undefined started with {tls_connection,start_link,undefined} at <0.16535.716> exit with reason no function clause matching tls_v1:enum_to_oid(28) line 404 in context child_terminated
 2015-10-21 15:55:13.205 [error] <0.16600.716> gen_fsm <0.16600.716> in state wait_for_tls terminated with reason: {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...}
 2015-10-21 15:55:13.206 [error] <0.16600.716> CRASH REPORT Process <0.16600.716> with 0 neighbours exited with reason: {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...} in gen_fsm:terminate/7 line 622
 2015-10-21 15:55:13.206 [error] <0.329.0> Supervisor riak_api_pb_sup had child undefined started with {riak_api_pb_server,start_link,undefined} at <0.16600.716> exit with reason {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...} in context child_terminated
 2015-10-21 15:55:18.354 [error] <0.16396.716> gen_fsm <0.16396.716> in state hello terminated with reason: no function clause matching tls_v1:enum_to_oid(28) line 404
 2015-10-21 15:55:18.354 [error] <0.16396.716> CRASH REPORT Process <0.16396.716> with 0 neighbours exited with reason: no function clause matching tls_v1:enum_to_oid(28) line 404 in gen_fsm:terminate/7 line 622
 2015-10-21 15:55:18.354 [error] <0.16641.716> gen_fsm <0.16641.716> in state wait_for_tls terminated with reason: {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...}
 2015-10-21 15:55:18.354 [error] <0.87.0> Supervisor ssl_connection_sup had child undefined started with {tls_connection,start_link,undefined} at <0.16396.716> exit with reason no function clause matching tls_v1:enum_to_oid(28) line 404 in context child_terminated
 2015-10-21 15:55:18.355 [error] <0.16641.716> CRASH REPORT Process <0.16641.716> with 0 neighbours exited with reason: {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...} in gen_fsm:terminate/7 line 622
 2015-10-21 15:55:18.355 [error] <0.329.0> Supervisor riak_api_pb_sup had child undefined started with {riak_api_pb_server,start_link,undefined} at <0.16641.716> exit with reason {{function_clause,[{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,[{file,"ssl_handshake.erl"},{line,1495}]},{ssl_handshake,dec_hello_extensions,2,[{file,"ssl_handshake.erl"},{line,1495}]},{tls_handshake,decode_handshake,3,[{file,"tls_handshake.erl"},{line,236}]},{tls_handshake,get_tls_handshake_aux,3,[{file,"tls_handshake.erl"},{...}]},...]},...} in context child_terminated

Output of application: (I did change the Ip adresses)

 error: [RiakConnection] Failed to connect: 1.1.1.1 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.2 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.3 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.3 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.1 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.3 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.2 port: 8087 error: RiakConnection Timed out trying to connect
 error: [RiakConnection] Failed to connect: 1.1.1.1 port: 8087 error: RiakConnection Timed out trying to connect
 ✖ testAuthenticatedConnection

 Assertion Message: Could not connect to Riak: RiakConnection Timed out trying to connect

[lukebakken]

Aha, I just diagnosed this issue the other day. You can read about it here: basho/riak-erlang-client#232

The default cipher list chooses a cipher that the Erlang VM running Riak can't use. I wonder how the other Node.js client you're using is choosing ciphers - can you provide a link to that client?

Here is an immediate workaround.

Change this:

https://github.com/basho/riak-nodejs-client/blob/master/lib/core/riakconnection.js#L55-L57

To this:

if (options.auth) {
    this.auth = options.auth;
    this.auth.ciphers = 'DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-SHA256:AES128-SHA:AES256-SHA256:AES256-SHA:RC4-SHA';
}

You can also use this method to set the cipher list via command-line arguments.

I will provide a fix for this in a future client release.

[hytvi]

Works like a charm! Thanks for helping me to debug this problem.

The other library can be found here:
https://github.com/nlf/riakpbc

Thanks again! We removed lots of insecure ciphers, maybe it had something to do with that?

(I will leave the issue open for the cipher fix)

[lukebakken]

We removed lots of insecure ciphers, maybe it had something to do with that?

Yep, I bet that was it. How did you remove the insecure ciphers?

The riakpbc library uses the same tls methods as the riak-nodejs-client and should have had the same connection issue if you used the same Node.js executable to run both.

[hytvi]

It has been quite a while ago since we have done that, but for what I remember we altered the ciphers in openssl somewhere.

I tried to google it again and I found out that I have clicked this page one day:
https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/

[lukebakken]

OK good to know. It was a global openssl change on the server running nodejs. Still, I have no idea how riakpbc was able to connect.