BUG: Using the 7th or higher registered token in Secretive appears to break SSH with "too many authentication failures"

[kmanwar89]

Hi,

I've been using Secretive since Day 1 of my recent M3 Pro purchase and I absolutely cannot stop raving about it to anyone I meet. Seriously, this is a grade-A piece of software (and has motivated me to emulate it for Linux hosts using a TPM....).

I believe there is a bug, however, in secrets that contain a number OR space in them. For example, I have secrets called "Pi 5", "Pi 4", etc. that, when I copy over to my devices (Pi 5 and Pi 4, respectively) and attempt to SSH into using Secretive, I am continuously met with the "too many authentication failures" error message.

Here is what my ~/.ssh/config file looks like:

Host *
        IdentityAgent /Users/kadar/Library/Containers/com.maxgoedjen.Secretive.SecretAgent/Data/socket.ssh

Host pi5
        HostName 192.168.##.##

Host prod
        HostName 192.168.##.##
        User my_user

Host prodts
        HostName 100.##.##.##
        User my_user

Host dev
        HostName 192.168.##.##

Host desk
        HostName 192.168.##.###
        User my_user

And the output from the command, showing the various keys I have created using Secretive (note the 123123 and "spaces in the name" keys:)

If I connect to my Pi5 using username password at first, copy over the public key associated with the Pi5 key and then try to SSH in from a separate terminal window, this error appears:

❯ ssh pi5
Received disconnect from 192.168.##.## port 22:2: Too many authentication failures
Disconnected from 192.168.##.## port 22

If I do the exact same thing but with any of the other keys, i.e. dev, prod, etc. the connection is fine! Notably, all other connections have been working flawlessly (dev, prod, desk, etc.) since I first installed Secretive. It was only after introducing these new filenames (spaces and/or numbers) that I've seen these failures.

Brief troubleshooting steps taken:

• Fresh OS wipe of various Linux machines (as a total nuclear option test)
• Un/re-install of SSH server
• Removal of ~/.ssh/known_hosts file to validate there was no duplicate or stale entry causing issues
• Comment out all of my SSH Config file except for the lines for secretive & a single host
• Reboots/restarts of SSH server on endpoints
• Manually trying the ssh -i command and specifying the full path to the secret works, which is interesting...

I'm happy to share more logs, or continue testing. I have access to....probably 15 different machines running Linux (or some variant thereof) that I'm happy to test this theory against. If there's something I'm fundamentally misunderstanding about how to use my configuration file (which I think might be the case?), I'd appreciate getting pointed in the right direction. Thanks!

Device: M3 Pro on Sonoma 14.5
Secretive Version 2.4.1 (1.7648958148)

[kmanwar89]

Another test - I just created a key called "M3_Mac" and tried to use it - same behavior. So seems like anything other than a single alphanumeric word (no spaces or special characters) causes the authentication to fail.

[kmanwar89]

I think the saga continues - the secret I created with "Pi 2" (both a number and a space) seem to work just fine!

So, now I've got conflicting behavior between machines - newer machines misbehaving while older machines are fine. Not entirely sure how to interpret this at this point!

[kmanwar89]

While investigating some more into this, this time on a totally different destination device (desktop, not a Pi-based device), I observed the following logs from the verbose SSH output:

debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug3: ssh_get_authentication_socket_path: path '/Users/kadar/Library/Containers/com.maxgoedjen.Secretive.SecretAgent/Data/socket.ssh'
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation <------------------------
debug1: get_agent_identities: agent returned 8 keys
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:scseUpzS/AH4V0E+nrwAgJa8fzHeY/fF9R3e9151wtI agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:NGJXDiSb9usouaGqHvl4vDqql2axyAm1EXBiZHBBMNo agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:hlnDtnCTR7Z3QGkXQxaiKZOZ1jICXHuN8LbVT3up7pc agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:nN4vrVNYsqYTCc/bBGfjMV2n7vmVPsb8+25SEO8O+wU agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:aDHIosixWxRWszENvaYFvwFy9Vm9/Bjqav0bm4Qwl6w agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:wbEEMMItCTtxrX6ZsC+xyPza1JfAWfTy42K1wRze0AI agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:e41o+zF5fcELN5iCua3n0rF6VMg1HcAfejwL5StLAiM agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:H6XLev8eUtyRn2SbPD70ZWO9qWqWadBt1C32slBkwOQ agent
debug1: Will attempt key: /Users/kadar/.ssh/id_rsa
debug1: Will attempt key: /Users/kadar/.ssh/id_ecdsa
debug1: Will attempt key: /Users/kadar/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/kadar/.ssh/id_ed25519
debug1: Will attempt key: /Users/kadar/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/kadar/.ssh/id_xmss
debug1: Will attempt key: /Users/kadar/.ssh/id_dsa
debug2: pubkey_prepare: done
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:scseUpzS/AH4V0E+nrwAgJa8fzHeY/fF9R3e9151wtI agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:NGJXDiSb9usouaGqHvl4vDqql2axyAm1EXBiZHBBMNo agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:hlnDtnCTR7Z3QGkXQxaiKZOZ1jICXHuN8LbVT3up7pc agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:nN4vrVNYsqYTCc/bBGfjMV2n7vmVPsb8+25SEO8O+wU agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:aDHIosixWxRWszENvaYFvwFy9Vm9/Bjqav0bm4Qwl6w agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:wbEEMMItCTtxrX6ZsC+xyPza1JfAWfTy42K1wRze0AI agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 1
Received disconnect from 192.168.1.181 port 22:2: Too many authentication failures
Disconnected from 192.168.1.181 port 22

This line stood out to me: debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation

It's not clear why this is happening, however. What's even stranger is the keys that I created nearly 1.5 months ago work just fine on almost any device I put them on - it seems to be only new(er) keys that are failing?

For instance:

1. I use my standard process to SSH into a machine that has one of the previously-created keys, works fine
2. I then generate a new key in Secretive on my Mac, and in the same window from step 1, add that key to ~/.ssh/authorized_keys
3. I open a separate terminal window and try to SSH in using the same process as step 1 - it fails.
4. I then comment out the key I just added, open a new SSH window - works totally fine.

[kmanwar89]

Hi @maxgoedjen , curious as to your thoughts on this one?

[CliqLabs]

Bump

[kmanwar89]

Bump

Are you experiencing this as well?

[maxgoedjen]

I'm not seeing this locally:

Semi-obvious question, but just to check: I notice there's no User specified in that SSH config for the Pi server. Does explicitly specifying the username change the behavior?

[kmanwar89]

I'm not seeing this locally:

Semi-obvious question, but just to check: I notice there's no User specified in that SSH config for the Pi server. Does explicitly specifying the username change the behavior?

No difference - here's some entries where the username is already predefined

I just did a bit more testing and think I may have found a more succinct error here - it appears that anything past the 6th entry created in Secretive is not accepted, regardless of the secret name.

I made a screen recording demonstrating this, but the file size is too large for me to upload to GitHub - if there is another way I can send this to you directly, I'm happy to do so. I'll keep trying to get the filesize down to a reasonable size in the meantime. The TL;DR to reproduce is:

1. Create 6 secrets in Secretive.
2. Login to another box using password, or one of these secrets, and edit ~/.ssh/authorized_keys
3. Create a 7th secret, of any name, in Secretive and add it to the remote box opened in step 2
4. Open another terminal and attempt to SSH in without closing the first terminal - it should fail
5. Comment out the 7th secret, add in any other secret from the list 1-6 and try again - it should succeed

This behavior appears to be consistent across versions of OpenSSH and is present on several different Linux machines - I've tried Raspberry Pi (2, 4 and 5), Beelink Mini PC running Debian Server, my Desktop running Zorin OS, and most recently this screen recording which is connecting to a Lenovo ThinkStation P7 Workstation. I also just tried it on my fiancé's M1 and had the exact same behavior - 7th secret failed to work, 1-6 worked fine.

Happy to continue to collaborate around this, but I think this is definitely the "smoking gun" - I'm really curious to see if you're able to reproduce this. Perhaps it's a limitation on the max # of entries in the secure enclave?

[Gellardo]

I ran into the same problem as well today. I don't think the "problem" is with the secure enclave but instead with the remote ssh server forcing a disconnect after 6 tries. See this stackoverflow question and the line from the above log:

Received disconnect from 192.168.1.181 port 22:2: Too many authentication failures

I have not yet had the time to figure out a workaround.

[Gellardo]

Workaround with a bit of configuration per host:

1. Store the public key in a file ~/.ssh/secretive.myname.pub
2. Configure ssh to explicitly use that key for the host (in ~/.ssh/config)

Host my.host.local
IdentityFile ~/.ssh/secretive.myname.pub
IdentitiesOnly true # optional, prevents other (default) identities being tried, only uses the configured one above

[kmanwar89]

Workaround with a bit of configuration per host:

1. Store the public key in a file ~/.ssh/secretive.myname.pub
2. Configure ssh to explicitly use that key for the host (in ~/.ssh/config)

Host my.host.local
IdentityFile ~/.ssh/secretive.myname.pub
IdentitiesOnly true # optional, prevents other (default) identities being tried, only uses the configured one above

Ty for sharing this. In the stackoverflow link you shared above, one comment states:

Second, if you use an ssh-agent, ssh will automatically try to use the keys in the agent, even if you have not specified them with in ssh_config's IdentityFile (or -i) option. This is a common reason you might get the Too many authentication failures for user error. Using the IdentitiesOnly yes option will disable this behavior.

So this appears a bit in contradiction to what you've stated - why would the server not try every key until it works? Or more specifically, and importantly, why is every key up to an arbitrary number able to work, but fails specifically on only the 7th iteration?

I asked a coworker to reproduce on his machine (Macbook Air), and he gets the same behavior. So, it is certainly reproducible.

Of course, one option could be to use a single, secure enclaved-backed keypair and secure it using TouchID, then share that with each endpoint, with the assumption that no one will break into your MacBook and bypass touchID/strong password, but that's assuming a lot, I think.

[kmanwar89]

I did a bit further testing, and confirmed that specifying the specific identity file per entry defined in the ~/.ssh/config file appears to work.

I also confirmed that the default # of Auth tries on the ssh server located in /etc/ssh/sshd_config defaults to 6, which explains why the 7th secret was failing. I increased this number to 20 and reproduce the same initially-reported behavior:

debug1: Next authentication method: publickey
debug3: ssh_get_authentication_socket_path: path '/Users/kadar/Library/Containers/com.maxgoedjen.Secretive.SecretAgent/Data/socket.ssh'
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation
debug1: get_agent_identities: agent returned 21 keys
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:scseUpzS/AH4V0E+nrwAgJa8fzHeY/fF9R3e9151wtI agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:NGJXDiSb9usouaGqHvl4vDqql2axyAm1EXBiZHBBMNo agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:hlnDtnCTR7Z3QGkXQxaiKZOZ1jICXHuN8LbVT3up7pc agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:nN4vrVNYsqYTCc/bBGfjMV2n7vmVPsb8+25SEO8O+wU agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:aDHIosixWxRWszENvaYFvwFy9Vm9/Bjqav0bm4Qwl6w agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:f9l9t4pjc27O1c4m4IGnMKjA/HEi+p5Ih87FnvC3HcU agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:SqJl7clekXZYz/mjQ0xrpb3hReTpTZnsTQvMFyQ9IVE agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:AbJvrL4Pc8leGq1SEEbl1O+c6ZvLTzPAVQF/m33QrAE agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:EE7saizJ+RYnd13uIC8SWh9Y7dfMi++Oy0q+LyB7sVA agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:MfXW2CJRTLM3HtdovO4SpWw8wIf/Bfgcn1CGtja18Ag agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:xxPPL1aposYQ5v4QRqhzUHoB0b4rqeEwDwSFrD2c/z8 agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:MT+nJelpWie5bxx5Ms4QnosnwtrK81VbBtGIpxkeJPk agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:yaC1fhQNlpnoAnFyC1toqEEx9+m7qJTdMhw58mF4V9o agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:Ae6CeiFmhNplPjnlYgWhvbseyiMm+mn0fofFOUY++JA agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:KpZlIpxw+TF+qEG8nen1A1uYkLjc9m122CSI0yN2OPM agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:XS6ULADSWN2pPwIx/Am3k/amFisGA5cQ+qwMnzfgLrw agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:bOR1xD4XuSWEjYiGutIbcEC68t9ElG2xIdqUd80JoFk agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:o9LN8GfaMtkB7GdaTnhuATA9+MGqaFqKGJ1X3vWQbig agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:wTDEfP4OVQJXxWmLRZEXJnWa2IHJRW584FTpj0ovzBg agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:HavHOjP9WhK8js24YVSE7HfAX9cEEAWdz5jQUbkL8ig agent
debug1: Will attempt key: ecdsa-sha2-nistp256 ECDSA SHA256:MiO9r31PO0r2Zei7opyTDxJ05hiXYSsS8Efr2u8lh1U agent
debug1: Will attempt key: /Users/kadar/.ssh/id_rsa
debug1: Will attempt key: /Users/kadar/.ssh/id_ecdsa
debug1: Will attempt key: /Users/kadar/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/kadar/.ssh/id_ed25519
debug1: Will attempt key: /Users/kadar/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/kadar/.ssh/id_xmss
debug1: Will attempt key: /Users/kadar/.ssh/id_dsa
---
Received disconnect from 192.168.1.10 port 22:2: Too many authentication failures
Disconnected from 192.168.1.10 port 22

So it indeed seems like it's more a core SSH behavior than it is tied to Secretive itself.