Failing authentication for jj git fetch ... / jj git clone ... / etc. with a FIDO2 (resident) key

[DustVoice]

Description

I know, I might have encountered a very special and specific scenario.
Nevertheless, I can't get any git remote command to work properly.

For SSH authentication, I'm using a Yubikey using a FIDO2 resident key.

In particular, the key was generated using the following command ssh-keygen -t ed25519-sk -O resident -O user=git -O application=ssh:git -O verify-required.
As I read on several occasions, the verify-required option can sometimes cause problems (it not only requires touch but also the entry of a PIN), so I also tried without: ssh-keygen -t ed25519-sk -O resident -O user=git -O application=ssh:git.
After that also didn't work, I tried a non-resident key with ssh-keygen -t ed25519-sk -O user=git -O application=ssh:git, still to no avail.

In every scenario, the key was present at ~/.ssh/id_ed25519_sk and present in the ssh-agent confirmed using ssh-add -L.
Each time I get prompted for touching the hardware token (or in case of verify-required also for the PIN) without issue but then it fails with

Error: failed to authenticate SSH session: ; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

Of course I also tried the mentioned command: ssh -F /dev/null [email protected] -vv without any issues.
It should also go without saying that doing things like git fetch or git pull work without any issues.

> ssh -F /dev/null [email protected] -vv
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
debug1: Reading configuration data /dev/null
debug2: resolving "github.com" port 22
debug1: Connecting to github.com [140.82.121.3] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\info/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\info/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa_sk type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\info/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ed25519_sk type 12
debug1: identity file C:\\Users\\info/.ssh/id_ed25519_sk-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\info/.ssh/id_xmss-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\info/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_9.5
debug1: Remote protocol version 2.0, remote software version babeld-3d60e350f
debug1: compat_banner: no match: babeld-3d60e350f
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to github.com:22 as 'git'
debug1: load_hostkeys: fopen C:\\Users\\info/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,[email protected]
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
debug2: MACs stoc: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
debug1: load_hostkeys: fopen C:\\Users\\info/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'github.com' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\info/.ssh/known_hosts:1
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug2: get_agent_identities: ssh_agent_bind_hostkey: invalid format
debug1: get_agent_identities: agent returned 1 keys
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ed25519_sk ED25519-SK SHA256:EiWoeWPBZv3JXnLqAJd+Y+1iHkutP7VXdmXzoj5EM9w authenticator agent
debug1: Will attempt key: C:\\Users\\info/.ssh/id_rsa
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ecdsa_sk
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\info/.ssh/id_xmss
debug1: Will attempt key: C:\\Users\\info/.ssh/id_dsa
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\info/.ssh/id_ed25519_sk ED25519-SK SHA256:EiWoeWPBZv3JXnLqAJd+Y+1iHkutP7VXdmXzoj5EM9w authenticator agent
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: C:\\Users\\info/.ssh/id_ed25519_sk ED25519-SK SHA256:EiWoeWPBZv3JXnLqAJd+Y+1iHkutP7VXdmXzoj5EM9w authenticator agent
Authenticated to github.com ([140.82.121.3]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug2: channel 0: send open
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: ENABLE_VIRTUAL_TERMINAL_INPUT is supported. Reading the VTSequence from console
debug1: ENABLE_VIRTUAL_TERMINAL_PROCESSING is supported. Console supports the ansi parsing
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: client_input_hostkeys: searching C:\\Users\\info/.ssh/known_hosts for github.com / (none)
debug1: client_input_hostkeys: searching C:\\Users\\info/.ssh/known_hosts2 for github.com / (none)
debug1: client_input_hostkeys: hostkeys file C:\\Users\\info/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug2: channel 0: request shell confirm 1
debug1: pledge: fork
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 32000 rmax 35000
debug2: channel_input_status_confirm: type 100 id 0
PTY allocation request failed on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug2: channel 0: rcvd ext data 3
debug2: channel 0: rcvd ext data 9
debug2: channel 0: rcvd ext data 2
Hi DustVoice! debug2: channel 0: written 2044404432910 to efd 6
debug2: channel 0: rcvd ext data 76
debug2: channel 0: rcvd ext data 1
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: rcvd close
debug2: chan_shutdown_read: channel 0: (i0 o1 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug1: channel 0: free: client-session, nchannels 1
Connection to github.com closed.
Transferred: sent 2192, received 2668 bytes, in 6.0 seconds
Bytes per second: sent 368.3, received 448.3

Executing jj with the --debug flag produces the following output

> jj git fetch --debug
2024-10-06T15:36:21.468008Z  INFO jj_cli::cli_util: debug logging enabled
2024-10-06T15:36:21.510254Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.download
2024-10-06T15:36:22.242154Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key_from_agent username="git"
2024-10-06T15:36:43.313220Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}:get_ssh_keys{_username="git"}: jj_cli::git_util: found ssh key path="C:\\Users\\info\\.ssh\\id_ed25519_sk"
2024-10-06T15:36:43.313429Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key username="git" path="C:\\Users\\info\\.ssh\\id_ed25519_sk"
Error: failed to authenticate SSH session: ; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

As everything works with git and even non-resident keys don't work with jj I'm assuming that the ed25519-sk format might somehow be the culprit?
It's weird as in my experience, a non-support of this SSH feature normally results in nothing working at all, but in this case, with jj the PIN entry opens, and I'm able to touch the key.
I also researched in the libssh2 repo, but every related issue I could find regarding FIDO2 and the key format has apparently been resolved and closed.

It is probably also worth noting that signing with this kind of SSH key is apparently working fine, as jj git init --colocate prompted me for my PIN/touch a couple of times and then successfully completed.

[signing]
sign-all = true
backend = "ssh"
key = "~/.ssh/id_ed25519_sk.pub"

[signing.ssh.backends.ssh]
allowed-signers = "~/.ssh/allowed_signers"

Thank you for your awesome work, I hope I will soon come to enjoy this remarkable piece of work.

Steps to Reproduce the Problem

1. Generate a ed25519-sk key with a hardware security key using ssh-keygen
2. Put the file into your ~/.ssh/ directory with the default name (id_ed25519_sk/id_ed25519_sk.pub)
3. Add it to your ssh-agent using ssh-add (ssh-add ~/.ssh/id_ed25519_sk)
4. Add the public key id_ed25519_sk.pub to your GitHub account as an Authentication key
5. Invoke jj git fetch
6. Confirm user presence on the key (often times touching and/or a PIN entry)
7. Watch jj fail

Expected Behavior

jj should successfully authenticate against the git server (GitHub in this instance) and be able to push, pull and clone without issue.

Actual Behavior

jj fails without any hints of why it fails after the user already confirmed their presence.
Locating the key in the ssh-agent, as well as the key file itself, seemed to be successful.

Specifications

• Platform: Windows 11 Pro (Version 10.0.22631 Build 22631), YubiKey 5 Nano (firmware version 5.4.3)
• Scoop versions: git-without-openssh (version 2.46.2.windows.1), openssh (version 9.5.0.0p1); (git and openssh are installed seperately as the ssh version bundled with git on Windows is too outdated to properly support FIDO2)
• Version: jj version 0.22.0, both from scoop and cargo with same result

[Amolith]

I'm experiencing almost the same behaviour on Arch Linux; I'm never prompted to touch the key and the error is slightly different. I do not have the PIN enabled, just touch.

Versions:

• jj: 0.22.0
• libssh2: 1.11.0-1
• libgit2: 1:1.8.1-1

$ ssh -F /dev/null [email protected]
Confirm user presence for key ED25519-SK SHA256:JBKEeoO/72Fz03rtlzeO49PATFT2maMancH3opcT0h0
User presence confirmed
PTY allocation request failed on channel 0
Hi amolith! You've successfully authenticated, but I do not provide an interactive shell. Bye!
Connection to git.sr.ht closed.

$ jj git fetch --debug                                                                                                                                                                                                                                17:38:39
2024-10-22T23:38:39.840277Z  INFO jj_cli::cli_util: debug logging enabled
2024-10-22T23:38:39.856667Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.download
2024-10-22T23:38:40.817321Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key_from_agent username="git"
2024-10-22T23:38:40.943956Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}:get_ssh_keys{_username="git"}: jj_cli::git_util: found ssh key path="/home/amolith/.ssh/id_ed25519_sk"
2024-10-22T23:38:40.943985Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key username="git" path="/home/amolith/.ssh/id_ed25519_sk"
Error: failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

[Amolith]

Read around some more and realised this might be a duplicate of #2958. Switching to OpenSSH via #3191 fixed it!

$ jj git fetch --debug                                                                                                                                                                                                                                18:08:38
2024-10-23T00:08:38.912583Z  INFO jj_cli::cli_util: debug logging enabled
2024-10-23T00:08:38.930913Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true } depth=None}: jj_lib::git: remote.download
2024-10-23T00:08:42.571281Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true } depth=None}: jj_lib::git: remote.prune
2024-10-23T00:08:42.571631Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true } depth=None}: jj_lib::git: remote.update_tips
2024-10-23T00:08:42.571732Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true } depth=None}: jj_lib::git: default_branch="main"
2024-10-23T00:08:42.571746Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true } depth=None}: jj_lib::git: remote.disconnect
2024-10-23T00:08:42.571840Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true } depth=None}: jj_lib::git: import_refs
Nothing changed.