pack doesn't honor `DOCKER_HOST` using `ssh` #1259

dmikusa · 2021-08-17T21:03:28Z

Summary

pack doesn't honor DOCKER_HOST that uses ssh.

Reproduction

Have docker running on a remote host that is accessible via ssh.
Optionally enable passwordless ssh access by setting an authorized_keys file on the remote host.
Run the following:

> set -x DOCKER_HOST 'ssh://[email protected]'  # chang the IP to the remote host you setup
> docker images  # confirm you see images from the remote host
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
ubuntu       focal     1318b700e415   3 weeks ago   72.8MB
> pack build maven   # fails, trying to connect over HTTP to the remote host, but the remote host uses SSH access
ERROR: failed to build: failed to fetch builder image 'index.docker.io/paketobuildpacks/builder:base': error during connect: Post "http://ubuntu%40192.168.1.130/v1.38/images/create?fromImage=paketobuildpacks%2Fbuilder&tag=base": dial tcp: lookup [email protected]: no such host

SSH is a convenient and secure way to access a remote docker host. It would be nice if this worked.

Environment

> pack --version
0.20.0+git-66a4f32.build-2668

> docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:22 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:33 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The text was updated successfully, but these errors were encountered:

joshuawhite929 · 2021-09-02T21:35:27Z

Just wanted to add that this prevents using Podman with pack on the Mac as Podman runs in a VM. With all of the noise around finding alternatives to Docker Desktop for the Mac, it would be REALLY nice to have this work.

matejvasek · 2021-09-02T22:48:40Z

It looks like Go's docker client library github.com/docker/docker/client doesn't support ssh. But docker CLI does.

matejvasek · 2021-09-02T22:53:20Z

I could work on this. I would take code from docker CLI and from podman to support ssh. Preferably with support for custom path is the uri the way podman supports it (while docker doesn't).

matejvasek · 2021-09-02T22:57:27Z

@jromero could you please assign this to me?

jromero · 2021-09-02T23:38:34Z

@jromero could you please assign this to me?

@matejvasek, done. I appreciate you volunteering.

joshuawhite929 · 2021-09-03T01:41:36Z

As we think about a solution for this, it would be nice if we can eliminate the need to specify the location of the remote docker host using the --docker-host=/run/user/1000/podman/podman.sock flag. Could another environment variable or other configuration mechanism be used for pack to specify this?

dmikusa · 2021-09-03T12:15:55Z

@joshuawhite929 +1 for that. I don't want to need an argument on every pack command I run. An env variable would be nice. For the Docker CLI, you can set DOCKER_HOST=ssh://... and it just works. Not sure if Podman has a similar convention, but it would be nice if pack looked for and followed existing conventions for env variables.

@matejvasek - Thanks so much for working on this!!

jromero · 2021-09-03T13:47:06Z

RE: --docker-host

We've had discussions in the past about reducing the complexity and/or eliminating the flag altogether. See #1093. @micahyoung was on the hook for creating an RFC 😉. We may need to take another closer look. I don't recall exactly where we last landed on that but I still feel like we should only have one flag/option for both pack and the lifecycle. I understand some of the intricacies of lifecycle running on a different network but I'm not sure providing two options is the best experience. Pack in theory do it's best to resolve the networking differences or at minimum output warnings (as I believe it's been previously suggested).

matejvasek · 2021-09-03T14:12:01Z

Also isn't there an attempt to avoid socket muting all together and export tar instead?

jromero · 2021-09-03T18:26:46Z

Also isn't there an attempt to avoid socket muting all together and export tar instead?

Yes. I'm not sure how far away from that we are though.

/cc @jjbustamante

matejvasek · 2021-09-10T02:59:15Z

@joshuawhite929 @dmikusa-pivotal I stared experimenting with SSH here https://github.com/matejvasek/sshdialer

example: https://github.com/matejvasek/sshdialer/blob/master/examples/docker-client/main.go

please try running the example locally to verify that it works on macOS

matejvasek · 2021-09-10T16:42:06Z

sample code: https://github.com/knative-sandbox/kn-plugin-func/pull/520/files#diff-6ab5fda3bf6386907a43c435b094df9024aba6999ce77a51f0b6fac5d09aae4bR110

dmikusa · 2021-09-12T02:31:35Z

@matejvasek I am getting this error:

2021/09/11 22:24:28 error during connect: Get "http://127.0.0.1:2376/v1.24/images/json?all=1": failed to dial ssh: ssh: handshake failed: server key not found in known_hosts

What seems to be happening is that I do have the server trusted in my known_hosts file, but the key type doesn't match. The code is looking for ssh-rsa but in my known_hosts file it's set as ecdsa-sha2-nistp256.

If I change the order of the host key algorithms, it will end up working:

		HostKeyAlgorithms: []string{
			ssh.KeyAlgoECDSA256,
			ssh.KeyAlgoRSA,
			ssh.KeyAlgoDSA,
			ssh.KeyAlgoECDSA384,
			ssh.KeyAlgoECDSA521,
			ssh.KeyAlgoED25519,
		},

that order works for me.

This is on a Mac (Big Sur) talking to an Ubuntu 20.04.3 VM running Docker.

I'm guessing my ssh client has a different order of preference? ~~I'm not sure where that is specified though. Sorry.~~

Nm, I found it. In man ssh_config, it gives this order.

     HostKeyAlgorithms
             Specifies the host key algorithms that the client wants to use in order of preference.  Alternately if the specified list begins with a `+' character, then
             the specified key types will be appended to the default set instead of replacing them.  If the specified list begins with a `-' character, then the specified
             key types (including wildcards) will be removed from the default set instead of replacing them.  If the specified list begins with a `^' character, then the
             specified key types will be placed at the head of the default set.  The default for this option is:

                [email protected],
                [email protected],
                [email protected],
                [email protected],
                [email protected],[email protected],
                [email protected],
                ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
                ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

             If hostkeys are known for the destination host then this default is modified to prefer their algorithms.

             The list of available key types may also be obtained using "ssh -Q key".

the list is slightly different on my Ubunt 20.04.3 VM.

                [email protected],
                [email protected],
                [email protected],
                [email protected],
                [email protected],
                [email protected],
                [email protected],
                [email protected],
                [email protected],
                ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
                [email protected],
                ssh-ed25519,[email protected],
                rsa-sha2-512,rsa-sha2-256,ssh-rsa

Maybe we could source this list from the system some how? or at least keep our list in the same order as OpenSSH. I think that would avoid issues like what I hit here.

matejvasek · 2021-09-13T18:27:31Z

@dmikusa-pivotal thanks for testing, I updated order could you please try it out? Also could you please download https://github.com/matejvasek/sshdialer and run tests?

EDIT: the test won't work on macOS because I am using Linux networking features in it.

dmikusa · 2021-09-16T12:50:04Z

I am unfortunately still getting the error about server key not found.

> ./main
2021/09/16 08:48:41 error during connect: Get "http://127.0.0.1:2376/v1.24/images/json?all=1": failed to dial ssh: ssh: handshake failed: server key not found in known_hosts

I'm not sure what else would be helpful to debug this, but let me know if there anything you want me to try.

matejvasek · 2021-09-16T12:53:20Z

@dmikusa-pivotal

try ssh into the VM first so key is stored in known_hosts
if it wouldn't work despite that the key is in the known_host file then try removing and repopulating it
could you please pull the library https://github.com/matejvasek/sshdialer and run make test?

matejvasek · 2021-09-16T12:54:43Z

ssh.KeyAlgoECDSA256 is first now

dmikusa · 2021-09-16T12:57:02Z

Your tests look good btw.

$ go test ./...
ok  	github.com/matejvasek/sshdialer	2.149s

That's on my Ubuntu 20.04.3 LTS VM.

dmikusa · 2021-09-16T13:06:00Z

@dmikusa-pivotal

try ssh into the VM first so key is stored in known_hosts

if it wouldn't work despite that the key is in the known_host file then try removing and repopulating it

dmikusa@dmikusa-a01 ~/D/s/e/docker-client (master)> grep ubuntu main.go
	host := "ssh://ubuntu@localhost:2222/var/run/docker.sock"
dmikusa@dmikusa-a01 ~/D/s/e/docker-client (master)> ssh ssh://[email protected]:2222
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-84-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu 16 Sep 2021 12:58:52 PM UTC

  System load:  0.15               Processes:                155
  Usage of /:   12.0% of 78.24GB   Users logged in:          1
  Memory usage: 5%                 IPv4 address for docker0: 172.17.0.1
  Swap usage:   0%                 IPv4 address for enp0s3:  10.0.2.15

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

5 updates can be applied immediately.
To see these additional updates run: apt list --upgradable


Last login: Thu Sep 16 12:50:32 2021 from 10.0.2.2
ubuntu@docker-vm:~$ exit
logout
Connection to 127.0.0.1 closed.
dmikusa@dmikusa-a01 ~/D/s/e/docker-client (master)> go build main.go
dmikusa@dmikusa-a01 ~/D/s/e/docker-client (master)> ./main
2021/09/16 09:02:19 error during connect: Get "http://127.0.0.1:2376/v1.24/images/json?all=1": failed to dial ssh: ssh: handshake failed: server key not found in known_hosts

and trying to delete & recreate.

> ssh ssh://[email protected]:2222
The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:eJIgdkmRg+RgZcWTCcQOwMsTr2XSustFAjrwnc2VSjs.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-84-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu 16 Sep 2021 01:01:53 PM UTC

  System load:  0.03               Processes:                159
  Usage of /:   12.0% of 78.24GB   Users logged in:          1
  Memory usage: 5%                 IPv4 address for docker0: 172.17.0.1
  Swap usage:   0%                 IPv4 address for enp0s3:  10.0.2.15

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

5 updates can be applied immediately.
To see these additional updates run: apt list --upgradable


Last login: Thu Sep 16 12:58:53 2021 from 10.0.2.2
ubuntu@docker-vm:~$ exit
logout
Connection to 127.0.0.1 closed.
dmikusa@dmikusa-a01 ~/D/sshdialer (master)> cd examples/docker-client/
dmikusa@dmikusa-a01 ~/D/s/e/docker-client (master)> ./main
2021/09/16 09:05:23 error during connect: Get "http://127.0.0.1:2376/v1.24/images/json?all=1": failed to dial ssh: ssh: handshake failed: server key not found in known_hosts
dmikusa@dmikusa-a01 ~/D/s/e/docker-client (master) [1]> cat ~/.ssh/known_hosts
[127.0.0.1]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8NpAvYnffHmRJf715vwIEqdg6e/kIDg+70d+pPXCkibNsgnYL9V4jGGdPCElIC+Rz6asQeghu1Vc09NB2MtV8=

could you please pull the library https://github.com/matejvasek/sshdialer and run make test?

This worked on my Linux box. On my mac, it fails with a port conflict.

> make test
go test -v ./...
failed to start container: Error response from daemon: driver failed programming external connectivity on endpoint sshdialer-test-container (cf2dcde6543e1a4ee7aa3b57e1a1b1275a88d2debd1063ad4bba188dd619580b): Error starting userland proxy: listen tcp4 0.0.0.0:22: bind: address already in use
FAIL	github.com/matejvasek/sshdialer	0.479s
FAIL
make: *** [test] Error 1

but nothing is listening on port 22 on my Mac.

dmikusa · 2021-09-16T13:20:21Z

Sorry, disregard my failures. Too early, not enough caffeine in my body.

I had used ssh://ubuntu@localhost:222 as my URL in the example but was running ssh://[email protected] to add the host key. Once I trusted the key for localhost it worked fine.

dmikusa added status/triage Issue or PR that requires contributor attention. type/bug Issue that reports an unexpected behaviour. labels Aug 17, 2021

jromero added status/ready Issue ready to be worked on. and removed status/triage Issue or PR that requires contributor attention. labels Aug 17, 2021

jromero assigned matejvasek Sep 2, 2021

matejvasek mentioned this issue Sep 6, 2021

Improve ssh connectivity for docker API knative/func#469

Closed

matejvasek mentioned this issue Sep 10, 2021

feat: ssh dockerd connectivity knative/func#520

Closed

matejvasek mentioned this issue Sep 13, 2021

Support for SSH connection #1282

Merged

2 tasks

jromero closed this as completed in #1282 Oct 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pack doesn't honor `DOCKER_HOST` using `ssh` #1259

pack doesn't honor `DOCKER_HOST` using `ssh` #1259

dmikusa commented Aug 17, 2021

joshuawhite929 commented Sep 2, 2021

matejvasek commented Sep 2, 2021

matejvasek commented Sep 2, 2021 •

edited

Loading

matejvasek commented Sep 2, 2021

jromero commented Sep 2, 2021

joshuawhite929 commented Sep 3, 2021

dmikusa commented Sep 3, 2021 •

edited

Loading

jromero commented Sep 3, 2021

matejvasek commented Sep 3, 2021

jromero commented Sep 3, 2021

matejvasek commented Sep 10, 2021

matejvasek commented Sep 10, 2021

dmikusa commented Sep 12, 2021 •

edited

Loading

matejvasek commented Sep 13, 2021 •

edited

Loading

dmikusa commented Sep 16, 2021 •

edited

Loading

matejvasek commented Sep 16, 2021

matejvasek commented Sep 16, 2021

dmikusa commented Sep 16, 2021

dmikusa commented Sep 16, 2021

dmikusa commented Sep 16, 2021

pack doesn't honor DOCKER_HOST using ssh #1259

pack doesn't honor DOCKER_HOST using ssh #1259

Comments

dmikusa commented Aug 17, 2021

Summary

Reproduction

Environment

joshuawhite929 commented Sep 2, 2021

matejvasek commented Sep 2, 2021

matejvasek commented Sep 2, 2021 • edited Loading

matejvasek commented Sep 2, 2021

jromero commented Sep 2, 2021

joshuawhite929 commented Sep 3, 2021

dmikusa commented Sep 3, 2021 • edited Loading

jromero commented Sep 3, 2021

matejvasek commented Sep 3, 2021

jromero commented Sep 3, 2021

matejvasek commented Sep 10, 2021

matejvasek commented Sep 10, 2021

dmikusa commented Sep 12, 2021 • edited Loading

matejvasek commented Sep 13, 2021 • edited Loading

dmikusa commented Sep 16, 2021 • edited Loading

matejvasek commented Sep 16, 2021

matejvasek commented Sep 16, 2021

dmikusa commented Sep 16, 2021

dmikusa commented Sep 16, 2021

dmikusa commented Sep 16, 2021

pack doesn't honor `DOCKER_HOST` using `ssh` #1259

pack doesn't honor `DOCKER_HOST` using `ssh` #1259

matejvasek commented Sep 2, 2021 •

edited

Loading

dmikusa commented Sep 3, 2021 •

edited

Loading

dmikusa commented Sep 12, 2021 •

edited

Loading

matejvasek commented Sep 13, 2021 •

edited

Loading

dmikusa commented Sep 16, 2021 •

edited

Loading