Support for SSH connection #1282

matejvasek · 2021-09-13T14:08:50Z

Summary

This allows for connection via SSH by setting DOCKER_HOST.

Documentation

Should this change be documented?
- Yes, see #___
- No

Codecov Report

Merging #1282 (224e0a4) into main (fc9cb50) will increase coverage by 0.35%.
The diff coverage is 91.43%.

@@            Coverage Diff             @@
##             main    #1282      +/-   ##
==========================================
+ Coverage   80.11%   80.46%   +0.35%     
==========================================
  Files         143      144       +1     
  Lines        8760     9039     +279     
==========================================
+ Hits         7017     7272     +255     
- Misses       1285     1301      +16     
- Partials      458      466       +8

Flag	Coverage Δ
os_linux	`79.08% <91.43%> (+0.36%)`	⬆️
os_macos	`73.63% <0.36%> (-2.34%)`	⬇️
os_windows	`80.29% <89.29%> (+0.29%)`	⬆️
unit	`80.46% <91.43%> (+0.35%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

matejvasek · 2021-09-23T18:30:36Z

cmd/cmd.go

+				Identity:           os.Getenv("DOCKER_HOST_SSH_IDENTITY"),
+				PassPhrase:         os.Getenv("DOCKER_HOST_SSH_IDENTITY_PASSPHRASE"),


How should we name these envvars?

Are these the same ones that Docker uses? If so, we should keep them

No they are not used by docker. It's my addition. With docker you need to put identity at default location with default name or add it to ssh-agent, you cannot specify identity file AFAIK.

matejvasek · 2021-09-23T18:31:23Z

@dmikusa-pivotal please try this out.

matejvasek · 2021-09-23T18:32:07Z

Also it would be great if somebody tried this on Windows.

cmd/cmd.go

matejvasek · 2021-09-23T18:38:42Z

@joshuawhite929 please try this out.

dmikusa · 2021-09-27T19:45:13Z

Sorry for the delay. It is working for me.

I checked out your code, make build and used ./out/pack to do a pack build. I had DOCKER_HOST set to ssh://user@my-vm:port. It didn't require setting --docker-host.

Looks good!

aemengo

(fearfully) attempting to validate on Windows here. But not really sure how these VMs are set up to be able to communicate over SSH.

@dmikusa-pivotal, do you have some sort of ssh proxy forwarding traffic from 22 to a Docker daemon listening on a port? Maybe we can look at your setup together to speed this along.

cmd/cmd.go

aemengo · 2021-09-28T01:25:16Z

cmd/cmd.go

+				Identity:           os.Getenv("DOCKER_HOST_SSH_IDENTITY"),
+				PassPhrase:         os.Getenv("DOCKER_HOST_SSH_IDENTITY_PASSPHRASE"),


Are these the same ones that Docker uses? If so, we should keep them

dmikusa · 2021-09-28T02:29:14Z

@aemengo

@dmikusa-pivotal, do you have some sort of ssh proxy forwarding traffic from 22 to a Docker daemon listening on a port? Maybe we can look at your setup together to speed this along.

You need a VM with two things to make this work:

OpenSSH installed. Basically, you should be able to ssh into the VM. For a Linux VM, just install your distro's openssh server package.
Docker Daemon running. You can use the Docker instructions for installing Docker community. If you can docker version on the remote VM as the user you're planning to connect with over ssh then you should be set (you may also need to add the user you're connecting with to the docker group so it can access the daemon without needing sudo).

There's nothing specific you have to do to Docker to make it support this. It's basically functionality in the docker client & now pack to run docker commands remotely.

Hope that helps!

aemengo · 2021-09-30T03:03:05Z

@dmikusa-pivotal Thanks for the help. I followed your instructions but am still having trouble using an ssh connection on a windows machine. I am able to use the ssh client directly.

Please let me know if I'm doing something wrong. Happy to look at this with someone.

PS> $env:DOCKER_HOST="ssh://[email protected]"
PS> $env:DOCKER_HOST_SSH_IDENTITY="C:\Users\aemengo\Desktop\rdp\id_ed25519"
PS> pack build aemengo/hello -p ..\hello -B cnbs/sample-builder:dotnet-framework-1809
ERROR: failed to build: failed to fetch builder image 'index.docker.io/cnbs/sample-builder:dotnet-framework-1809': error during connect: Post "http://example.com%2F/v1.38/images/create?fromImage=cnbs%2Fsample-builder&tag=dotnet-framework-1809": ssh: rejected: connect failed (open failed)

dmikusa · 2021-09-30T12:23:39Z

@aemengo - FYI, I only tested with password-less login. I'm not sure if it works if you have to login over ssh.

Are you able to ssh [email protected] without any additional prompts? Password most notably, but also make sure you've trusted the host key.

Also, can you docker images and see images from the remote docker? Docker should also work over ssh. If that works but pack doesn't, then it would be something specific with pack that's failing.

matejvasek · 2021-09-30T12:28:18Z

There should already be implemented prompt for password and for temporary trusting the server.

matejvasek · 2021-09-30T12:32:47Z

So ssh "ssh://[email protected]" -i "C:\Users\aemengo\Desktop\rdp\id_ed25519" does work?
Is aemengo user in docker group?

matejvasek · 2021-09-30T12:37:20Z

@aemengo Does something like:
ssh -NL localhost:2374:/var/run/docker.sock "ssh://[email protected]" -i "C:\Users\aemengo\Desktop\rdp\id_ed25519"
work?

matejvasek · 2021-09-30T13:08:52Z

also what Daniel said: try whether docker CLI works with DOCKER_HOST="ssh://[email protected]"

matejvasek · 2021-10-06T02:54:05Z

@aemengo could you please try Windows again, I did some changes.

matejvasek · 2021-10-06T02:54:23Z

@dmikusa-pivotal I did some changes could you please re-test it?

matejvasek · 2021-10-06T12:13:40Z

How can I rerun Codecov?

matejvasek · 2021-10-11T09:15:55Z

@aemengo @jromero PTAL

aemengo · 2021-10-11T14:02:01Z

@matejvasek Don't know your magic, but things seems to be working well on my windows machine 👍🏾
Please give me a few more to finish reviewing.

matejvasek · 2021-10-11T14:04:59Z

@matejvasek Don't know your magic, but things seems to be working well on my windows machine 👍🏾 Please give me a few more to finish reviewing.

I allowed use of standard docker system dial-stdio mechanism if the commands happens to be in the remote machine. If it's not it will use tunneling. I mean I call https://pkg.go.dev/github.com/docker/cli/cli/connhelper#GetConnectionHelper if it should work in given machine.

matejvasek · 2021-10-11T14:12:10Z

I allowed use of standard docker system dial-stdio mechanism if the commands happens to be in the remote machine. If it's not it will use tunneling.

I could also do it other way around: use tunneling if possible (i.e. with everything but npipe) and use docker system dial-stdio as the fallback.

aemengo

To be honest, I'm a little afraid of introducing something that a select few maintainers are likely to debug. But it did work stellarly during my testing.

Once again, we can't thank you enough for a PR of such depth.

cmd/cmd.go

n1hility · 2021-10-21T04:03:25Z

Nice work on buildpack ssh support @matejvasek

dfreilich · 2021-10-11T14:21:19Z

cmd/cmd.go

 	if err != nil {
 		return pack.Client{}, err
 	}

 	return *client, nil
 }
+
+func tryInitSSHDockerClient() (dockerClient.CommonAPIClient, error) {


Could we move these functions to a separate file in this directiory, maybe docker_init.go? I like keeping cmd/cmd.go simple and self-contained

cmd/cmd.go

internal/sshdialer/ssh_dialer.go

dfreilich · 2021-10-21T13:19:27Z

This is wildly impressive work, @matejvasek . I am a bit nervous about the future upkeep for this, so I would definitely rather the ssh_dialers functionality comes through an imported function from podman; if they'd consider exposing the required api and we could use that, I think it would definitely make it a lot safer for us to import it.

jromero · 2021-10-22T15:15:45Z

cmd/cmd.go

+func newHostKeyCbk() sshdialer.HostKeyCallback {
+	var trust []byte
+	return func(hostPort string, pubKey ssh.PublicKey) error {
+		if bytes.Equal(trust, pubKey.Marshal()) {


I don't see trust being persisted. Should it be? What's the experience connecting to the same connection a second time?

It could be written into ~/.ssh/known_hosts. But I am not too eager to write into system files. I could however print hint, something like: "add this line {key record here} into your ~/.ssh/known_hosts, or "to trust the key connect to the host via ssh first". BTW standard docker CLI is not solving this at all, you must add trust by yourself.

So right now you are asked each time.

BTW standard docker CLI is not solving this at all, you must add trust by yourself.

to be more specific you will get:

~> docker image ls error during connect: Get "http://docker.example.com/v1.24/images/json": command [ssh -l root -p 45383 -- localhost docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory Host key verification failed.

Also standard docker CLI does not support password authentication.
And you cannot specify identity file (and its passphrase) -- it either has to be at standard location with standard name,
or it has to be added to ssh-agent. Passphrase reading also has to be handled by ssh-agent.

cmd/cmd.go

matejvasek · 2021-10-22T16:19:14Z

@jromero wrt host key callbck

Where are these keys persisted?

We are not persisting it anywhere yet. I could print instruction what line should be appended to ~/.ssh/known_hosts to persist trust.

What/How do we handle non-interactive scenarios?

You can forward output of the yes utility to input of pack command: yes | pack build ....
Or you can add the host key into ~/.ssh/know_hosts somehow else.

matejvasek · 2021-10-22T16:32:57Z

This is wildly impressive work, @matejvasek . I am a bit nervous about the future upkeep for this, so I would definitely rather the ssh_dialers functionality comes through an imported function from podman; if they'd consider exposing the required api and we could use that, I think it would definitely make it a lot safer for us to import it.

@dfreilich I totally agree this is not best place to keep it. Initial I created separate repo: https://github.com/matejvasek/sshdialer.
Especially because this code might be useful in another projects. But then I would be responsible for maintaining it.

So latter I moved it here, one of the reasons was that I really liked your GithubActions. You even have custom runner for Windows with Linux containerization which I found really useful.

I could talk to people from https://github.com/containers about it. But I am afraid I won't be able to run WIndows tests in CI there.

matejvasek · 2021-10-22T18:04:37Z

cmd/cmd.go

+
+		if answer == "yes" || answer == "y" {
+			trust = pubKey.Marshal()
+			fmt.Fprintf(os.Stderr, "To avoid this in future add following line into your ~/.ssh/known_hosts:\n%s %s %s\n",


@jromero here I added message for user with instruction how to make server trusted. Is it OK?

matejvasek · 2021-10-22T22:03:43Z

@jromero @dfreilich PTAL

dfreilich

LGTM!

I appreciate the rigorous tests, which definitely makes it a lot easier to support. At the same time, if the podman team is open to supporting this, that would be great, and I'd appreciate adding a link to an issue on their repo so we can track this integration.

@jromero , what do you think? Holding it off until your approval.

cmd/docker_init.go

internal/sshdialer/ssh_dialer.go

jromero

Minor nits.

Overall good. Excited to get this in.

~~Could you resolve the merge conflict? I can then give it a quick test and merge this in.~~

EDIT: Looks like conflict is resolved.

internal/sshdialer/windows_test.go

internal/sshdialer/posix_test.go

internal/sshdialer/ssh_dialer.go

matejvasek · 2021-10-25T15:26:23Z

I am going to squash review requested fixups.

Signed-off-by: Matej Vasek <[email protected]>

github-actions bot added this to the 0.22.0 milestone Sep 13, 2021

github-actions bot added type/chore Issue that requests non-user facing changes. type/enhancement Issue that requests a new feature or improvement. labels Sep 13, 2021

matejvasek force-pushed the add-ssh-support branch 2 times, most recently from d1bc4f0 to aa529c0 Compare September 13, 2021 14:20

matejvasek commented Sep 13, 2021

View reviewed changes

cmd/cmd.go Outdated Show resolved Hide resolved

matejvasek force-pushed the add-ssh-support branch from bf687e0 to a999d08 Compare September 21, 2021 18:45

matejvasek force-pushed the add-ssh-support branch from f8f7fff to 38b1b11 Compare September 23, 2021 18:16

matejvasek commented Sep 23, 2021

View reviewed changes

matejvasek marked this pull request as ready for review September 23, 2021 18:32

matejvasek requested a review from a team as a code owner September 23, 2021 18:32

matejvasek commented Sep 23, 2021

View reviewed changes

cmd/cmd.go Outdated Show resolved Hide resolved

jromero self-assigned this Sep 27, 2021

jromero removed the type/chore Issue that requests non-user facing changes. label Sep 27, 2021

aemengo reviewed Sep 28, 2021

View reviewed changes

aemengo approved these changes Oct 11, 2021

View reviewed changes

cmd/cmd.go Outdated Show resolved Hide resolved

matejvasek mentioned this pull request Oct 20, 2021

Compatability with docker client containers/podman#11668

Closed

dfreilich reviewed Oct 21, 2021

View reviewed changes

jromero reviewed Oct 22, 2021

View reviewed changes

matejvasek force-pushed the add-ssh-support branch from ab7c7ec to 60c9002 Compare October 22, 2021 17:02

matejvasek commented Oct 22, 2021

View reviewed changes

matejvasek force-pushed the add-ssh-support branch from 118639b to 712ff06 Compare October 22, 2021 22:02

dfreilich approved these changes Oct 25, 2021

View reviewed changes

cmd/docker_init.go Outdated Show resolved Hide resolved

cmd/docker_init.go Outdated Show resolved Hide resolved

internal/sshdialer/ssh_dialer.go Show resolved Hide resolved

jromero suggested changes Oct 25, 2021

View reviewed changes

internal/sshdialer/windows_test.go Show resolved Hide resolved

internal/sshdialer/posix_test.go Show resolved Hide resolved

internal/sshdialer/ssh_dialer.go Outdated Show resolved Hide resolved

Support for SSH connection

224e0a4

Signed-off-by: Matej Vasek <[email protected]>

matejvasek force-pushed the add-ssh-support branch from 81503c3 to 224e0a4 Compare October 25, 2021 15:28

matejvasek requested a review from jromero October 25, 2021 15:28

jromero removed the type/chore Issue that requests non-user facing changes. label Oct 25, 2021

jromero approved these changes Oct 25, 2021

View reviewed changes

jromero merged commit 28598a1 into buildpacks:main Oct 25, 2021

This was referenced Aug 24, 2023

Pack 0.30.0 fails to fetch builder with Docker engine 24.0.5 on remote host over SSH #1870

Closed

Fixing error connecting to a remote daemon over ssh #1876

Merged

		Identity: os.Getenv("DOCKER_HOST_SSH_IDENTITY"),
		PassPhrase: os.Getenv("DOCKER_HOST_SSH_IDENTITY_PASSPHRASE"),

Support for SSH connection #1282

Support for SSH connection #1282

Conversation

matejvasek commented Sep 13, 2021 • edited by jromero Loading

Summary

Documentation

Related

matejvasek commented Sep 13, 2021

matejvasek commented Sep 13, 2021

matejvasek commented Sep 21, 2021

codecov bot commented Sep 21, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matejvasek commented Sep 23, 2021

matejvasek commented Sep 23, 2021

matejvasek commented Sep 23, 2021

dmikusa commented Sep 27, 2021

aemengo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmikusa commented Sep 28, 2021

aemengo commented Sep 30, 2021

dmikusa commented Sep 30, 2021

matejvasek commented Sep 30, 2021 • edited Loading

matejvasek commented Sep 30, 2021 • edited Loading

matejvasek commented Sep 30, 2021 • edited Loading

matejvasek commented Sep 30, 2021 • edited Loading

matejvasek commented Oct 6, 2021

matejvasek commented Oct 6, 2021

matejvasek commented Oct 6, 2021

matejvasek commented Oct 11, 2021

aemengo commented Oct 11, 2021

matejvasek commented Oct 11, 2021 • edited Loading

matejvasek commented Oct 11, 2021

aemengo left a comment

Choose a reason for hiding this comment

n1hility commented Oct 21, 2021

Choose a reason for hiding this comment

dfreilich commented Oct 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matejvasek commented Oct 22, 2021

matejvasek commented Oct 22, 2021 • edited Loading

Choose a reason for hiding this comment

matejvasek commented Oct 22, 2021

dfreilich left a comment

Choose a reason for hiding this comment

jromero left a comment • edited Loading

Choose a reason for hiding this comment

matejvasek commented Oct 25, 2021

matejvasek commented Sep 13, 2021 •

edited by jromero

Loading

codecov bot commented Sep 21, 2021 •

edited

Loading

matejvasek commented Sep 30, 2021 •

edited

Loading

matejvasek commented Sep 30, 2021 •

edited

Loading

matejvasek commented Sep 30, 2021 •

edited

Loading

matejvasek commented Sep 30, 2021 •

edited

Loading

matejvasek commented Oct 11, 2021 •

edited

Loading

matejvasek commented Oct 22, 2021 •

edited

Loading

jromero left a comment •

edited

Loading