I probably should have just created a PR for this and we could discuss this further there. Let me know if you wish for me to do so and I'll find some time.

I'm not sure if the approach here is correct, since the DNS entry for accessing the host should now work for all recent docker versions for example:

package testcontainers_test

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"net"
	"net/http"
	"net/http/httptest"
	"net/url"
	"testing"

	"github.com/docker/docker/api/types/container"
	"github.com/stretchr/testify/require"
	"github.com/testcontainers/testcontainers-go"
	"github.com/testcontainers/testcontainers-go/wait"
)

const caddyFileContent = `
listen :80

reverse_proxy /api/* {
	to {$API_SERVER}

	health_uri /health
	health_status 200
	health_interval 10s
}
`

func TestCaddyfile(t *testing.T) {
	ctx := context.Background()
	apiServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "Hello, World!")
	}))
	t.Cleanup(apiServer.Close)
	u, err := url.Parse(apiServer.URL)
	require.NoError(t, err)
	_, port, err := net.SplitHostPort(u.Host)
	require.NoError(t, err)
	u.Host = "host.docker.internal:" + port

	caddyContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{
			Image:        "caddy:2.8.4",
			ExposedPorts: []string{"80/tcp"},
			WaitingFor:   wait.ForLog("server running"),
			Env: map[string]string{
				"API_SERVER": u.String(),
			},
			Files: []testcontainers.ContainerFile{
				{
					Reader:            bytes.NewReader([]byte(caddyFileContent)),
					ContainerFilePath: "/etc/caddy/Caddyfile",
				},
			},
			HostConfigModifier: func(hc *container.HostConfig) {
				hc.ExtraHosts = append(hc.ExtraHosts, "host.docker.internal:host-gateway")
			},
		},
		Started: true,
	})
	testcontainers.CleanupContainer(t, caddyContainer)
	require.NoError(t, err)

	caddyURL, err := caddyContainer.PortEndpoint(ctx, "80/tcp", "http")
	require.NoError(t, err)

	resp, err := http.Get(caddyURL + "/api/test")
	require.NoError(t, err)
	defer resp.Body.Close()

	body, err := io.ReadAll(resp.Body)
	require.NoError(t, err)

	require.Equal(t, http.StatusOK, resp.StatusCode)
	require.Equal(t, "Hello, World!\n", string(body))

	lr, err := caddyContainer.Logs(ctx)
	require.NoError(t, err)
	lb, err := io.ReadAll(lr)
	require.NoError(t, err)
	fmt.Printf("== Caddy Logs ==\n%s================\n\n", string(lb))
}

Thanks @stevenh!

I like this approach better for my use case but I'll keep this issue and PR open and let the project maintainers decide if it's worth pulling in? Given your solution, I'm not sure what the use case is for HostAccessPorts and if my change would make a difference?

Thank you for your insight!

One of the benefits of the HostAccessPorts is that it can expose ports listening on 127.0.0.1 which is the case of the httptest.NewServer which was originally why I was swapping out the httptest.Server.Listener. In my example this was just junk left over from trying to come up with a solution, but listening on 0.0.0.0 wasn't technically needed with my fix. In your example you still need to swap out the Listener as talking to host-gateway talks to the bridge interface which would require the Listener to listen on all interfaces or for me to determine the bridge interface address to listen on.

One of the benefits of the HostAccessPorts is that it can expose ports listening on 127.0.0.1 which is the case of the httptest.NewServer which was originally why I was swapping out the httptest.Server.Listener. In my example this was just junk left over from trying to come up with a solution, but listening on 0.0.0.0 wasn't technically needed with my fix. In your example you still need to swap out the Listener as talking to host-gateway talks to the bridge interface which would require the Listener to listen on all interfaces or for me to determine the bridge interface address to listen on.

I'm not sure I understand, httptest.NewServer in the example above listens on 127.0.0.1 and works fine. Could you clarify what you believe HostAccessPorts solves?

Thanks @stevenh!

I like this approach better for my use case but I'll keep this issue and PR open and let the project maintainers decide if it's worth pulling in? Given your solution, I'm not sure what the use case is for HostAccessPorts and if my change would make a difference?

As one of the community maintainers, happy to consider your PR, I'm just trying to understand the use case, and if its really needed anymore?

I'm wondering if HostAccessPorts pre-dates the internal DNS entries working on all platforms, as I know it was flaky for some time on Windows as an example. @mdelapenya do you know the history on this?

I just came across this issue yesterday as I was exploring a similar problem with HostAccessPorts and the SSH tunnel not being set up in time - I too was depending on the host network being accessible during container startup.

The DNS solution works great and is certainly simpler, so thanks for that @stevenh !

My $.02 --

IMO, if the support for HostAccessPorts is to be kept around in spite of this, then I do think the fix provided by @hathvi is a logical one - I had implemented a similar custom container lifecycle hook to get things working the way we needed before coming across this issue.

If indeed there is no justification for continuing to support HostAccessPorts other than backwards compatibility, then it would be great to see the docs updated to point out the availability of host.docker.internal and perhaps even to illustrate its use with an example like the one here.

Thanks for your thoughts @jeremyg484, can anyone think of a situation which HostAccessPorts can handle which the DNS route can't?

Are you two by chance on Mac or Windows? I think Windows, under Docker Desktop, routes traffic to the host a bit differently then Mac or Linux. I use Linux as my primary OS and am not super familiar with Windows. The test that @stevenh provided does not work for me, which I'll describe why below.

I'd preface that my original problem with testcontainers was that it was non obvious how to access the host using testcontainers. I however was also not aware of the docker extra hosts config and my searching around testcontainers lead me to HostAccessPorts which I found to be setting up the listeners to late.

Docker by default uses a bridge network. It creates a new interface on the host named docker0 with it's own subnet, in my case this is 172.17.0.1/16 with the interface address being 172.17.0.1. When you configure extra hosts to host-gateway it creates a DNS record in /etc/hosts to host.docker.internal 172.17.0.1.

When you create a new httptest.Server it by default binds to 127.0.0.1 on a random port of the machine you are running the app on, there's no direct way of changing this behavior and you must swap out the httptest.Server.Listener to bind to a different interface. In order for the DNS solution to work the httptest.Server would need to bind to 172.17.0.1 ( the container gateway / bridge interface ).

HostAccessPorts requests the SSH server bind the remote port in the container to 127.0.0.1:PORT and tunnel traffic back to the client which in turn dials 127.0.0.1:PORT ( the httptest.Server ) on the client machine and copies those bytes from the tunnel to the server. If I were to swap out the httptest.Server.Listener to bind to 172.17.0.0.1 I'd have the same issue with HostAccessPorts but DNS would work.

My guess would be that you guys are likely on Windows and/or running Docker Desktop which likely does something similar to what HostAccessPorts is doing and/or Windows allows connecting to a port bound on 127.0.0.1 from any interface. I don't have a Windows VM currently to do any testing.

I would say HostAccessPorts does solve a problem, especially since people are generally going to be using this for testing and I assume httptest.Server would be a somewhat common use case for people testing HTTP clients, however I don't think this is a problem testcontainers necessarily needs to resolve as it feels out of scope, it's just helpful that it does.

I would also say that if HostAccessPorts were kept it might be nice to extend its functionality to change which interface the local client dials, working more inline with how the ssh -R works. e.g., ssh -R [local-addr:]local-port:remote-addr:remote-port destination. Right now testcontainers doesn't expose the ability to set local-addr or remote-addr and hard codes these to localhost ( here for the remote listener, and here for the local forwarding client ). I assume this would result in a breaking change however as you'd likely need something like HostAccessPorts []string instead of []int and support values like local-addr:port/tcp, port/tcp ( defaults to 127.0.0.1 ), local-addr:port ( defaults to tcp ), port ( defaults to 127.0.0.1 and tcp )

I don't contribute as much to open source as I should mainly due to anxiety socializing with others, but I'd be down to contributing the suggested changes if that seems like a good idea to you guys.

Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to HostAccessPorts really make sense as this would only be a problem for servers bound to 127.0.0.1. If the server was bound to any other interface other then a loopback then the server should be accessible to the container as long as the host has a route to it.

Thanks for all the extra info. I wonder what the expected behaviour should be. Could you confirm what the name resolves to?

Oh to confirm my test host was Windows under WSL. So would be interested to understand that OSs work and what doesn't

Tested on Mac and it also works fine.

Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to HostAccessPorts really make sense as this would only be a problem for servers bound to 127.0.0.1. If the server was bound to any other interface other then a loopback then the server should be accessible to the container as long as the host has a route to it.

Thats still a legitimate problem to solve.

Thanks for all the extra info. I wonder what the expected behaviour should be. Could you confirm what the name resolves to?

If we're referring to DNS using the extra hosts set to host-gateway then it's expected that an entry in /etc/hosts is added that points to the docker0 interface, which is what I get.

$ docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.1	host.docker.internal
172.17.0.2	7c7f24b879a3

If I listen on 127.0.0.1 on the host and try curling from a container using the DNS solution then I would not expect to be able to as the server is not listening on that interface, which is what I get.

$ nc -l 127.0.0.1 8888

$ docker run --rm -it --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://host.docker.internal:8888
curl: (7) Failed to connect to host.docker.internal port 8888 after 0 ms: Could not connect to server

If I listen on 172.17.0.1 then I would expect to be able to access the server from the container using the DNS solution as it's listening on that interface, which is what I get.

$ nc -l 172.17.0.1 8888
HEAD / HTTP/1.1
Host: host.docker.internal:8888
User-Agent: curl/8.10.1
Accept: */*

^C

$ docker run --rm -it --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://host.docker.internal:8888
curl: (52) Empty reply from server

Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to HostAccessPorts really make sense as this would only be a problem for servers bound to 127.0.0.1. If the server was bound to any other interface other then a loopback then the server should be accessible to the container as long as the host has a route to it.

Thats still a legitimate problem to solve.

To clarify, I was only referring to my suggested changes on adding the ability to specify the local address to connect to in HostAccessPorts, I believe that suggestion was redundant.

It should only be a problem if the server you're trying to talk to is bound to an address on the host that is not routable from the container. This is the case for 127.0.0.1 as the container also creates an loopback interface with an addr of 127.0.0.1, so talking to 127.0.0.1 would just send to the container loopback interface and not the host.

For example I can connect to my tailscale interface from a container just fine as the container doesn't have a route for this address and sends the request out the default gateway to docker0 on the host, and the host has a route to the tailscale address so sends it out that interface.

$ ip -4 addr show tailscale0
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 100.78.128.77/32 scope global tailscale0
       valid_lft forever preferred_lft forever

$ nc -l 100.78.128.77 8888
HEAD / HTTP/1.1
Host: 100.78.128.77:8888
User-Agent: curl/8.10.1
Accept: */*

^C

$ docker run --rm -it curlimages/curl curl --head http://100.78.128.77:8888
curl: (52) Empty reply from server

I think there's a legitimate helpful use case for HostAccessPorts but only for the following cases

1. When the server is bound to an address not accessible to the container ( 127.0.0.1, which is the case of httptest.NewServer )
2. When the user doesn't wish to expose their server to the network by binding to all interfaces ( which brings us back to 1 as you'd bind to 127.0.0.1 to resolve this )
3. When the user doesn't wish to determine the IP of docker0 to bind to that address or have tests break if ran with a different network mode.

I'm not sure how/why this is working on windows and macos unless binding a port on 127.0.0.1 for some results in it listening on all interfaces. On your Mac are you also using Docker Desktop? I still speculate that Docker Desktop is doing something to expose the host to the VM that results in ports bound on 127.0.0.1 working from the container.

What does host.docker.internal resolve to in your container? Only Linux, regardless of which network mode I use, it's always set to the docker0 IP of the host. However, if I set the network mode to host I can talk to ports bound to 127.0.0.1 on the host but only if the request is to 127.0.0.1 in the container and not host.docker.internal. I wonder if possibly host.docker.internal:host-gateway on windows is setting the address to 127.0.0.1 in the container and you have docker desktop set to use a host network by default?

$ docker run --rm -it --network host --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://host.docker.internal:8888
curl: (7) Failed to connect to host.docker.internal port 8888 after 0 ms: Could not connect to server

$ docker run --rm -it --network host --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://127.0.0.1:8888
curl: (52) Empty reply from server

$ docker run --rm -it --network bridge --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://127.0.0.1:8888
curl: (7) Failed to connect to 127.0.0.1 port 8888 after 0 ms: Could not connect to server

To add some further examples of usage to the discussion - I have not tested on Windows, but my specific usage of host.docker.internal does work on both Mac (our dev machines, using Docker Desktop) and Ubuntu (our CI runners).

I am setting ExtraHosts to host.docker.internal:host-gateway just as in the example from @stevenh above. I then have our app running in the container set to use host.docker.internal as a substituted address specifically for this testing scenario, replacing the actual address of an external 3rd-party service for which we're providing a test double using the httptest package.

I am setting a custom net.Listener on the httptest.Server, because our scenario requires talking to specific predefined ports instead of the random open port that gets used by default in httptest.NewUnstartedServer. Note (and I would think this relates to what @hathvi has explored above) that I am creating the listener with

l, err := net.Listen("tcp", ":"+port)

according to the docs for net.Listen, "For TCP networks, if the host in the address parameter is empty or a literal unspecified IP address, Listen listens on all available unicast and anycast IP addresses of the local system".

I just checked, and if I specify the loopback address 127.0.0.1 explicitly in net.Listen, it still works on my Mac, but fails in our Ubuntu CI runner.

That's what I see too, Linux seems to be the odd one out.

I wouldn't call Linux the odd one out, it's working as it should, I think Mac, Windows or something else is just configured slightly different within these test environments and those are the odd ones out. Docker is Linux native, and Docker for me is working just as I would expect given the defaults the Docker documentation define and how networking in general works both inside and outside containers.

Can you run a few commands @stevenh?

1. docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts so we can see if it's getting set to the docker0 gateway address like it does on Linux
2. Block the test before terminate and run docker inspect CONTAINER_NAME/ID -f '{{json .NetworkSettings.Networks}}' so we can see what network mode is being used. Since you're not explicitly setting it in your test, it should in theory be bridge according to the docs, but I suspect it's host in your case.
3. It would also be interesting to see what address the port is being bound to on windows, which looks like you can see with netstat -a -n or you can open the resource manager and see all listeners there.
4. docker version - I'm running 27.3.1 for reference

I can set up a Windows VM later tonight ~8PM PT w/ Docker Desktop and WSL, just so I can explore more mainly out of curiosity now. I think given the differences we see just in this thread gives HostAccessPorts a use. If you can talk to a server on the host bound to 127.0.0.1 from a container then great, if not you can use HostAccessPorts as a solution. We just need to resolve the late listening issue this issue is for. I think there's value in continuing to explore and potentially updating the docs to reflect our findings to better guide people in the future so they don't try hacking their own solutions.

1. docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts so we can see if it's getting set to the docker0 gateway address like it does on Linux

 docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.65.254  host.docker.internal
172.17.0.2      32db41e7dd87

2. Block the test before terminate and run docker inspect CONTAINER_NAME/ID -f '{{json .NetworkSettings.Networks}}' so we can see what network mode is being used. Since you're not explicitly setting it in your test, it should in theory be bridge according to the docs, but I suspect it's host in your case.

docker inspect 020989b419e6 -f '{{json .NetworkSettings.Networks}}'
{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"02:42:ac:11:00:03","DriverOpts":null,"NetworkID":"f4e593f77e57e8d759b34578d47ca89583c3cae50f4ac18b5cf9d447a97cebb2","EndpointID":"0dcb292a563a3041263672990cce0c49efcb28b60f3eca4e6d3614d82ac7d31b","Gateway":"172.17.0.1","IPAddress":"172.17.0.3","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}

3. It would also be interesting to see what address the port is being bound to on windows, which looks like you can see with netstat -a -n or you can open the resource manager and see all listeners there.

netstat -na |grep 45757 |grep LIST
tcp       30      0 127.0.0.1:45757         0.0.0.0:*               LISTEN

docker version - I'm running 27.3.1 for reference

docker version
Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:14:20 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop  ()
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:15 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I wouldn't call Linux the odd one out, it's working as it should, I think Mac, Windows or something else is just configured slightly different within these test environments and those are the odd ones out. Docker is Linux native, and Docker for me is working just as I would expect given the defaults the Docker documentation define and how networking in general works both inside and outside containers.

We shouldn't assume that because docker is native on Linux that the intent wasn't that seen on Windows or Mac with regards being able to connect to loopback address. It could be a bug with either, as we should be able to rely on consistent behaviour across all platforms.

I was not able to get WSL2 running within VirtualBox last night, but I got things set up on a laptop this AM and can confirm your test works for me on Windows as well.

Based on the outputs you provided in your last post this should technically not be allowed unless something in the middle is doing something with the network packets such as NAT or proxying to the host. It looks like WSL2 uses NAT by default, but doesn't allow this behavior itself.

package main

import (
	"fmt"
	"net"
	"net/http"
	"net/http/httptest"
	"os"
	"os/signal"
	"syscall"
)

func main() {
	l, err := net.Listen("tcp", "127.0.0.1:51403")
	if err != nil {
		panic(err)
	}

	s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		w.Write([]byte("Hello, world!"))
	}))
	s.Listener.Close()
	s.Listener = l
	s.Start()

	fmt.Printf("Listening: %s\n", s.Listener.Addr())

	sc := make(chan os.Signal, 1)
	signal.Notify(sc, syscall.SIGTERM)
	<-sc
}

PS C:\Users\justin\workspace\test> go run server.go
Listening: 127.0.0.1:51403

PS C:\Users\justin\workspace\test> wsl -d ubuntu

justin@jlap:/mnt/c/Users/justin/workspace/test$ ip route
default via 172.26.192.1 dev eth0 proto kernel
172.26.192.0/20 dev eth0 proto kernel scope link src 172.26.196.73

justin@jlap:/mnt/c/Users/justin/workspace/test$ curl -m 5 172.26.192.1:51403
curl: (28) Connection timed out after 5002 milliseconds

The WSL2 documentation even makes explicit mention of this https://learn.microsoft.com/en-us/windows/wsl/networking#connecting-via-remote-ip-addresses

When using remote IP addresses to connect to your applications, they will be treated as connections from the Local Area Network (LAN). This means that you will need to make sure your application can accept LAN connections.

For example, you may need to bind your application to 0.0.0.0 instead of 127.0.0.1. In the example of a Python app using Flask, this can be done with the command: app.run(host='0.0.0.0'). Keep security in mind when making these changes as this will allow connections from your LAN.

I tried digging more into how Docker is running under WSL2 but at first glance doesn't seem as straight forward as I was hoping. I'll try digging more into this later tonight.