Back to all tech blogs

Oops, I forgot to –publish! How can I connect to the container then?

  • Backend
A fun way to connect over the network to an unpublished port of a process running inside a Docker container

At Adevinta, we’re continually shipping new containers to production, forming a constellation of services owned by multiple teams collaborating to provide our customers with delightful features.

As we deploy those containers, we typically want to ensure that all relevant behaviours can be reproduced between the artefacts we develop locally and/or test on CI servers and those landing on production servers. Crucially, we should be able to experiment with such artefacts and troubleshoot them.

Sometimes we get too immersed in the development/testing cycle and may realise that we’ve missed an important step only when it’s pretty inconvenient to restart the whole process.

Let’s say we have a networked service running inside a Docker container. Here exemplified by Apache’s httpd with the following Dockerfile:

FROM httpd@sha256:518e9447a236de47cc2fb4f2dbf06466b6cd5cf50f9951742f9a20af76e8118d
COPY ./public-html/ /usr/local/apache2/htdocs/
Dockerfile

And ./publish-html/index.html:

<html>
The answer?
</html>
HTML

After building the image:

λ docker build -t httpd-fun .
Bash

We spin up a container:

λ docker run --rm --name httpd-fun httpd-fun
Bash

However, as we play with the application, we need to make a couple of in-place changes to the container for experimentation. Here exemplified by “revealing the answer”:

λ docker exec httpd-fun sed -i '/^The answer\?/ s/$/ <strong>42<\/strong>/' /usr/local/apache2/htdocs/index.html
Bash

After all that, we try to fetch the HTML page from our host machine:

λ curl http://localhost:80
Bash

And… Oops:

curl: (7) Failed to connect to localhost port 80 after 0 ms: Connection refused
Bash

It didn’t work. There’s nothing listening on localhost:80!

Of course, that’s to be expected. We didn’t publish the container’s port 80 onto our host and as a result, we couldn’t reach the service.

At this point, we could simply re-run the container with the port published, say -p 8080:80 to expose 80 inside the container to 8080 on our host, or even build a new image with the changes we wanted and start a new container from that. However, we’d lose the changes we’ve made so far and that might not be acceptable. We’d prefer to keep the changes throughout the experimentation session.

There are different options to proceed (commit the container, shell out to iptables and manually add the necessary rules etc.), but we’ll limit ourselves to just two in this blog.

Connecting to the container’s IP

To start off, perhaps the container’s IP is routable from the host machine. If that’s the case, then we can:

1. Find the IP of httpd-fun:

λ docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd-fun


172.17.0.2
Bash

2. Send an HTTP request to the service to its IP as opposed to localhost:

λ curl http://172.17.0.2:80
Bash

3. Profit:

<html>
The answer? <strong>42</strong>
</html>
HTML

Although straightforward, that’s not always feasible.

Connecting through a proxy container

Another option is to run a second container with the port published, from which, by default, there’s connectivity with httpd-fun, and then proxy connections from the former to the latter:

1. Find the network name of httpd-fun (to ensure there exists connectivity between the containers):

λ docker container inspect -f '{{range $net,$v := .NetworkSettings.Networks}}{{printf "%s" $net}}{{end}}' httpd-fun


bridge
Bash

2. Run a secondary container with the published port mapped to the host:

λ docker run -it --rm -p 8080:8080 --network bridge --name httpd-fun-proxy alpine@sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 /bin/sh
Bash

This provides a shell in an Alpine container where 8080 on the host goes to 8080 in the container.

3. For the proxy, we can install socat:

# apk update && apk add socat
Bash

And spin it up:

# socat -v TCP-LISTEN:8080,fork,reuseaddr TCP-CONNECT:172.17.0.2:80
Bash

This starts socat, enables verbose logging, binds a socket at localhost:8080 for multiple TCP connections and forwards incoming connections to 172.17.0.2:80 (the container’s IP and the port where httpd listens).

4. Send an HTTP request to localhost:8080:

λ curl http://localhost:8080
Bash

5. Profit:

<html>
The answer? <strong>42</strong>
</html>
HTML

Automating the process

We can even take this all a step further and put together a shell script to automate the spinning of the proxy container (that we can tweak according to our needs):

#!/bin/sh


# proxy-container
#
# USAGE
# proxy-container target-container-name target-container-port proxy-container-port


TARGET_CONTAINER_NAME="${1}"
TARGET_CONTAINER_PORT="${2}"
PROXY_CONTAINER_PORT="${3}"


target_container_ip="$(docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${TARGET_CONTAINER_NAME})"
target_container_net="$(docker container inspect -f '{{range $net,$v := .NetworkSettings.Networks}}{{printf "%s" $net}}{{end}}' ${TARGET_CONTAINER_NAME})"


docker run -it --rm \
    -p "${PROXY_CONTAINER_PORT}:${PROXY_CONTAINER_PORT}" \
    --network "${target_container_net}" \
    --name "${TARGET_CONTAINER_NAME}-proxy" \
    alpine@sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 \
    /bin/sh -c "apk update && apk add socat && socat -v TCP-LISTEN:${PROXY_CONTAINER_PORT},fork,reuseaddr TCP-CONNECT:${target_container_ip}:${TARGET_CONTAINER_PORT}"
Bash

And use it to achieve the same result we did previously:

λ ./proxy-container httpd-fun 80 8080
Bash

Conclusion

We’ve played with ways to reach out via the network on a container where we’d forgotten to publish a necessary port. This can be particularly helpful during one-shot debugging sessions as part of the broader development process.

To achieve our goal, we’ve used the versatile socat, a powerful tool in our toolbox. As a bonus, we’ve hacked a script to automate a large portion of the process, which should streamline our experimentation sessions.

That came in handy when I had a service running locally as a Docker container and after doing a couple of experiments with it, I realised that I hadn’t published the port. Although there were other means to achieve the same goal, it was entertaining nonetheless.

Finally, it’s important to keep track of what we changed and, once the session is over, declare the necessary modifications in the Dockerfile (or similar system). You also need to build a new image with what is necessary to restore immutability and make it go through the standard integration process (checks, reviews, etc) before going to production.

Originally published at https://rvarago.github.io.

Related techblogs

Discover all techblogs

Java plugins with isolating class loaders

Read more about Java plugins with isolating class loaders
Java plugins with isolating loaders

Testing Mastery: Fast and Secure Application Tests

Read more about Testing Mastery: Fast and Secure Application Tests
Lens

GitHub Actions — from Usable to Useful

Read more about GitHub Actions — from Usable to Useful
Design