Yet another Docker 'A ha!' moment

tl;dr; To build once and run Docker containers with different files use a volume mount. If that's not an option, like in CircleCI, avoid volume mount and rely on container build every time.

What the heck is a volume mount anyway?

Laugh all you like but after almost year of using Docker I'm still learning the basics. Apparently. This, now, feels laughable but there's a small chance someone else stumbles like I did and they might appreciate this.

If you have a volume mounted for a service in your docker-compose.yml it will basically take whatever you mount and lay that on top of what was in the Docker container. Doing a volume mount into the same working directory as your container is totally common. When you do that the files on the host (the files/directories mounted) get used between each run. If you don't do that, you're stuck with the files, from your host, from the last time you built.

Consider...:

# Dockerfile
FROM python:3.6-slim
LABEL maintainer="mail@peterbe.com"
COPY . /app
WORKDIR /app
CMD ["python", "run.py"]

and...:

#!/usr/bin/env python
if __name__ == '__main__':
    print("hello!")

Let's build it:

$ docker image build -t test:latest .
Sending build context to Docker daemon   5.12kB
Step 1/5 : FROM python:3.6-slim
 ---> 0f1dc0ba8e7b
Step 2/5 : LABEL maintainer "mail@peterbe.com"
 ---> Using cache
 ---> 70cf25f7396c
Step 3/5 : COPY . /app
 ---> 2e95935cbd52
Step 4/5 : WORKDIR /app
 ---> bc5be932c905
Removing intermediate container a66e27ecaab3
Step 5/5 : CMD python run.py
 ---> Running in d0cf9c546fee
 ---> ad930ce66a45
Removing intermediate container d0cf9c546fee
Successfully built ad930ce66a45
Successfully tagged test:latest

And run it:

$ docker container run test:latest
hello!

So basically my little run.py got copied into the container by the Dockerfile. Let's change the file:

$ sed -i.bak s/hello/allo/g run.py
$ python run.py
allo!

But it won't run like that if we run the container again:

$ docker container run test:latest
hello!

So, the container is now built based on a Python file from back of the time the container was built. Two options:

1) Rebuild, or
2) Volume mount in the host directory

This is it! That this is your choice.

Rebuild might take time. So, let's mount the current directory from the host:

$ docker container run -v `pwd`:/app test:latest
allo!

So yay! Now it runs the container with the latest file from my host directory.

The dark side of volume mounts

So, if it's more convenient to "refresh the files in the container" with a volume mount instead of container rebuild, why not always do it for everything?

For one thing, there might be files built inside the container that cease to be visible if you override that workspace with your own volume mount.

The other crucial thing I learned the hard way (seems to obvious now!) is that there isn't always a host directory to mount. In particular, in tecken we use a base ubuntu image and in the run parts of the CircleCI configuration we were using docker-compose run ... with directives (in the docker-compose.yml file) that uses volume mounts. So, the rather cryptic effect was that the files mounted into the container was not the files checked out from the git branch.

The resolution in this case, was to be explicit when running Docker commands in CircleCI to only do build followed by run without a volume mount. In particular, to us it meant changing from docker-compose run frontend lint to docker-compose run frontend-ci lint. Basically, it's a separate directive in the docker-compose.yml file that is exclusive to CI.

In conclusion

I feel dumb for not seeing this clearly before.

The mistake that triggered me was that when I ran docker-compose run test test (first test is the docker compose directive, the second test is the of the script sent to CMD) it didn't change the outputs when I edit the files in my editor. Adding a volume mount to that directive solved it for me locally on my laptop but didn't work in CircleCI for reasons (I can't remember how it errored).

So now we have this:

# In docker-compose.yml

  frontend:
    build:
      context: .
      dockerfile: Dockerfile.frontend
    environment:
      - NODE_ENV=development
    ports:
      - "3000:3000"
      - "35729:35729"
    volumes:
      - $PWD/frontend:/app
    command: start

  # Same as 'frontend' but no volumes or command
  frontend-ci:
    build:
      context: .
      dockerfile: Dockerfile.frontend