This section introduces to volumes and multi-staged builds, two of the most concepts in Docker.
Let's plan on what and how our app should run by enumerating :
- Get a good image by running JavaScript applications like Node
- Set the default working directory inside the image
- copy the package.json file into the image
- Install necessary dependancy
- Copy the rest of the project files
- Start the vite development server by executing npm run dev command
Our Dockerfile.dev having the above plan should then look like:
FROM node:lts-alpine
EXPOSE 3000
USER node
RUN mkdir -p /home/node/app
WORKDIR /home/node/app #set the default working directory
COPY ./package.json .
RUN npm install
COPY . .
CMD [ "npm", "run", "dev" ]
The FORM instruction sets the the official Node.js image as the base giving us all the goodness of Node .js necessary to run any JavaScript application.
The lts-alpine indicate that we want to use the Alpine variant, long term support version of the image.
The USER instruction sets the default user for the image to node. By default Docker run container as the root user. But according to Docker and Node.js best practice, this can pose a security threat. So it better to run a non-root user whenever possible. The node image comes with a non-root user named node whoch we can set as the default user using the USER instruction.
WORKDIR set the default directory to the specified with this instruction. By default the working image of any image is the root, but we don't want unecessary files sprayed all over our root directory. The working directory will be applicable to any subsequent COPY, ADD, RUN and CMD instructions.
The second COPY instruction copies the rest of the content from the current directory (.) of the host filesystem to the working directory (.) inside the image.
Finally, the CMD instruction here sets the default command for this image which is npm run dev written in exec form.
The vite development server by default runs on port 3000 , and adding an EXPOSE command seemed like a good idea, so there you go.
To build an image from our Dockerfile we'll use the command below:
docker image build --file Dockerfile.dev --tag hello-dock:dev .
Given the filename is not a Dockerfile, we have to explicitly pass the filename using the --file option.
Let's execute the following commad to run a container using the iamge we just created.
docker container run --rm --detach --publish 3000:3000 --name hello-dock-dev hello-dock:dev
What we did is okay but there is one big issue with it and a few places to be improved.
Normally, the development features in JS Framework usually come with a hot reload i.e the server reload and reflect each change made locally immediately. Unfortunately, docker is not working that way because the server is running in a container and changes are made locally.
TO solve that issue we can make use of bind mounts, using so, we can easily mount one of our local file system directory inside a container. Instead of making a whole other copy of the local syste, the bind mount can reference the local file system directly inside a container.
This way the hot reload can work as a charm and with any change made locally the server will immediately reflect them.
As seen in part I, bind mounts are achieved by using the --volume or -v option for the container run or container start commands.
--volume <local file system directory absolute path>:<container file system directory absolute path>:<read write access>
Let's apply bind mounts to our vite js app
docker container run --rm --publish 3000:3000 --name hello-dock-dev --volume $(pwd):/home/node/app hello-dock:dev
Although the usage of volume solve the issue of hot reloads, it introduces another problem.
While the dependancies of a node project live inside the nodes_modules directory on the project root.
docker container run --rm --publish 3000:3000 --name hello-dock-dev --volume $(pwd):/home/node/app hello-dock:dev
> [email protected] dev /home/node/app
> vite
sh: vite: not found
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! [email protected] dev: `vite`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?
npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2021-07-05T03_24_54_158Z-debug.log
After we mount the project root on our local file system as a volume inside the container, the content inside the container gets replaced along with the node_modules directory containing all the dependencies. This means that the vite package has gone missing.
To solve the above problem we use anonymous volume. An anonymous volume is identical to a bind mount except that we don't need to specify the source directory. The generic syntax is as follow
--volume <container file system directory absolute path>:<read write access>
The final command for starting the hello-dock container is as follow:
docker container run --rm --detach --publish 3000:3000 --name hello-dock-dev --volume $(pwd):/home/app/node --volume /home/node/app/node_modules hello-dock:dev
7546a5587d28a1d2b47bfe3ad53da10e24a1fcb9504dcc6740eb071626d5fe21
Here, Docker will take the entire node_modules directory from inside the container and tuck it away in some other directory managed by the Docker daemon on our host file system and will mount that directory as node_modules inside the container.
#Performing a Multi-Stages build in docker The image we've built so far was in a development mode. To build one in a production mode, we gon have to face some challenges.
In production mode, the npm run build compile all the JS code to HTML/CSS a JS files. No other runtinme is needed to run those files but a server like apache or nginx
We may take the following step to create an image to be run in a production mode:
- Using node as the base image to build the application
- Install nginx inside the node image and use that to serve the static files. One problem here is that the node image to too heavy and most of the stuff in there aren't needed for the static genereated files. A best approach is :
- Use node image as the base and build the application.
- Copy the files created using the node image to an nginx image.
- Create the final image based on nginx and discard all node related stuff.
This way the image contains only what needed and become very handy.
This approach is a
multi-staged build. To perform a such build, we create a Dockerfile as follow:
FROM node:lts-alpine as builder ## First stage of the build using node:lts-alpine as the base image, this stage is assigned with as builder
WORKDIR /app
COPY ./package.json ./
RUN npm install
COPY . .
RUN npm run build
FROM nginx:stable-alpine ## second stage of the build using nginx:stable-alpine as the base image
EXPOSE 80 ## nginx server runs at port 80 by default
COPY --from=builder /app/dist /usr/share/nginx/html
- The last line is a
COPYinstruction. The--from=builderpart indicates that you want to copy some files from thebuilder stage. After that it's a standard copy instruction where/app/distis the source and/usr/share/nginx/htmlis the destination. The destination used here is the default site path forNGINXso any static file you put inside there will be automatically served. Let's run our new container by executing the following command:
docker container run --rm --detach --name hello-dock-prod --publish 8080:80 hello-dock:prod
Multi-staged builds can be very useful when building large applications with a lot dependancies. If configured properly, images built in multiple stages can be very optimized and compact.
With .dockerignore files and directories can be ignored and/or excluded from the image builds. The .dockerignore file contains a list of files and directories to be ignored. Docker uses the same concept as .git. Example of the content of such file:
*Dockerfile
.git
*docker-compse*
node_modules
The .dockerignore has to be in the build context. Files and directories mentionned here will be ignored by the COPY instruction. When using a bind mount, the .dockerignorefile won't have any effect.
Working with more than one container can be a little bit difficult, if we don't grasp the nuances of container isolation. The question here is : How to connect two completely isolated containers to each other? Some Approaches:
- Accessing one container using an exposed port (NOT RECOMMANDED)
- Accessing one container using its IP adress and defalut port (in case of server) [NOT RECOMANDED]
Best Approach : connect them by putting them under a user-defined bridge network.
A network in docker is another logical object like a contianer or image. There is a plethora of commands under the docker network group for manipulating network.
docker network ls # List the networks in our system
# NETWORK ID NAME DRIVER SCOPE
# c2e59f2b96bd bridge bridge local
# 124dccee067f host host local
# 506e3822bf1f none null local
By default, Docker has five networking drivers. They are as follows:
bridge- The default networking driver in Docker. This can be used when multiple containers are running in standard mode and need to communicate with each other.host- Removes the network isolation completely. Any container running under a host network is basically attached to the network of the host system.none- This driver disables networking for containers altogether. I haven't found any use-case for this yet.overlay- This is used for connecting multiple Docker daemons across computers and is out of the scope of this book.macvlan- Allows assignment of MAC addresses to containers, making them function like physical devices in a network.
As you can see, Docker comes with a default bridge network named bridge. Any container you run will be automatically attached to this bridge network:
docker container run --rm --detach --name hello-dock --publish 8080:80 fhsinchy/hello-dock
# a37f723dad3ae793ce40f97eb6bb236761baa92d72a2c27c24fc7fda0756657d
docker network inspect --format='{{range .Containers}}{{.Name}}{{end}}' bridge
# hello-dock
Containers attached to the default bridge network can communicate with each others using IP addresses which is NOT RECOMMENDED AT ALL
A user-defined bridge, however, has some extra features over the default one. According to the official docs on this topic, some notable extra features are as follows:
-
User-defined bridges provide automatic DNS resolution between containers: This means containers attached to the same network can communicate with each others using the container name.
-
User-defined bridges provide better isolation: All containers are attached to the default bridge network by default which can cause conflicts among them. Attaching containers to a user-defined bridge can ensure better isolation.
-
Containers can be attached and detached from user-defined networks on the fly: during a container’s lifetime, you can connect or disconnect it from user-defined networks on the fly. To remove a container from the default bridge network, you need to stop the container and recreate it with different network options.
A network can be created using the network create command. The generic syntax for the command is as follows:
docker network create <network name>
## Example: command to create skynet network
docker network create skynet
228a8a20f776a975eb1da55c2c9669ae6920bdcd8718132e139b87c28d14e651
Two possible ways to attach a container to a network.
-
Using the network command:
docker network connect <network identifier> <container identifier>We might the inspect command to check if the network is successfully connected to our container:
docker network inspect --format='{{range .Containers}} {{.Name}} {{end}}' skynet hello-dock ### docker network inspect --format='{{range .Containers}} {{.Name}} {{end}}' bridge hello-dock -
Using the
--networkoption for thecontainer runorcontainer create--network <network identifier>Let's run another container and attach it to skynet and ping
hello-dockfrom it to see if the two can communicatedocker container run --network skynet --rm --name alpine-box -it alpine sh / # ping hello-dock PING hello-dock (172.19.0.2): 56 data bytes 64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.141 ms 64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.255 ms 64 bytes from 172.19.0.2: seq=2 ttl=64 time=0.234 ms --- hello-dock ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.141/0.210/0.255 msAs you can see, running ping hello-dock from inside the alpine-box container works because both of the containers are under the same user-defined bridge network and automatic DNS resolution is working.
NOTE: Keep in mind, though, that in order for the automatic DNS resolution to work you must assign custom names to the containers. Using the randomly generated name will not work.
#####DETACHING A CONTAINER TO A NETWORK
We use the network disconnect command for this task.
docker network disconnect <network identifier> <container identifier>
###
docker network disconnect skynet hello-dock
Network can be removed using the network rm command with the syntax:
docker network rm <network identifier>
###
docker network rm skynet
We may use the network prune command to remove any unused networks from our system; this command also has the -f or --force and a or --all options.
The JS project will consist of a notes-api powered by Express.js and PostgreSQL.
The project comprise of two containers in total that will be connected using a network. Concepts about environments variables and named volumes are also discussed in this.
The database of the notes-api is a simple PostgreSQL that uses the official postgres image.
According to the official docs, in order to run a container with this image, you must provide the POSTGRES_PASSWORD environment variable. Apart from this one, I'll also provide a name for the default database using the POSTGRES_DB environment variable. PostgreSQL by default listens on port 5432, so you need to publish that as well.
Let's run the DB server by executing the following:
docker network create notes-api-network # creating the network
#
docker container run --detach --name=notes-db --env POSTGRES_DB=notesdb --env POSTGRES_PASSWORD=secret --network=notes-api-network postgres:12
The --env option for the container run and container create commands can be used for providing environment variables to a container. Databases like PostgreSQL, MongoDB, and MySQL persist their data in a directory. PostgreSQL uses the /var/lib/postgresql/data directory inside the container to persist data.
Now what if the container gets destroyed for some reason? You'll lose all your data. To solve this problem, a named volume can be used. This is where named volume come in hand.
Similar to anonymous volume, except that you can refer to a named volume by using its name.
generic syntax:
docker create volume <volume name>
Let's create a volume named notes-db-data
docker create volume notes-db-data
notes-db-data
#
docker volume ls
DRIVER VOLUME NAME
local notes-db-data
Now let's mount this volume to the /var/lib/postgresql/data inside the notes-db container:
We've got to first stop and remove the old notes-db container.
docker stop container notes-db
docker rm container notes-db
Let's run a new container and assign the volume using the --volume option:
docker container run --detach --volume notes-db-data:/var/lib/postgresql/data --name notes-db --env POSTGRES_DB=notesdb --env POSTGRES_PASSWORD=secret --network=notes-api-network postgres:12
Let's do some inspection and see if the mounting was sucessful.
docker container inspect --format='{{range .Mounts}} {{ .Name }} {{end}}' notes-db
notes-db-data
Now the data will safely be stored inside the notes-db-data volume and can be reused in the future. A bind mount can also be used instead of a named volume here, but I prefer a named volume in such scenarios.
Command:
docker container logs <container identifier>
#
docker cotainer logs notes-db
Command:
docker network create notes-api-network
#
docker network connect notes-apu-network notes-db
Our Dockerfile inside the notes-api is set to be:
# stage one uses lts-alpine as its base and uses builder as the stage name
FROM node:lts-alpine as builder
# install dependencies for node-gyp
RUN apk add --no-cache python make g++
WORKDIR /app # setting our working directory
COPY ./package.json .
RUN npm install --only=prod
# stage two uses also lts-alpine as it base
FROM node:lts-alpine
EXPOSE 3000
ENV NODE_ENV=production # important for the api to run properly
USER node
RUN mkdir -p /home/node/app
WORKDIR /home/node/app
COPY . .
COPY --from=builder /app/node_modules /home/node/app/node_modules
CMD [ "node", "bin/www" ]
This is a multi-staged build. The first stage is used for building and installing the dependencies using node-gyp and the second stage is for running the application.
Let's build an image from this Dockerfile by execting:
docker image build --tag notes-api build .
We need to inspect the database and see if it status is running:
docker inspect notes-db
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 12466,
"ExitCode": 0,
"Error": "",
"StartedAt": "2021-07-11T16:35:28.39422633Z",
"FinishedAt": "2021-07-11T05:16:54.752809858Z"
},
"Mounts": [
{
"Type": "volume",
"Name": "notes-db-data",
"Source": "/var/lib/docker/volumes/notes-db-data/_data",
"Destination": "/var/lib/postgresql/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Networks": {
"notes-api-network": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"3cf117853c70"
],
"NetworkID": "f68e7b0f1b8f3714bf57e98012328e7a319fd22c2440d6d60a63d9ac014a8eca",
"EndpointID": "bcee6097053903307652fe5bad9b49450cc564ffab912d9e0d3577f0072b3e7c",
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:13:00:02",
"DriverOpts": null
}
The notes-db container is running, uses the notes-db-data volume, and is attached to the notes-api-network bridge.
Now we can run the container and start using our notes-fb application.
docker container run --detach --name=notes-api --env DB_HOST=notes-db --env DB_DATABASE=notesdb --env DB_PASSWORD=secret --publish=3000:3000 --network=notes-api-network notes-api
b0acfa02d79aac8203d9f83cabab3d2e496a9799404392151454964466f79c72
The notes-api application requires three environment variables to be set. They are as follows:
DB_HOST - This is the host of the database server. Given that both the database server and the API are attached to the same user-defined bridge network, the database server can be refereed to using its container name which is notes-db in this case.
DB_DATABASE - The database that this API will use. On Running the Database Server we set the default database name to notesdb using the POSTGRES_DB environment variable. We'll use that here.
DB_PASSWORD - Password for connecting to the database. This was also set on Running the Database Server sub-section using the POSTGRES_PASSWORD environment variable.
Although the container is running, before starting using it, we have to run the database migration necessary for setting up the database tables by executing npm run db:migrate inside the container.
We use the exec command inside a running container to execute a custom command.
Syntax:
docker container exec <container identifier> <command>
To run npm run db:migrate inside our container :
docker container exec notes-api npm run db:migrate
> notes-api@ db:migrate /home/node/app
> knex migrate:latest
Using environment: production
Batch 1 run: 1 migrations
All the steps seen in the previous section, from networking container, creating volumes, databases ...require writing a lot of commands. The process can be simplified by using shell scripts and Makefile
In the notes-api there is shell scripts as follow:
boot.sh- Used for starting the containers if they already exist.build.sh- Creates and runs the containers. It also creates the images, volumes, and networks if necessary.destroy.sh- Removes all containers, volumes and networks associated with this project.stop.sh- Stops all running containers.
There is also a Makefile that contains four targets named start, stop, build and destroy, each invoking the previously mentioned shell scripts.
If the container is in a running state in your system, executing make stop should stop all the containers. Executing make destroy should stop the containers and remove everything. Make sure you're running the scripts inside the notes-api directory:
make destroy
./shutdown.sh
stopping api container --->
notes-api
api container stopped --->
stopping db container --->
notes-db
db container stopped --->
shutdown script finished
./destroy.sh
removing api container --->
notes-api
api container removed --->
removing db container --->
notes-db
db container removed --->
removing db data volume --->
notes-db-data
db data volume removed --->
removing network --->
notes-api-network
network removed --->
destroy script finished