-
Notifications
You must be signed in to change notification settings - Fork 5
Networking
These configurations are for a Linux V7 installation, other distributions are very similar except for Mac and Windows.
Note: Docker for Mac and Docker for Windows has a different networking configurations.
In general, and for MarkLogic clusters espeicaly, you should configure distinct named ‘networks’ – both in docker and your OS.
The –bip flag configures dockers default network – which is where contains will run if you don’t specify otherwise. You can create as many additional networks as you like. Its very easy, and the key to being able to keep a consistant hostname and IP address mappings to containers.
The end goal of using your own network explicitly is to allow the following
Distinct and isolated networks for any group of releated containers (e.g. a MarkLogic cluster). You can achieve the following in isolated single host environments, purely remote docker hosts, and hybrid local and remote hosts.
-
Static/Fixed IP addresses for every container
-
Static/fixed hostnames for every container
-
Static DNS/Host resolution for every container
-
Docker container name matching its hostname
The ideal configuration provides consistency of hostname, networks, ports, and IP addreses such that the same names and values are used.
-
From within the container
-
To and from other containers in the same network
-
From the host OS to any container
-
From other hosts to your docker containers
The Simplified View:
A ‘Network’ is a group of IP addresses with a name (for the network). Each IP address should also have a name (hostname).
Processes in a ‘network’ should be able to talk to each other using ‘usual’ means consistently, namely by ‘name’. Very useful, but optional, is for the ‘outside’ (the docker host, or another host) to talk to the containers using the same name.
Docker manages networking for all containers, and to some extent on the Host OS. If you do nothing, it can be very confusing – container names may not be their hostnames, IP addresses and ports appear random. Often people try to manage this ‘after the fact’ by complex query scripts to figure out what docker did and store that in files, environment variables etc. Its much easier to decide yourself and tell docker what to do.
To achive this you first need a network toplogy plan. Basicaly that means to pick a set of address ranges and IP addresses and name them. Optional, if you need to communicate between hosts, is configuring routing. The OS has a few config files we can ‘borrow’ to make this easy.
You need to choose at least 2, preferably 3 or more ‘networks’. One for the default docker network, and the rest for your own use. These networks are composed of IP address ranges that do not overlap with other uses. The following examples are likely to be just fine for you. For each subnet, make an entry in /etc/networks. This file is not used much by the OS, you wont break anything.
Example: add the following in /etc/networks , make sure these do not conflict with other entries.
docker0 10.250
ml-docker 192.168.200
mlr-docker 192.168.201
This corresponds to the following subnets in CIDR format, it gives docker a large address range for its own use (64k IP addresses), and 2 named networks of 255 IPs’
- 10.250.0.0/16
- 192.168.200/24
- 192.168.201/24
Now edit /etc/hosts – its good to predefine some named hosts to use for containers later. How many is up to you, you can always add more. Leave room at the ‘bottom’ of the network for docker to use for DHCP (containers which you don’t specify the IP address are assigned from the DHCP range).
Example of /etc/hosts – ADD these to the end. It helps if you keep to a simple naming pattern.
# network ml-docker
192.168.200.128 ml-run-1
192.168.200.129 ml-run-2
192.168.200.130 ml-run-3
192.168.200.131 ml-run-4
192.168.200.132 ml-run-5
192.168.200.133 ml-run-6
192.168.200.134 ml-run-7
192.168.200.135 ml-run-8
192.168.200.136 ml-run-9
192.168.200.137 ml-run-10
# network mlr-docker
192.168.201.128 mlr-run-1
192.168.201.129 mlr-run-2
192.168.201.130 mlr-run-3
192.168.201.131 mlr-run-4
192.168.201.132 mlr-run-5
192.168.201.133 mlr-run-6
192.168.201.134 mlr-run-7
192.168.201.135 mlr-run-8
192.168.201.136 mlr-run-9
192.168.201.137 mlr-run-10
You will need this information later so that the docker configuration and the ml-docker scripts will match the OS configurations.
See Systemctl for how to add these options to Docker startup
-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock
This enables the Docker REST API over tcp as well as local socket.
-
port 2376 is the default for SSLS/TLS
-
port 2375 is the default for non-TLS
Exposing the REST API over TCP is critical for programs that use the docker API instead of calling 'docker' as a subprocess
the --bip flag changes the default Docker subnet range used for the default bridge network. This is the address range used by the internal DHCP for new containers.
By luck, Docker defaults to the same IP range as used in many VPNs, so its useful to explicitly set the subnet. The following uses a subnet that Cisco VPN does not use. You should pick a 'Non Routable' IP range, you will need one more for your own networks.
https://en.wikipedia.org/wiki/Private_network
Using the ‘docker0’ network in /etc/networks the subnet 10.250.0.0 to 10.250.255.255
--bip=10.250.0.1/16
Note: you MUST use a .1 not .0 for –bip !
--tlsverify --tlscacert /etc/docker/ca.pem
--tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem
If you want to configure TLS use the following guides ... or skip ...
http://tech.paulcz.net/2016/01/secure-docker-with-tls/
There is a good docker image that does most of the work for you.
https://hub.docker.com/r/paulczar/omgwtfssl/
You should configure environment variables before running docker. By default, docker will attempt to use the local connection with default values – that’s not always what you want.
If your only going to run one docker server from one host then you can hard-code these into your login scripts (.profile / .bashrc / .kshrc etc)
I recommend that you make script file just for setting these variables so that in the future you can easily switch between docker environments.
At minimum, you should set DOCKER_HOST to the URL to your docker engine, even if its local. Use the port configured in docker.conf * port 2376 is the default for SSL/TLS * port 2375 is the default for non-TLS. An example from my .bashrc
DOCKER_HOST=tcp://localhost:2376
# If using TLS set the following
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/home/dlee/.docker/machine/machines/default
# if not using TLS
DOCKER_TLS_VERIFY=0
If you use docker-machine, you should use the ‘docker-machine env’ command to set these. Or you can write a simple script to set the values according to your own installation.
After you have the basic docker up and running and fully configured, the last step is to configure a ‘docker network’ for each of the networks you defined in /etc/networks (except docker0).
There is a script to do this for you in ml-docker/bin/create-network.
Usage
bin/create-network network-name
This creates a docker network configured using the data from /etc/networks and /etc/hosts. The result is a docker network of the same name as the OS network name, with the same IP range for the subnet. The lower 128 IP addresses are dedicated for DCHCP, the upper 128 are for static IP addresses (in /etc/hosts).
At the end it runs ‘docker network inspect’ which will print out your network configuration.
The script calculates the correct values for the network name, subnet, gateway, drive and ip range and runs this command:
docker network create --subnet $sub --driver bridge --gateway $gateway --ip-range $range $net
#!/bin/bash
# Example of creating a named network
# This allocates a named subnet based on a network defined in /etc/networks
# It assumes a /24 ( 255.255.255.0 ) network ( 254 IPs)
# Example: /etc/networks:
# ml-docker 192.168.66
###########################################################
# create-network ml-docker
#
# Creates a network named 'ml-docker' with a /24 subnet
# Configures DHCP for the high 128 IPs leaving the low 128 for static IPs
#
# Notes on docker network cli ---
# --ip-range is like dhcp, if you dont specify an ip in docker run then it used the ip-range
# Run with --ip=192.168.66. 1-127 , docker will allocate .128+
# Gateway is the outbound gateway IP, it should end in 1+ the loweset subnet IP,
# Note: the network will stay around until you remove it with 'docker network rm '
# You can see it with 'docker network ls' as well as 'route' or 'routel' or 'iproute'
#
net=${1:-${DOCKER_NETWORK:-ml-docker}}
subnet=$(getent networks $net | awk '{print $2;}')
[ -z "$subnet" ] && { echo "Create network $net in /etc/networks" ; exit 1; }
sub=${subnet%.0}.1/24
range=${subnet%.0}.128/25
gateway=${subnet%.0}.1
docker network create --subnet $sub --driver bridge --gateway $gateway --ip-range $range $net
docker network inspect $net
# Remove with
# docker network rm ml-docker
#
If all goes well it should look like this bin/create-network ml-docker
[
{
"Name": "ml-docker",
"Id": "7c4650337c5d1a072ceec0e3cd26874a950bdd5dba0994582ac53da68e4ab6e6",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {
},
"Config": [
{
"Subnet": "192.168.200.1/24",
"IPRange": "192.168.200.128/25",
"Gateway": "192.168.200.1"
}
]
},
"Internal": false,
"Containers": {
},
"Options": {
},
"Labels": {
}
}
]
You are now ready to build and run docker containers, and clusters of MarkLogic.