This is #26 in the Windows Dockerfile series, where I look at running, managing and upgrading distributed apps in Docker using Docker Compose.
Organizing Distributed Apps with Docker Compose
I spend the first five chapters of Docker on Windows running existing .NET Framework applications in Docker, packaging new .NET Core applications in Docker, and design strategies using third-party apps in containers.
You can use Docker to decompose monoliths and add new features to your apps, and the end result is lots of containers to manage which have dependencies between them. #25 in this series finished with 8 containers running a modernized version of the classic Nerd Dinner ASP.NET app.
I use a PowerShell script to run up all the containers, but that's just a simple option that keeps the focus on what the containers are doing. As soon as you have more than one container in your app, you'll be using Docker Compose to manage it.
Docker Compose is a separate client from the normal
dockerCLI. It uses a YAML file to define desired application state and makes calls to the Docker API to deploy apps.
The Docker Compose syntax is very simple and it's a great way to define the structure of a distributed app. Think of the Dockerfile as the deployment guide for each component of your app, and the Compose file as the deployment guide for the whole application.
You can use Docker Compose on a single-node Docker environment and in a cluster. So the same application definition gets used by developers with Docker Desktop right through to production with Docker Enterprise.
You can learn all about using Docker Compose in production with Linux and Windows containers in my Pluralsight course Managing Load Balancing and Scale in Docker Swarm Mode Clusters.
Defining Distributed Applications with Docker Compose
You define applications in Compose in terms of services rather than containers. Services will be deployed as containers - but a service could be multiple instances of a single container, so it's an abstraction beyond managing containers on hosts.
Here's a snippet of the full Compose file for the modernized Nerd Dinner app running in Windows containers:
version: '3.3' services: message-queue: image: nats:nanoserver networks: - nd-net elasticsearch: image: sixeyed/elasticsearch:nanoserver environment: - ES_JAVA_OPTS=-Xms512m -Xmx512m volumes: - es-data:c:\data networks: - nd-net
And here's how that breaks down:
versionis the Docker Compose schema version - different versions of Docker and Docker Compose support different features, and the Compose schema version helps enforce that
servicesare the top-level collection for all the components in the application. Each component has its own service definition
natsservice is the simplest - it defines the Docker image to use, and it connects the service containers to a Docker network.
natsis the service name, and that gets used in Docker's DNS server, so when containers lookup the
natshostname, Docker will return the IP address of this service's container
elasticsearchservice has the Docker image to use, and it connects to the same Docker network, but it also specifies extra options for
environmentvariables to surface in the containers, and Docker
volumesto attach. These are equivalent to the
-vflags in the
docker container runcommand
The next service in the Compose file is for the database:
nerd-dinner-db: image: dockeronwindows/ch03-nerd-dinner-db env_file: - db-credentials.env volumes: - db-data:C:\data networks: - nd-net
nerd-dinner-db is the SQL Server database for the Nerd Dinner app, with the schema packaged back in Windows Dockerfile #16. It has a volume for data, and it uses an environment file for environment variables.
Compose is powerful because it lets you configure services with all the options you need, but the YAML files are still simple and readable. The longest service definition is for the Nerd Dinner ASP.NET website:
nerd-dinner-web: image: dockeronwindows/ch05-nerd-dinner-web ports: - "80:80" environment: - HOMEPAGE_URL=http://nerd-dinner-homepage - MESSAGE_QUEUE_URL=nats://message-queue:4222 env_file: - api-keys.env - db-credentials.env depends_on: - nerd-dinner-homepage - nerd-dinner-db - message-queue networks: - nd-net
This also includes the
ports to publish, and the other services which this service needs, in the
In all the Compose file comes in at 87 lines, including whitespace. That's a pretty efficient way of describing a distributed app which runs across 8 services, using Nano Server and Windows Server Core, and running Go, Java, Node.js and .NET components. It's a standard format too which means you can use tools like docker-compose-viz to create visualizations:
Dependencies are worth a mention. On the desktop, Compose will start containers in the correct order to honour the dependencies. In a dynamic, clustered environment that doesn't apply - it limits the cluster too much if certain containers have ordering requirements.
Check out my session from DockerCon 2018: Five Patterns for Success for App Transformation to learn the correct way to manage dependencies for distributed apps in Docker.
This compose file is all you need to run the whole Nerd Dinner stack locally.
Deploying and Managing Apps with Compose
You use the
docker-compose command line with your Compose YAML file to manage your app. The key commands are
up to deploy the app,
down to stop the app and remove all containers, and
build to build the images.
To run the app using Compose, just clone the repo, navigate to the directory and use
git clone https://github.com/sixeyed/docker-on-windows.git cd ./docker-on-windows/ch06/ch06-docker-compose docker-compose up -d
Compose looks for a file called
docker-compose.yml if you don't specify a filename. The YAML file in this directory defines images which are all public on Docker Hub, so Compose will pull those images if you don't have them locally.
Then Compose starts containers for all the services, in the right order to maintain the dependencies. The
-d flag in Compose is the same as in
docker container run - it just starts containers in the background.
This docker-compose.yml doesn't specify the scale for any services, so they'll all launch with the default - one container per service. The Compose file is the desired state, and when you run
up Compose looks at the actual state in the Docker engine and creates what it needs to get to the desired state.
Check the running containers with
docker container ls and you'll see the whole application stack is there, all in containers with names generated by Compose - which prepends the current directory name to the service name:
You can browse to the Nerd Dinner app container now, and use the app in the same way I described back in #25.
You can use Compose to scale the application components - provided the components are able to run in multiple instances without affecting each other. The message handlers are designed to run at scale in a dynamic environment, so they can be easily scaled up:
docker-compose up -d ` --scale nerd-dinner-save-handler=3 ` --scale nerd-dinner-index-handler=2
This will add a second container for the Elasticsearch index handler, and two more containers for the SQL Server save handler. They're running message handlers which connect to NATS and because they're designed for scale, they'll share the message processing load.
Upgrading Apps with Compose
This is built as the Docker image
dockeronwindows/ch06-nerd-dinner-db, and is an update to the Chapter 3 SQL Server database that's currently running.
You can deploy the update using Docker Compose and the new docker-compose-db-upgrade.yml file:
docker-compose -f docker-compose-db-upgrade.yml up -d
Compose recreates the database service - removing the old container and running a new one from the new image tag. The new container attaches to the same volume as the old container, so all the data in SQL Server is preserved, and the new column gets added when the database container startup script runs.
There are other services defined as being dependent on the database service - and the database service has changed, so those services get recreated too. And in this case, Compose also scales down the message handler services to a single container each.
Why does Compose scale down services which I've explicitly scaled up? Because the Compose file is the desired state - and my updated file doesn't specify any service scales, so the default is 1. Compose sees the running state with a greater scale and it removes containers to bring the service in line with the desired state.
This is a side-effect from mixing declarative deployment with the Compose file and imperative deployment with the
It's better to stick to declarative deployment and make all updates through the Compose file - which lives in source control with your Dockerfiles and your app source.
Separating Concerns with Compose Overrides
You can also split your app definition across multiple Compose files. That's very handy to separate concerns - so you can include deployment options for dev and production in separate files, and have a central file for the core application definition.
I cover all that in Chapter 6 too using some sample Compose files:
docker-compose.yml defines the core application services, with options that apply in every environment
docker-compose.build.yml adds the build definitions for the custom Docker images. This gets used in
docker-compose buildby devs and in the CI pipeline, but not in other scenarios. Putting it in a separate file keeps the core Compose file clean
docker-compose.local.yml adds options for running the application locally, publishing container ports to random ports on the host
docker-compose.production.yml adds configuration for production deployments, specifying explicit port publishing and host directory mapping for the volumes.
You combine multiple files together with the
-f flag - and if several files have values for the same configuration section, the latest overrides the earliest.
So to build the whole stack I'd run:
docker-compose ` -f docker-compose.yml ` -f docker-compose.build.yml ` build
And to run locally:
docker-compose ` -f docker-compose.yml ` -f docker-compose.local.yml ` up -d
This is a nice way of structuring your application definition, so the setup for every environment lives in one place, but the details for each environment are separated.
You'll deploy to production using Compose files, but when you run in a cluster using Docker Swarm, there's extra features available to use.
Next I'll look at using some of the features in Docker Swarm which make it such a great option for production deployments: Docker secrets.
You need to some extra setup steps to get ASP.NET apps reading configuration files from Docker secrets, which I'll cover with the Dockerfile for ch07-nerd-dinner-web.