In this post we will go through the process of building bcardiff/miniserver: a 12MB docker image that runs the github:bcardiff/miniserver Crystal app.
Soon(ish) a great feature will be arriving to your Docker tool:
Happy day: https://t.co/WyXdLexRBq
— Darren Shepherd (@ibuildthecloud) March 24, 2017
Essentially, it will simplify the process of building a Docker image that will contain just the required files. This leads to smaller images and reduced download times, letting us save energy and a few trees in the process.
So, let’s move ahead in time and see what we can do to save those trees.
Our time machine in this case will be docker@master. At the very end of this article there is a simple guide on how to do that. But, for now, back to the future.
Building our app
Let’s say we have created a Crystal app with the following structure:
$ tree .
.
├── shard.yml
└── src
└── miniserver.cr
The shard.yml file will contain a description saying that the miniserver executable should be generated from src/miniserver.cr:
# ...
targets:
miniserver:
main: src/miniserver.cr
# ...
In our development environment $ shards build will fetch all dependencies and generate the bin/miniserver executable.
Building the app in Docker
Let’s create a Dockerfile to ship this thing. Our goal here is to be able to run this Dockerfile in an automated way. So, we will start from a Docker image that ships with a Crystal compiler. Do we really need to ship the compiler itself with the miniserver? No, we won’t need it in the final image.
FROM crystallang/crystal:latest
ADD . /src
WORKDIR /src
RUN shards build --production
If we run $ docker build . it will create a image with the /src/bin/miniserver binary but the image is ~400MB, this Docker image is based on ubuntu after all.
If we extract the binary from this new image we will be halfway there: the generated binary depends on shared libraries.
Building a small Docker image
You can follow some manual procedure to find which are the runtime dependencies or, as we will do here, use our favorite language to simplify the task, even if this task is meant to be done once (per project). Help yourself to a copy of the list-deps.cr gist and leave it in ./support/list-deps.cr if you want to be tidy. This script will collect the libs your project depends on, and output the very contents of the second-stage that you’ll need to append to your Dockerfile. So we would have something like this in our working copy:
.
├── Dockerfile
├── shard.yml
├── src
│ └── miniserver.cr
└── support
└── list-deps.cr
Let’s update the Dockerfile to run the list-deps.cr script again ./bin/miniserver:
FROM crystallang/crystal:latest
ADD . /src
WORKDIR /src
RUN shards build --production
RUN crystal run ./support/list-deps.cr -- ./bin/miniserver
This time after running docker build . we will have some output that will help us in the next step including the new features for multi-stage builds:
$ docker build .
Sending build context to Docker daemon 2.454 MB
Step 1/5 : FROM crystallang/crystal:latest
---> 8ceae3fe1276
Step 2/5 : ADD . /src
---> 3af380f48502
Removing intermediate container 953cf6b11ae1
Step 3/5 : WORKDIR /src
---> cda0b4db245e
Removing intermediate container e639c5e9e720
Step 4/5 : RUN shards build --production
---> Running in a9ecd9fb6eea
Dependencies are satisfied
Building: miniserver
---> 895e4e9924a2
Removing intermediate container a9ecd9fb6eea
Step 5/5 : RUN crystal run ./support/list-deps.cr -- ./bin/miniserver
---> Running in 07806672cdcd
Extracting libraries for /src/bin/miniserver ...
Generating Dockerfile
==============================
FROM scratch
COPY --from=0 /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1
COPY --from=0 /lib/x86_64-linux-gnu/libz.so.1.2.8 /lib/x86_64-linux-gnu/libz.so.1.2.8
COPY --from=0 /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libssl.so.1.0.0
COPY --from=0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
COPY --from=0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6
COPY --from=0 /lib/x86_64-linux-gnu/libm-2.19.so /lib/x86_64-linux-gnu/libm-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=0 /lib/x86_64-linux-gnu/libpthread-2.19.so /lib/x86_64-linux-gnu/libpthread-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt.so.1
COPY --from=0 /lib/x86_64-linux-gnu/librt-2.19.so /lib/x86_64-linux-gnu/librt-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so.2
COPY --from=0 /lib/x86_64-linux-gnu/libdl-2.19.so /lib/x86_64-linux-gnu/libdl-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=0 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
COPY --from=0 /lib/x86_64-linux-gnu/libc-2.19.so /lib/x86_64-linux-gnu/libc-2.19.so
COPY --from=0 /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=0 /lib/x86_64-linux-gnu/ld-2.19.so /lib/x86_64-linux-gnu/ld-2.19.so
COPY --from=0 /src/bin/miniserver /miniserver
ENTRYPOINT ["/miniserver"]
==============================
---> 92557005cfbe
Removing intermediate container 07806672cdcd
Successfully built 92557005cfbe
Append all these Dockerfile commands the output suggested to your Dockerfile, and add the ENV, EXPOSE, etc. your app needs to properly work.
With these copied lines, we don’t need to run the list-deps.cr again. So the Dockerfile could be:
FROM crystallang/crystal:latest
ADD . /src
WORKDIR /src
RUN shards build --production
# RUN crystal run ./support/list-deps.cr -- ./bin/miniserver
FROM scratch
COPY --from=0 /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1
COPY --from=0 /lib/x86_64-linux-gnu/libz.so.1.2.8 /lib/x86_64-linux-gnu/libz.so.1.2.8
COPY --from=0 /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libssl.so.1.0.0
COPY --from=0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
COPY --from=0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6
COPY --from=0 /lib/x86_64-linux-gnu/libm-2.19.so /lib/x86_64-linux-gnu/libm-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=0 /lib/x86_64-linux-gnu/libpthread-2.19.so /lib/x86_64-linux-gnu/libpthread-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt.so.1
COPY --from=0 /lib/x86_64-linux-gnu/librt-2.19.so /lib/x86_64-linux-gnu/librt-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so.2
COPY --from=0 /lib/x86_64-linux-gnu/libdl-2.19.so /lib/x86_64-linux-gnu/libdl-2.19.so
COPY --from=0 /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=0 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
COPY --from=0 /lib/x86_64-linux-gnu/libc-2.19.so /lib/x86_64-linux-gnu/libc-2.19.so
COPY --from=0 /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=0 /lib/x86_64-linux-gnu/ld-2.19.so /lib/x86_64-linux-gnu/ld-2.19.so
COPY --from=0 /src/bin/miniserver /miniserver
ENV WWW=/www
EXPOSE 80
ENTRYPOINT ["/miniserver"]
It looks like two Dockerfiles in one and that’s exactly the idea of multi-stage builds. The first FROM is the building environment. The second FROM allows us to start from scratch and will grab just the required resources to run the miniserver.
Note: if you aren’t in the future yet, here you might need to perform the jumping ahead in time steps now.
Now we finally build the image we wanted with $ docker build .. This time ~12 MB are used. And thanks to the order of the dependencies 10MB probably will be reused across Docker images. And these numbers are even before compression.
After your image is published you can tell everybody about it. In this case, go ahead and run:
$ docker run -v `pwd`:/www -p 8080:80 bcardiff/miniserver
Listening on http://0.0.0.0:80
You can see the content served by the built-in static file handler of Crystal when you navigate to http://localhost:8080.
Jumping ahead in time
If you want to do all the cool things exposed in this post today, we need to work with docker@master. That is bleeding edge. Let’s be sure we are on the working copy of our miniserver project:
$ ls
Dockerfile bin shard.yml src support
$ pwd
/path/to/miniserver
Let’s checkout and build Docker
$ cd ..
$ git clone git@github.com:docker/docker.git
$ cd docker
$ make tgz
… grab a cup of coffee …
Now we need to have an inception moment. In order to use the recently compiled Docker we need to use it in a Docker image. But we want the miniserver source code available so we can build the Dockerfile with the multi-stage commands.
$ docker run --rm -v `pwd`/../miniserver:/src -v `pwd`/bundles:/go/src/github.com/docker/docker/bundles --privileged -ti docker-dev:master bash
root@f8b6a482ed7a:/go/src/github.com/docker/docker# export PATH=$PATH:`pwd`/bundles/latest/dynbinary-daemon:`pwd`/bundles/latest/binary-client/
root@f8b6a482ed7a:/go/src/github.com/docker/docker# dockerd &
[1] 12
… (press enter to get a console prompt) ....
root@f8b6a482ed7a:/go/src/github.com/docker/docker# cd /src
root@f8b6a482ed7a:/src# docker build . -t bcardiff/miniserver:latest
… grab a small snack …
Successfully tagged miniserver:latest
root@f8b6a482ed7a:/src# docker login
Username: bcardiff
Password: ******
Login Succeeded
root@f8b6a482ed7a:/src# docker push bcardiff/miniserver:latest
The push refers to a repository [docker.io/bcardiff/miniserver]
ed6705234e71: Pushed
bf067b612ffb: Pushed
…
root@f8b6a482ed7a:/src# exit
exit
$
The Docker image is now ready to be used by everybody, even if they are not running a bleeding edge Docker.