Docker Container

There are some challenges when running application on Mesos which Docker can address quite nicely:

App distribution: You need to package/version your application and make it available for all Mesos slave. Possible solutions are:
- Using standard OS package like rpm, deb and install on all server. Pros: proper application versioning & metadata. Cons: doesn't scale
- Storing application in centralized storage like NFS or HDFS and use Artifact Store to retrieve in your job. Pros: centralized storage. Cons: not a proper packaging mechanism, limited support.
Resource control and isolation: By default Mesos doesn't impose resource constraint on Mesos containers. You will have to enable various configurations to achieve total isolation from your host namespaces. There is also no per-app constraints, all configurations are applied on slave level.

mesos-execute --master=$(mesos-resolve `cat /etc/mesos/zk` 2>/dev/null) --containerizer=docker --docker_image=python:2.7-alpine --name='dockertest' --command='python -m SimpleHTTPServer'

Mesos will delegate the task to Docker daemon. The daemon will then pull the Docker image and launch a container to execute the command specified in --command. Sometimes the Docker image can be quite big that it takes a while to download. You will notice from the Mesos/Marathon UI that your task state is set to STAGING during the download. By default Mesos only wait 1 minute before killing off STAGING tasks therefore you will have to configure Mesos slave to wait for longer by setting --executor_registration_timeout to longer period (default is 1mins). A good practice is to ensure your image is as small as possible e.g. using minimum Docker image such as Alpine Linux so that your application can be deployed faster.

Docker containerizer

To enable Docker on Mesos, you must launch the agent with "docker" as one of the containerizer option i.e. --containerizer = docker. This requires a Docker daemon running on the Mesos slave.

Running Docker images from private registry

https://github.com/apache/mesos/blob/master/docs/docker-containerizer.md#private-docker-repository

https://github.com/apache/mesos/blob/master/src/docker/docker.cpp#L1477

Running Docker on Marathon

  "id": "/python-simple-server-docker",
  "cmd": "python -m SimpleHTTPServer 5000",
  "cpus": 0.1,
  "mem": 32,
  "disk": 0,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "python:2.7-alpine",
      "network": "HOST",
      "portMappings": null,
      "privileged": false,
      "parameters": [],
      "forcePullImage": true
    }
  },
  "healthChecks": [
    {
      "path": "/",
      "protocol": "HTTP",
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3,
      "ignoreHttp1xx": false,
      "port": 5000
    }
  ],
  "portDefinitions": [
    {
      "port": 5000,
      "protocol": "tcp",
      "labels": {}
    }
  ]
}

Let's look closer at the Marathon app definition. Here we have added the "container" configuration which defines

image: we use the minimalistic alpine-based python image here. More on Alpine: https://alpinelinux.org/
network: we use HOST (docker option: --net host) which put our container in the same network as the host network. Other options are BRIDGE (docker default isolated network) and USER (user-defined network)
forcePullImage: true always pull image for updates before start. false will always launch container from local image

Just like our previous example, our Python app will serve static content on port 5000 from one of Mesos slave since HOST networking was used for the Docker container. Doing a query however returns different contents

curl mesos-slave-002.com:5000
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><html>
<title>Directory listing for /</title>
<body>
<h2>Directory listing for /</h2>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a>
<li><a href="bin/">bin/</a>
<li><a href="dev/">dev/</a>
<li><a href="etc/">etc/</a>
<li><a href="home/">home/</a>
<li><a href="lib/">lib/</a>
<li><a href="linuxrc">linuxrc@</a>
<li><a href="media/">media/</a>
<li><a href="mnt/">mnt/</a>
<li><a href="proc/">proc/</a>
<li><a href="root/">root/</a>
<li><a href="run/">run/</a>
<li><a href="sbin/">sbin/</a>
<li><a href="srv/">srv/</a>
<li><a href="sys/">sys/</a>
<li><a href="tmp/">tmp/</a>
<li><a href="usr/">usr/</a>
<li><a href="var/">var/</a>
</ul>
<hr>
</body>
</html>

Since we are running on Docker container not Mesos container anymore, all host namespaces has been replaced by the container namespace. We still however can view the stdout and stderr produced by container's command.

Stdout

--container="mesos-0e06779f-f1ad-46df-b47e-f06209ae9361-S3.cfe6695c-be97-482f-b06a-b791c36f3a91" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/data/mesos/work/slaves/0e06779f-f1ad-46df-b47e-f06209ae9361-S3/frameworks/b0468ff3-bac1-475a-8695-5ddf225f7f9f-0000/executors/test-docker.0596972c-1ed2-11e7-be77-005056ab487f/runs/cfe6695c-be97-482f-b06a-b791c36f3a91" --stop_timeout="0ns"
--container="mesos-0e06779f-f1ad-46df-b47e-f06209ae9361-S3.cfe6695c-be97-482f-b06a-b791c36f3a91" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/data/mesos/work/slaves/0e06779f-f1ad-46df-b47e-f06209ae9361-S3/frameworks/b0468ff3-bac1-475a-8695-5ddf225f7f9f-0000/executors/test-docker.0596972c-1ed2-11e7-be77-005056ab487f/runs/cfe6695c-be97-482f-b06a-b791c36f3a91" --stop_timeout="0ns"
Registered docker executor on mesos-slave-002.com
Starting task test-docker.0596972c-1ed2-11e7-be77-005056ab487f
Serving HTTP on 0.0.0.0 port 5000 ...

Stderr (notice the full command used by docker run)

I0411 16:15:15.402621 15434 exec.cpp:162] Version: 1.1.0
I0411 16:15:15.416131 15439 exec.cpp:237] Executor registered on agent 0e06779f-f1ad-46df-b47e-f06209ae9361-S3
I0411 16:15:15.418313 15439 docker.cpp:811] Running docker -H unix:///var/run/docker.sock run --cpu-shares 102 --memory 33554432 -e PORT_5000=31000 -e MARATHON_APP_VERSION=2017-04-11T16:15:04.342Z -e HOST=mesos-slave-002.com -e MARATHON_APP_RESOURCE_CPUS=0.1 -e MARATHON_APP_RESOURCE_GPUS=0 -e MARATHON_APP_DOCKER_IMAGE=python:2.7-alpine -e MESOS_TASK_ID=test-docker.0596972c-1ed2-11e7-be77-005056ab487f -e PORT=31000 -e MARATHON_APP_RESOURCE_MEM=32.0 -e PORTS=31000 -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/test-docker -e PORT0=31000 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-0e06779f-f1ad-46df-b47e-f06209ae9361-S3.cfe6695c-be97-482f-b06a-b791c36f3a91 -v /data/mesos/work/slaves/0e06779f-f1ad-46df-b47e-f06209ae9361-S3/frameworks/b0468ff3-bac1-475a-8695-5ddf225f7f9f-0000/executors/test-docker.0596972c-1ed2-11e7-be77-005056ab487f/runs/cfe6695c-be97-482f-b06a-b791c36f3a91:/mnt/mesos/sandbox --net host --entrypoint /bin/sh --name mesos-0e06779f-f1ad-46df-b47e-f06209ae9361-S3.cfe6695c-be97-482f-b06a-b791c36f3a91 python:2.7-alpine -c python -m SimpleHTTPServer 5000
10.32.19.193 - - [11/Apr/2017 16:16:09] "GET / HTTP/1.1" 200 -
10.32.19.193 - - [11/Apr/2017 16:17:09] "GET / HTTP/1.1" 200 -

More on Docker on Marathon: https://github.com/mesosphere/marathon/blob/master/docs/docs/native-docker.md

Building Docker Image

We have seen that Docker can streamline Mesos deployment. No longer we need to install binaries on all Mesos slaves. This has a huge advantage of making our infrastructure immutable which is essential for reliability and scalability. With great power comes great responsibility however, and Docker is not without its limitation. While it is fairly easy to write some Dockerfiles and push to some Docker repo, there are challenges such as

Image size: you don't want to package the whole Centos image to deploy a 10 Mb go binary
What to include: this links to both image size and security. Putting more softwares in a Docker image means more security risks. You want your Docker image to contain just enough to run your application.

Docker Builder Pattern

The Builder Pattern separate the build process into 2 different stages: one for producing the actual binary and the second to copy the binary into a production image that provide the right runtime to run that binary. These 2 processes can be linked together by some scripts or automation tools such as make.

# Example Makefile
build-binary:
    docker build -t YOUR_APP:build -f Dockerfile.build .
    docker create --name YOUR_APP_BUILD YOUR_APP:build  /bin/bash
    docker cp YOUR_APP_BUILD:PATH_TO_BINARY .
    docker rm YOUR_APP_BUILD

build-image:
    docker build --no-cache -t YOUR_APP:latest .

build-docker: build-binary build-image

Here we use Dockerfile.build for building our application. The .build image usually contains language compilers e.g. golang:1.7 for golang or openjdk:8-jdk for java.

For example, the following Dockerfile.build is used to build a Maven project

FROM anapsix/alpine-java:8_jdk

# Download maven
RUN MAVEN_VERSION=3.0.5 \
    && cd /usr/share \
    && wget http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz -O - | tar xzf - \
    && mv /usr/share/apache-maven-$MAVEN_VERSION /usr/share/maven \
    && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn

ENV PATH_TO_BINARY=/usr/share/lib

ADD . /build
RUN cd /build && mvn clean package -P standalone -DskipTests

To build a image which in turn build our binary run make build-binary. Once finish, we will have our application inside the newly create Docker image which we can copy it over to the next build step (by launching a Docker container and run docker cp). The final build step is just copy the binary over and preparing some launch scripts.

# Note that we only use the JRE version here
FROM anapsix/alpine-java:8e

# Add binaries and launch script
ADD ....

ENTRYPOINT ["..."]

Docker Multi-stage build (New)

Starting from Docker 17.05 you can use a single Dockerfile to achieve the same result as described in the Builder Pattern.

FROM ubuntu AS build-env
RUN apt-get install make
ADD . /src
RUN cd /src && make

FROM busybox
COPY --from=build-env /src/build/app /usr/local/bin/app
EXPOSE 80
ENTRYPOINT /usr/local/bin/app

We can now source from multiple base images but only the last one with be used for the final output image. We can also copy files from one image to another. In the above example, we start off with a build image (ubuntu) and copy the build output over to a busybox image which will be our runtime base image. The result is just the busybox image with the ready-to-run binary and thus the size is minimum.

Passing Parameter to Docker container

Very often you will need to pass some arguments to your Docker container. These values can be passed in command line arguments or stored in a config file. Docker supports passing in environment variables when launching containers so you can substitute your app arguments with these variables. If your application read settings from a configuration file however, you will need to produce the file from a template, usually by text manipulation program like sed or envsubst

To use envsubst in your container:

ENV BUILD_DEPS="gettext"  \
    RUNTIME_DEPS="libintl"

# install envsubst
RUN apk add --update $RUNTIME_DEPS && \
    apk add --virtual build_deps $BUILD_DEPS &&  \
    cp /usr/bin/envsubst /usr/local/bin/envsubst && \
    apk del build_deps

# Add the config template
ADD server.properties.template /APP_DIR

Your template file will look like this

port = ${PORT}
bind_address = ${BIND_ADDRESS}

In your docker-entry.sh

cd /APP_DIR
cat server.properties.template | envsubst > server.properties
./APP_BINARY --config server.properties

Docker Container

Docker Container

Docker containerizer

Running Docker images from private registry

Running Docker on Marathon

Building Docker Image

Docker Builder Pattern

Docker Multi-stage build (New)

Passing Parameter to Docker container

results matching ""

No results matching ""