Running Mesos

We will look at how to deploy a Mesos cluster. Below is the system after we finish.

We want to have high availability on everything, therefore we run multiple instances for Mesos Master, Marathon and Chronos on multiple host. Three or five is usually a good number. Note that we need an odd number of instances to avoid split brain problem when failure happens. Only one instance can be leader (green color) and that leader is determined by Zookeeper.

Setting up Mesos Master

First of all we need to install the repo which contains all mesos packages. Assuming we are using Red hat, install the following packages on all servers:

yum installhttp://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm

On (assuming) 3 zookeeper server

yum install mesos  mesosphere-zookeeper
echo <ZOOKEEPERID>  > /var/lib/zookeeper/myid     # ZOOKEEPERID is id of your server i.e. 1/2/3

Add the following lines to /etc/zookeeper/conf/zoo.cfg

# The port at which the clients will connect
clientPort=2181
clientPortAddress=0.0.0.0
# The ports for follower connections and leader election respectively
server.1=<SERVER1HOSTNAME>:2888:3888
server.2=<SERVER2HOSTNAME>:2888:3888
server.3=<SERVER3HOSTNAME>:2888:3888

Start Zookeeper

systemctl start zookeeper

Once all 3 Zookeeper server has started successfully, we can now setup mesos master and tell it to connect to Zookeeper. On each master server:

yum install mesos marathon chronos

# Starting mesos
/usr/bin/mesos-init-wrapper master

# Or start as a systemd service
systemctl enable mesos-master
systemctl start mesos-master

You can configure mesos with different configurations by supplying --key value into mesos-init-wrapper script. Mesos also stores config in /etc/mesos/ or /etc/mesos-master/ or /etc/mesos-slave. Let say you want to start mesos-master on a different port (default is 5050)

echo 6060 > /etc/mesos-master/port

# Alternatively you can run /usr/bin/mesos-init-wrapper master --port 6060
# To set zookeeper URL:
echo 'zk://10.193.154.120:2181,10.193.154.122:2181,10.193.154.140:2181/mesos' > /etc/mesos/zk

# We expect at least 2 mesos masters to be up all the times 
echo 2 > /etc/mesos-master/quorum

# Mesos work dir
echo '/tmp/mesos/work' > /etc/mesos-master/work_dir
echo '/tmp/mesos/work' > /etc/mesos-slave/work_dir

# Optionally, we can also add attributes to slave(to be used later for constraints)
echo north-west > /etc/mesos-slave/attributes/rack

Setting isolation mechanism for running tasks (on slave), by default posix/cpu,posix/mem is used which only report cpu/memory usage without taking any action. We are going to use cgroups since it is the de facto for controlling resource usage.

echo 'cgroups/cpu,cgroups/mem' > /etc/mesos-slave/attributes/isolation

Starting allmesos-master and mesos-slave service, go to any http://mesos-master-host:5050 and you should see that we have a cluster of masters and agents ready to do some cool stuffs.

TODO: Add picture of cluster

Testing Mesos

On any mesos node, run

# Get Mesos master IP:
export MASTER=$(mesos-resolve `cat /etc/mesos/zk` 2>/dev/null)

# Run a Mesos task
mesos-execute  --master=$MASTER --name="cluster-test" --command="sleep 40"

You then should be able to see the task running on Mesos UI and finish after 40 seconds. Congratulations! You just launched your first Mesos application.