Using Spotinst Elastigroup to manage stateful containers with Docker Swarm & Cloudstor plugin

Nowadays, while containers are gaining popularity and becoming a commodity for organizations to run their workloads, containers orchestration platforms are becoming essential. One of the more popular platforms is Docker Swarm.  

In this blog, we’ll be talking about how to utilize persistent data volumes in Docker Swarm on top of AWS, using the Cloudstor plug-in, whilst utilizing Spotinst Elastigroup to manage Docker Swarm Workers. Cloudstor provides a solution for managing persistent data volumes on top of cloud platforms within Docker Swarm, and Spotinst Elastigroup provides a solution for running Docker Swarm on Spot Instances in AWS. Using this combination will ensure that the required infrastructure for the containers will be always available and sourced in the most cost-effective way while maintaining persistent data volumes.

Installation Procedure

Create an IAM Role with the following Policy:

“Action”: [
  “ec2:CreateTags”,
  “ec2:AttachVolume”,
  “ec2:DetachVolume”,
  “ec2:CreateVolume”,
  “ec2:DeleteVolume”,
  “ec2:DescribeVolumes”,
  “ec2:DescribeVolumeStatus”,
  “ec2:CreateSnapshot”,
  “ec2:DeleteSnapshot”,
  “ec2:DescribeSnapshots”
],
“Effect”: “Allow”,
“Resource”: “*”

Make sure to attach this IAM role to your instances (Manager & Workers).

Install Docker Swarm Server

Use the following commands in order to install Docker Swarm Server (this example is for CentOS/Amazon Linux)

#! /bin/bash
yum update -y
yum install docker -y
service docker start
docker swarm init # copy these values and save them, we will use the token in order to connect
docker node ls # this is to validate that the swarm creates and there is one node: the manager itself

# Exposing an API port for the integration with Elastigroup
yum install socat
socat -d TCP-LISTEN:2376,reuseaddr,fork  UNIX:/var/run/docker.sock &

Install Cloudstor:AWS plugin on your Swarm Manager

# Use this command if you want to support EBS as well as EFS
docker plugin install --alias cloudstor:aws --grantall-permissions docker4x/cloudstor:18.03.0-ce-aws1 CLOUD_PLATFORM=AWS EFS_ID_REGULAR=<YOUR_EFS_ID> EFS_ID_MAXIO=<YOUR_MAXIO_EFS_ID> AWS_REGION=<REGION_NAME EFS_SUPPORTED=1 DEBUG=1

# Use this command if you only want to support EBS
docker plugin install --alias cloudstor:aws --grant-all-permissions docker4x/cloudstor:18.03.0-ce-aws1 CLOUD_PLATFORM=AWS AWS_REGION=<REGION_NAME> EFS_SUPPORTED=0 DEBUG=1

mkdir /dev/mqueue

# The EFS_ID_MAXIO value could be similar to the EFS_ID_REGULAR value

Install Docker Swarm Workers

Use the following User-Data script in order to install Docker Swarm Workers 

#!/bin/bash
yum update -y
yum install docker -y
usermod -a -G docker ec2-user
service docker start
docker swarm join \
    --token {{workerToken}} \
    {{managerHost}}:2377 # Make sure you use the private ip of the managerHost

# Use this command if you want to support EBS as well as EFS
docker plugin install --alias cloudstor:aws --grantall-permissions docker4x/cloudstor:18.03.0-ce-aws1 CLOUD_PLATFORM=AWS EFS_ID_REGULAR=<YOUR_EFS_ID> EFS_ID_MAXIO=<YOUR_MAXIO_EFS_ID> AWS_REGION=<REGION_NAME EFS_SUPPORTED=1 DEBUG=1

# Use this command if you only want to support EBS
docker plugin install --alias cloudstor:aws --grant-all-permissions docker4x/cloudstor:18.03.0-ce-aws1 CLOUD_PLATFORM=AWS AWS_REGION=<REGION_NAME> EFS_SUPPORTED=0 DEBUG=1

mkdir /dev/mqueue

# The EFS_ID_MAXIO value could be similar to the EFS_ID_REGULAR value

Cloudstor plugin use cases

Cloudstor plugin gives a solution for having persistent data volumes with Docker Swarm. Cloudstor:aws provides solutions working with AWS EBS (Elastic Block Store) and EFS (Elastic File System) storage solutions.

This section will go through the three available options and how they work.

EBS Volume

Amazon Elastic Block Store (Amazon EBS) provides persistent block storage volumes for use with Amazon EC2 instances.

# Creating a service using EBS volume
docker service create \
--replicas 1 \
-p 8081:80 \
--constraint 'node.role != manager' \
--name nginx1 \
--mount type=volume,volume-driver=cloudstor:aws,source={{.Service.Name}}-{{.Task.Slot}}-vol,target=/mydata,volume-opt=backing=relocatable,volume-opt=size=5,volume-opt=ebstype=io1,volume-opt=iops=100 \
nginx

While working in the same AZ, the plugin makes sure to reattach the volume to another available node and reschedule the container on it.

While working in different AZs, the plugin makes sure to recreate the EBS volume and attach it to the available node, and then reschedule the container on it.

The procedure of migrating a snapshot between AZ’s could take a while (depending on the size of the volumes) therefore it is not recommended to run the plugin using EBS volumes on production environments running on multiple AZ’s, without planning the architecture accordingly in advance.

EFS

Amazon Elastic File System (Amazon EFS) provides simple, scalable, elastic file Storage.

Unique EFS Volume

Using this method we use EFS to create a volume for a specific container.

# Creating a service using EFS unique volume
docker service create \
--replicas 1 \
-p 8082:80 \
--name nginx2 \
--mount type=volume,volume-driver=cloudstor:aws,source={{.Service.Name}}-{{.Task.Slot}}-vol,destination=/mydata \
nginx

AZ difference is irrelevant in this use case. since EFS is exposed to all AZs, the container was rescheduled and the plugin made sure the volume was attached to it.

Shared EFS Volume

In this method, we use EFS to create a shared volume that will serve more than one container.

# Creating a service using EFS shared volume
docker service create \
--replicas 3 \
--name nginx3 \
--mount type=volume,volume-driver=cloudstor:aws,source=sharedvol1,destination=/shareddata \
nginx

AZ’s are irrelevant in this use case as well, all the containers scheduled were exposed to this shared volume.

Using Spotinst Docker Swarm Integration & Autoscaler

You can create your cluster of Swarm workers using Spotinst Elastigroup, the integration will work this way:

1. Log in to your Spotinst console. (https://console.spotinst.com)
2. Create a new elastigroup or edit an existing group, in the compute page in additional configuration section make sure to choose the IAM role you’ve created, scroll down to the Integrations section.
3. Select Docker Swarm and enter the API URL (either its DNS record or public IP of the Swarm manager) and Port.
4. Create an ingress rule allowing communication from spotinst SaaS platform (34.226.61.134/32) to your Swarm manager in the API port you exposed earlier.
5. Click Test connection to validate connectivity.
   

Please note: You have to expose an additional port for this integration, you can do that using the following command in the Swarm manager: 

sudo socat -d TCP- LISTEN:2376,reuseaddr,fork UNIX:/var/run/docker.sock &

Spotinst Elastigroup allows you to create a Docker Swarm nodes cluster using different instance types out of the box.

Spotinst’s Docker Swarm AutoScaler uses Tetris Scaling to schedule your workloads in the most cost-effective way (Both scaling up and down), the scaling is driven by the needs of the services and containers you deploy on top of it and not by nodes resources thresholds you have to set in advance.

On top of that, by using Spotinst AutoScaler, you can set up an upfront spare capacity that will be available for very quick scaling of new resources, this is possible due to the configuration of Headroom capacity, in general, you configure a building block consistent of CPU and memory and set the number of these building blocks you want to maintain available in your cluster.

Read more of Spotinst Docker Swarm Auto-Scaler and how it works.

Conclusion

Using the Cloudstor plugin provides an easy way to deploy consistent storage in Docker Swarm environments. When integrating this over Elastigroup with the Docker Swarm integration & Spotinst Autoscaler, you can achieve maximum cost efficiency and ease of management for Docker Swarm.

For more information about Spotinst Elastigroup visit – https://spot.io/products/elastigroup/

For more information on Spotinst Elastigroup integration with Docker Swarm visit – https://help.spot.io/integration-docs/elastigroup/container-management/docker-swarm/

For more information of the Cloudstor plugin visit – https://docs.docker.com/docker-for-aws/persistent-data-volumes/#use-cloudstor

For more information on EBS and EFS visit – https://aws.amazon.com/ebs https://aws.amazon.com/efs