One of key areas when running containers using Kubernetes is managing the stateful applications such as a database. We use Gluster along with Kubernetes in this post and demonstrate how you can run stateful applications on Kubernetes
GlusterFS is an open-source scalable network FileSystem which can be created using off the shelf hardware. Gluster allows creation of various types of volumes such as Distributed, Replicated, Striped, Dispersed and many combinations of these which are described in detail here.
Heketi is a Gluster Volume manager that provides RESTful interface to create/manage Gluster volumes. Heketi makes it easy for cloud services such as Kubernetes, OpenShift, OpenStack Manila to interact with Gluster clusters and provision volumes as well as manage brick layout.
The purpose of this post is to demonstrate how to create a GlusterFS cluster, manage this cluster using Heketi to provision volumes and then install a demo application to use the Gluster volume.
We shall create a 4 node Kubernetes cluster with two unformatted disks on each node. We shall then install glusterFS as DaemonSet and heketi as a service to create gluster volumes which will be consumed by a Postgres database using StatefulSet. We shall have another application which adds 1 entry per second to the Postgres DB and a flask application that will allow us to view the contents of the Database. We shall also move the postgresdb from one node to another and still be able to access our data.
git clone https://github.com/infracloudio/gluster-heketi-k8s.git .
This will create a 4 node GKE cluster and each node will have 2 unformatted disks attached to it. This script will also generate a topology file which is going to be used as a kubernetes configmap. The topology contains details of kubernetes nodes in the cluster and their mounted disks.
Note: Following these steps will create resources which are chargeable by GCP. Ensure to follow cleanup steps mentioned below to delete all resources once done with the demo.
Run following command to import the configmap in kubernetes
kubectl create -f heketi-turnkey-config.yaml
Run following command to start a gluster daemon set and install heketi service. This creates a temporary pod which is based on container created by Janos Lenart
kubectl create -f heketi-turnkey.yaml
kubectl logs heketi-turnkey -f
Now run “kubectl create -f storage_class.yaml” so kubernetes can talk to heketi in order to create volumes as needed.
Next, run “kubectl create -f postgres-srv-ss.yaml” this will create a postgres pod in a StatefulSet. This Stateful set is backed by a gluster volume. The configuration to achieve this is shown below: . This StatefulSet is exposed by a service name called “postgres”.
Run “kubectl get pvc” to check if the volume got created and bound to the pod as expected. The result should look like below:
Run “kubectl create -f pgwriter-deployment.yaml”. This will create a deployment for a pod which will write 1 entry per second to the postgres db created above.
Run “frontend.yaml”. This will create a deployment of a flask-based frontend which will allow querying the postgres DB. A external LoadBalancer is also created to front the service.
Enter the LoadBalancer IP in browser to access the front end. Enter HH:MM of previous minute (system time) and click Submit. A list of entries with timestamps and counter values is shown.
Run “kubectl describe pod postgres-ss-0” and note the Node on which the pod is running.
Now we will delete the statefulset (effectively the DB) and then create it again. Run following commands:
Run “kubectl describe pod postgres-ss-0” and observe that the pod is assigned to a new node.
kubectl delete -f frontend.yaml
Heketi is a simple and convenient way to create and manage gluster volumes. It pairs very well with Kubernetes storage classes to give an on-demand volume creation capability which can be used with other gluster features such as replication, striping etc to handle many complex storage use cases.
Looking for help with Kubernetes adoption or Day 2 operations? learn more about our capabilities and why startups & enterprises consider as one of the best Kubernetes consulting services companies.