An introduction to cluster provisioning using Crossplane

An introduction to cluster provisioning using Crossplane

An introduction to cluster provisioning using Crossplane

(image source)

Introduction

What if you could create a Kubernetes cluster across major cloud providers like Google Cloud Platform (GCP), Microsoft Azure or Amazon Web Services (AWS)  through a resource like Deployment or a PersistentVolumeClaim (PVC) and manage it like you manage any other Kubernetes resource? That’s what you can do through Crossplane (among many other things).

Okay, what’s with the PersistentVolumeClaim (PVC) analogy? PersistentVolumeClaim (PVC) requests a PersistentVolume (PV), which under the hood provisions a storage volume according to whatever kind of storage you specify in the StorageClass. More about PV and PVCs can be read here and here.

Crossplane and Stacks

Imagine a Kubernetes resource pair like PVC-PV but for provisioning a Kubernetes cluster instead of storage.
That is, if I create a claim (like PVC) which says I want a cluster, it would provision a cluster resource (like PV) in Kubernetes which in turn would create a cluster (actual cluster,  in as in a GKE cluster) in whichever cloud provider I want based on the storage class, except in this case this storage class would hold the information about the cloud provider where I want to provision the cluster (and it’s called a Provider instead of StorageClass). This is roughly what Crossplane does.

Shows architectural overview of Crossplane

Architecture

Crossplane is basically a bunch of core controllers with core CustomResources (claims) which can be extended using Stacks. Stack is another bunch of controllers with CustomResourceDefinitions (CRDs) which plug into the core library to satisfy the claim. Of course I am oversimplifying here. It offers lot more than just a bunch of controllers with CRDs but for the time being, let’s assume that’s all it is. So, you have Crossplane core which is a bunch of controllers containing generic resource claims like (e.g., KubernetesCluster, analogous to PersistentVolumeClaim) while stacks on the other hand implement the specific resources (e.g., GKECluster in GCP stack, analogous to PersistentVolumes) in Crossplane. Let’s take the example of GCP stack to understand this better.

Installation

Let’s try provisioning a cluster in GKE (Google Kubernetes Engine) through Crossplane.

We will be using Minikube to install Crossplane but you can install it in Kind or whichever cluster you want to install it in (as long as you can use kubectl and you have the permissions to install CRDs aka Custom Resource Definitions).

Let’s fire up Minikube first:

We will be using helm 3  to install Crossplane but you should  be fine using helm 2 as well (installation using helm 2) with slight modifications in the commands. Note that we will be installing Crossplane in the default namespace.

Let’s first add the helm repo for Crossplane:

Now we can check the Crossplane versions available for installation:

We will be installing the latest release of Crossplane when writing this post, which is 0.6.

You should see two running pods:

GCP Stack (which gives us the Custom Resources to provision resources in GCP) does not come in-built in our Crossplane installation. Let’s install it first. You can install the Stack either through Crossplane’s own cli tool (kubectl plugin) or if you don’t want to do that, just kubectl apply -f stack.yaml.

Here’s our stack.yaml (yes, you install stack using a custom resource. Crossplane’s cli tool does the same thing under the hood):

Version v0.4.1 of GCP stack, was the latest version at the time of writing this post. You can check all the available versions at the Crossplane’s GCP Stack repo here. If all goes well, you can check if the stack-gcp pods are running using kubectl get po. You should see something similar to this:

You can also check if the GCP custom resources are available:


Custom Resources which are of interest to us here are gkeclusters.container.gcp.crossplane.io (aka GKECluster resources) , gkeclusterclasses.container.gcp.crossplane.io (aka GKEClusterClass resource) and providers.gcp.crossplane.io (aka Provider resource).

Sidenote: <resource>.compute.gcp.crossplane.io is the older version of <resource>.container.gcp.crossplane.io so you will see same resources in both the APIs. We will be using resources from *.container.gcp.crossplane.io wherever available.

Alright, now that Crossplane is up and running, let’s go a little deeper

Provisioning the Cluster

Here’s a TL:DR; image of what we are going to do:

Explains the connection between resources used to provision the cluster and the real text is also present nearby.

Relation between resources used to provision a cluster in Crossplane

A KubernetesCluster uses a GKEClass using cloud provider information from Provider to provision a GKECluster (this is the resource which represents an actual cluster. Deleting this deletes the actual cluster).

Here’s another image to give better context. Here, KubernetesCluster is our Resource Claim, GKEClass is our Resource Class and the resource which is provisioned by KubernetesCluster using GKEClass is GKECluster Resource (more commonly,  Managed Resource) in the above image. External Resource is the actual cluster which sits in our GKE and is accessible through GCP Console.

If you don’t want to copy and paste things, here‘s the repo for all the templates used.

Creating the Provider Resource

Provider is the custom resource which holds information about the GCP Project we are connecting to (including reference to the project credentials aka GCP service account). Crossplane uses Provider to provision our cluster and communicate with it. Please make a note that you need to have a GCP project with owner or editor access.

We will setup theProvider by creating a file named provider.yaml. Now, there are two ways we can go about configuring this. If you have gcloud cli installed (or if you want to install it), you can just use this:

OR

If you don’t have gcloud installed or you don’t want to install it, you can do the same thing as above manually (Crossplane has excellent documentation on setting up Provider which you can find here. Following is a shortened and a hopefully simplified copypasta from that link):

Set PROJECT_ID env variable to your GCP Project ID and BASE64ENCODED_GCP_PROVIDER_CREDS to the contents of your service account json encoded in base64.

For service account, let’s go with Kubernetes Engine Admin role in GCP (here’s the doc for how to grant a role and here’s what Kubernetes Engine Admin can do). 

Aand with that painful thing of setting up service account and all those environment variables done, let’s create our Provider resource (environment variables we set before replace the ${} thingies).

Notice that the Provider does not hold the credentials to our GCP account. Instead, it delegates that to a Secret which in turn is referenced in the Provider.

Once you’re all set, apply the the provider.yaml with

Note that we are using providers.gcp.crossplane.io instead of simply providers because there are multiple resources with the name providers in Crossplane (e.g., Azure Kubernetes Service aka AKS has its own Provider resource which points to providers.azure.crossplane.io ).

Now that we’re all set with the credentials (if you face any problems, feel free to comment below or just skip to the end to see where you can get support if you get stuck).

Creating GKEClusterClass Resource

With that out of the way, let’s get to the heart of what this post is all about. Configuring a cluster! This is taken care of through GKEClusterClass Kubernetes resource. Here’s what I am talking about:

This is where we specify the details for our cluster. Think of this is a template that you can re-use to create many clusters with the same settings. Notice the label here. This is going to be important later.

Let’s kubectl apply -f the above GKEClusterClass template first:

Here’s what it should look like:

If you take a closer look at the yaml template above, you might ask, hey you said I can configure my cluster in GKEClusterClass but where do I specify how many nodes I want machine type, disk size and all the important things? Well, up until recently, specifying what kind of nodepools you want was a part of the yaml template above but it was recently moved into a separate NodePool resource which has to be created separately (what a pain right? well, it’s kinda complicated. You can read more about that here. You might have to login to the Crossplane slack channel to view this. Sorry 🙁 but to summarize, this pain is a part of a grand plan to something good. It’s a secret, well not really).

Here’s what our NodePool resource looks like (except you will have to replace my-project with your GCP project ID):

Let’s save this to a file nodepool.yaml and apply it:

If we kubectl get nodepools, you should see something like this:

Note that the Nodepool has no State right now because our cluster has not been provisioned yet. State field will be updated once our NodePool resource starts provisioning a node pool (GKE nodepool) for us which happens once the cluster has provisioned.

Creating KubernetesCluster Resource

There’s only one more thing left to do now (phew!) and that is our KubernetesCluster resource. KubernetesCluster is the *claim* resource I was talking about at the beginning of this post. This is the analogue of PersistentVolumeClaim in our PVC-PV analogy and the very last piece in the puzzle to create our very own cluster using Crossplane.

Aaand, once this is done, all you have to do is sit and watch . Ahem, I meant:

Also, remember the label we talked about when adding GKEClusterClass, well that is used by our KubernetesCluster to match the class for itself. You might ask, why not just allow adding the class name instead? Well, label makes things more flexibile. You don’t have to depend on the name of the class anymore but a label which can be present on any class (if that’s not very convincing, you can use the classRef instead to specify your class name). Note that Crossplane does not use the name of KubernetesCluster resource to name the provisioned cluster in GCP Console. You have to explicitly specify what you want to name your provisioned cluster. Notice the crossplane.io/external-name: my-cluster here. This is used to specify the name of the actual cluster being provisioned in GCP console. If you don’t specify this, Crossplane will just pick an arbitrary name for the cluster being provisioned.

Let’s check how our KubernetesCluster resource is doing first.

You should see something like this (if you don’t see CLASS-NAME name populated with RESOURCE-NAME name, something is wrong. Go to the end of this post to see how you can debug this).

default-example-cluster-mq75g is the GKECluster resource that Crossplane provisioned for us. These kind of resources which are managed by another resource are called Managed Resources in the Crossplane lingo. If you notice, NodePool represents an actual NodePool resource which seems similar to GKECluster as in, it is the resource which actually represents an external resource in the cloud provider, you might wonder if NodePool might be a Managed Resource and you would be right but the thing is, the resource which would actually manage this NodePool is not there. You read that right. It’s something that the Crossplane community is working on right now.

You can go and check the status of provisioning by doing kubectl get gkecluster default-example-cluster-mq75g -o yaml:

So the cluster is being provisioned but the STATUS is empty. This will be populated once the cluster is fully provisioned.

Provisioning a cluster might take some time, so if you are bored or want to stretch a bit or grab a cup of coffee maybe, now’s the time.

If everything goes well, you should see Bound status in your KubernetesCluster resource.

Troubleshooting

Well, that’s all! But oh wait, what do you do if you run into any problems?

The first place to look at is your GKECluster. Just do kubectl get container.gcp.crossplane.io <resource-name> -o yaml. For example, if we take the above case:

You can find what went wrong in the conditions section.

If you still can’t fix the issue,

Where to go from here?

  • If you are interested in what else Crossplane can do, besides provisioning cluster (trust me, crossplane can do much much more than just provisioning clusters), check the docs.
  • If you’re interested in the general architecture of Crossplane, look here.
  • Also, Crossplane examples directory can be super helpful (it contains examples for most of the resources that you can create through Crossplane).
  • If you want to understand how Crossplane fares with other similar tools, check this.
  • Crossplane hosts a fortnightly podcast TBS on every alternate Thursday (check #tbs on slack for this). You can check the past recordings here.
  •  
Suraj Banakar

Author Suraj Banakar

More posts by Suraj Banakar

Leave a Reply