It’s been about a year when we wrote about Crossplane. The previous post explains how Crossplane helps you provision and manage infrastructure using the Kubernetes API. Crossplane has since evolved and reached v1.0! Big congratulations to the Crossplane community! Besides being an interface to cloud providers, Crossplane now has mechanisms to create and publish your own custom planes, without writing code. It offers ways to define your own infrastructure resources declaratively, in form of CRDs on top of cloud providers APIs. In this post, we’ll see how you can use Crossplane to create your own control plane on top of cloud providers. Let’s begin!
Let’s take a look at how Crossplane allows us to provision cloud resources.
How to install Crossplane? The below commands will install Crossplane in a Kubernetes cluster using Helm 3
$ kubectl create namespace crossplane-system $ helm repo add crossplane-stable https://charts.crossplane.io/stable $ helm repo update $ helm install crossplane --namespace crossplane-system crossplane-stable/crossplane
Crossplane CLI is kubectl plugin to manage Crossplane packages. To install, run:
curl -sL https://raw.githubusercontent.com/crossplane/crossplane/release-1.0/install.sh | sh
Working with a cloud provider A “provider” in Crossplane is a bunch of CRDs and their controllers, creating an interface to a cloud provider API (in fact, any API). We’ll use AWS for demonstration here.
Install the provider
kubectl crossplane install provider crossplane/provider-aws:v0.16.0
Create a provider secret from the AWS credentials by running the following. This uses the default profile from AWS CLI credentials.
$ AWS_PROFILE=default && echo -e "[default]\naws_access_key_id = $(aws configure get aws_access_key_id --profile $AWS_PROFILE)\naws_secret_access_key = $(aws configure get aws_secret_access_key --profile $AWS_PROFILE)" > creds.conf $ kubectl create secret generic aws-creds -n crossplane-system --from-file=key=./creds.conf
And a ProviderConfig to configure credentials for the AWS provider:
$ cat > providerconfig.yaml <<EOF apiVersion: aws.crossplane.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: aws-creds key: key EOF $ kubectl apply -f providerconfig.yaml
Below is a Crossplane representation of a AWS RDS instance, called a “managed resource”. A Crossplane managed resource is a representation of a cloud provider resource and in fact, any API.
$ cat > rds.yaml <<EOF apiVersion: database.aws.crossplane.io/v1beta1 kind: RDSInstance metadata: name: rdspostgresql spec: forProvider: region: us-east-1 dbInstanceClass: db.t2.small masterUsername: masteruser allocatedStorage: 20 engine: postgres engineVersion: "9.6" skipFinalSnapshotBeforeDeletion: true writeConnectionSecretToRef: namespace: crossplane-system name: aws-rdspostgresql-conn EOF $ kubectl apply -f rds.yaml
This will create an RDS instance and a Kubernetes Secret with the credentials for it.
While managed resources are great for managing cloud resources, it can soon become overwhelming to manage lots of them each time. For instance, an application developer might not be interested in the details of how a RDS instance is being created and managed with firewall rules, database configuration, etc. The only thing of interest might be a database and credentials to access it. On the contrary, an infrastructure operator may not want to expose all of the RDS configurations to a developer, but instead restrict only to some necessary parameters.
Crossplane offers mechanisms to compose managed resources where users can create their own abstractions in a declarative way.
Composite resources: A Composite Resource (XR) is a custom resource that is composed of managed resources allowing you to abstract the infrastructure details. A CompositeResourceDefinition (XRD) defines a new kind of composite resource. XRDs are cluster scoped. To create a namespaced XR, the respective XRD may optionally offer a Composite Resource Claim (XRC).
Composition: A Composition specifies which resources a XR will be composed of i.e what happens when you create XR. There can be multiple compositions for one XR. For example, for a
CompositeDatabase XR, you may have a composition which will create a AWS RDS instance, a security group and a MySQL database. Another composition can define GCP CloudSQL instance and a PostgreSQL database.
Configuration: A configuration is a package of XRDs and Compositions that can be then published using the Crossplane CLI to an OCI image registry and installed into a Crossplane cluster by creating a declarative Configuration resource
The below figure shows how the Crossplane resources might look like in a Kubernetes cluster. The XRDs are cluster scoped with compositions satisfying them. These XRDs offer a XRC each, which are namespace scoped. Offering a XRC is optional and is done to allow users to create XRs in their own namespace. The XRDs also are referring to a managed resource in the cluster scope, so that the XRC can use it.
Let’s see a quick example to understand these concepts better. We’ll create a
CompositePostgreSQLInstance XRD for PostgreSQL database , a composition for satisfying this XRD, and see how we can use the XRC that gets created.
You can leverage the Crossplane CLI to install a configuration package:
kubectl crossplane install configuration registry.upbound.io/xp/getting-started-with-aws:v1.0.0
Otherwise, you can define the XRD and composition yourself as explained below. (The examples are taken from official Crossplane docs)
The XRD is defined below. Note how a claim is offered under
claimNames. The only configurable field in the spec is
// xrd.yaml apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: compositepostgresqlinstances.database.example.org spec: group: database.example.org names: kind: CompositePostgreSQLInstance plural: compositepostgresqlinstances claimNames: kind: PostgreSQLInstance plural: postgresqlinstances connectionSecretKeys: - username - password - endpoint - port versions: - name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: parameters: type: object properties: storageGB: type: integer required: - storageGB required: - parameters
The composition defined below defines creation of a AWS RDS instance. Notice how it references the XRD using the
compositeTypeRef field. The
resources define a set of
base resources that get created when this composition is used. And under that, the
patches field is used to fetch the property values from the XRD.
// composition.yaml apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: compositepostgresqlinstances.aws.database.example.org labels: provider: aws guide: quickstart vpc: default spec: writeConnectionSecretsToNamespace: crossplane-system compositeTypeRef: apiVersion: database.example.org/v1alpha1 kind: CompositePostgreSQLInstance resources: - base: apiVersion: database.aws.crossplane.io/v1beta1 kind: RDSInstance spec: forProvider: region: us-east-1 dbInstanceClass: db.t2.small masterUsername: masteruser engine: postgres engineVersion: "9.6" skipFinalSnapshotBeforeDeletion: true publiclyAccessible: true writeConnectionSecretToRef: namespace: crossplane-system patches: - fromFieldPath: "metadata.uid" toFieldPath: "spec.writeConnectionSecretToRef.name" transforms: - type: string string: fmt: "%s-postgresql" - fromFieldPath: "spec.parameters.storageGB" toFieldPath: "spec.forProvider.allocatedStorage" connectionDetails: - fromConnectionSecretKey: username - fromConnectionSecretKey: password - fromConnectionSecretKey: endpoint - fromConnectionSecretKey: port
Install these two manifests
$ kubectl apply -f xrd.yaml && kubectl apply -f composition.yaml
You can now see a new XRC created,
postgresqlinstances.database.example.org. Use this XRC to create a AWS RDS instance.
// xr.yaml apiVersion: database.example.org/v1alpha1 kind: PostgreSQLInstance metadata: name: my-db namespace: default spec: parameters: storageGB: 20 compositionSelector: matchLabels: provider: aws vpc: default writeConnectionSecretToRef: name: db-conn
$ kubectl apply -f xr.yaml
This will result in creation of RDS PostgreSQL instance. Similarly, you can have another Composition for GCP CloudSQL and create a XR by modifying the
compositionSelector field in the above manifest.
As we saw, Crossplane helps to build custom resources on top of cloud resources and in fact on top of any API. This is great to build a control plane with required abstractions and restrictions using XRD, composition and XRC.
The below depiction shows an example of how an infrastructure operator might allow other people to create their database instances which are attached to VPCs which they don’t have control of.
The image shows a VPC at cluster scope. Also at the cluster level are two database XRDs offering a claim each. Other users can then create their own database XRs using the namespace scoped XRC and still refer to the VPC. There can be multiple XRDs with different configurations on the cluster scope, allowing users to choose from. Also, different compositions for different cloud providers to satisfy those XRDs. The defined resource definitions can then be packaged and distributed as a configuration. You can read more how the Crossplane CLI helps you do this.
Crossplane helps to provision infrastructure using the Kubernetes API. Moreover, it helps to build custom abstractions and restrictions over any API. This makes Crossplane a great tool to build a single control plane to manage multiple cloud providers.
Update: I talked about Provisioning Infrastructure with Crossplane in Kubernetes Community Days Bengaluru 2021, here is the recorded video