Deploying your workload on the Kubernetes cluster using Fission involves creating some Fission resources for example Fission environment and functions. If you are not familiar with the basic constructs of Fission I would suggest you go through this article in order to have that basic understanding. When we create a Fission environment we can provide the details of which runtime the functions are going to run on, and how to build the provided source code to create a Fission function. We use --image
and --buider
flags to specify the environment runtime and builder images respectively.
Below command that helps us create a Fission environment to deploy python code,
» fission env create --name python --image fission/python-env:latest --builder fission/python-builder:latest
would eventually create some Fission function runtime pods in the fission-function
namespace and builder pod in the fission-builder
namespace as shown below.
» kubectl get pods -n fission-function NAME READY STATUS RESTARTS AGE poolmgr-python-default-34042-658df7c758-c9mr5 2/2 Running 0 4m55s poolmgr-python-default-34042-658df7c758-lqq2c 2/2 Running 0 4m55s poolmgr-python-default-34042-658df7c758-nfxqd 2/2 Running 0 4m55s » kubectl get pods -n fission-builder NAME READY STATUS RESTARTS AGE python-34042-54c7ccdc9-vqwnd 2/2 Running 0 5m10s
The pods in the fission-function
namespaces are the pods that are actually going to serve the requests to your function and the pod in the fission-builder
namespace is going to help us build the source code with the dependencies if there is a need.
Now, the pods that get created after running that env create
command, in fission-function
and fission-builder
are being run with some default settings or spec we could say. For example the pods in the fission-function
namespace have this toleration by default,
tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300
this simply means that the pods have toleration if the node is tainted with node.kubernetes.io/not-ready
or node.kubernetes.io/unreachable
.
This is where we can leverage Fission podspec
to customize the spec
of the pods that get created in the fission-function
and fission-builder
namespaces.
Let’s take an example of a Kubernetes cluster that is set up using kind with a single node. And we don’t want to schedule pods on this node, we can taint this node and after that, the pods will only be scheduled on this node if and only if they have toleration for specified taint on the node. Let’s take a look at the node that we have in the cluster and then we can taint the node to show, how the pods will not be scheduled on the node.
» kubectl get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready master 8h v1.15.3
And if we describe the node we don’t see any taint set to the node and that’s why when created the Fission environment all the pods were scheduled on this node correctly. Let’s taint the node and create the environment once again to check if the pods would be scheduled on this node or not. Run below command to taint the node
» kubectl taint node kind-control-plane com.ic.priority=high:NoSchedule node/kind-control-plane tainted
Now if we go ahead and describe the node once again we would be able to see that the node has been tainted with the provided key and value pairs, like below
Name: kind-control-plane Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=kind-control-plane kubernetes.io/os=linux node-role.kubernetes.io/master= ... Taints: com.ic.priority=high:NoSchedule Unschedulable: false ... ...
This means that Kubernetes scheduler will not be able to schedule the pods on this node if they don’t have toleration for the specific taints that have been set. Now let’s delete the Fission environment that we created earlier and create the environment again to check if the pods would get scheduled on the node.
» fission env list NAME IMAGE BUILDER_IMAGE POOLSIZE MINCPU MAXCPU MINMEMORY MAXMEMORY EXTNET GRACETIME python fission/python-env:latest fission/python-builder:latest 3 0 0 0 0 false 0 » fission env delete --name python environment 'python' deleted » fission env create --name python --image fission/python-env:latest --builder fission/python-builder:latest environment 'python' created
Now to check if the pods have been scheduled on the node successfully, let’s list all the pod from fission-function
and fission-builder
namespaces.
» kubectl get pods -n fission-function NAME READY STATUS RESTARTS AGE poolmgr-python-default-37512-76b7fbd65f-jpfm5 0/2 Pending 0 2m5s poolmgr-python-default-37512-76b7fbd65f-snbr4 0/2 Pending 0 2m5s poolmgr-python-default-37512-76b7fbd65f-w6r2s 0/2 Pending 0 2m5s » kubectl get pods -n fission-builder NAME READY STATUS RESTARTS AGE python-37512-bfbb8f9f6-stzkp 0/2 Pending 0 2m15s
And as you can see the pods have not been scheduled on the node because we have tainted the node and describing the pod would clearly tell us that,
» kubectl describe pods -n fission-function poolmgr-python-default-37512-76b7fbd65f-w6r2s Name: poolmgr-python-default-37512-76b7fbd65f-w6r2s Namespace: fission-function ... ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 79s (x4 over 4m5s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Using Fission to have custom tolerations in the builder or runtime pods
In that case, we can use Fission podspec
to specify the spec for the pods that would get created when we create the Fission environment. And in podspec
we can specify the toleration for the taint that the node has. To specify the podspec
we will have to create the Fission environment using manifest instead of the command line.
Below is an example of Fission environment manifest that can be used to create a Fission environment that would spin up the pods with specified podspec
.
apiVersion: fission.io/v1 kind: Environment metadata: name: python namespace: default spec: builder: command: build image: fission/python-builder:latest podspec: tolerations: - key: "com.ic.priority" value: "high" operator: "Equal" effect: "NoSchedule" imagepullsecret: "" keeparchive: false poolsize: 3 resources: {} runtime: image: fission/python-env:latest podspec: tolerations: - key: "com.ic.priority" value: "high" operator: "Equal" effect: "NoSchedule" version: 2
You can create the above spec using –spec flag with the fission env create
command, please take a look at this documentation for details on Fission spec.
If we have a closer look into the above manifest, what we are trying to say is the pods that are going to be created for environment runtime and builder should have specified podspec
. And if we just apply the above manifest to create the environment and then describe pods we should be able to see that the pods are created with these specified tolerations.
» kubectl create -f env.yaml environment.fission.io/python created » kubectl describe pods -n fission-function poolmgr-python-default-41586-585dbdbb49-r5htw Name: poolmgr-python-default-41586-585dbdbb49-r5htw Namespace: fission-function Priority: 0 Node: kind-control-plane/172.17.0.2 ... ... QoS Class: Burstable Node-Selectors: <none> Tolerations: com.ic.priority=high:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s ... ... » kubectl describe pod -n fission-builder python-41586-6d795c5744-9jdrg Name: python-41586-6d795c5744-9jdrg Namespace: fission-builder Priority: 0 ... ... QoS Class: Burstable Node-Selectors: <none> Tolerations: com.ic.priority=high:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s ... ...
As we can see in the above snippet now the pods have been created with modified PodSpec and not the default ones, that includes the new toleration for the taint that the node has. And if we list the pods now, we should see that all the pods have successfully been scheduled on the node.
» kubectl get pods -n fission-function NAME READY STATUS RESTARTS AGE poolmgr-python-default-41586-585dbdbb49-nzk4l 2/2 Running 0 6m11s poolmgr-python-default-41586-585dbdbb49-p9djs 2/2 Running 0 6m11s poolmgr-python-default-41586-585dbdbb49-r5htw 2/2 Running 0 6m11s » kubectl get pods -n fission-builder NAME READY STATUS RESTARTS AGE python-41586-6d795c5744-9jdrg 2/2 Running 0 6m40s
This was how we can leverage Fission podspec
to actually specify the Kubernetes’ pod’s PodSpec that will be applied to the Fission function and builder pods that would be created. Like we specified the toleration
of the pods that would be created, we can specify any PodSpec that we want to, using Fission environment manifest.
Using Fission to have volume attached to the builder or runtime pods’ containers
The other example that we are going to see in this post is how we can leverage Fission podspec
to mount a volume into the runtime container of the pods that get created after creating the Fission environment. If we take a look into the pods that get created as result of creating an environment
apiVersion: v1 kind: Pod metadata: ... name: poolmgr-python-default-45111-55c9f78877-h6wm9 namespace: fission-function ... spec: containers: - image: fission/python-env imagePullPolicy: IfNotPresent ... name: python ... volumeMounts: - mountPath: /userfunc name: userfunc - mountPath: /secrets name: secrets - mountPath: /configs name: configmaps - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: fission-fetcher-token-2d6kf readOnly: true ... volumes: - emptyDir: {} name: userfunc - emptyDir: {} name: secrets - emptyDir: {} name: configmaps - name: fission-fetcher-token-2d6kf secret: defaultMode: 420 secretName: fission-fetcher-token-2d6kf ...
You can see that the main container (there would be another one named fetcher
) of the pod has some volumes mounted in it, now our motive is to mount another volume into this container at the path, let’s say /var
. To mount a volume, we will have to create a volume and we are going to use hostpath
volume type in this example. To mount the volume into a container we will have to specify the container name and we will be getting that name from above pod manifest that we showed because that is the name of the container that Fission creates.
Now let’s look into the below manifest of a python environment
apiVersion: fission.io/v1 kind: Environment metadata: name: python namespace: default spec: builder: command: build image: fission/python-builder imagepullsecret: "" keeparchive: false poolsize: 3 resources: {} runtime: image: fission/python-env podspec: containers: - name: python volumeMounts: - name: vol mountPath: /var volumes: - name: vol hostPath: path: /var version: 2
As you can see, in podspec
its pretty easy to specify the volume and we are doing that using hostpath
type. The question that arrives is where/which container we are going to mount this volume in. Now since our requirement is to mount the volume in the runtime environment’s pod’s python
container, we can easily figure out, from the pods manifest above, the container name is going to be python
and that’s why we have mentioned name of container to be python
where we want to mount the volume and then specified the name of the volume that should be mounted and the path where that volume should be mounted.
Let’s go ahead and create a Fission environment using above manifest
» kubectl create -f python-env.yaml environment.fission.io/python created
and now if we go ahead and take a look at the container of any of the runtime pods that have been created for this environment, we would be able to see we were successfully able to mount another volume in the container.
apiVersion: v1 kind: Pod metadata: ... name: poolmgr-python-default-46843-5cc5fbdb6c-lf6wq namespace: fission-function ... spec: containers: - image: fission/python-env imagePullPolicy: IfNotPresent ... name: python ... volumeMounts: - mountPath: /userfunc name: userfunc - mountPath: /secrets name: secrets - mountPath: /configs name: configmaps - mountPath: /var name: vol - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: fission-fetcher-token-2d6kf readOnly: true ... volumes: - emptyDir: {} name: userfunc - emptyDir: {} name: secrets - emptyDir: {} name: configmaps - hostPath: path: /var type: "" name: vol - name: fission-fetcher-token-2d6kf secret: defaultMode: 420 secretName: fission-fetcher-token-2d6kf ...
And as you can see we were able to mount a new volume in the container python at the path /var
. To verify that the things are working correctly let’s exec into this pod and create a file at the /var location and we should be able to see that file in the hostpath
.
» kubeclt exec -it -n fission-function poolmgr-python-default-46843-5cc5fbdb6c-lf6wq bash Defaulting container name to python. Use 'kubectl describe pod/poolmgr-python-default-46843-5cc5fbdb6c-lf6wq -n fission-function' to see all of the containers in this pod. bash-5.0# cd /var/ bash-5.0# mkdir volume-test bash-5.0# cd volume-test/ bash-5.0# echo 'This file contains some data, that will be persisted in hostpath' > datafile bash-5.0# ls -l total 4 -rw-r--r-- 1 root root 65 Jul 3 19:52 datafile bash-5.0# pwd /var/volume-test bash-5.0#
So, we have created a file inside a dir /var/volume-test
, since hostpasth
‘s /var
is mounted on container’s /var
the new dir volume-test
and its content should be available on host at the pat /var/volume-test/
. Since we are running the Kubernetes cluster using kind, that eventually runs the things inside docker container, let’s figure out which container is running this cluster.
» docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eb16e7bc094e kindest/node:v1.15.3 "/usr/local/bin/entr…" 11 hours ago Up 11 hours 45923/tcp, 127.0.0.1:45923->6443/tcp kind-control-plane
Now, exec into this docker container to check if we have the /var/volume-test/datafile
that contains the text that we have written.
» docker exec -it eb16e7bc094e bash root@kind-control-plane:/# cd /var/volume-test/ root@kind-control-plane:/var/volume-test# ls -l total 4 -rw-r--r-- 1 root root 65 Jul 3 19:52 datafile root@kind-control-plane:/var/volume-test# cat datafile This file contains some data, that will be persisted in hostpath root@kind-control-plane:/var/volume-test#
and here we go as we can see we are able to see the content in the host path.
Caveat:
As of now we can not specify the ServiceAccount of the pods using Fission podspec
in environment manifest, we are having discussion if we want to support that and if we agree on that, the support would be merged soon.