Deploying your workload on the Kubernetes cluster using Fission involves creating some Fission resources for example Fission environment and functions. If you are not familiar with the basic constructs of Fission I would suggest you go through this article in order to have that basic understanding. When we create a Fission environment we can provide the details of which runtime the functions are going to run on, and how to build the provided source code to create a Fission function. We use --image and --buider flags to specify the environment runtime and builder images respectively.
Below command that helps us create a Fission environment to deploy python code,
» fission env create --name python --image fission/python-env:latest --builder fission/python-builder:latest
would eventually create some Fission function runtime pods in the fission-function namespace and builder pod in the fission-builder namespace as shown below.
» kubectl get pods -n fission-function NAME READY STATUS RESTARTS AGE poolmgr-python-default-34042-658df7c758-c9mr5 2/2 Running 0 4m55s poolmgr-python-default-34042-658df7c758-lqq2c 2/2 Running 0 4m55s poolmgr-python-default-34042-658df7c758-nfxqd 2/2 Running 0 4m55s » kubectl get pods -n fission-builder NAME READY STATUS RESTARTS AGE python-34042-54c7ccdc9-vqwnd 2/2 Running 0 5m10s
The pods in the fission-function namespaces are the pods that are actually going to serve the requests to your function and the pod in the fission-builder namespace is going to help us build the source code with the dependencies if there is a need.
Now, the pods that get created after running that env create command, in fission-function and fission-builder are being run with some default settings or spec we could say. For example the pods in the fission-function namespace have this toleration by default,
tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300
this simply means that the pods have toleration if the node is tainted with node.kubernetes.io/not-readyor node.kubernetes.io/unreachable.
This is where we can leverage Fission podspec to customize the spec of the pods that get created in the fission-function and fission-builder namespaces.
Let’s take an example of a Kubernetes cluster that is set up using kind with a single node. And we don’t want to schedule pods on this node, we can taint this node and after that, the pods will only be scheduled on this node if and only if they have toleration for specified taint on the node. Let’s take a look at the node that we have in the cluster and then we can taint the node to show, how the pods will not be scheduled on the node.
» kubectl get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready master 8h v1.15.3
And if we describe the node we don’t see any taint set to the node and that’s why when created the Fission environment all the pods were scheduled on this node correctly. Let’s taint the node and create the environment once again to check if the pods would be scheduled on this node or not. Run below command to taint the node
» kubectl taint node kind-control-plane com.ic.priority=high:NoSchedule node/kind-control-plane tainted
Now if we go ahead and describe the node once again we would be able to see that the node has been tainted with the provided key and value pairs, like below
Name: kind-control-plane
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=kind-control-plane
kubernetes.io/os=linux
node-role.kubernetes.io/master=
...
Taints: com.ic.priority=high:NoSchedule
Unschedulable: false
...
...This means that Kubernetes scheduler will not be able to schedule the pods on this node if they don’t have toleration for the specific taints that have been set. Now let’s delete the Fission environment that we created earlier and create the environment again to check if the pods would get scheduled on the node.
» fission env list NAME IMAGE BUILDER_IMAGE POOLSIZE MINCPU MAXCPU MINMEMORY MAXMEMORY EXTNET GRACETIME python fission/python-env:latest fission/python-builder:latest 3 0 0 0 0 false 0 » fission env delete --name python environment 'python' deleted » fission env create --name python --image fission/python-env:latest --builder fission/python-builder:latest environment 'python' created
Now to check if the pods have been scheduled on the node successfully, let’s list all the pod from fission-function and fission-builder namespaces.
» kubectl get pods -n fission-function NAME READY STATUS RESTARTS AGE poolmgr-python-default-37512-76b7fbd65f-jpfm5 0/2 Pending 0 2m5s poolmgr-python-default-37512-76b7fbd65f-snbr4 0/2 Pending 0 2m5s poolmgr-python-default-37512-76b7fbd65f-w6r2s 0/2 Pending 0 2m5s » kubectl get pods -n fission-builder NAME READY STATUS RESTARTS AGE python-37512-bfbb8f9f6-stzkp 0/2 Pending 0 2m15s
And as you can see the pods have not been scheduled on the node because we have tainted the node and describing the pod would clearly tell us that,
» kubectl describe pods -n fission-function poolmgr-python-default-37512-76b7fbd65f-w6r2s Name: poolmgr-python-default-37512-76b7fbd65f-w6r2s Namespace: fission-function ... ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 79s (x4 over 4m5s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Using Fission to have custom tolerations in the builder or runtime pods
In that case, we can use Fission podspec to specify the spec for the pods that would get created when we create the Fission environment. And in podspec we can specify the toleration for the taint that the node has. To specify the podspec we will have to create the Fission environment using manifest instead of the command line.
Below is an example of Fission environment manifest that can be used to create a Fission environment that would spin up the pods with specified podspec.
apiVersion: fission.io/v1
kind: Environment
metadata:
name: python
namespace: default
spec:
builder:
command: build
image: fission/python-builder:latest
podspec:
tolerations:
- key: "com.ic.priority"
value: "high"
operator: "Equal"
effect: "NoSchedule"
imagepullsecret: ""
keeparchive: false
poolsize: 3
resources: {}
runtime:
image: fission/python-env:latest
podspec:
tolerations:
- key: "com.ic.priority"
value: "high"
operator: "Equal"
effect: "NoSchedule"
version: 2You can create the above spec using –spec flag with the fission env create command, please take a look at this documentation for details on Fission spec.
If we have a closer look into the above manifest, what we are trying to say is the pods that are going to be created for environment runtime and builder should have specified podspec. And if we just apply the above manifest to create the environment and then describe pods we should be able to see that the pods are created with these specified tolerations.
» kubectl create -f env.yaml
environment.fission.io/python created
» kubectl describe pods -n fission-function poolmgr-python-default-41586-585dbdbb49-r5htw
Name: poolmgr-python-default-41586-585dbdbb49-r5htw
Namespace: fission-function
Priority: 0
Node: kind-control-plane/172.17.0.2
...
...
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: com.ic.priority=high:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
...
...
» kubectl describe pod -n fission-builder python-41586-6d795c5744-9jdrg
Name: python-41586-6d795c5744-9jdrg
Namespace: fission-builder
Priority: 0
...
...
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: com.ic.priority=high:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
...
...As we can see in the above snippet now the pods have been created with modified PodSpec and not the default ones, that includes the new toleration for the taint that the node has. And if we list the pods now, we should see that all the pods have successfully been scheduled on the node.
» kubectl get pods -n fission-function NAME READY STATUS RESTARTS AGE poolmgr-python-default-41586-585dbdbb49-nzk4l 2/2 Running 0 6m11s poolmgr-python-default-41586-585dbdbb49-p9djs 2/2 Running 0 6m11s poolmgr-python-default-41586-585dbdbb49-r5htw 2/2 Running 0 6m11s » kubectl get pods -n fission-builder NAME READY STATUS RESTARTS AGE python-41586-6d795c5744-9jdrg 2/2 Running 0 6m40s
This was how we can leverage Fission podspec to actually specify the Kubernetes’ pod’s PodSpec that will be applied to the Fission function and builder pods that would be created. Like we specified the toleration of the pods that would be created, we can specify any PodSpec that we want to, using Fission environment manifest.
Using Fission to have volume attached to the builder or runtime pods’ containers
The other example that we are going to see in this post is how we can leverage Fission podspec to mount a volume into the runtime container of the pods that get created after creating the Fission environment. If we take a look into the pods that get created as result of creating an environment
apiVersion: v1
kind: Pod
metadata:
...
name: poolmgr-python-default-45111-55c9f78877-h6wm9
namespace: fission-function
...
spec:
containers:
- image: fission/python-env
imagePullPolicy: IfNotPresent
...
name: python
...
volumeMounts:
- mountPath: /userfunc
name: userfunc
- mountPath: /secrets
name: secrets
- mountPath: /configs
name: configmaps
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: fission-fetcher-token-2d6kf
readOnly: true
...
volumes:
- emptyDir: {}
name: userfunc
- emptyDir: {}
name: secrets
- emptyDir: {}
name: configmaps
- name: fission-fetcher-token-2d6kf
secret:
defaultMode: 420
secretName: fission-fetcher-token-2d6kf
...You can see that the main container (there would be another one named fetcher) of the pod has some volumes mounted in it, now our motive is to mount another volume into this container at the path, let’s say /var. To mount a volume, we will have to create a volume and we are going to use hostpath volume type in this example. To mount the volume into a container we will have to specify the container name and we will be getting that name from above pod manifest that we showed because that is the name of the container that Fission creates.
Now let’s look into the below manifest of a python environment
apiVersion: fission.io/v1
kind: Environment
metadata:
name: python
namespace: default
spec:
builder:
command: build
image: fission/python-builder
imagepullsecret: ""
keeparchive: false
poolsize: 3
resources: {}
runtime:
image: fission/python-env
podspec:
containers:
- name: python
volumeMounts:
- name: vol
mountPath: /var
volumes:
- name: vol
hostPath:
path: /var
version: 2
As you can see, in podspec its pretty easy to specify the volume and we are doing that using hostpath type. The question that arrives is where/which container we are going to mount this volume in. Now since our requirement is to mount the volume in the runtime environment’s pod’s python container, we can easily figure out, from the pods manifest above, the container name is going to be python and that’s why we have mentioned name of container to be python where we want to mount the volume and then specified the name of the volume that should be mounted and the path where that volume should be mounted.
Let’s go ahead and create a Fission environment using above manifest
» kubectl create -f python-env.yaml environment.fission.io/python created
and now if we go ahead and take a look at the container of any of the runtime pods that have been created for this environment, we would be able to see we were successfully able to mount another volume in the container.
apiVersion: v1
kind: Pod
metadata:
...
name: poolmgr-python-default-46843-5cc5fbdb6c-lf6wq
namespace: fission-function
...
spec:
containers:
- image: fission/python-env
imagePullPolicy: IfNotPresent
...
name: python
...
volumeMounts:
- mountPath: /userfunc
name: userfunc
- mountPath: /secrets
name: secrets
- mountPath: /configs
name: configmaps
- mountPath: /var
name: vol
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: fission-fetcher-token-2d6kf
readOnly: true
...
volumes:
- emptyDir: {}
name: userfunc
- emptyDir: {}
name: secrets
- emptyDir: {}
name: configmaps
- hostPath:
path: /var
type: ""
name: vol
- name: fission-fetcher-token-2d6kf
secret:
defaultMode: 420
secretName: fission-fetcher-token-2d6kf
...And as you can see we were able to mount a new volume in the container python at the path /var. To verify that the things are working correctly let’s exec into this pod and create a file at the /var location and we should be able to see that file in the hostpath.
» kubeclt exec -it -n fission-function poolmgr-python-default-46843-5cc5fbdb6c-lf6wq bash Defaulting container name to python. Use 'kubectl describe pod/poolmgr-python-default-46843-5cc5fbdb6c-lf6wq -n fission-function' to see all of the containers in this pod. bash-5.0# cd /var/ bash-5.0# mkdir volume-test bash-5.0# cd volume-test/ bash-5.0# echo 'This file contains some data, that will be persisted in hostpath' > datafile bash-5.0# ls -l total 4 -rw-r--r-- 1 root root 65 Jul 3 19:52 datafile bash-5.0# pwd /var/volume-test bash-5.0#
So, we have created a file inside a dir /var/volume-test, since hostpasth‘s /var is mounted on container’s /var the new dir volume-test and its content should be available on host at the pat /var/volume-test/. Since we are running the Kubernetes cluster using kind, that eventually runs the things inside docker container, let’s figure out which container is running this cluster.
» docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eb16e7bc094e kindest/node:v1.15.3 "/usr/local/bin/entr…" 11 hours ago Up 11 hours 45923/tcp, 127.0.0.1:45923->6443/tcp kind-control-plane
Now, exec into this docker container to check if we have the /var/volume-test/datafile that contains the text that we have written.
» docker exec -it eb16e7bc094e bash root@kind-control-plane:/# cd /var/volume-test/ root@kind-control-plane:/var/volume-test# ls -l total 4 -rw-r--r-- 1 root root 65 Jul 3 19:52 datafile root@kind-control-plane:/var/volume-test# cat datafile This file contains some data, that will be persisted in hostpath root@kind-control-plane:/var/volume-test#
and here we go as we can see we are able to see the content in the host path.
Caveat:
As of now we can not specify the ServiceAccount of the pods using Fission podspec in environment manifest, we are having discussion if we want to support that and if we agree on that, the support would be merged soon.
