Shared Disk for your Pods: PersistentVolumes for Oracle Kubernetes Engine (OKE) Implemented as NFS File Storage in Oracle Cloud Infrastructure (OCI)

When you deploy in k8s a pod depending in persistent volume attached to block storage (for example this post), the volume created is mounted on a specific node. If that node fails or is stopped, the pods running on it fail when trying to be created in other node according to the replication policies they have, because other nodes do not have the disk mounted.

Oracle Cloud Infrastructure (OCI) File Systems are shared storage that you can easily expose/attach to your pods for those use cases where shared persistent data is needed.

Of course, you can still mount the disk in each node, but this is not a good approach because we have a better way to achieve it.

So, let’s get started

Go to OCI dashboard, create a new File System and a new Export called /myexport. Click on the mount target link and take note of the File System IP address.

This slideshow requires JavaScript.

Download and deploy the following yaml, it creates:

• a persistentVolume
• a persistentVolumeClaim
• a deployment with 3 replicas of a container image with nginx in it
• a service with public IP and LoadBalancer with round-robin policy

kubectl apply -f k8spod2nfs.yaml

Get a list of the pods and “ssh” to one of them

kubectl get pods

NAME READY STATUS RESTARTS AGE
k8spod2nfs-xxx 1/1 Running 0 99s
k8spod2nfs-yyy 1/1 Running 0 99s
k8spod2nfs-zzz 1/1 Running 0 99s

kubectl exec -it k8spod2nfs-xxx bash

cd /usr/share/nginx/html/

echo hola > index.html

You can also “ssh” to other pod and verify that you see the same file

Now get the list of services and grab the public IP of the k8spod2nfs service

kubectl get services

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8spod2nfs LoadBalancer 10.96.128.9 x.y.z.t 80:31014/TCP 10m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d4h

Go to http://yourserviceip in your browser:

Change the content of the index.html file:

echo adios >> index.html

Go to http://yourserviceip in your browser again, the data changes

Delete all pods of the k8spod2nfs deployment and wait until at lest one of them are recreated again:

kubectl delete pod -l app=k8spod2nfs

Go to http://yourserviceip in your browser again, the data is still in there!

Delete the deployment and create it again:

kubectl delete -f k8spod2nfs.yaml

kubectl apply -f k8spod2nfs.yaml

Wait until al least one of the ñods are ready and go to http://yourserviceip in your browser again, the data is still in there!

As you can see, the data is shared across all pods and is persistent! (unless you delete it or destroy the OCI FIleSystem)

Hope it helps! 🙂