Velero backups in Oracle Cloud Kubernetes Engine Clusters powered by Verrazzano

Verrazzano is a general-purpose container platform that developers and DevOps engineers can use to deploy, secure, and manage container workloads, including microservices and traditional applications. Verrazzano is built from a familiar collection of open source components and supports Kubernetes clusters running on-premises or in the cloud.

Although not the one and only kuebrnetes backup tool in town, backup operations in OKE+Verrazzano can be done with Velero easily.

In this particular case we are gonna be backing up a couple of namespaces:

• a Siebel environment running in OKE. Siebel pods are statefulsets with persisten volumes
• a WebLogic domain migrated to OKE+Verrazzano using the WebLogic Kubernetes Toolkit UI*

Installing AND CONFIGURING velero

Goto OCI Web Console, create a “Customer Secret Key” and grab the key and password generated:

Next grab the name of the storage namespace and your home region, using the oct cli for example:

# get home region
oci iam region-subscription list | jq -r '.data[0]."region-name"'

eu-madrid-1

# get storage namespace name
oci os ns get | jq -r .data

ax...p7 

Next create an object storage bucket in a compartment and grab the name, in this example velero-backups

Next create a file named what you want (in this case credentials-velcro) with the following content:

[default]
aws_access_key_id=<the id of the key created above>
aws_secret_access_key=<the value of the secret>

In your laptop or in cloud shell or from a bastion (from a Mac in this case):

brew install velero

Deploy velero in Kubernetes cluster (assuming you already have the cluster context in your kubeconfig and kubectl installed):

NOTE: Replace values in bold with those obtained up above:

./velero install \
    --provider aws \
    --bucket <bucket-name> \
    --prefix <whatever prefix literal> \
    --use-volume-snapshots=false \
    --secret-file credentials-velero \
    --plugins velero/velero-plugin-for-aws:v1.0.0 \
    --use-node-agent \
    --backup-location-config region=<home-region>,s3ForcePathStyle="true",s3Url=https://<namespace>.compat.objectstorage.<home-region>.oraclecloud.com

Example:

./velero install \
    --provider aws \
    --bucket velero-backups \
    --prefix v \
    --use-volume-snapshots=false \
    --secret-file credentials-velero \
    --plugins velero/velero-plugin-for-aws:v1.0.0 \
    --use-node-agent \
    --backup-location-config region=eu-madrid-1,s3ForcePathStyle="true",s3Url=https://ax...p7.compat.objectstorage.eu-madrid-1.oraclecloud.com

SIEBEL BACKUP

In case you want to see the PV’s associated to pods run something like:

kubectl get po -o json -n desa | jq -j '.items[] | "\(.metadata.namespace), \(.metadata.name), \(.spec.volumes[].persistentVolumeClaim.claimName)\n"' | grep -v null

Execute the following (note the –default-volumes-to-fs-backup flag):

./velero backup create <backup name> --default-volumes-to-fs-backup --include-namespaces <namespaces to backup>

For example:

./velero backup create desa --default-volumes-to-fs-backup --include-namespaces desa

Backup request "desa.202303170024" submitted successfully.
Run `velero backup describe desa.202303170024` or `velero backup logs desa.202303170024` for more details.

Check the status:

./velero backup get

Wait until it hopefully finishes or check details with message retrieved when backup command was issued:

Backup request "desa.202303170024" submitted successfully.
Run `velero backup describe desa.202303170024` or `velero backup logs desa.202303170024` for more details.

TIP: If you have pods that are not running, label those pods to get rid of them in the backup, because if not you’ll get errors, for example:

kubectl label -n desa pod/configure-hw5v9 velero.io/exclude-from-backup=true
kubectl label -n desa pod/postinstalldb-8zbdm velero.io/exclude-from-backup=true

Goto object storage bucket and see what happened:

WEBLOGIC DOMAIN ON VERRAZZANO BACKUP

Same thing, in this case, namespace does not contain TVs:

kubectl get po -o json -n tef-domain-ns | jq -j '.items[] | "\(.metadata.namespace), \(.metadata.name), \(.spec.volumes[].persistentVolumeClaim.claimName)\n"' | grep -v null

<nothing found>

./velero backup create tefdomain.202303170035 --default-volumes-to-fs-backup --include-namespaces tef-domain-ns

./velero describe backup  tefdomain.202303170035
Name:         tefdomain.202303170035
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.24.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=24

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  tef-domain-ns
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:  10m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-03-16 23:35:42 +0000 UTC
Completed:  2023-03-16 23:37:45 +0000 UTC

Expiration:  2023-04-15 23:35:42 +0000 UTC

Total items to be backed up:  68
Items backed up:              68

Velero-Native Snapshots: <none included>

restic Backups (specify --details for more information):
  Completed:  42

That’s all, hope it helps! 🙂

(*) A desktop app that puts together open source tools to migrate WebLogic to Kubernetes clusters (with or without Verrazzano in it) such as:

• WebLogic Deploy Tooling (WDT)
• WebLogic Image Tool (WIT) 
• WebLogic Kubernetes Operator (WKO)
• WebLogic Remote Console