Can I Use Google Cloud Storage Buckets for Kubernetes Persistent Storage in Hybrid Deployment?

Question

Can I use Google Cloud Storage buckets for Kubernetes persistent storage in Hybrid Deployment?

Environment

Container platform: Kubernetes

Answer

Yes, you can use Google Cloud Storage (GCS) buckets as persistent storage for Kubernetes in Hybrid Deployment. To enable this setup, ensure your Google Kubernetes Engine (GKE) cluster has the Cloud Storage FUSE CSI driver enabled. For more information about enabling the driver, see Google Cloud documentation.

Once the driver is enabled, configure a Persistent Volume (PV) and a Persistent Volume Claim (PVC) that use the GCS FUSE CSI driver. For sample PV and PVC configurations, see Google Cloud documentation.

Example of Persistent Volume using GCS FUSE CSI driver:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: YOUR_PERSISTENT_VOLUME_NAME
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 1Ti
  storageClassName: YOUR_STORAGE_CLASS_NAME   -----> `storageClassName` does not need to refer to an existing StorageClass object
  mountOptions:
    - implicit-dirs
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: YOUR_BUCKET_NAME
  claimRef:
    name: YOUR_PERSISTENT_VOLUME_CLAIM_NAME
    namespace: default

Example of Persistent Volume Claim using GCS FUSE CSI driver:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: YOUR_PERSISTENT_VOLUME_CLAIM_NAME
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Ti
  storageClassName: YOUR_STORAGE_CLASS_NAME

Configure GCS FUSE annotation

When deploying the Hybrid Deployment Agent with GCS FUSE, you must enable the FUSE sidecar container by adding the gke-gcsfuse/volumes: true annotation. You can configure this annotation in one of the following ways:

• Configure using values.yaml file
• Configure inline with Helm command

Configure annotation using values.yaml file

1. Add the following configuration to your Helm values file:

   agent:
     image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production"
     image_pull_policy: "Always"
   config:
     namespace: "default"
     token: YOUR_TOKEN_HERE
     data_volume_pvc: YOUR_PERSISTENT_VOLUME_CLAIM_NAME
     use_empty_dir_for_temp: true
     annotations:
       gke-gcsfuse/volumes: true
   

   You must set use_empty_dir_for_temp to true to enable the agent to use a local temporary directory instead of the mounted GCS volume. Ensure that there is sufficient space available on the Kubernetes node for writes to the /tmp directory, as this directory is used for temporary storage during processing.

2. Deploy the agent.

   helm upgrade --install hd-agent oci://us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/helm/hybrid-deployment-agent -f values.yaml/ --version=<agent_helm_chart_version>
   

Configure annotation inline with Helm command

Set the annotation from the command line.

helm upgrade --install hd-agent oci://us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/helm/hybrid-deployment-agent \
  --set config.annotations."gke-gcsfuse/volumes"=true \
  --set config.token="YOUR_TOKEN_HERE" \
  --set config.namespace="default" \
  --set config.data_volume_pvc="YOUR_PERSISTENT_VOLUME_CLAIM_NAME" \
  --set config.use_empty_dir_for_temp="true" \
  --version=<agent_helm_chart_version>