To manage stateful services in KOTS, check out the KOTS persistent datastores guide.
Kubernetes applications often rely on persistent volumes (PVs) and persistent volume claims (PVCs) to manage data.
When Replicated creates a Kubernetes appliance, a custom storage class named
default is automatically created that will be available to the application for persistent volumes provisioned by Rook.
Replicated will set the
storageClassName on all PersistentVolumeClaims in your application, allowing customers to use an alternative provisioner, such as
standard on GKE.
replicated.com/no-rewrite-storage-class annotation to any PVC or StatefulSet’s volumeClaimTemplate to prevent Replicated from rewriting the storageClassName of that volume. You will need to ensure that an alternative storageClassName is set and a provisioner for that class is running in the customer’s environment.
Replicated 2.36.0 to 2.38.2 had a race condition that could cause Pods to run after failing to mount the shared filesystem. Affected Pods would be writing to the ephemeral container filesystem rather than the shared filesystem.
Rook is not able to provision PVCs with an access mode of
ReadWriteMany, but it does support a shared filesystem that can be mounted as a flexVolume.
This can be enabled in the
kubernetes section of your yaml.
kubernetes: shared_fs: enabled: true mount_paths: - /subdir1
Templating is supported on the
Once enabled, you can mount the shared fileystem in any of your pods, either from the root or at a subpath as shown in this example:
--- #kind: scheduler-kubernetes apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment spec: replicas: 2 selector: matchLabels: app: my-deployment template: metadata: labels: app: my-deployment spec: containers: - name: ubuntu image: index.docker.io/ubuntu:16.04 command: - /bin/sh - -c - 'sleep 3600' volumeMounts: - name: shared mountPath: /var/lib/shared volumes: - name: shared flexVolume: driver: ceph.rook.io/rook fsType: ceph options: fsName: rook-shared-fs clusterNamespace: rook-ceph path: /subdir1 #optional
When mounting a subdirectory as shown in the example above, you must add the path you are mounting to
Paths within the shared filesystem can be included in snapshots by adding them to the backup section of your yaml.
Replicated injects an initContainer into any pods that mount the shared filesystem to verify the mount succeeded.
This will be the first initContainer in the pod.
In the case of a failed mount, the initContainer will force delete the pod with grace-period=0.
In order to delete the pod, Replicated binds a Role,
pod-deleter, to the default service account in the application namespace that allows “delete” on pods in that namespace.
This will cause kubelet to rebuild the pod sandbox and retry the mount.
This will prevent the application from losing data by inadvertently writing to the ephemeral container filesystem instead of the shared filesystem.
For more information on using Persistent Volumes with Kubernetes, see the Kubernetes Documentation.
For information on snapshotting Kubernetes volumes, see the Replicated snapshots documentation for Kubernetes.
For more information on managing the storage needs of Kubernetes in customer environments, see Managing Storage.