Resize disk

Last updated: Aug 09, 2022

Requirements

You need to run a Kubernetes cluster with the Volume Expansion feature enabled. This feature is supported on the majority of volume types since Kubernetes version >= 1.11 (see docs).

Details

PersistentVolumes can be configured to be expandable. This feature when set to true, allows the users to resize the volume by editing the corresponding PersistentVolumeClaims object.

This can become useful in case your storage usage grows and you want to resize the disk on-the-fly without having to resync data across PVCs.

To verify if your storage class allows volume expansion you can run:

Terminal

kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'
true

In case it returns false, you can enable volume expansion capabilities for your storage class by running:

Terminal

DEFAULT_STORAGE_CLASS=$(kubectl get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}')
kubectl patch storageclass "$DEFAULT_STORAGE_CLASS" -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched

N.B:
expanding a persistent volume is a time consuming operation
some platforms have a per-volume quota of one modification every 6 hours
not all the volume types support this feature. Please take a look at the official docs for more info

How-to

Connect to the Clickhouse container to verify the data directory filesystem size (in this example 20G)

Terminal

kubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bash
clickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c   20G   50M   19G   1% /var/lib/clickhouse

In your Helm chart configuration, update the clickhouse.persistence.size value in value.yaml to the target size (40G in this example)
Run a helm upgrade

Connect to the ClickHouse container to verify the new filesystem size

Terminal

kubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bash
clickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c   40G  186M   38G   1% /var/lib/clickhouse

Troubleshooting

If the resize didn't work check for errors in the ClickHouse operator pod.

This procedure doesn't work to decrease a volume. If you try, the disk won't be resized and the following errors can be seen in the ClickHouse operator pod

posthog/posthog:ERROR unable to reconcile PVC(posthog/data-volumeclaim-template-chi-posthog-posthog-0-0-0) err: PersistentVolumeClaim "data-volumeclaim-template-chi-posthog-posthog-0-0-0" is in valid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

For increases double check what the storage class is that's used for ClickHouse and make sure it's expandable as mentioned in requirements above.

Terminal

kubectl get pvc data-volumeclaim-template-chi-posthog-posthog-0-0-0 -o json | jq '.spec.storageClassName'

If a storage class with that name doesn't exist (kubectl get storageclass), then it's probably because major upgrade steps from 9.x.x weren't completed. Set installCustomStorageClass: true & run a helm upgrade first (without the storage increase request).

Questions?

Was this page useful?

Restore

A backup is worthless if the restoration process hasn’t been tested. If you backed something up and never tried to restore it, chances are that restore will not work properly when you actually need it or it will take longer than business can tolerate. Perform regular test restores to ensure your data will be there when you need it! Automated using clickhouse-backup The tool provides an automatic restore operation that you can invoke by running: clickhouse-backup restore <BACKUP NAME> . For…