Disk space management

This page gives brief overview for actions in case of running out of disk space. Default OpenCRVS server disk layout has large root (/) disk partition with encrypted files mounted as /data folder.

Disk space issues

Root filesystem (/)

Reasons:

• System upgrades accumulated packages or kernels.

• Logs or temporary files filled the disk.

• Docker / Kubernetes artifacts accumulated.

• Unexpected files written to the root filesystem.

Impact When available disk space drops below 10 GB, Kubernetes will begin stopping containers to protect the node.

This may cause:

• OpenCRVS services to stop

• Kubernetes pods to be evicted

• system upgrades to fail

• logging to stop working

Diagnose the issue

Diagnose the issue

If disk encryption is enabled, the /data directory is backed by: /cryptfs_file_sparse.img Do not modify or delete /cryptfs_file_sparse.img.

Check disk usage:

df -h

Find large directories:

sudo du -xh / --max-depth=1 | sort -h

Common locations to check:

/home
/var/log
/var/cache
/tmp
/var/lib/containerd

Free disk space

Clean package cache

sudo apt clean
sudo apt autoremove -y

Remove old kernel versions

List installed kernels and identify old kernes:

dpkg --list | grep linux-image

Example output:

ii  linux-image-6.8.0-101-generic   6.8.0-101.101                           amd64        Signed kernel image generic
ii  linux-image-6.8.0-90-generic    6.8.0-90.91                             amd64        Signed kernel image generic
ii  linux-image-virtual             6.8.0-101.101                           amd64        Virtual Linux kernel image

Remove unused kernels (keep the currently running one):

apt purge <package name>

Example command:

apt remove linux-image-6.8.0-90-generic

Clean system logs

Check journal size:

journalctl --disk-usage

Reduce logs:

sudo journalctl --vacuum-time=7d

Clean temporary files

sudo rm -rf /tmp/*

/data partition

This partition stores OpenCRVS application (citizens) data.

Possible reasons

1. Elasticsearch monitoring and logging indices filled the disk.

2. Citizen data growth.

3. Backup files accumulation.

4. Attachments stored in MinIO.

Impact

When /data is full:

• PostgreSQL will produce write errors

• OpenCRVS may stop operating

• Elasticsearch will stop indexing

• Monitoring and logging will break

Diagnose the issue

Check partition usage:

df -h /data

Find largest directories:

sudo du -xh /data --max-depth=1 | sort -h

Common directories:

• /data/elasticsearch, see Elasticsearch disk cleanup

• /data/postgres, see PostgreSQL storage

• /data/minio, see MinIO storage cleanup

Elasticsearch disk cleanup

Check index sizes:

curl -X GET localhost:9200/_cat/indices?v&s=store.size

Delete old indices using:

• Kibana Dev Tools, or

• Elasticsearch API

Example:

curl -X DELETE "localhost:9200/index-name"

If Elasticsearch APIs are not available:

1. Stop Elasticsearch

2. Remove oldest index folders

/data/elasticsearch/nodes/0/indices

As a last resort:

rm -rf /data/elasticsearch/*

This removes all logs and monitoring data.

MinIO storage cleanup

Check size:

du -sh /data/minio

Possible actions:

• Remove old attachments

• Enable MinIO lifecycle rules to automatically delete old objects.

---

PostgreSQL storage

Check database size:

du -sh /data/postgres

If PostgreSQL data is the main consumer:

Possible solutions:

• Increase /data partition size, see Increase /data partition

• Move database storage

• Disable encryption and expand the partition if needed

Move datastore storage

Default OpenCRVS setup uses host_path storage type and stores all data in /data folder on Kubernetes master node.

OpenCRVS dependencies helm chart has an options to change data location for particular datastore outside of /data folder or on another node:

• node_selector : Label(s) to properly select node for data store. All Kubernetes nodes have label role with value dataX , X is taken from infrastructure/server-setup/inventory/<env>.yaml. Value is auto-generated by environment:init script.

• host_data_path: Path to data folder on the Kubernetes node.

For storage type pvc data location is managed by Kubernetes CSI (storage class) and underlaying persistent storage implementation

An example for MinIO bellow shows how to modify data location for storage_type: host_path, but same configuration options also apply to Elasticsearch and Postgres. In this example MinIO data is stored on fist worker node (data2) instead of default master node (data1):

minio:
  storage_type: host_path # Store data on filesystem (default)
  node_selector:
    role: data2 # Store data on worker node instead of master, default is 'data1'
  host_data_path: /data/minio  # default value

To change data location on live environment operator (SysAdmin, DevOps) needs to modify helm chart values and copy existing data manually to new location.

Increase /data partition

/data is backed by a file-based encrypted container (/cryptfs_file_sparse.img), the procedure is slightly different from resizing a normal partition. Most safest way is to backup all data from /data partition and then create new larger `/cryptfs_file_sparse.img file.

Digram shows encrypted partition stack:

VM disk
 └ root filesystem
     └ /cryptfs_file_sparse.img
         └ LUKS encrypted device
             └ ext4 filesystem
                 └ /data

Use steps with caution

Before resizing an encrypted container:

cryptsetup luksHeaderBackup /cryptfs_file_sparse.img \
  --header-backup-file luks-header-backup.img

This protects against catastrophic header corruption.

To increase /data, the typical sequence is:

1. Increase VM disk

2. Increase root filesystem

3. Increase sparse container file:

   fallocate -l <new size>

4. Resize encrypted device:

   cryptsetup resize <mapper_name>

5. Resize filesystem:

   resize2fs /dev/mapper/<mapper_name>