# Disk space management

This page gives brief overview for actions in case of running out of disk space. Default OpenCRVS server disk layout has large root (`/`) disk partition with encrypted files mounted as `/data` folder.

## Disk space issues

### Root filesystem (`/`)

**Reasons:**

* System upgrades accumulated packages or kernels.
* Logs or temporary files filled the disk.
* Docker / Kubernetes artifacts accumulated.
* Unexpected files written to the root filesystem.

**Impact**\
When available disk space drops below **10 GB**, Kubernetes will begin **stopping containers** to protect the node.

This may cause:

* OpenCRVS services to stop
* Kubernetes pods to be evicted
* system upgrades to fail
* logging to stop working

Diagnose the issue

#### Diagnose the issue

{% hint style="danger" %}
If disk encryption is enabled, the `/data` directory is backed by: `/cryptfs_file_sparse.img` **Do not modify or delete `/cryptfs_file_sparse.img`.**
{% endhint %}

Check disk usage:

```
df -h
```

Find large directories:

```
sudo du -xh / --max-depth=1 | sort -h
```

Common locations to check:

```
/home
/var/log
/var/cache
/tmp
/var/lib/containerd
```

#### Free disk space

**Clean package cache**

```
sudo apt clean
sudo apt autoremove -y
```

**Remove old kernel versions**

List installed kernels and identify old kernes:

```
dpkg --list | grep linux-image
```

Example output:

```
ii  linux-image-6.8.0-101-generic   6.8.0-101.101                           amd64        Signed kernel image generic
ii  linux-image-6.8.0-90-generic    6.8.0-90.91                             amd64        Signed kernel image generic
ii  linux-image-virtual             6.8.0-101.101                           amd64        Virtual Linux kernel image
```

Remove unused kernels (keep the currently running one):

```
apt purge <package name>
```

Example command:

```
apt remove linux-image-6.8.0-90-generic
```

**Clean system logs**

Check journal size:

```
journalctl --disk-usage
```

Reduce logs:

```
sudo journalctl --vacuum-time=7d
```

**Clean temporary files**

```
sudo rm -rf /tmp/*
```

### `/data` partition

This partition stores OpenCRVS application (citizens) data.

#### Possible reasons

1. Elasticsearch monitoring and logging indices filled the disk.
2. Citizen data growth.
3. Backup files accumulation.
4. Attachments stored in MinIO.

#### Impact

When `/data` is full:

* PostgreSQL will produce **write errors**
* OpenCRVS may stop operating
* Elasticsearch will stop indexing
* Monitoring and logging will break

#### Diagnose the issue

Check partition usage:

```
df -h /data
```

Find largest directories:

```
sudo du -xh /data --max-depth=1 | sort -h
```

Common directories:

* /data/elasticsearch, see [#elasticsearch-disk-cleanup](#elasticsearch-disk-cleanup "mention")
* /data/postgres, see [#postgresql-storage](#postgresql-storage "mention")
* /data/minio, see [#minio-storage-cleanup](#minio-storage-cleanup "mention")

#### Elasticsearch disk cleanup

Check index sizes:

```
curl -X GET localhost:9200/_cat/indices?v&s=store.size
```

Delete old indices using:

* **Kibana Dev Tools**, or
* Elasticsearch API

Example:

```
curl -X DELETE "localhost:9200/index-name"
```

If Elasticsearch APIs are not available:

1. Stop Elasticsearch
2. Remove oldest index folders

```
/data/elasticsearch/nodes/0/indices
```

As a last resort:

```
rm -rf /data/elasticsearch/*
```

{% hint style="warning" %}
This removes **all logs and monitoring data**.
{% endhint %}

#### MinIO storage cleanup

Check size:

```
du -sh /data/minio
```

Possible actions:

* Remove old attachments
* Enable **MinIO lifecycle rules** to automatically delete old objects.

***

### PostgreSQL storage

Check database size:

```
du -sh /data/postgres
```

If PostgreSQL data is the main consumer:

Possible solutions:

* Increase `/data` partition size, see [#increase-data-partition](#increase-data-partition "mention")
* Move database storage
* Disable encryption and expand the partition if needed

## Move datastore storage

Default OpenCRVS setup uses `host_path` storage type and stores all data in `/data` folder on Kubernetes master node.

OpenCRVS dependencies helm chart has an options to change data location for particular datastore outside of `/data` folder or on another node:

* `node_selector` : Label(s) to properly select node for data store. All Kubernetes nodes have label `role` with value `dataX` , `X` is taken from `infrastructure/server-setup/inventory/<env>.yaml`. Value is auto-generated by `environment:init` script.
* `host_data_path`: Path to data folder on the Kubernetes node.

For storage type `pvc` data location is managed by Kubernetes CSI (storage class) and underlaying persistent storage implementation

An example for MinIO bellow shows how to modify data location for `storage_type: host_path`, but same configuration options also apply to Elasticsearch and Postgres. In this example MinIO data is stored on fist worker node (`data2`) instead of default master node (`data1`):

```yml
minio:
  storage_type: host_path # Store data on filesystem (default)
  node_selector:
    role: data2 # Store data on worker node instead of master, default is 'data1'
  host_data_path: /data/minio  # default value
```

To change data location on live environment operator (SysAdmin, DevOps) needs to modify helm chart values and copy existing data manually to new location.

## Increase /data partition

`/data` is backed by a **file-based encrypted container (`/cryptfs_file_sparse.img`)**, the procedure is slightly different from resizing a normal partition. Most safest way is to backup all data from `/data` partition and then create new larger \``/cryptfs_file_sparse.img` file.

Digram shows encrypted partition stack:

```
VM disk
 └ root filesystem
     └ /cryptfs_file_sparse.img
         └ LUKS encrypted device
             └ ext4 filesystem
                 └ /data
```

{% hint style="danger" %}
Use steps with caution

Before resizing an encrypted container:

```
cryptsetup luksHeaderBackup /cryptfs_file_sparse.img \
  --header-backup-file luks-header-backup.img
```

This protects against **catastrophic header corruption**.
{% endhint %}

To increase `/data`, the typical sequence is:

1. Increase VM disk
2. Increase root filesystem
3. Increase sparse container file:

   ```
   fallocate -l <new size>
   ```
4. Resize encrypted device:

   ```
   cryptsetup resize <mapper_name>
   ```
5. Resize filesystem:

   ```
   resize2fs /dev/mapper/<mapper_name>
   ```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://documentation.opencrvs.org/v2.0/technical/guides/installation/advanced-topics/disk-space-management.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.