Manual restore / Disaster recovery

Before you begin

Manual restore process assumes automated backup workflow was already configured and at least one working backup copy exists.

Following components are supported by restore process:

• MongoDB

• PostgreSQL

• MinIO

• InfluxDB

Manual restore process invokes same Kubernetes cronjobs as for automated process. Manual restore process can be executed on environment with configured automated restore or even without automated restore configured.

Usually reasons for manual restore:

• After environment reset

• After OpenCRVS or dependencies upgrade/rollback when data was corrupted

• Migration to new hardware or cloud

• Disaster recovery

Following restore scenarios are available:

• Manual restore when Automated restore is configured on the cluster and operator needs to manually restore datastores. Scenario applies to staging environment. Restore

• Manual restore / Disaster recovery. Scenario applies to any environment.

Manual restore when Automated restore is configured

Its recommended to configure connection to cluster as described at Add new cluster to your default kubeconfig before running this task.

1. From your laptop SSH (login) to backup server and make sure latest backup exists, if for some reasons backup is not there, please follow Manual restore scenario.

2. Connect to your cluster with kubectl

3. Change namespace to opencrvs-deps-<environment>:

   kubectl config set-context --current --namespace=opencrvs-deps-<environment>

4. Run following command to trigger restore jobs for all components:

   for cj in $(kubectl get cronjob -l job-type=restore -o jsonpath='{.items[*].metadata.name}'); do \
     kubectl delete job $cj-job --ignore-not-found; \
     kubectl create job --from=cronjob/$cj $cj-job; \
   done

   Example output:

   job.batch/influxdb-restore-job created
   job.batch/minio-restore-job created
   job.batch/mongodb-restore-job created
   job.batch/postgres-restore-job created

5. Verify all jobs completed without issues:

   kubectl get job -ljob-type=restore

   Example output: Check job names like *-restore-job without ending number, jobs with number were triggered per schedule:

   NAME                        STATUS     COMPLETIONS   DURATION   AGE
   influxdb-restore-29384640   Complete   1/1           10s        7h16m
   influxdb-restore-job        Complete   1/1           9s         2m3s
   minio-restore-29384640      Complete   1/1           11s        7h16m
   minio-restore-job           Complete   1/1           8s         2m2s
   mongodb-restore-29384640    Complete   1/1           16s        7h16m
   mongodb-restore-job         Complete   1/1           15s        2m1s
   postgres-restore-29384640   Complete   1/1           11s        7h16m
   postgres-restore-job        Complete   1/1           10s        2m

Manual restore / Disaster recovery

Its recommended to configure connection to cluster as described at Add new cluster to your default kubeconfig before running this task.

• From your laptop SSH (login) to backup server

• Find backup that needs to be restored in remote directory /home/backup/<environment>/

• Note backup folder name, it should have following format: YYYY-MM-DD, e/g 2025-10-23 (Year-Month-Day).

• From your laptop connect to your cluster with kubectl

• Change namespace to opencrvs-deps-<environment>:

  kubectl config set-context --current --namespace=opencrvs-deps-<environment>

• Make sure following kubernetes secrets exist:

  • backup-server-ssh-credentials

  • backup-encryption-secret

• Generate values files from helm release:

  helm get values opencrvs-deps > /tmp/opencrvs-deps.$(date +%F).yaml

• Run following command to trigger restore jobs for all components:

  jobs=(influxdb minio mongo postgres)
  for job in ${jobs[@]}; do
    echo "Starting $job service..."
    kubectl delete job ${job}-restore --ignore-not-found;
    helm template \
    --set restore.enabled=true \
    --set restore.cronjob=false \
    --set backup_encryption_secret=backup-encryption-secret \
    --set env.RESTORE_DATE=<YYYY-MM-DD: PUT YOUR DATE> \
    -s templates/${job}-restore-cronjob.yaml \
    -f /tmp/opencrvs-deps.$(date +%F).yaml \
    oci://ghcr.io/opencrvs/opencrvs-dependencies-chart | k apply -f -
  done

  Example output:

  job.batch/influxdb-restore-job created
  job.batch/minio-restore-job created
  job.batch/mongodb-restore-job created
  job.batch/postgres-restore-job created