Automated restore configuration
Automated restore configuration
Usually restore job is configured on staging environment to restore data from production backups stored on backup server. Backup server is provisioned as part of production infrastructure.
Complete following steps to configure automated restore on staging
Copy secrets from production to staging
Before setting up the restore jobs, make sure you copy both required secrets — backup-server-ssh-credentials and restore-encryption-secret — from the production cluster to the staging cluster.
The restore job needs SSH credentials to access the backup server and download backup files. These credentials are stored in the production environment in the opencrvs-deps-<environment> namespace under the secret name backup-server-ssh-credentials.
It also requires the backup encryption key, which is stored in the same namespace in the backup-encryption-secret secret.
The steps for copying these secrets from the production cluster, adjusting them as needed, and creating them in the staging cluster are described below.
Connect to production cluster with
kubectl:kubectl config use-context <production environment context>Change namespace to opencrvs-deps-<environment>, where
<environment>is production environment name:kubectl config set-context --current --namespace=opencrvs-deps-<environment>Get secrets as yaml manifest:
kubectl get secret backup-encryption-secret \ backup-server-ssh-credentials -oyaml > /tmp/production-backup-secrets.yamlUpdate file
/tmp/production-backup-secrets.yaml:Delete following fields:
creationTimestamp
resourceVersion
uid
annotations
namespace
Update
name: backup-encryption-secrettoname: restore-encryption-secret
If you are working remotely, copy
/tmp/production-backup-secrets.yamlto staging serverConnect to staging cluster with
kubectl:kubectl config use-context <staging environment context>Change namespace to opencrvs-deps-<environment>, where
<environment>is staging environment name:kubectl config set-context --current --namespace=opencrvs-deps-<environment>Create secrets on staging cluster:
kubectl apply -f /tmp/production-backup-secrets.yamlExample output:
secret/restore-encryption-secret created secret/backup-server-ssh-credentials created
Example how your temporal file will look before your apply changes:
Enable restore in Dependencies helm chart values
Add following section to environments/<environment>/dependencies/values.override.yaml and run "Deploy dependencies" workflow:
Update
backup_server_dirvalue to match with your environment name, e/g/home/backup/productionUpdate
scheduleto reflect best time backup job to be started,Update
hostto the backup server IP address or hostnameSet
enabledtotrue
Push your changes to github and Re-Deploy Dependencies
Enable reindex in OpenCRVS helm chart values
Add following section to environments/<environment>/opencrvs-services/values.override.yaml and run "Deploy OpenCRVS" workflow:
enabled: Enable reindexschedule: Cronjob schedule for reindex, see https://crontab.guru/
Push your changes to github and Re-Deploy OpenCRVS
Verify restore configuration
Usually to make sure restore was successful you need to verify data from Production is present on Staging environment. Additional verification steps are described here.
Verify kubernetes jobs are present:
Connect to your cluster with
kubectlRun following command:
Example output:
Verify Kubernetes jobs were executed per schedule:
Wait at least for first job execution, usually takes at to 24 hours
Connect to your cluster with
kubectlRun following command:
Example output:
Disable/Enable restore
In same cases during upgrade or testing restore should be disabled.
Steps to disable:
Connect to your cluster with
kubectlChange namespace:
Run following command to verify cronjob status:
Example output:
Disable all restore cronjobs:
Verify status (SUSPEND should be set to
true):(Optionally) Disable particular cronjob:
Last updated