3.3.8 Automated & manual backup and manual restore
Critical information required to understand how to regularly backup and restore your citizen registration data in case of a server problem.
Last updated
Critical information required to understand how to regularly backup and restore your citizen registration data in case of a server problem.
Last updated
If you configured a separate backup server and used the external backup server optional SSH properties in the Ansible script in step , then every day OpenCRVS automatically backs up all databases to the following directories on the manager node.
Every 7 days the Mongo data is deleted to save disk space. The Ansible playbook sets this up as an option in this . The files are backed up to that server in the middle of the night.
TO AVOID CITIZEN DATA LOSS... You must configure and test this automated backup process in production. Operationallly, we highly recommend that once a week, these files should be saved to a password protected and encrypted external harddrive and stored in a secure and approved location.
Mongo backups are saved as mongo .gz zip files using the date of the backup here:
Elasticsearch snapshot files and indices are saved here.
The entire elasticsearch folder contains all snapshots and must be preserved indefinitely
InfluxDB backup files are saved into a date named directory here:
Commencing with OpenCRVS v1.2, Minio attachments are saved as a .gz zip file using the date of the backup here:
You can manually run the automated backup script at any point.
SSH into your server and navigate to the following directory:
3. The backup script is run like this, with the parameters explained below. Even if you have not set up an external server that the manager node can SSH into, the files will still backup to the directory locations above. Just the SSH part will fail. So once the script finishes you can use rsync to copy the files manually off your server and into a local directory.
Replace and separate the <parameters> with a space when calling the script
SSH_USER
Mandatory
This is the SSH user for your backup server. If running manually you can enter anything, such as: root
SSH_HOST
Mandatory
This is the IP address for your backup server. If running manually you can enter anything, such as: 132.42.41.15
SSH_PORT
Mandatory
This is the SSH port for your backup server. If running manually you can enter anything, such as: 22
PRODUCTION_IP
Mandatory
This is the public IP address for the manager server node for your OpenCRVS cluster. It must be entered correctly.
REMOTE_DIR
Mandatory
This is the directory path on the backup server that you wish to copy the backup files into. E.G. /root/opencrvs-backups. If running manually you can enter anything.
REPLICAS
Mandatory
The number of servers in your OpenCRVS cluster. This value can only be equal to: 1, 3 or 5
LABEL
Optional
Normally, today's date is used as a suffix to name the backup files in the directories as explained above. If you want to name the backup files something different, you can enter a lowercase string here that will be used as a replacement file name suffix.
If using a custom LABEL above, it must be lowercase, otherwise Elasticsearch cannot make a snapshot.
4. When complete, exit your server:
5. This command copies files from a server to your current local directory using rsync:
Before you attempt a data restore, please ensure that you have read and understand these 2 warnings ...
Warning 1: Ensure you are restoring to a new, CLEAN, OpenCRVS installation that has NOT BEEN SEEDED.
You should only restore to a clean installation of OpenCRVS: TO AVOID CITIZEN DATA LOSS and avoid potential issues that may be very difficult to debug. THESE COMMANDS ENTIRELY REPLACE ALL DATA AND BACKUPS ON YOUR OPENCRVS SERVER! THEY CANNOT BE REVERTED!
Warning 2: Prepare operationally for a restore. Test a restore on a test server first. Schedule the production restore when staff can cease operations.
If you are restoring a backup from a previous version of OpenCRVS, we have some large data migrations to run. These migrations may take several hours to complete. In this case, we recommend performing a data restore on an entirely new set of servers. When the restore and subsequent migrations are complete, you can then change your DNS settings to point to your new server with confidence. CEASE CIVIL REGISTRATION ACTIVITIES DURING A RESTORE. CONSIDER PERFORMING A RESTORE DURING NATIONAL HOLIDAYS TO AVOID RISK OF DATA LOSS.
To perform a restore, ensure that you have backup files in the day's folders you wish to restore from. If this is a new environment, you would need to copy the backed up files and folders into the locations using scp or rsync.
Copy an Elasticsearch backup onto the server like this:
Copy a Mongo backup onto the server like this:
Copy an InfluxDB backup onto the server like this:
Copy a Minio backup onto the server like this:
Copy a VS Export backup onto the server like this:
As of OpenCRVS v1.3.0, copy a VS Export backup onto the server like this:
2. SSH into your server and navigate to the following directory:
Restart Elasticsearch to pick up the new snapshots availables to it
4. The script is run like this, with the parameters explained below. Replace and separate the <parameters> with a space when calling the script
LABEL
Mandatory
Normally, a date is used as a suffix to name the backup files. E.G. 01-01-2023 but if you named the suffix in a custom way using the LABEL parameter when creating the backup, enter the lowercase file suffix for your backups here.
REPLICAS
Mandatory
The number of servers in your OpenCRVS cluster. This value can only be equal to: 1, 3 or 5
The script will prompt you to answer "Yes" or "No" to continue as this is a destructive and irreversible action.
You next need to take down and re-deploy OpenCRVS. This is because, every time an OpenCRVS deployment is made, database secrets that microservices use are rotated. But as you have just restored and migrated from a backup, the database secrets are now out of date. You must take OpenCRVS down. SSH into the server and run:
6. Exit the server:
8. Once data has been restored to your server, migrations will optionally run on your data depending on how old the backup is. You can check the progress status of your migrations in Kibana.
We have an example production server, in which we have created 50,000 test birth and death registrations. Migrating the data on that server between a backup made using v1.1.* and v1.2.* took 5 hours to complete. DO NOT ATTEMPT TO USE OPENCRVS WHILE MIGRATIONS ARE RUNNING
In the left navigation, select "Observability", then "Logs"
In the search bar, enter this text: "tag: migration"
Click the "Stream live" button for live output from the migration microservice. You will see output like this, potentially for many hours!!! NOTE: After a few minutes the "Stream live" feature stops working to save resources. To get latest results, try refreshing the page in this scenario.
When the migrations are completed, you will see the word "Done".
9. You can login to OpenCRVS and test that all your data has restored successfully.
If for whatever reason, migrations do not run due to an error, you will see the word "PENDING" next to each migration. Please contact us on this occurance as there may be a bug requiring a hotfix. Once a hotfix has been applied, you can always re-run migrations at any time by running this command on the server:
2. Ensure that your database are available to the script as environment variables. You can do this by running:
7. Now, re-deploy following the with the --clear_data property set to "no" of course, so as not to clear the databases.
To login to Kibana, visit https://kibana.<your_domain> and login using the username: "elastic" and the ELASTICSEARCH_SUPERUSER_PASSWORD you configured in .
