3.3.2 Provision environment

In OpenCRVS v1.3.1 hotfix scheduled for November 6th 2023, we will be releaseing a Github Action to automate this process. :-)

Ansible is required to be installed on your local development machine in order to provision the server with required software. Installation instructions are here. Ensure that you have ssh access using the root user to all the servers that you are trying to configure.

Ansible is an IT automation tool that you install locally and run from your local computer. It uses SSH to connect to your servers automatically and installs all the supporting software and configures Ubuntu for you to be able to deploy OpenCRVS. The Ansible script we provide will install all the dependencies onto your server nodes, configure a secure firewall, open required ports and provision the optional automated backup of OpenCRVS for use in production. You may need to customise our Ansible playbooks depending on your hosting provider and the access you have.

You will need a docker container registry account on Dockerhub to build and push your country configuration image, in order to use our Ansible script. This is because the server needs access to your containrer registry account to pull images from it. Dockerhub is a free containerisation repository. You can customise our Ansible script if you wish to use a different registry.

In OpenCRVS v1.3, all server setup configuration and deployment files exist in the country configuration repository - e.g. opencrvs-countryconfig. This allows you to completely customise them for your needs without needing to fork opencrvs-core

1. Create an account on Dockerhub as Ansible and Github Actions will require your Dockerhub username and password in order to login.\

2. Referring to the country configuration repository e.g. opencrvs-countryconfig, duplicate the example-X.ini inventory_file of choice where X is relative to the number of servers. These can be found in the infrastructure/server-setup directory, depending upon whether or not you are deploying to 1, 3 or 5 servers. For example: If you are only deploying to 1 server, you need to make a copy of the inventory_file: example-1.ini file to run with the Ansible playbook_file: playbook-1.yml explained below. If you are deploying to a standard production deployment of 3 servers, you need to make a copy of the inventory_file: example-3.ini file to run with the Ansible playbook_file: playbook-3.yml explained below. If you are deploying to 5 servers, you need to make a copy of the inventory_file: example-5.ini file to run with the Ansible playbook_file: playbook-5.yml explained below.

Deploying to only 1 server is not recommended and is CONSIDERABLY RISKY for a production installation. YOU COULD LOSE CITIZEN DATA IF THE SERVER CRASHES. Docker Swarm's power is that it can load balance between servers in a cluster. If one of the servers goes down, the others will still be operational.

You will be required to uncomment some lines to enter the IP addresses and hostnames, e.g.:\

;manager1 ansible_host="ENTER YOUR MANAGER HOST IP"

becomes:

manager1 ansible_host="159.223.11.243"

... and:

;data1_hostname=ENTER_HOSTNAME_1

becomes:

data1_hostname=<your server hostname>

4. Using a strong password generator, such as 1Password you should create and safely store the following parameters.:

We suggest that your passwords are unguessable RFC 4122 alphanumeric version 1 or 4 UUIDs . If you use special characters in the passwords, they will need to be escaped.

The passwords and secret keys that you make in this section are critically important to be saved and stored securely. They are the master keys to access your citizen data! Consider ISO270001 Password Management Policies!! Tools such as 1Password can help you here. You can customise our playbook to store the keys in a hardware security module.

mongodb_admin_username

mongodb_admin_password

disk_encryption_key (The LUKS Disk encryption secret, encrypting your citizen data at rest.)

elasticsearch_superuser_password

You must store these keys safely for future use, as you will need the details in the deploy process.

5. You are now ready to call the Ansible command passing these required parameters and additionally some optional parameters.

Required parameters:

dockerhub_username

dockerhub_password

mongodb_admin_username

mongodb_admin_password

elasticsearch_superuser_password

disk_encryption_key

encrypted_disk_size

Depending on the size of your server, encrypted_disk_size should be equal to about 80% of your available disk. E.G. for a 320GB disk, encrypt 256GB to store OpenCRVS data. You need to leave enough unencrypted space for your operating system and software. The encrypted_disk_size prop for 256GB is: 256g

PRODUCTION NOTE: In production, we advise that you provision a Hardware Security Module and amend the country configuration playbook.yml, decrypt.sh, emergency-backup-metadata.sh and emergency-restore-metadata.sh scripts at the linked locations in order to change the approach to storing and accessing the disk_encryption_key and MongoDB and Elasticsearch passwords. Secure secret storage is currently outside the scope of OpenCRVS.

MOSIP's documentation on the requirements of a Hardware Security Module is useful reading.

Optional parameters:

For the optional automated daily external data backup to another server, these parameters must be prepared:

external_backup_server_ip

external_backup_server_user

external_backup_server_ssh_port

external_backup_server_remote_directory

You must ensure that you are in your local computer and that both your local directories opencrvs-core and opencrvs-<your country> are on the same release version v* or master branch before running Ansible. Ansible is run from your local machine and it connects to your server using SSH and automatically runs commands on it.

Ansible playbooks are run like this from your local machine:

1. If you are on the root directory of the country configuration repository - e.g. opencrvs-countryconfig, navigate to the server-setup folder:

cd infrastructure/server-setup

Now you can run the playbook like this, substituting the parameters as required:

ansible-playbook -i <inventory_file> <playbook_file> -e " \
dockerhub_username=<your dockerhub username> \
dockerhub_password=<your dockerhub password> \
mongodb_admin_username=<mongo username> \
mongodb_admin_password=<mongo password you generated> \
elasticsearch_superuser_password=<elastic password you generated> \
disk_encryption_key=<a strong disk encryption password> \
encrypted_disk_size=<your available disk size for encryption, e.g. 256g>"

Or with all the possible optional props:

ansible-playbook -i <inventory_file> <playbook_file> -e " \
dockerhub_username=<your dockerhub username> \
dockerhub_password=<your dockerhub password> \
mongodb_admin_username=<mongo username> \
mongodb_admin_password=<mongo password you generated> \
elasticsearch_superuser_password=<elastic password you generated> \
disk_encryption_key=<a strong disk encryption password> \
encrypted_disk_size=<your available disk size for encryption, e.g. 256g> \
external_backup_server_ip=<your_external_backup_server_ip> \
external_backup_server_user=<your_external_backup_server_user> \
external_backup_server_ssh_port=<your_external_backup_server_ssh_port> \
manager_production_server_ip=<your_manager_production_server_ip> \
external_backup_server_remote_directory=<your_external_backup_server_remote_directory>"

Once this command is finished the servers are prepared for an OpenCRVS deployment. You can read more about how the external backups work in the Automated & manual backup... section