3.3.6 Deploy (Automated & Manual)

Steps required to build your country configuration to a Docker Container Registry and Deploy your OpenCRVS to a server hosted environment.

Automated continuous deployment

The easiest way to deploy OpenCRVS to your servers is to use automated continuous deployment.

Refer to our supplied "Publish image to Dockerhub" Github Action and our supplied "Deploy" Github Action in the country configuration repo to set up appropriate environments for your use case or customise your own delivery pipeline. You could rewrite our Actions for use in Gitlab, Jenkins, Travis, CircleCI or Azure Pipelines for example.

In our example, we have 3 environments provisioned in Github. These environments allow you to provision different subdomains, secrets and optional deployment properties depending on your chosen environment when running the action.

All of our workflows can be manually run from a shell script, so you can deploy from your local machine too if you like. Manual steps are explained below each automated step.

To use our automated Github Action workflow, first you need to ensure that you have set up at least one, or optionally all, of the following Git environments:

a) Staging - A useful environment for testing, where the environment variable NODE_ENV is set to development and you can create test user accounts with a 6 zero "000000" 2FA code during login. This allows us to see a debug experience.

b) QA - A quality assurance/pseudo production environment for software testers, where the environment variable NODE_ENV is set to production and a secondary exception variable QA_ENV is set to true. This allows us to see a production like experience, but with the capability of still creating test user accounts with a 6 zero "000000" 2FA code during login.

c) Production - A live environment, where NODE_ENV is set to production & QA_ENV is set to false and random 2FA is enabled so a comms Gateway must be active as in step 3.3.3.

Our supplied Github Actions are examples that cannot be edited from a fork. You should duplicate these Github Actions files if you want to make changes for your infrastructure and update the branches that dispatch them (here for example) from master, develop to master-<your country alpha3 code>, develop-<your country alpha3 code> as we described in the fork section 3.2.1.

Configure docker-compose files

There are a few environment variables you need to manually set in docker compose files for your installation. These are:

docker-compose.countryconfig.staging-deploy.yml

docker-compose.countryconfig.qa-deploy.yml

docker-compose.countryconfig.prod-deploy.yml

a) Replace any instance of the comma separated available language strings for your localisation needs. E.G.:

LANGUAGES=en,fr

b) Replace any instance of the COUNTRY code. E.G.:

COUNTRY=FAR

Publishing your country configuration

Before you can deploy, you need to make sure that your country configuration Docker image has compiled and has been pushed to your container registry (E.G. Dockerhub).

Option 1: Using our automated action to publish your country configuration

We supply an automated action to do this for you. Our "Publish image to Dockerhub" Github Action is set to automatically run on your country configuration repository whenever code is pushed to a branch named master, main or develop. This action will build and push your Docker image to Dockerhub. You must edit this action if you wish to use a different container registry.

The image will be tagged with the short Git commit hash. This hash is important to refer to and use in the deploy step.

Global repository secrets

In OpenCRVS v1.3.1 hotfix scheduled for November 6th 2023, we will be releaseing a Github Action to automate the creation of environments and secrets :-)

These secrets below must be set as global repository secrets in Github as they apply to all environments and are used by the "Publish image to Dockerhub" Github Action👍

Parameter	Description
DOCKER_USERNAME	Your Dockerhub username to access the container registry. If you are using a different container registry, you will need to manually edit the deploy.yml appropriately.
DOCKER_PASSWORD	Your Dockerhub password.
DOCKERHUB_ACCOUNT	The name of your Dockerhub account or organisation that forms the URL to your country config docker image on Dockerhub before the slash. e.g: opencrvs
DOCKERHUB_REPO	The name of your Dockerhub repository that forms the URL to your country config docker image on Dockerhub after the slash.. e.g. ocrvs-farajaland

In Github, navigate to "Actions" and click "Publish image to Dockerhub" to see the output of the action that automates whenever code merges to "develop", "main", "master" branches. Copy the git commit hash in the Action logs to see how the Docker image was tagged. You will use it in the deploy step.

Option 2: Manually publishing your country configuration

You can navigate to the country configuration repository to build your image from the command line if you wish:

cd opencrvs-countryconfig

Export all of the required secrets as environment variables:

export DOCKER_PASSWORD=<your-docker-password> \
export DOCKER_USERNAME=<your-docker-username> \
export DOCKERHUB_ACCOUNT=<your-dockerhub-account> \
export DOCKERHUB_REPO=<your-dockerhub-repo>

Run the following script:

yarn compose:push:version

Take note of the githash at the end of the process to use in the next step ...

Deploying

Next, you need to deploy to your server environments.

1. First create appropriate secrets and environments in Github

Create the following Github secrets for the usernames and passwords you created earlier when provisioning the servers using Ansible in step 3.3.2, along with other secrets Github will use to SSH into your servers, set the Traefik SSL hostname and connect to your container registry (E.G. Dockerhub) etc.

Note: Using a strong password service such as 1Password you should store the usernames and passwords you create in this section as you will need them regularly.

Global repository secrets

These secrets below must be set as global repository secrets in Github as they apply to all environments and are used by the Github Action:

Secret	Description
SUPER_USER_PASSWORD	This is an OpenCRVS superuser National System Admnistrator password used when seeding the databases in deployed environments.
ELASTICSEARCH_SUPERUSER_PASSWORD	Created in 3.3.2. The Elasticsearch superuser password. You can also use this to login to Kibana with the username "elastic" and you have superuser Elastic privileges. Kibana URL: https://kibana.<your_domain>
KIBANA_USERNAME	A username for a regular Kibana user to login and monitor OpenCRVS stack health. Useful for developers as this user will have no superuser privileges.
KIBANA_PASSWORD	A password for a regular Kibana user to login and monitor OpenCRVS stack health
MONGODB_ADMIN_USER	Created in 3.3.2. The MongoDB superuser admin username. A powerful account that has all rights to OpenCRVS data
MONGODB_ADMIN_PASSWORD	Created in 3.3.2. The MongoDB superuser admin password.
MINIO_ROOT_USER	A username for a Minio superuser admin to login to the Minio console to view supporting document attachments submitted during registrations. https://minio-console.<your_domain>
MINIO_ROOT_PASSWORD	A password for a Minio superuser admin
DOCKER_USERNAME	Your Dockerhub username to access the container registry. If you are using a different container registry, you will need to manually edit the deploy.yml appropriately.
DOCKER_PASSWORD	Your Dockerhub password. (Docker doesnt like special characters in its passwords when we tested this recently. So choose a long, obfuscated alphanumeric password of 13 chars or more.)
DOCKERHUB_ACCOUNT	The name of your Dockerhub account or organisation that forms the URL to your country config docker image on Dockerhub before the slash. e.g: opencrvs
DOCKERHUB_REPO	The name of your Dockerhub repository that forms the URL to your country config docker image on Dockerhub after the slash.. e.g. ocrvs-farajaland
SMTP_HOST	Described in step 3.3.4
SMTP_PORT	Described in step 3.3.4
SMTP_USERNAME	Described in step 3.3.4
SMTP_PASSWORD	Described in step 3.3.4
ALERT_EMAIL	Described in step 3.3.4

Copy your id_rsa.pub key into the authorized_keys file

The deploy Github Action uses this library to SSH into your environments. This library depends on an PEM(RSA), PKCS8, and RFC4716(OpenSSH) SSH key. The id_rsa.pub associated with the same key must be saved into the authorized_keys file here along with the id_rsa.pub for any other developer who requires SSH access to your server. Be aware that these developers have root level access to your system and must be trusted. If they leave your organisation, it is your responsibility to remove their id_rsa.pub from the authorized_keys file in Git and on the servers.

Environment secrets

The following secrets are likely to be unique for each environment so they should be duplicated as environment secrets in Github and are used by the Github Action:

Secret	Description	Example
KNOWN_HOSTS	You will need the lines in the .ssh/known_hosts file relevant to the environments that the SSH Key uses. This can be generated using a test SSH connection using your key. From your local computer, SSH in to your manager node using your domain name. E.G.: ssh root@<your_domain> This will save an entry into your ~/.ssh/known_hosts. You will need a copy of the line in your ~/.ssh/known_hosts relevant to the host domain name for your environment in this secret so that Github can access your server using your SSH_KEY below.
SSH_KEY	This is a copy of the id_rsa file for the SSH Key, not the id_rsa.pub! Described here.

Environment variables

The following variables are likely to be unique for each environment so they should be duplicated as environment variables in Github and are used by the Github Action:

Variable	Description	Example
REPLICAS	The number of replicas: 1, 3 or 5 depending on how many servers are in the environment cluster as explained in step 3.3.1
DOMAIN	The host domain name (without www!) for your environment. You must make sure that you can ping this domain and that the ping resolves to your manager server's IP address. If this does not resolve, there must be a problem with your A record configuration explained in the previous step 3.3.5.
CONTENT_SECURITY_POLICY_WILDCARD	This string is supplied to the clients and nginx config and ensures that the format of your domain above can be configurable for CORS purposes.	*.<DOMAIN> or *<DOMAIN>

Customisable, optional environment secrets

The following secrets depend on your choice of communications provider. The Github deploy action and the deploy.sh script that the action calls will likely need to be edited according to your needs in order to customise these values.

Parameter	Description
EMAIL_API_KEY	If you are using Email for communications, then you will likely require an API key. We use this secret for the SendGrid API. Its likely you will have your own custom requirements depending on your chosen provider. Amend the handler appropriately to suit your needs.
SENDER_EMAIL_ADDRESS	If you are using Email for communications, then the sender email address that appears in all emails will need to be configured. Its likely you will have your own custom requirements depending on your chosen provider. Amend the handler appropriately to suit your needs.
SENTRY_DSN	OpenCRVS can report application errors to Sentry in order to help you debug any issues in production.
INFOBIP_API_KEY	If you are using SMS for communications, then you will likely require an API key. We use this secret for the Infobip API. Its likely you will have your own custom requirements depending on your chosen provider. Amend the handler appropriately to suit your needs.
INFOBIP_SENDER_ID	If you are using SMS for communications, then the sender id that appears in all SMS messages will need to be configured. Its likely you will have your own custom requirements depending on your chosen provider. Amend the handler appropriately to suit your needs.
INFOBIP_GATEWAY_ENDPOINT	We use this prop to store the endpoint URL for the Infobip API. Its likely you will have your own custom requirements depending on your chosen provider. Amend the handler appropriately to suit your needs.

1. Run the Deploy action

You can deploy to your server using the automated "Deploy" Github Action. Navigate to "Actions", and click "Deploy."

a) You will be required to select the environment that you wish to deploy to.

b) You will be required to enter the OpenCRVS Core Dockerhub image tag for any tagged build on Dockerhub) to refer to the OpenCRVS Core release of choice. Usually this will be an official release if you have performed no customisations to core. E.G. v1.3.0

c) You will be required to enter the OpenCRVS Country Configuration version (or short Git hash tag for any tagged custom country configuration build on Dockerhub) to refer to your country configuration image and githash created by the previous "Publish image to Dockerhub" action. E.G. 4e39a2a

d) Click Run workflow, and watch the output to make sure that the deployment was successful.

Once the deployment is complete, you should be able to navigate to your OpenCRVS URLs.

To login to OpenCRVS: https://login.<your_domain>

To login to OpenHIM: https://openhim.<your_domain>

To login to Kibana: https://kibana.<your_domain>

To login to Minio: https://minio-console.<your_domain>

Option 2: Manual deploy

You can navigate to the country configuration repository to deploy from the command line if you wish:

cd opencrvs-countryconfig

Export all of the required secrets as environment variables, and repeat lines for any customisable secrets you require:

export DOCKER_PASSWORD=<your-docker-password> \
export DOCKER_USERNAME=<your-docker-username> \
export DOCKERHUB_ACCOUNT=<your-dockerhub-account> \
export DOCKERHUB_REPO=<your-dockerhub-repo> \
export SUPER_USER_PASSWORD=<your-super-user-password>
export ELASTICSEARCH_ADMIN_USER=elastic \
export ELASTICSEARCH_SUPERUSER_PASSWORD=<your-ELASTICSEARCH_SUPERUSER_PASSWORD> \
export MONGODB_ADMIN_USER=<your-MONGODB_ADMIN_USER> \
export MONGODB_ADMIN_PASSWORD=<your-MONGODB_ADMIN_PASSWORD> \
export MINIO_ROOT_USER=<your-MINIO_ROOT_USER> \
export MINIO_ROOT_PASSWORD=<your-MINIO_ROOT_PASSWORD> \
export KIBANA_USERNAME=<your-KIBANA_USERNAME> \
export KIBANA_PASSWORD=<your-KIBANA_PASSWORD> \
export SMTP_HOST=<your-SMTP_HOST> \
export ALERT_EMAIL=<your-ALERT_EMAIL> \
export SMTP_USERNAME=<your-SMTP_USERNAME> \
export SMTP_PASSWORD=<your-SMTP_PASSWORD> \
export SMTP_PORT=<your-SMTP_PORT>

Run the following script substituting the parameters as explained in the table below:

yarn deploy --environment=<ENVIRONMENT> --host=<DOMAIN> --version=<CORE_VERSION> --country_config_version=<COUNTRY_CONFIG_VERSION> --replicas=<REPLICAS> --clear_data=no

Parameter	Description	Example
ENVIRONMENT	Can be set to "staging", "qa" or "production" with the impact on 2FA and comms delivery as previously explained in step 3.3.3. "staging" = a development environment where NODE_ENV is equal to "development". "qa" = a quality assurance environment where NODE_ENV is equal to "production", but all users will be test users and comms will not be sent - 2FA is disabled. "production" = a production environment where NODE_ENV is equal to "production" and comms will be sent - 2FA is enabled.	staging
DOMAIN	The host domain name (without www!) for your environment. You must make sure that you can ping this domain and that the ping resolves to your manager server's IP address. If this does not resolve, there must be a problem with your A record configuration explained in the previous step 3.3.5.
CORE_VERSION	The OpenCRVS Core Dockerhub image tag for any tagged build on Dockerhub) to refer to the OpenCRVS Core release of choice. Usually this will be an official release if you have performed no customisations to core. E.G. v1.3.0	v1.3.0
COUNTRY_CONFIG_VERSION	The OpenCRVS Country Configuration version (or short Git hash tag for any tagged custom country configuration build on Dockerhub) to refer to your country configuration image and githash created by the previous "Publishing your country configuration" step.	4e39a2a
REPLICAS	The number of replicas: 1, 3 or 5 depending on how many servers are in the environment cluster as explained in step 3.3.1	1
--clear_data=no	THIS IS A DESTRUCTIVE ACTION!!! - This parameter should be set to "no". Unless you wish to entirely clear all data on your server. That may be useful each time you deploy to a development environment. To PERMANENTLY DELETE ALL DATA on a server after deployments, set this to "yes"	no