4.3.4 Create a Github environment

We have an automated script to generate Github environments for you along with all the application secrets that Github needs to run the continuous provisioning and deployment scripts.

The environments will be named according to the naming convention identical to that we have described here.

Specifically: qa, staging, production, backup and for training purposes a development (optional) server.

Github Actions use these environments to access the secret keys and configurations used when installing software on servers and deploying OpenCRVS.

Before running the script, you must prepare some secrets that the script requires.

While we would like to show you a video of this process, it involves a lot of organisation secrets. Therefore, we have provided as many obfuscated screenshots as possible :-)

1. Set up an individual and an organisation account on Dockerhub

Firstly, you will need a companion container registry account. Our scripts are hardcoded to use Dockerhub.

You will need a docker container registry organisation account on Dockerhub to build and push your country configuration container (image) in order to use our scripts. An organisation is required with all of your developers (including yourself) individual accounts added to the organisation's "members" (or teams) list, so that each developer can access the container.

This is because your full team and all the servers will need access to your forked countryconfig docker container. The servers will use your personal Dockerhub credentials to access the container. Dockerhub is a free containerisation repository that will provide you with 1 free private repository, and that is all you need. You will have to customise our infrastructure scripts if you wish to use a different container registry provider.

Then create an empty private repository to store your configuration...

Ensure that the Dockerhub members have permissions to write to the repository:

You will need your Dockerhub username and a personal Dockerhub account access token in order to create the Github environment. Our scripts use these credentials to login to Dockerhub programmatically. This is how you create a Dockerhub access token: https://docs.docker.com/security/for-developers/access-tokens/

2. Create a Github Personal Access Token

You need to create a Github Personal Access Token with the required permissions in order for the script to programmatically create Github environments on your forked countryconfig repository.

The only required scope for the token is "repo".

Set the expiration as you wish. Note that the token secret will need to be updated regularly for deployment actions to function when it expires.

3. According to your requirements, decide your VPN approach

OpenCRVS should be installed behind a VPN. You will need to prepare access secrets and variables for your VPN of choice. This script will ask you for VPN information when provisioning a production or staging environment. There are a few things to consider here.

You have many choices depending on your preferred way for Github Actions to access the target machine you are provisioning or deploying to.

You will need to edit Github Action files for all scripts that require access to your network using whatever "recipe" is required for the Github Action runner. Consider the Github Action runner as a client for your VPN. Github's IP addresses should be whitelisted as authentic sources for VPN client connections. They are listed here: https://api.github.com/meta

4. Prepare VPN variables for the script

Therefore depending on your choice above, you will need to define some variables that you will either use when running the script, or set as secrets manually in your environment.

For option a), Wireguard / Jump VPN server, you will be asked to set SSH_ARGS & VPN_HOST_ADDRESS:

VariableExampleDescription

SSH_ARGS

-J jump@<Insert IP address you use as the VPN_HOST_ADDRESS>

Arguments that are passed to the SSH command in Github Actions CI/CD pipelines

VPN_HOST_ADDRESS

An IP address for the VPN server. In our example, we use the QA server IP

For option b) your own VPN and openconnect, you will be manually adding the codeblock mentioned previously above in Github Actions. When the script has finished, you will need to manually add extra secrets into your chosen Github environment. You will need to add:

VPN_HOST_ADDRESS

An IP address for the VPN server. In our example, we use the QA server IP

VPN_PROTOCOL

e.g. fortinet

Openconnect supported protocol

VPN_PORT

The port used for VPN client connections

VPN_USER

The username for VPN client connections

VPN_PWD

The password for VPN client connections

VPN_SERVERCERT

Follow the below instructions to retrieve your cert

Use the following command to get the VPN_SERVERCERT value:

echo "$VPN_PWD" | sudo openconnect $VPN_HOST_ADDRESS --user=$VPN_USER --passwd-on-stdin --no-dtls

You have to copy the cert from the output which will look something like this:

...
SSL negotiation with xxxxxxx
Server certificate verify failed: signer not found

Certificate from VPN server "xxxxxxx" failed verification.
Reason: signer not found
To trust this server in future, perhaps add this to your command line:
    --servercert pin-sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  <--- Here it is
Enter 'yes' to accept, 'no' to abort; anything else to view: fgets (stdin): Operation now in progress

Our code is hardcoded to track bugs in Sentry and then these can be redirected to an email address or Slack channel. Sentry and Slack are extremely low cost tools (companion service costs are listed in the gather requirements section). However, a self-hosted Apache 2 licensed version of Sentry exists https://github.com/getsentry/self-hosted if you wish to use it. Our documentation includes only instructions for the paid-for, Sentry cloud account.

We can use an same email address or Slack channel to receive any uncaught errors or infrastructure monitoring alerts from Kibana: explained further in the Monitoring section.

Create a NodeJS project in Sentry for your chosen environment

In the Sentry project settings, select "Client Keys", and copy the DSN property. You will need it later.

If you wish to use Slack to receive Sentry and Kibana alerts you can create a dedicated Slack Channel and set up a Sentry Alert Rule like this:.

Create a Slack integration with Sentry. Then you can configure an alert like this example that directs alerts to a Slack Channel called "#sentry-madagascar":

Any uncaught errors that are not tracked by Sentry, will be tracked by Kibana as an uncaught error. In order to receive Kibana errors, you must set the ALERT_EMAIL property and Kibana will email these errors to you via your SMTP service. You can get a unique email address for a Slack Channel by clicking this button in your channel "Integrations" tab. In this way, all your Kibana errors can direct to the same Slack channel as used by Sentry:

Sentry & Kibana alerts can be configured to be broadcast to any email address. The benefits of using Slack, are that your entire development and quality assurance team can receive these notifications without a single individual being a gatekeeper, thus improving your process.

6. You are now ready to run the create github environment script

To run the script, cd into your forked countryconfig repository and run the following command:

yarn environment:init --environment=<name: e.g. development, qa, production or staging>

The environment name parameter will become the name of your Github environment. The script will ask you questions to connect to your Github repository, create an environment and populate that environment with all of the secrets and variables.

The Docker Hub secrets will be saved as repository secrets that are available for all environments to use. As you create additional environments, you do not need to update these as they will work across all environments. You will see a prompt each time asking if you mean to update them. You can just say no.

The final question you can see in this screenshot is asking for the SSH_KEY associated with the SSH_USER "provision", which we created in Step 3.3.1. Normally Vim will be the default editor which opens in Linux, and you should paste in the corresponding key after you click "Enter". After pasting in the key, you can type the following command to save & exit Vim:

:wq!

The script will continue like so:

In the above example, we set the amount of DISK_SPACE which will be set aside for encrypted civil registration data. In this example the total available diskspace 250g, therefore we save 50g aside for the operating system and Docker images.

In the "Database & monitoring" section, the script will generate for you strong, random passwords for all the databases. Our recommendation is that you should simply hit "Enter" for each of these and accept the usernames and passwords which are created.

The script will continue ...

If you are setting up a "qa" environment, which could optionally act as your Wireguard VPN & Jump server, or ..

if you are setting up a "staging" or "production" environment where "personally identifiable information" or (PII) for citizens is stored and accessed, by default the script will ask you if you are utilising the default Wireguard VPN.

It can automatically generate a Wireguard UI Admin password for you which you will use to create VPN client users.

if you are setting up a "staging" or "production" environment where "personally identifiable information" or (PII) for citizens is stored and accessed, by default the script will ask you for your backup server IP.

It can automatically generate a backup file encryption passphrase.

You must use the same BACKUP_ENCRYPTION_PASSPHRASE in the production and staging environments! Don't generate 2 different passwords.

The script will continue ...

You will be asked to enter the SMTP server details and choose if your default notification transport will be Email or SMS. You can always change this later. If you select SMS, the script will ask you for Infobip SMS information. You can customise these scripts to use a different SMS engine.

Finally the script will complete and if you look in your Github repository settings, you will notice that an environment has been created with all the secrets and variables automatically created like this:

You will also noticed that a file: .env.<your environment> has been created. This file contains all the secrets that were created. It is essential that the contents of this file be stored safely as it contains all your passwords for accessing your server and all the data on it.

The .env.<your environment> file is not required by OpenCRVS. It simple exists for your reference. Once you have copied the contents into a secure password manager such as Bitwarden or 1Password, you can delete this file. YOU MUST NEVER SHARE THIS FILE, NOR COMMIT IT TO GIT!!!

Run the script again to create the qa, staging, production & development environments that you require.

When all the environments are prepared, you should see something like this:

If you are using LetsEncrypt to generate TLS using the DNS challenge method as mentioned in the previous section, you will need to manually add the secret you plan to use in docker-compose files for that environment.

infrastructure/known-hosts

You will notice that a known-hosts file has been created by the script and is updated each time you create a new environment. This fie must always be committed to Git. The Gihub Action requires access to this file in order to SSH into the environments.

Note, that if you merge your PR into the develop branch, the Docker container image for the server should now build and push successfully as Dockerhub repository secrets exist. This is explained later.

You are now ready to proceed to the next step to provision the servers used in the environments.

Last updated