Create a Github environment
Before you begin
In this section you will run the yarn environment:init script in your forked infrastructure repository which will help you to configure OpenCRVS environments. This command:
Creates GitHub environments
Drafts commands to bootstrap new servers
Prepare inventory files for ansible for environment provision
Prepares Helm chart values for deployment
GitHub environments will host all secrets and variables required for successful infrastructure configuration (users, filesystem, kubernetes cluster, etc) and OpenCRVS deployment.
Make sure you completed environment preparation steps and have all required information:
GitHub Organisation
Own GitHub organisation with Team subscription.
Country config repository
Refer to Quick Start
Virtual machines (servers) created
Preparation steps: Verify you have IP addresses or DNS names for each environment and you are able to login on those VMs
Domain names are registered
Verify DNS names are pointed to appropriate IP addresses of VMs.
SSL Certificates issued or one of the available Let’s Encrypt is considered to be used
Infrastructure repository forked
Refer to Quick Start
GitHub Token with full code access and workflow permissions created
Personal access token (Fine grained token) with access to Country config and Infrastructure repositories. Refer to Quick Start
DockerHub Account, token and repository are created
Make sure Country config image was built and pushed to DockerHub
Users with their public keys to grant remote access to the servers
Refer to Advanced Topics > SSH access
SMTP server configured
Refer to Setup Infrastructure
Optionally Third-party accounts created (sentry, slack, etc)
Refer to prerequisite accounts
Create github environments
Environments are managed by yarn environment:init script. The script will create files that must be pushed to Git, so it is advisable to run the script in a new branch in order to open a pull request.
Re-run script for each environment:
development and (or) qa
staging
production
Migrating from an earlier version? ... Read the Migration from Docker swarm guide.
Configuring a staging or prodution Github environment requires you to have a backup server environment in place.
No explicit backup Github environment is created by this script anymore as of OpenCRVS v2.0.
To run the script, open terminal window and cd into your forked infrastructure repository and run the following command to start the configuration wizard:
You may notice that the same commands exist in an infrastructure folder in forked countryconfig repo but these exist only for backwards compatibility, e.g. OpenCRVS v1.8 or below and should no longer be used. Older installations should follow the Migration from Docker swarm guide
You will be asked to provide values to configure key OpenCRVS components. Some actions can be automated and the script will guide you to the next steps.
Environments init script questions
Intro questions
Purpose for the environment?
The script will ask you to select the type of environment that you wish to create:
Depending on your anwer different logic will be executed and some features might be not available. For example Approval workflow work only on environments with PII data.
What is the name of your environment?
Environment name is used later to create Kubernetes namespaces and other configuration objects like backup and restore folders. Environment name is read only property.
We recommend you use following pre-configured environment names:
development
qa
staging
production
GitHub
What is the name of your Github organisation? Type your organisation.
Note that for personal repositories organisation is your GitHub login.
What is your Github infrastructure repository? Repository name where your infrastructure code lives.
What is your Github token? Classic or Fine-grained token with access to infrastructure workflows and code.
Once you provide answers to the questions script will connect GitHub:
If environment doesn't exist, script will continue it's execution
GitHub environment doesn't exist If environment exists, script will fetch information about existing secrets and variables for particular environment and repository.
GitHub environment already exists The script will fail if it cannot connect to Github for whatever reason.
For environments with PII data script will ask you to answer additional questions:
GH_APPROVERS: Comma separated list of GitHub accounts responsible for Reviews and Approvals on OpenCRVS GitHub Actions workflows. This variable is defined at repository level.APPROVAL_REQUIRED: Require approval for particular environment. This variable is defined at environment level.
It is strongly recommended to Approval requirement in production environments to mitigate the risk of accidental deployments or environment resets, which may lead to the deletion of citizen data.
Docker Hub
The script will ask for your Dockerhub credentials or skip if they already exist. GitHub doesn't allow you to fetch secret values so you will not be able to check the current value, only updating the value is possible.
Kubernetes & Runtime
The script will ask you to provide Kubernetes and Runtime options:
DOMAIN: Domain name to expose the OpenCRVS application frontend and APIs. It will be the domain after the subdomains that you configured when setting DNS.KUBE_API_HOST: IP address or domain address for the Kubernetes master node. Provision script will generate Kubernetes config files for each user defined in users section of inventory file.WORKER_NODES: (Options property) Comma separated list of additional Kubernetes cluster members (Virtual Machines). Leave empty for a single node setup. Worker nodes can be added later.
SSH Users
Script will ask you to create users with remote SSH access, answer following questions:
User name: Remote user name to login
Public ssh keys: Add public key(s) for remote login. Login by password is disabled by default, keys must be provided
Role: User role on remote system
Check documentation for more examples and detailed instructions how to manage remote access: Advanced topics > SSH Access
Once all the users are added, select "Save & Exit" in order to continue with the script.
Traefik SSL Certificate
Installation script does configuration for traefik helm deployment.
Following options are available at configuration time:
Lets Encrypt certificate: No additional input is required. The installation script will automatically configure Traefik to obtain and use Let’s Encrypt SSL certificates for you. NOTE: Your server must be accessible from the public internet for this process to work. This option is recommended only for testing and demonstration purposes, not for production environments.
Static SSL certificate: The script will prompt you to provide an SSL certificate and key for the frontend. The provided values will be securely stored as GitHub secrets.
Custom configuration: No additional questions will be asked. A preconfigured file (without the Let’s Encrypt section) is used, and Traefik will use its default certificate. Choose this option if you plan to configure Traefik yourself later using Helm chart values.
Questions for Static SSL certificate
SSL_CRT: SSL Certificate or Certificate chainSSL_KEY: SSL Certificate private keys
Full documentation about traefik configuration can be found at:
Official documentation page: https://github.com/traefik/traefik-helm-chart
Storage
Only enable disk encryption if your data centre is equivalent to a Tier 2 or lower, where physical security may not be at its optimum. If your data centre tier is higher, and extremely secure, there should be no need to encrypt the disk. It is vital to safely store the encryption key that is generated by the script to avoid loss of data. Losing the encryption password is not recoverable.
Disk Encryption: If disk encryption is enabled, provision GitHub Actions workflow will create encrypted file in the root (/) directory and mount it as /data. Disk encryption is optional. Disk encryption can't be enabled later.
DISK_SPACE: Amount of disk space that should be dedicated to OpenCRVS data (logs, monitoring and citizens crvs records). Specified value will become the size of an encrypted cryptfs/datadirectory. Specified disk space will be allocated on each cluster node for multi-node kubernetes cluster and this behaviour may change in the future.
Backup
Its possible for this environment to back up its data to another server e.g. backup every night. It is strongly recommended if you are provisioning a production environment to enable backup.
BACKUP_HOST: Backup server IP address or hostname.BACKUP_SERVER_USER: User to connect to backup server. At this point you may choose any username, provision script will create user for you. E/g If you would like to use shared server to store backups from qa and production, you may want to have separate users for security reasons.
Script will add backup host to ansible inventory files and configure appropriate values for helm release. Script will create private/public key-pair for backup server user.
By default backup is configured to run at 01:00 AM by UTC, if you need to adjust backup schedule, update configuration manually at environments/<env>/dependencies/values.yaml
More information about backup server configuration can be found at Backup and Restore section.
Restore
Its possible for this environment to restore backed-up data from another environment every night. It is strongly recommended if you are provisioning a staging (pre-prod/mirror) environment to enable restore.
Restore configuration will ask you to provide the restore environment name from the existing environment list. If the environment doesn't exist and will be created later, feel free to type future environment name here, but don't forget to create that environment BEFORE running OpenCRVS dependencies deployment.
By default restore is configured to run at 00:00 AM by UTC, if you need to adjust the restore schedule, update configuration manually at environments/<env>/dependencies/values.yaml
Databases & monitoring
The script will proceed to ask you to set database and monitoring passwords. Strong passwords are suggested. It is advisable to accept the defaults presented to you.
KIBANA_USERNAME: (Default: opencrvs-admin): Kibana user nameKIBANA_PASSWORD: (Default: random value): Kibana password
Sentry
SENTRY_DSN: The DSN tells the SDK where to send bug events to. OpenCRVS application has built-in Sentry support.
Metabase admin
Metabase provides a public web interface to access an analytics editor.
OPENCRVS_METABASE_ADMIN_EMAIL: Metabase Admin user (only emails are allowed)OPENCRVS_METABASE_ADMIN_PASSWORD: Metabase Admin password
SMTP
At this point smtp server should be configured. If you are using third-party email providers like Sendgrid, please make sure that your OpenCRVS domain is in whitelist and issued credentials are correct.
SMTP_HOST: Hostname or IP address of your smtp serverSMTP_PORT: Port where smtp server is listeningSMTP_SECURE: Use TLS for connectionSMTP_USERNAME: Username or email used to authenticate as a email client on smtp serverSMTP_PASSWORD: Password or API token depend on your email providerSENDER_EMAIL_ADDRESS: All emails will be sent with this email in sender fieldALERT_EMAIL: Email address for alerting, this field is often used to integrate with Slack, Google Chart or any other corporate communication tool.
Review step
Once all questions are answered, the script will generate strong database passwords for all the database technologies used in OpenCRVS. It will display all the secrets that the script will create and ask you if you want to continue to create the environment on Github.
The script will slowly create the Github environment and upload all the secrets OpenCRVS requires to provision and deploy OpenCRVS from Github Actions. It will create Helm chart and values files ready for committing to your repository.
On the final step the script will provide a command to bootstrap a self-hosted runner on your server. Save the command from script output to a temporal file, you will need it later.
This is explained in the section Bootstrap servers:
Run following command on your infrastructure repository:
You should get a number of files modified:
Usually review is not required for files under the .github folder.
Review modified files:
infrastructure/server-setup/inventory/<environment name>.yml: Configuration file for Ansible playbook responsible for server provision. For more information please follow hints inside file and SSH Access section.environments/<environment name>: Folder withvalues,yamlfiles for helm charts:environments/<environment name>/traefik/values.yaml: Update this file with proper configuration to handle SSL certificate. Please follow documentation under TLS / SSL & DNSenvironments/<environment name>/opencrvs-services/values.yaml: Review configuration and adjust according to your needs, usually defaults are good for initial deploymentenvironments/<environment name>/dependencies/values.yaml: Review configuration and adjust according to your needs, usually defaults are good for initial deployment.
Final notice
The later provision script will disable password SSH access for all users on the server and create new users from the infrastructure/server-setup/inventory/<environment name>.yml file. After provisioning, SSH will only be possible using public/private key pairs.
Users will be required to use Google Authenticator to SSH in after provisioning. This 2FA approach is an important step to secure your infrastructure.
If any user is utilising the 1000 group, the script will fail. Modify your available user groups on the server to ensure this one is available.
The .env.<your environment> file outputted by this script contains sensitive information about your environment configuration. Copy the content of this file into a secure place or password manager such as Bitwarden or 1Password and delete this file. YOU MUST NEVER SHARE THIS FILE, NOR COMMIT IT TO GIT!!! This file is not required by OpenCRVS.
You will notice that an environment now exists in your Github repo containing all the secrets required.
If you made a mistake and wish to run the script for this environment again, you must delete the environment on Github by clicking the trash icon first. The environment and all secrets will be deleted and recreated, enforcing you to start over.
Repeat the process for all your required environments, e.g. qa, production, staging ...
Once all environments are setup, feel free to continue.
Last updated