4.3.1 Verify servers & create a "provision" user

These are the steps you need to perform after receiving a server IP address and an SSH user before you can run the provisioning scripts for any given environment. E.G: qa, staging, production (1, 2, 3 or 5 server cluster).

Verify the Ubuntu version is >22.04

First, login as root, or if you only have sudoer access, do sudo su root.

riku@farajaland-prod:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04
Release:	22.04
...

If not, either recreate the server or upgrade Ubuntu.

Verify the disk has been partitioned correctly and that you have enough space for your chosen environment

riku@farajaland-prod:~$ df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/vda1            311G  206G  105G  67% /
/dev/vda15           105M  6.1M   99M   6% /boot/efi

We want to ensure the partition mounted to / has enough disk space. Like we se here, we have 311G in total which would be enough for a qa, production or staging environment.

Check the internet connection from the servers.

Check you can ping google.com from the servers.

Create a user named provision

The next commands will create a user named provision, make it a sudoer (needed for provisioning) and finally generate an SSH key for logging in as the user. The SSH private key will not persist on the server as it should only be stored in Github Secrets.

adduser --gecos "OpenCRVS Provisioning user" --disabled-password provision
usermod -aG sudo provision
echo 'provision ALL=(ALL) NOPASSWD:ALL' | sudo tee -a /etc/sudoers
su provision

Create SSH keys for each environment for provision

The deploy Github Action uses this library to SSH into your environments. This library depends on an PEM(RSA), PKCS8, and RFC4716(OpenSSH) SSH key.

For production servers, SSH keys should only be created for the manager node.

mkdir -p /home/provision/.ssh
ssh-keygen -f /tmp/ssh-key -N ""
cat /tmp/ssh-key.pub >> /home/provision/.ssh/authorized_keys
chmod 600 /home/provision/.ssh/authorized_keys
echo -e "\n\nThis is the SSH_KEY you add to Github Environments:\n\n"
cat /tmp/ssh-key
rm /tmp/ssh-key*

After running the commands, you see the SSH private key in the terminal window. It will look like this:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXkt ...

...uY3J2cy1tb3NpcAEC
-----END OPENSSH PRIVATE KEY-----

Copy this key and save it into secure password manager software. It is the private key used by the provision user to SSH into the servers automatically from Github environments. We will use it when setting up our Github environments.

Note: You will need password manager software such as Bitwarden or 1Password to safely store OpenCRVS secrets and manage them in line with your internal data security policies.

For additional replicas (worker) servers in a production cluster

SSH into the production manager node and copy the public key for the provision user.

cat /home/provision/.ssh/authorized_keys

SSH into the worker node and create the provision user, but DO NOT create new SSH keys for the provision user. Instead make the user and then the .ssh directory to store the public key above in an authorized_keys text file:

adduser --gecos "OpenCRVS Provisioning user" --disabled-password provision
usermod -aG sudo provision
echo 'provision ALL=(ALL) NOPASSWD:ALL' | sudo tee -a /etc/sudoers
su provision
mkdir /home/provision/.ssh
nano /home/provision/.ssh/authorized_keys

Paste in the public key for the manager node provision user. Exit & save.

Add the staging and production "provision" user's id_rsa.pub to the backup server

So that the Github Action can provision the staging and production server to backup in the correct way, their public keys must be also added to the backup server's provision users' authorized_keys file.

SSH into the production manager node and copy the public key for the provision user.

cat /home/provision/.ssh/authorized_keys

SSH into the staging manager node and copy the public key for the provision user.

cat /home/provision/.ssh/authorized_keys

SSH into the backup server as the provision user and paste these keys at the bottom of the /home/provision/.ssh/authorized_keys file.

Next, you need to consider how your servers are networked, and how you plan to generate TLS. A lot depends on your VPN approach.