4.2.7 Configure Metabase Analytics

OpenCRVS provides rich analytics functionality using Metabase, an open-source business intelligence platform. Metabase enables stakeholders to explore civil registration data through interactive dashboards, charts, tables, and maps.

Analytics helps countries:

• Monitor vital events (births, deaths, custom events)

• Analyse registration performance and bottlenecks

• Support planning and evidence-based decision making

• Export non-PII datasets as CSV for further analysis

---

🏗️ System Architecture

The analytics system is made up of three main components:

┌──────────────────┐    ┌────────────────────┐    ┌──────────────────┐
│  OpenCRVS Core   │───▶│  Country Config     │───▶│     Metabase     │
│  Event Actions   │    │  Analytics Pipeline │    │   Dashboards     │
└──────────────────┘    └────────────────────┘    └──────────────────┘

1. OpenCRVS Core

Captures all actions performed on events (e.g., declarations, validations, registrations) and sends them to countryconfig to be stored in he database below via this endpoint.

2. Analytics Database (PostgreSQL)

The countryconfig analytics pipeline stores processed analytics in the analytics schema, primarily in the event_actions table.

3. Metabase

Connects directly to PostgreSQL to power dashboards and reports.

---

📥 Data Sources

Analytics data is generated from several inputs.

Event Data

OpenCRVS processes:

• Event registrations — demographics, locations, timeliness

• Registration states — draft, validated, registered, certified

• Action histories — every workflow action and its timestamp

Countryconfig receives full event documents and determines:

• Which fields should be written to analytics

• Which fields contain PII and must be excluded

Configuration-Driven Analytics

Only fields marked with analytics: true in form configuration are extracted into analytics:

field: {
  id: 'childDetails.firstName',
  analytics: true, // Included in analytics pipeline - Do not set on any field containing PII e.g. names, ids, phone numbers, email addresses etc
}

This gives full control over what data is used for analytics.

---

🔄 Data Flow to Metabase

1. Event action occurs Any action on an event triggers countryconfig’s analytics pipeline via HTTP action hooks.

2. Analytics service extracts data Only fields marked analytics: true are considered.

3. Transformation & enrichment The service calculates secondary metrics, such as:

   • Registration delay

   • Age at event

   • Aggregated demographic stats

4. Storage All processed analytics fields are written to:

   analytics.event_actions

5. Metabase connects to PostgreSQL Metabase queries the analytics schema using configured DB credentials.

6. Dashboards render data Preconfigured dashboards visualise the output as charts, tables, metrics, or maps.

---

🚀 Deployment

Production Deployment

Metabase is deployed as a Docker service using:

infrastructure/docker-compose.deploy.yml

Example:

dashboards:
  image: metabase/metabase:v0.56.4
  volumes:
    - /opt/opencrvs/infrastructure/metabase/metabase.init.db.sql:/metabase.init.db.sql
    - /opt/opencrvs/infrastructure/metabase/run.sh:/run.sh
  environment:
    - METABASE_DATABASE_HOST=${METABASE_DATABASE_HOST:-postgres}
    - METABASE_DATABASE_NAME=${METABASE_DATABASE_NAME:-events}
    - METABASE_DATABASE_USER=${METABASE_DATABASE_USER:-events_analytics}

Environment Configuration Files

• infrastructure/metabase/environment-configuration.sql Contains database connection details and admin users. This file never needs to be modified.

• Environment variables configure:

  • DB host, port, user, password

  • Map visualisation settings

  • Site name and branding

---

🛠️ Development & Dashboard Management

⚠️ Important: Do Not Edit Metabase Directly in Production

Any dashboard or query changes made directly in deployed Metabase (staging or production) will be overwritten on the next deployment.

How to Make Permanent Changes

1. Work locally In your countryconfig service, run Metabase in development mode:

   yarn metabase

2. Modify dashboards using the local UI Visit http://localhost:4444 Default credentials:

   • Email: [email protected]

   • Password: m3tabase

Metabase login page

1. Configure UI components from database connections in Metabase Follow the official Metabase documentation to create UI components that satisfy your reporting requirements. Control which events you wish to track in Metabase here.

2. Persist your changes When Metabase stops (Ctrl+C), all updates are written to:

   infrastructure/metabase/metabase.init.db.sql

3. Commit the updated file This ensures the dashboards are version-controlled.

4. Deploy The new dashboards appear automatically in all environments.

---

🔧 Other Development Commands

# Clear all analytics data (development only)
yarn db:clear:all

---

🗃️ Why Analytics Data Is Not Backed Up

Analytics tables are not included in backups for three reasons:

1. Regenerable

The analytics database is derived from event data. If lost, it can be fully rebuilt from the source records. This rebuild occurs on every OpenCRVS deployment via the reindex database migration.

2. Performance

Analytics tables can grow very large (GB → TB). Backing them up would increase:

• Backup time

• Storage costs

• Restore time

3. Separation of concerns

Operational data is the source of truth; analytics is a read-optimised mirror that can be safely rebuilt.

---

⚙️ Configuration Reference

Database Connection Variables

• METABASE_DATABASE_HOST

• METABASE_DATABASE_PORT

• METABASE_DATABASE_NAME

• METABASE_DATABASE_USER

• METABASE_DATABASE_PASSWORD

Analytics Schema

Primary table:

analytics.event_actions

Contains:

• Processed event data

• Action timestamps

• Calculated metrics

• Non-PII fields only

Maps Configuration (Geographical Dashboards)

• OPENCRVS_METABASE_MAP_NAME

• OPENCRVS_METABASE_MAP_URL

• OPENCRVS_METABASE_MAP_REGION_KEY

• OPENCRVS_METABASE_MAP_REGION_NAME

These settings allow Metabase to render geographic dashboards (e.g., registration heatmaps by district).

Updating the map used in geographical visualisations

Several dashboard visualisations rely on regional maps. We provide an example map located at src/api/dashboards/file/map.geojson.

The simplest method to customise this map is by modifying the GeoJSON file's content. In local development environment, you should see the map changing right away in Metabase. To verify this, proceed to the Admin Settings by clicking the cog-icon at the top right corner of Metabase, and then navigate to "Maps". In "Custom maps", click on the map labeled "Full country".

If you are not seeing your updates, click on "Refresh" and try directly navigating to the default address http://localhost:3040/content/map.geojson which should now serve the updated GeoJSON.

Your map geojson file must be a single line, not multi-line file. This command can convert a multi-line json into single line file:

echo `cat src/api/dashboards/file/<your multiline geojson input file>.geojson` > src/api/dashboards/file/<your single line geojson output file>.geojson

Notice that changing the map name or url will have no effect after you stop and restart the Metabase environment once. For these to be changed, the Metabase development environment needs to be started using the following environment variables

export OPENCRVS_METABASE_MAP_URL=http://localhost:3040/content/map.geojson
export OPENCRVS_METABASE_MAP_NAME="Full country"
yarn start

Without the variables, these values will always change back to the default ones.

More information about customising dashboards and creating visualisations can be found from the Metabase documentation.