Production incident

The Empty Dashboard

Freejunior· 0 of 9 min· 0 clues
09:05

Marketing dashboard shows 0 signups for today; CMO is asking why.

  • The marketing site is up and accepting traffic.
  • Yesterday's signup count was normal (2,140).
  • The dashboard reads from an analytics warehouse populated by a nightly ETL job.

Graded on 5 review questions (70%), 3 key clues pinned to the evidence board (15%), and finishing within 9 min (15%).

System map

Services

marketing-site writes signups to signups-db, and each night at 02:00 airflow-scheduler starts etl-runner, which copies them from signups-replica into the warehouse that dashboard-service reads.

  • marketing-site

    Public website. Accepts the signup form and writes each signup to signups-db.

    Calls: signups-db

  • signups-db

    Production Postgres primary that stores every signup.

    No downstream calls

  • signups-replica

    Read replica of signups-db at host signups-replica.prod. The ETL job reads from it so it does not load the primary.

    Calls: signups-db

  • airflow-scheduler

    Starts the daily_signups_etl job at 02:00 and records the state of each run.

    Calls: etl-runner

  • etl-runner

    Runs the nightly ETL jobs. daily_signups_etl copies new signups from signups-replica into the warehouse.

    Calls: signups-replica, warehouse

  • warehouse

    Analytics warehouse. Holds the marketing.signups_daily table.

    No downstream calls

  • dashboard-service

    Serves the Marketing Dashboard by querying the warehouse. This is the dashboard the CMO opened.

    Calls: warehouse

Evidence board

0 / 8

Collect outputs from tools to build a defensible RCA.