Production incident

The Missing Orders

Freejunior· 0 of 10 min· 0 clues
14:18

Paid checkouts are no longer creating orders for some customers.

  • Payment success rate is normal.
  • Order creation rate dropped by 31%.
  • API error rate remains below 0.5%.

Graded on 5 review questions (70%), 3 key clues pinned to the evidence board (15%), and finishing within 10 min (15%).

System map

Services

checkout-api charges the card through payment-provider, which sends payment_succeeded to payments-webhook, then payments-webhook calls order-service, which checks duplicate-detector, writes orders-db and calls email-service.

  • checkout-api

    Runs checkout. Starts the payment with payment-provider and shows the customer their order number.

    Calls: payment-provider, order-service

  • payment-provider

    External card processor. Sends a payment_succeeded webhook after each successful charge.

    Calls: payments-webhook

  • payments-webhook

    Receives payment_succeeded events and hands each one to order-service. Health check: payments/health.

    Calls: order-service

  • order-service

    Creates one order for each paid event, then asks email-service to confirm it. Health check: orders/health.

    Calls: duplicate-detector, orders-db, email-service

  • duplicate-detector

    Tells order-service whether a payment event was already processed, so a retried webhook does not create a second order.

    Calls: orders-db

  • orders-db

    Postgres database that stores orders and the payments already processed.

    No downstream calls

  • email-service

    Sends the order confirmation email after an order is created.

    No downstream calls

Evidence board

0 / 8

Collect outputs from tools to build a defensible RCA.