Production incident

The Sluggish Feed

Freejunior· 0 of 9 min· 0 clues
11:15

Home feed p95 latency increased from 320ms to 8.4s.

  • Users report the feed takes about eight seconds to load.
  • Error rate is normal.
  • The feed shipped a new feature this morning: live reaction counts per post.

Graded on 5 review questions (70%), 3 key clues pinned to the evidence board (15%), and finishing within 9 min (15%).

System map

Services

The app calls frontend, which calls feed-api, which checks auth-service and then calls user-service, posts-service, ranking-service and reaction-counts-service, which reads reactions-db.

  • frontend

    Edge service for the mobile and web apps. Receives the home feed request and calls feed-api.

    Calls: feed-api

  • feed-api

    Builds one home feed page by calling the services below. The service that paged.

    Calls: auth-service, user-service, posts-service, ranking-service, reaction-counts-service

  • auth-service

    Validates the session token on every feed request.

    No downstream calls

  • user-service

    Returns the viewer's profile and the accounts they follow.

    No downstream calls

  • posts-service

    Returns the posts for one feed page, up to 25 per page.

    No downstream calls

  • ranking-service

    Orders the posts on the page for this viewer.

    No downstream calls

  • reaction-counts-service

    Returns the reaction count shown under each post on the page.

    Calls: reactions-db

  • reactions-db

    Postgres database that stores one row per reaction.

    No downstream calls

Evidence board

0 / 8

Collect outputs from tools to build a defensible RCA.