Read Replicas & Read/Write Separation

Tier 1 — Building Block

1. What Is It?

A read replica is a copy of a primary (leader) database that receives a continuous stream of changes from the primary and can serve read queries. The primary handles all writes; replicas handle reads. This is also called read/write separation or leader-follower replication.

Without read replicas, a single database node handles both writes (which require durable commits and locking) and reads (which scan indexes and return results). At scale, these workloads compete for the same CPU, memory, and I/O budget. Read replicas separate these concerns: writes go to the primary, reads are distributed across one or more replicas, multiplying read throughput horizontally.

The tradeoff is replication lag: replicas receive changes asynchronously (usually), meaning they may lag the primary by milliseconds to seconds. Reads from replicas can return stale data.

2. How It Works

Replication Mechanism

1. Primary receives a write (INSERT, UPDATE, DELETE).
2. Primary writes the change to its Write-Ahead Log (WAL) (PostgreSQL) or binlog (MySQL) before committing.
3. A replication stream ships WAL/binlog entries to each replica in near-real-time.
4. Each replica applies the log entries to its own data files, staying in sync with the primary.

-+ResetFullscreen

Read/Write Routing

The application (or a proxy like PgBouncer, ProxySQL, RDS Proxy) must route queries to the correct endpoint:

• INSERT/UPDATE/DELETE/BEGIN TRANSACTION → Primary connection string
• SELECT (that tolerates stale data) → Replica load balancer
• SELECT (immediately after a write, must be fresh) → Primary, or use read-your-writes consistency technique

-+ResetFullscreen

Replication Modes

• Asynchronous (default in MySQL, PostgreSQL): Primary commits immediately; WAL is sent to replicas without waiting for acknowledgment. Fastest writes; small lag window (~10ms–1s typically, can be longer under load).
• Synchronous: Primary waits for at least one replica to durably persist the change — and, in the strictest mode (PostgreSQL synchronous_commit = remote_apply), to apply it so it is visible to queries on that replica — before committing. Zero data loss; higher write latency (~+10ms RTT per synchronous replica).
• Semi-synchronous (MySQL): Primary waits only for at least one replica to receive the entry into its relay log (not apply it) before committing. Balances durability and latency.

3. Variants & Comparisons

Specific Technologies:

• PostgreSQL Streaming Replication: Native async/sync physical replication. WAL-level replication. Exact byte-for-byte copy of primary. Managed versions: AWS RDS/Aurora (Aurora replicas have ~10ms lag), GCP Cloud SQL, Azure Database.
• MySQL Group Replication / InnoDB Cluster: Semi-sync replication, supports automatic failover. Used by MySQL Router for read/write splitting.
• AWS Aurora: Custom replication protocol with shared storage layer. Up to 15 read replicas with <10ms lag. Failover in ~30 seconds. Aurora Global Database for cross-region replicas (60ms+ lag typical).
• Vitess (YouTube/PlanetScale): Sharding + replication management layer for MySQL. Used by YouTube, Slack, GitHub. Handles read/write routing and pooling at massive scale.
• PgBouncer / ProxySQL: Connection poolers that also route queries — ProxySQL can automatically send writes to primary and reads to replicas using query analysis rules.

4. When to Use It (and When NOT To)

Use Read Replicas When:

• Read-to-write ratio > 5:1: Most web apps are read-heavy (product listings, profile pages, feed queries)
• Primary DB CPU > 60% from reads: Reads are starving write throughput
• Analytics/reporting queries: Long-running SELECT scans can block or compete with OLTP; route them to a dedicated reporting replica
• Geographic distribution: Put a replica in each region to serve local reads with low latency (e.g., US replica for US users, EU replica for EU users)
• Disaster Recovery: A synchronous standby replica means zero data loss on primary failure

Decision triggers:

• "If your read QPS > 70% of DB capacity → add read replicas"
• "If you have reporting workloads running on the same DB as OLTP → add a dedicated analytics replica"
• "If you need <10ms reads from multiple geographies → add regional replicas"

Do NOT Use Read Replicas When:

• Strong read-after-write consistency required: After writing a record, user immediately reads it — if routed to a replica, they may see stale data. Fix: route such reads to primary, or use sync replica.
• Replication lag is unacceptable: Financial balances, inventory counts during checkout — even 100ms stale data causes correctness issues. Use primary for these reads.
• Write-heavy workload: Replicas help with reads; if your bottleneck is write throughput, you need sharding, not replicas.

Anti-patterns:

• Reading from replica immediately after write without a read-your-writes mechanism — users see their own write disappear.
• Ignoring replication lag monitoring — lag can grow under write spikes; replica may fall significantly behind without alerting.
• Using replicas for distributed transactions — transactions that span primary + replica are not atomic.

5. Real-World Usage

GitHub (MySQL with Vitess): GitHub serves ~100M read queries/day from MySQL read replicas. They use ProxySQL to route reads to replicas and writes to the primary. A critical pattern they document: "read-your-writes" is implemented by recording the binlog position after each write; subsequent reads are directed to the primary until the replica reports it has caught up to that position.

Instagram (PostgreSQL + Django): Instagram used Django's multiple database routing to direct reads to PostgreSQL replicas. Their 2012 engineering blog describes scaling to ~25M users with a primary + 2 async replicas per shard, routing 90% of reads to replicas. Replication lag was acceptable because most reads (photo feeds, profile views) tolerate eventual consistency.

AWS Aurora Multi-AZ: Aurora maintains one primary and up to 15 replicas using a shared distributed storage layer (not traditional WAL shipping). Because all replicas read from the same storage layer, replication lag is <10ms — much lower than traditional async replication. This is how Aurora achieves "fast local reads with replica consistency."

6. Interview Cheat Sheet

Key sentences to demonstrate depth:

1. "Read replicas multiply read throughput horizontally — each replica is an independent read node, so N replicas give you N× read capacity."
2. "The fundamental tradeoff is replication lag: async replicas are ~10ms–1s behind the primary. For most reads (feed queries, product listings, profile views), this is acceptable. For reads immediately after writes, I route to the primary."
3. "Replication lag can become a serious problem under write bursts — if the primary is writing 50K rows/sec and a replica falls behind, it could be minutes behind. You need lag monitoring and alerting."
4. "A synchronous standby replica costs you write latency (must wait for replica ACK) but gives you zero data loss on failover — worth it for financial or critical data."
5. "Read replicas solve read throughput, not write throughput. If writes are the bottleneck, I need sharding — replicas don't help there."

Common follow-up questions:

Connections to other building blocks:

• Caching Strategies: Complementary to caching. Cache handles repeated reads of the same objects; replicas scale diverse query patterns. Use both: cache for hot keys, replicas for query load.
• Sharding & Partitioning: Read replicas scale read throughput; sharding scales write throughput. A fully scaled system often uses both: shards, each shard having replicas.
• Write-Ahead Log (WAL): Replication is fundamentally built on the WAL. The replica is a consumer of the primary's WAL stream.
• CAP Theorem: Async replicas = AP (high availability, eventual consistency). Sync replicas with quorum = CP (strong consistency, lower availability under network partition).
• Load Balancing: The read load balancer in front of replicas distributes read traffic; it needs health checking to avoid routing to a lagged or failed replica.