Unique ID Generator in Distributed Systems (Snowflake, UUID)

Tier 1 — Building Block

1. What Is It?

A unique ID generator produces globally unique identifiers for records in a distributed system where multiple servers create records simultaneously. Unlike a single-machine database with an AUTO_INCREMENT primary key, distributed systems cannot rely on a central counter — doing so creates a bottleneck and single point of failure.

The problem: if ten app servers are simultaneously inserting rows into a sharded database, each server needs a unique ID for its row that will not collide with IDs generated by the other nine servers. The ID must be unique across all shards, ideally sortable by time (so you can query "records created after X"), and generated without cross-server coordination.

Without a good solution, you either have a central database counter (bottleneck), random UUIDs (non-sortable, large), or per-node counters (collision risk). Each has serious drawbacks at scale.

2. How It Works

Twitter Snowflake (the canonical approach)

Snowflake IDs are 64-bit integers composed of three fields:

 63 62       22 21   12 11       0
 |--|---------|------|----------|
 sign timestamp worker  sequence
  1    41 bits  10 bits  12 bits

Max throughput per node: 4,096 IDs/millisecond = 4.096 million IDs/second per worker. With 1,024 workers: ~4 billion IDs/second globally.

-+ResetFullscreen

Generation pseudocode:

function next_id(worker_id):
    timestamp = current_time_ms() - EPOCH
    if timestamp == last_timestamp:
        sequence = (sequence + 1) & 0xFFF  # 12-bit mask
        if sequence == 0:
            wait until next millisecond  # Overflow: all 4096 IDs used this ms
    else:
        sequence = 0
    last_timestamp = timestamp
    return (timestamp << 22) | (worker_id << 12) | sequence

-+ResetFullscreen

UUID v4 (Random)

128-bit randomly generated identifier. Example: 550e8400-e29b-41d4-a716-446655440000

• Pros: Zero coordination needed; works completely offline; no centralized service
• Cons: 128-bit (vs. 64-bit Snowflake) — larger indexes, worse B-tree performance (B-trees are the sorted tree structures databases use for indexes); not sortable by time; random distribution causes B-tree page splits on inserts (index fragmentation)

UUID v7 (Time-Ordered, RFC 9562)

A newer UUID variant: first 48 bits are millisecond Unix timestamp, remaining bits are random. Gives time-ordering while remaining decentralized.

• Pros: Time-sortable like Snowflake; no coordination; 128-bit compatibility with UUID infrastructure
• Cons: Still 128-bit (larger than 64-bit Snowflake); slightly less compact

Database Auto-Increment (Not Distributed)

AUTO_INCREMENT in MySQL / SERIAL in PostgreSQL — the simplest approach for single-database systems.

• Pros: Dead simple; guaranteed sequential; perfect B-tree locality
• Cons: Single point of failure; bottleneck for high write throughput; doesn't work across multiple shards

3. Variants & Comparisons

ULID (Universally Unique Lexicographically Sortable Identifier)

01ARZ3NDEKTSV4RRFFQ69G5FAV
|----------|--------------|
 timestamp    randomness
  48 bits      80 bits
  10 chars     16 chars

• 128-bit, Crockford Base32 encoded
• Monotonically sortable within the same millisecond
• No central coordination

4. When to Use It (and When NOT To)

When to Use Each Approach:

• Snowflake-style: You need 64-bit integer IDs (small index size, fast comparisons), time-ordered inserts (good B-tree locality), and can assign worker IDs at startup. Ideal for: social media posts, messages, events, orders.
• UUID v7: You need IDs compatible with UUID infrastructure (128-bit), time-ordered, but want to avoid central coordination. Ideal for: microservices where each service generates its own IDs without central ID service.
• UUID v4: You need zero-infrastructure uniqueness. Acceptable when: the table is small, random access patterns make B-tree page splits acceptable, or you're in a system that already uses UUID format.
• DB auto-increment: Single-database system, sequential IDs required, no sharding needed.

Decision triggers:

• "If inserting > 10K rows/sec into a sharded database → Snowflake or UUID v7 (avoid random UUID — index fragmentation kills write throughput)"
• "If you need IDs sortable by creation time (for pagination, feeds) → Snowflake or UUID v7"
• "If you have no central coordination and need maximum simplicity → UUID v4 or v7"

Anti-patterns:

• Random UUIDs as B-tree primary keys at scale: Every insert goes to a random page in the index. At tens of millions of rows, >90% of inserts cause page faults (non-sequential I/O). Use Snowflake or UUID v7 for write-heavy tables.
• Using wall clock time without monotonic check: If the server clock moves backward (NTP adjustment), Snowflake can generate duplicate IDs for the same worker. Fix: detect clock drift and wait or throw an error.
• Exposing Snowflake IDs in APIs: Snowflake IDs embed your worker topology and timestamp — leaks information. Consider obfuscating (e.g., XOR with a secret) if the ID is exposed publicly.
• Single Snowflake ID service: The ID generation service itself becomes a SPOF. Deploy one ID generator per app server (embedded library), not as a centralized service.

5. Real-World Usage

Twitter Snowflake: Twitter open-sourced the original Snowflake in 2010. Tweet IDs are Snowflake IDs — they are 64-bit integers, sortable by time, and generated at each tweet creation service instance without coordination (worker IDs are assigned at startup via ZooKeeper, a distributed coordination service). This is why tweet IDs increase monotonically over time: tweet 1234567890 was created before tweet 2345678901. Generating ~6K tweets/sec at peak required a system that could never bottleneck on ID generation.

Discord Snowflakes: Discord uses a Snowflake variant for message IDs, user IDs, channel IDs, and server IDs. Discord's API exposes a since_id pagination pattern — "give me all messages with ID > X" — which only works because Snowflake IDs are time-ordered. Discord's worker ID encodes epoch differently (Discord epoch = 2015-01-01) but the structure is otherwise identical to Twitter's.

Instagram (64-bit IDs with PL/pgSQL): Instagram used a different approach: a PostgreSQL stored procedure (id_generator) that uses a combination of timestamp + shard ID + sequence number, generating 64-bit IDs within the database using PL/pgSQL. This avoids a separate ID service entirely; each PostgreSQL shard generates IDs for its own data. The approach produces ~1000 IDs/sec per DB connection — sufficient for their write rate.

6. Interview Cheat Sheet

Key sentences to demonstrate depth:

1. "Snowflake IDs are 64-bit integers: 41-bit millisecond timestamp + 10-bit worker ID + 12-bit sequence. This gives 4,096 IDs per millisecond per worker with no cross-node coordination — just assign unique worker IDs at startup."
2. "The key advantage of time-ordered IDs (Snowflake, UUID v7) over random UUIDs is B-tree index locality: sequential inserts fill pages from left to right without splits, maintaining near-100% page utilization and avoiding write amplification."
3. "The Snowflake clock drift problem: if a server's clock moves backward, the same worker might generate the same sequence for the same apparent millisecond. Mitigation: detect clock drift, refuse to generate IDs until clock catches up."
4. "For an interview question about ID generation, I'd default to Snowflake-style: embedded in each app server, ZooKeeper or a config service assigns worker IDs at startup, no centralized ID service needed."
5. "UUID v4 is fine for low-write-rate tables or when you need UUID format compatibility. At high write rates (>10K rows/sec) on a B-tree primary key, random UUIDs cause severe index fragmentation — switch to Snowflake or UUID v7."

Common follow-up questions:

Connections to other building blocks:

• Sharding & Partitioning: In a sharded database, IDs must be globally unique across shards. Snowflake IDs include a worker ID (often includes shard ID) to guarantee uniqueness.
• Distributed Locking: Assigning Snowflake worker IDs at startup requires a distributed lock or coordination service (ZooKeeper, etcd) to prevent two nodes from claiming the same worker ID.
• Message Queues: Kafka message offsets and Cassandra's TIMEUUID serve similar purposes — time-ordered unique identifiers for events in a distributed log.
• CAP Theorem: Centralized ID services (Ticket Server) are CP — if the service is down, ID generation stops. Snowflake (embedded per node) is AP — each node generates IDs independently, tolerating network partitions.