Time-Series DB (InfluxDB, TimescaleDB)

1. What Is It?

A time-series database (TSDB) is a database optimized for storing and querying data points indexed by time — metrics, events, sensor readings, and telemetry. The distinguishing characteristic is that timestamps are the primary dimension: data is always written with a timestamp and queried over time ranges, often with aggregations (avg, max, min, rate-of-change) across fixed time buckets.

Without time-series databases, engineers store metrics in general-purpose databases (PostgreSQL, MySQL, MongoDB). These work at small scale but degrade badly at high write rates: a metrics system writing 100K data points/second would require constant schema migrations, has no built-in data compression for repeating float sequences, and no automatic data expiration (old metrics accumulate forever). Time-series databases are purpose-built for append-only, time-indexed workloads with automatic downsampling, columnar compression, and TTL-based data expiration.

2. How It Works

Core Data Model

A time-series record has:

• Timestamp: nanosecond precision (InfluxDB) or microsecond (TimescaleDB)
• Metric name / Measurement: e.g., cpu_usage, http_request_count
• Tags (indexed dimensions): e.g., host=web-01, region=us-east
• Fields (unindexed values): e.g., value=72.4, count=4521

-- InfluxDB Line Protocol
cpu_usage,host=web-01,region=us-east value=72.4 1699920000000000000
              ^                                ^             ^
           tags (indexed)               field value      nanosecond timestamp

-- TimescaleDB (PostgreSQL hypertable)
INSERT INTO cpu_usage (time, host, region, value)
VALUES ('2024-01-15 08:30:00', 'web-01', 'us-east', 72.4);

InfluxDB Storage Engine (TSM)

InfluxDB 1.x/2.x uses TSM (Time-Structured Merge Tree) — a purpose-built storage engine inspired by LSM Trees but optimized for time-series:

1. Write: Data is appended to WAL (Write-Ahead Log) + in-memory Cache
2. Cache flush: When Cache reaches size limit or WAL reaches a threshold, data is written as a new TSM file
3. TSM files: Read-only, columnar files. Data is organized by (measurement, tag set, field key) → sorted timestamp arrays. Uses type-specific compression: Gorilla XOR encoding for floats (stores only the changing bits between consecutive values — highly efficient for slowly-changing metrics), ZigZag for integers, bit-packing for booleans
4. Compaction: Background process merges TSM files (reduces file count, removes deleted data, improves query performance)
5. Shards: Data is organized into time-based shards (e.g., 7-day boundaries). When a shard exceeds its retention period, the entire shard is deleted in O(1) — no row-by-row tombstoning

InfluxDB 3.0 (complete rewrite): The TSM engine was replaced by Apache Parquet on disk, with Apache DataFusion as the query engine and Apache Arrow Flight as the network protocol (the "FDAP stack"). The codebase was rewritten from Go to Rust. Flux query language was deprecated; SQL is now the primary query language.

TimescaleDB Architecture

TimescaleDB is a PostgreSQL extension that adds time-series optimizations transparently:

1. Hypertables: User-visible table that automatically partitions data into "chunks" by time interval (e.g., 7-day chunks). Partitioning is transparent — queries use standard SQL.
2. Chunk pruning: The query planner skips chunks outside the requested time range, reducing I/O dramatically for time-bounded queries.
3. Columnar compression: Older chunks are automatically converted to columnar format. Values for the same field across many rows are stored together, enabling type-specific compression. Real-world results: 90–95% size reduction (150 GB → 15 GB documented in production).
4. Continuous aggregates: Pre-computed materialized views that auto-refresh as new data arrives — e.g., hourly averages pre-computed and stored for fast dashboard queries.

Mermaid: Write Path Comparison

-+ResetFullscreen

Mermaid: System Architecture

-+ResetFullscreen

3. Variants & Comparisons

Time-Series Database Options

InfluxDB vs TimescaleDB

Prometheus vs InfluxDB (Architecture Comparison)

4. When to Use It (and When NOT To)

Use Time-Series DBs When:

• High write rates of timestamped data: 10K–1M+ data points/second from sensors, metrics agents, or telemetry streams. General-purpose databases struggle with this write volume without careful optimization.
• Time-range queries with aggregation are the dominant access pattern: SELECT avg(cpu_usage) WHERE time > now()-1h GROUP BY time(5m). This is what TSDBs are optimized for; SQL databases require full-table scans or complex index strategies.
• Automatic data expiration: Metrics older than 90 days are irrelevant but must be deleted. TSDBs handle this automatically via retention policies (shard-level deletion is O(1)).
• Downsampling / continuous aggregates: Keep raw 1-second data for 7 days, hourly averages for 1 year, daily averages forever. TimescaleDB continuous aggregates and InfluxDB tasks handle this natively.
• IoT / industrial telemetry: Thousands of devices each sending data every second. InfluxDB's tag-based data model and high-cardinality handling are purpose-built for this.

Do NOT Use Time-Series DBs When:

• Data is not primarily time-indexed: If time is just one of many fields (not the primary query dimension), a general-purpose database is more appropriate.
• You need complex JOINs across non-time entities: "Find all users who had CPU spikes AND submitted a support ticket in the same hour" — TimescaleDB can do this (full SQL), but InfluxDB cannot without external joins.
• Your data model is highly relational: Graph of users, orders, products — time-series databases have no concept of foreign keys or multi-table relational queries (InfluxDB) or limited support (TimescaleDB extends PostgreSQL).
• Write rate is modest (<1K/sec): PostgreSQL with a time-indexed table and pg_partman for partitioning handles this perfectly. Don't add a specialized system for low-volume time-series data.

Decision Triggers

5. Real-World Usage

Tesla → InfluxDB (Manufacturing Telemetry)

Tesla uses InfluxDB Enterprise as its primary time-series database to collect and store data from manufacturing assets — assembly robots, industrial equipment, and production line sensors. The use case is classic IoT telemetry: each device emits data points at high frequency, data needs retention policies (recent = full resolution, old = downsampled), and dashboards aggregate metrics across device fleets. InfluxDB's tag-based data model naturally handles the host=machine-42, plant=fremont grouping for fleet-level queries.

InfluxDB in Observability Stacks (TICK Stack)

The TICK stack — Telegraf (agent), InfluxDB (storage), Chronograf (visualization), Kapacitor (alerting) — became a popular open-source alternative to commercial APM systems. Thousands of teams use Telegraf to collect server metrics (CPU, memory, disk, network) and push them to InfluxDB, with Grafana as the dashboard layer. The push-based model fits monitoring agents that run on each server and report outward.

TimescaleDB in Financial Tick Data

TimescaleDB's hypertable + continuous aggregates pattern is well-suited for financial tick data: millions of price ticks per day, needing both raw access (specific trade at 14:23:17.042) and aggregated views (OHLCV candles by minute/hour/day). The combination of PostgreSQL SQL expressiveness (window functions, complex JOINs with order/instrument metadata) and time-series optimizations (columnar compression for old ticks, chunk pruning for time-range queries) makes it a natural fit for financial analytics where SQL flexibility is non-negotiable.

6. Interview Cheat Sheet

5 Sentences to Show Deep Understanding

1. "Time-series databases are optimized for three things that general-purpose databases do poorly: high-throughput append-only writes, time-range queries with aggregation, and automatic data expiration — a metrics system writing 100K points/second with 90-day retention would overwhelm PostgreSQL without specialized handling."

2. "InfluxDB's TSM engine (Time-Structured Merge Tree) is like an LSM tree tuned for time-series: data is organized by (measurement, tag set, field), sorted by timestamp, and compressed with type-specific codecs — Gorilla XOR for floats, ZigZag for integers — achieving much better compression than generic gzip on the repeating float sequences typical of metrics."

3. "TimescaleDB's key insight is that time-series is just PostgreSQL with time-based partitioning — by hiding hypertable chunk management behind a standard SQL interface, teams get 90-95% compression and time-range query acceleration while keeping the full PostgreSQL ecosystem: JOINs, foreign keys, pg_stat, pg_dump."

4. "Cardinality is the Achilles heel of InfluxDB 1.x/2.x: too many unique tag combinations (e.g., user_id as a tag on a billion-user system) causes memory exhaustion in the time-series index. InfluxDB 3 addresses this by removing the in-memory cardinality index entirely, using Parquet files that handle high cardinality natively."

5. "Prometheus is pull-based (server scrapes targets) and optimized for Kubernetes monitoring, where service discovery and scrape configs are natural. InfluxDB is push-based and optimized for IoT and industrial telemetry, where sensors push outbound to a central collector. Choosing between them comes down to collection model and query complexity."

Common Follow-Up Questions

Q: What is the cardinality problem in time-series databases? A: Cardinality refers to the number of unique time-series (unique combinations of metric name + all tag values). In InfluxDB 1.x/2.x, each unique series required an entry in an in-memory inverted index. If you use user_id as a tag in a billion-user system, the index holds a billion entries — causing OOM. The fix: use user_id as a field (unindexed) not a tag, or upgrade to InfluxDB 3.0 which removes the cardinality limit.

Q: How does data retention work in InfluxDB? A: InfluxDB organizes data into time-based shards (e.g., 7-day boundaries). A retention service runs every 30 minutes and deletes entire shards that have exceeded the retention period. This is O(1) file deletion — much more efficient than row-level tombstoning in general-purpose databases.

Q: When would you use TimescaleDB instead of InfluxDB? A: When you need: (1) full SQL with JOINs across relational and time-series data; (2) PostgreSQL tooling (pg_dump, logical replication, existing drivers); (3) complex analytics that go beyond simple aggregations; (4) an existing PostgreSQL team that doesn't want to learn a new database. InfluxDB wins on: pure write throughput, native high-cardinality support (v3), and the IoT/line-protocol ecosystem.

Q: What is downsampling and why does it matter? A: Downsampling is the process of aggregating high-resolution data into lower-resolution summaries as data ages. Example: keep 1-second raw metrics for 7 days, then downsample to 1-minute averages for 6 months, then to 1-hour averages forever. This dramatically reduces storage: 86,400 points/day at 1-second → 1,440 at 1-minute → 24 at 1-hour. Both InfluxDB (tasks) and TimescaleDB (continuous aggregates) support this natively.

Connections to Other Building Blocks

• Write-Ahead Log (WAL): TSDBs use WALs for durability, same as PostgreSQL and Kafka. InfluxDB's WAL is an append-only log flushed before acknowledging a write.
• LSM Trees: InfluxDB's TSM is conceptually an LSM tree adapted for time-series. The same compaction, level, and bloom filter concepts apply.
• Message Queues (Kafka): High-volume telemetry pipelines typically flow: sensors → Kafka (buffer/durability) → Telegraf/consumer → InfluxDB. Kafka provides backpressure protection; InfluxDB provides queryable storage.
• Caching: Grafana dashboards often add Redis in front of TSDB queries for dashboard-level caching — the same time-range query (last 24h CPU avg) hits the cache on repeated loads.
• Sharding: TSDBs use time-based sharding (temporal partitioning) rather than key-based sharding. Query pruning is trivial: WHERE time > T skips all shards before T.