ML Feature Pipeline

Opening Scenario

You're on the ML platform team. You support a dozen model owners across recommendations, ranking, search, fraud, pricing, and churn. Today, each team owns its own feature engineering. One team computes "user 7-day clicks" in a nightly Spark job and loads it to Redis. Another computes the same concept in a Flink job and writes it to DynamoDB. A third computes it in their training notebook from raw logs and never serves it online. Training/serving skew is a recurring incident. A model just shipped that scored a 0.92 AUC offline and dropped to 0.71 in production. The cause was the "average order value" feature. In training, it was computed over the full history of the user. In serving, it was computed from a cache that excluded the last 30 minutes of activity. The director wants a unified feature pipeline. Before designing it, you need to understand what an ML feature pipeline actually is. You also need to understand why parity is the hard problem. Finally, you need to know what the canonical Kafka + Flink + feature-store shape looks like.