Opening Scenario
The data team needs the contents of a Postgres orders table streamed into Kafka in near-real time, with downstream consumers also writing aggregates back into a Snowflake warehouse. A junior engineer's first proposal is to write a custom service that polls Postgres every second, publishes to Kafka, and a second service that consumes from Kafka and inserts into Snowflake — two new microservices to operate, monitor, deploy, and on-call. Your tech lead frowns and says "use Kafka Connect." You vaguely remember Connect is "the thing that does sources and sinks" but you've never run it in production. Before approving the design, you need to understand what Connect actually is, when it's a better choice than writing the code yourself, and what its operational footprint looks like.
Question Progression
1. What is Kafka Connect, structurally?
Question. "Kafka Connect" sounds like a feature, but it shows up in deploys as a separate set of processes. What is it actually?
In the full tutorial
- When to Reach for This