Knowing when not to reach for Kafka

At work I’ve built pipelines where Kafka earns every bit of its weight: change data capture off a MySQL binlog, a durable bus in the middle, an adaptive consumer and a dead-letter queue, over hundreds of millions of records. At home I run a platform for my own household — money, vehicles, appliances, the kids’ vaccination schedule, a dozen self-hosted services — and it has no Kafka, no CDC, and no microservices. That’s not a contradiction. It’s the same decision made twice.

What the machinery is for

Kafka, CDC and service boundaries buy specific things: absorbing bursts you can’t control, decoupling teams that deploy independently, replaying history when a consumer was wrong, surviving a slow downstream without slowing the source. At the scale of a brokerage’s trade history, each of those is a real requirement, and the cost of the machinery is small next to the cost of not having it.

A two-person household has none of those problems. The Laravel scheduler running a morning digest and dispatching reminders is the entire event system, and it’s enough. Adding a broker would add an outage surface, an upgrade burden and a debugging story to protect against load that will never arrive.

Reliability without the machinery

Restraint isn’t the same as carelessness. The household hub reads my finance app’s database and calls its API, and a vehicle service cost posts straight through as an expense. That sync is idempotent by a stable key, so a retry can never double-charge. When the finance app is unreachable, the record degrades to Pending instead of failing or vanishing, and goes through on the next attempt.

Those are the same properties a CDC pipeline gives you — no lost writes, no duplicates, safe retries — achieved with a key and a status column instead of a cluster.

One edge, one identity

The rest of the platform follows the same instinct. A single Caddy edge terminates TLS for every service, a single Authentik login covers all of them, and no service publishes a public port. A dozen services, one way in.

The principle

Knowing when to reach for Kafka and knowing when not to are the same skill: name the property you need — durability, replay, decoupling, burst absorption — and pay for the cheapest thing that provides it. Sometimes that’s a cluster. Sometimes it’s a unique key.