Laith Al Enooz
Résumé
← Logs

2 min read

Measure before you rewrite

  • performance
  • php
  • mobility

Every online captain in a ride-hailing platform keeps sending its position, so the GPS ping is the hottest endpoint in the system by a wide margin. On FuratGo it cost 62ms. I had a confident plan for that: the endpoint was PHP, the obvious fix was to move the ping path to Go, and I’d already started.

Then I measured where the 62ms actually went.

The work was 2.6% of the request

The ping exists to do three things: write the captain’s position to Redis, look up whether the captain is on an active trip, and queue a broadcast if they are. Together, those came to 1.6ms — about 2.6% of the request. The other 97% was the framework around it.

The thing I would have cached first, the Postgres lookup for the active trip, was 0.47ms. Optimising it would have been real effort spent on 0.7% of a request. It is exactly the kind of change that feels productive, shows up nicely in a diff, and moves nothing.

The bottleneck was a default

The number that mattered wasn’t in the request at all. PHP-FPM’s pm.max_children was set to 5 — the stock default, never tuned — on an eight-core machine. Five concurrent requests, for the entire API. A rider’s booking could queue behind GPS pings, and no amount of shaving the ping would fix that, because the limit wasn’t how long a request took; it was how many could run at once.

I sized it from measured worker memory instead of guessing: 24 workers. The ping fell to 55ms, and more importantly, throughput now plateaus CPU-bound rather than worker-bound. The system stopped waiting in line and started using the machine it was paying for.

The Go service still exists

I did build the Go ping service. It’s deployed, and it’s off by default. It’s the right destination — when the fleet grows, the ping path is the first thing that should leave the framework — but it was the wrong thing to have built first. Built first, it would have hidden the real problem behind a faster endpoint, and the booking would still have queued behind it.

What I’d tell myself earlier

  • Profile the request before you profile the code. The surprise is usually in the part you didn’t write.
  • Check the boring knobs. Worker counts, connection pools, queue concurrency. Defaults are chosen to be safe on a laptop, not right on your server.
  • A rewrite is a hypothesis. Measure what it would save before you pay for it. Sometimes the answer is one line in a config file.