Laith Al Enooz
← MT Sync Service

INGOT Brokers · MetaTrader sync pipeline

Trace a row through the sync pipeline

A résumé bullet says “full resync from 48 hours to six, across hundreds of millions of records.” Here is the pipeline that did it — one change at the source, followed all the way to a queryable row, with the pieces that make it fast, durable, and self-healing. Scroll, and the pipeline assembles around the change as it moves.

consume poison MetaTrader MySQL binlog Maxwell CDC Kafka durable transport Adaptive consumer PostgreSQL COPY bulk load Dead-letter queue

the system, at rest

  1. 01 / 06

    Tail the source — don’t poll it

    MetaTrader (MT4/MT5) is the source of truth for trades, and every change is already written to its MySQL binlog. Maxwell reads that log as change data capture, so we observe every row change without polling the source or adding load to a system traders depend on.

    Maxwell CDC

  2. 02 / 06

    A durable bus in the middle

    Each change lands on Kafka as an event. The bus is what makes the pipeline survivable: a consumer can be slow, crash, or be redeployed mid-stream and no change is lost, because the event is still on the log waiting to be read.

    durable transport

  3. 03 / 06

    A consumer that adapts to load

    An adaptive Python consumer reads from the bus and scales its batch size to throughput — small batches when the stream is quiet, large ones when it floods. A circuit breaker trips on downstream trouble instead of hammering a struggling database.

    adaptive batch size

  4. 04 / 06

    Bulk-load, don’t drip-feed

    The initial sync lands in PostgreSQL through COPY-based bulk loading rather than row-by-row inserts — the single biggest lever on the rebuild. A full resync that took more than 48 hours came down to six to eight, and incremental sync became continuous.

    48h → 6–8h

  5. 05 / 06

    Quarantine poison, don’t drop it

    A single malformed record used to be able to stall the whole stream. Now it’s routed to a dead-letter queue: the pipeline keeps moving, and the bad record becomes a visible, replayable event. That turns “the numbers look off” into “here is exactly what failed, and why.”

    visible · replayable

  6. 06 / 06

    The whole pipeline, at rest

    Rebuilt as an eight-phase migration — an ordered project, not a rewrite-and-pray — the pipeline moves hundreds of millions of records across broker servers and keeps them in sync continuously. The dead-letter queue and durable bus are what make it self-healing instead of fragile.