Laith Al Enooz
← All work

INGOT Brokers

MT Sync Service

A MetaTrader database sync pipeline rebuilt across eight phases, taking a 48-hour resync down to six.

Full resync 48h → 6–8h · hundreds of millions of records

  • Python
  • Kafka
  • PostgreSQL
  • Maxwell CDC
consume poison MetaTrader MySQL binlog Maxwell CDC Kafka durable transport Adaptive consumer PostgreSQL COPY bulk load Dead-letter queue
Follow it through the system, step by step an interactive teardown — scroll, and the architecture assembles →

The problem

MetaTrader (MT4/MT5) is the source of truth for trades, but its data has to land in our own databases for reporting, risk, and CRM. The original pipeline was fragile and slow: a full resync took 48+ hours, and a failure mid-run meant starting over. At hundreds of millions of records across multiple broker servers, that was untenable.

What I built

I rebuilt the pipeline from scratch as an eight-phase project, a real ordered migration rather than a rewrite-and-pray. The load-bearing pieces:

  • Change data capture with Maxwell CDC off the MetaTrader MySQL binlog.
  • Kafka as the durable transport, so a slow or failed consumer never loses events.
  • Migration from MySQL to PostgreSQL with COPY-based bulk loading for the initial sync.
  • An adaptive Python consumer that scales its batch size to throughput, with circuit breakers and dead-letter queues so a single poison record can’t stall the stream.

The result

Full resync dropped from 48+ hours to 6–8 hours, and incremental sync became continuous and self-healing. The dead-letter queue turned silent data-loss bugs into visible, replayable events: the difference between “the numbers look off” and “here is exactly what failed, and why.”