{hr.work}

Streaming a live order book with Kafka

Nordic Analytics, fintech platform for traders and analysts

Live Level II market depth from Interactive Brokers, streamed through a durable event log to multiple independent consumers, with no data lost on a restart.

Level IIlive order-book depth
Multipleindependent consumers
Durable logevents retained on disk
Replayconsumers resume after a restart

01Live order-book depth

Downstream services needed live Level II order-book updates, the bids and asks at each price. The stream also needed to remain available for independent processing and later analysis.

02Feed live, store everything, lose nothing

Build one service that streams updates reliably to multiple consumers, without losing data when a process restarts.

03One stream, multiple readers

I changed the design from a nightly-reset in-memory store to a durable Kafka log, so multiple consumers could receive depth updates and recover independently.

Live order-book streaming architecture Interactive Brokers sends Level II updates through a Python WebSocket ingestion service to a durable Kafka topic on Redpanda. The disk-backed log retains events. Multiple independent readers keep their own offsets, can replay events after a restart, and operate at their own pace. DURABLE EVENT STREAM · SYSTEM DESIGN EXTERNAL PROVIDER INGESTION BOUNDARY DURABLE STREAM READER BOUNDARY IBKR Web APILevel II depthWebSocket feed bid / ask updates WebSocket clientPython ingest servicereceives every update Event producerwrites updates to topic Kafka topicRedpandadurable event log Disk-backed logretained events Multiplereadersindependent pace Reader offset Reader offset Reader offset own offsets Restart → replay from saved offset Readers recover independently New readers do not affect others
Multiple consumers read the same topic, each at its own pace.

04Running in production

It's running in production. Consumers receive live market depth, restarts do not lose data, and a new consumer can be added without touching the others.