Flat search across 10 years of financial news
Nordic Analytics, fintech platform for traders and analysts
A topic timeline built from about 25 million articles now comes back in under five seconds. It used to take about a minute.
01A minute per timeline
The platform builds a timeline for any topic from more than ten years of financial news, hour by hour or day by day. Every article has to be scored against the topic, because every period needs its own count. On Elasticsearch, one timeline took about a minute.
02Make it usable next to a live market
Make timelines fast enough to use next to a live market. There was no spec beyond that.
03One flat index per year, all scanned at once
- Timelines need every article scored, so I chose an exact flat search over a top-k engine: FAISS
IndexFlatIPon 1024-dimensional normalised vectors, with no cap on results. - Split the index by year. Each year sits in memory in its own Ray actor, and a query scans all years at once, so the wait is roughly one year's scan.
- Kept Elasticsearch for article metadata. Polars joins it to the scores and groups them into the timeline, served through FastAPI.
- Made it safe to update every minute: write-ahead log, atomic saves, hot reload and health checks.
04Under five seconds, in production
A timeline went from about a minute to under five seconds, and it's running in production. The same search powers a chat on the platform's price chart: pick a time range, ask what moved the price, and get an answer in under five seconds.