₿
After hours Real-time Trading Platform
A real-time platform where players bet on which price band BTC lands in at the end of short recurring rounds. Odds are parimutuel, rounds settle automatically, and everything streams live. The project where I taught myself distributed systems and performance engineering under real load.
What I used
.NET 10.NET AspirePostgreSQLRedisRabbitMQNext.jsWebSocketsk6Grafana / OpenTelemetryDockerEFCore
01
What I did
- Built a betting platform on .NET Aspire — separate microservices for the game engine, accounts, and price streaming.
- Stream simulated BTC prices and live parimutuel odds to the frontend over WebSockets (MessagePack).
- Implemented atomic money operations (refund-on-failure, idempotent payouts) and auto-settling rounds.
- More than doubled sustained throughput (~1.8k → 4k+ bets/s) at a fraction of the latency, load test by load test.
02
What I learned
- Performance engineering as a method — drive load → measure → find the bottleneck → fix → repeat.
- Reading distributed traces (Tempo) and metrics to locate the real bottleneck instead of guessing.
- Redis data modeling for the hot path — atomic Lua scripts, avoiding read-modify-write, hot-key awareness.
- .NET internals under load (thread pool, GC, async, connection multiplexing) — and that the measurement rig matters as much as the system.
03
Challenges
- A latency tail that turned out to be the load generator stealing CPU from the app on the same box — a methodology trap that cost real time.
- A single Redis connection plus per-round settlement (reading ~14.5k bets at once) head-of-line-blocking the hot bet path.
- Keeping money correct — no double-charge, idempotent payouts — under high concurrency and partial failures.
- Observability that caused its own incidents — a telemetry collector OOM froze the whole host; Prometheus silently stopped ingesting metrics.
04
Key optimizations
- Moved bets out of one growing JSON blob into per-key Redis structures with atomic HSET/HINCRBYFLOAT — placing a bet went from a read-modify-write CAS storm to O(1) (bet-engine 395% → 40% CPU, 6 GB → 370 MB RAM).
- Made balances Redis-authoritative with an atomic Lua debit and write-behind persistence to Postgres — the synchronous per-bet SQL UPDATE left the hot path.
- Replaced per-bet WebSocket broadcasts with a periodic projector (100 ms snapshots), which also let the bet write drop its read-back — the hot path became a single Redis round-trip.
- Rewrote the bet write as one atomic Lua script instead of WATCH/MULTI/EXEC, removing optimistic-lock contention under load.
- Cached column validation in memory, so placing a bet needs zero Redis reads to validate.
- Switched per-round settlement to a paged HSCAN instead of a single ~14.5k-row HGETALL that head-of-line-blocked the live bet path.
- Added admission control — a concurrency limiter shedding graceful 429s — so overload degrades cleanly instead of collapsing.
- Tuned Postgres (targeted indexes, synchronous_commit=off, batched per-user payouts) and pre-grew the .NET thread pool to flatten burst-latency tails.