Lessons From a Quiet Night: What Our C++20 Trading Bots Taught Us About High-Frequency Trading
A forensic analysis of overnight execution logs from a live C++20 / Redis / Rithmic trading stack — July 23–24, 2026
---
## 1. Executive Summary
Between 22:10 on the evening of July 23 and roughly 11:00 the following morning, a fleet of nine C++20 strategy bots ran against a Rithmic market-data gateway fronted by a Redis pub/sub bus. The headline number is simple: across more than 15,000 log lines, the system recorded zero trades, zero fills, and zero realized profit and loss. Every heartbeat, in every bot, read `position:0, cumulative_pnl:0.000000`.
A casual reader might call the night a failure. That conclusion would be wrong. Buried inside the structured JSON logs is a rich, honest record of how a low-latency trading system actually behaves when it is healthy enough to ingest live market data but disciplined enough not to fire orders into a market that did not meet its entry criteria — and how it degrades, reconnects, and recovers when its data gateway stutters. For anyone building or operating a high-frequency trading (HFT) system, this overnight run is a goldmine of operational lessons.
This article dissects the logs line by line, reconstructs the timeline of what really happened, extracts the quantitatively meaningful signals hidden in the data, and translates them into concrete engineering and strategy recommendations for a production HFT platform. The analysis is grounded entirely in observed evidence: real timestamps, real prices, real volatility readings, and real failure modes.
The three most important takeaways are these. First, a night with no trades can still be a successful night if the system's risk gates are doing their job; the absence of P&L is not the absence of information. Second, the single largest source of alpha leakage in this stack is not strategy logic — it is data-path reliability, and the logs prove it with timestamped precision. Third, the bots are already emitting the exact telemetry needed to build a world-class monitoring and auto-tuning loop; the opportunity is to start consuming that telemetry systematically rather than letting it accumulate unexamined.
---
## 2. The System Under Test
To interpret the logs correctly, it helps to understand the architecture that produced them. The platform is a ground-up C++20 rewrite of a legacy Java trading system, designed around ultra-low-latency principles: cache-line-aligned data structures, zero-allocation hot paths, lock-free inter-thread communication, and `consteval`/`constexpr` compile-time computation wherever possible. The market-data path follows a classic decoupled design:
- A Rithmic gateway process connects to the exchange-grade Rithmic feed and normalizes raw ticks.
- The gateway publishes normalized ticks onto Redis pub/sub channels of the form `rithmic:md:<SYMBOL>-<strategy_name>`.
- Each strategy bot is an independent native executable (`CPP_REDIS_GATEWAY` mode) that subscribes to its own channel, maintains internal state, and applies its entry/exit logic tick by tick.
- Bots emit structured JSON log events for every lifecycle milestone, market-data batch, diagnostic snapshot, and heartbeat.
Nine bots ran during the window, spanning a deliberately diversified set of asset classes and strategy families: gold momentum (COMEX GCM6), gold-versus-real-yields stagflation, crude-oil geopolitical momentum (NYMEX CL), a Bitcoin futures regulatory hedge (CME BTC/BTCN6), a natural-gas LNG bull spread, a yen BoJ-intervention play, a copper AI-demand call-spread with a gold hedge (COMEX HG/HGN6), an NQ vol-reversion strategy, and an ES AI-rotation hedge. This breadth is itself a strength: it lets us compare how the same data infrastructure behaves across instruments with very different liquidity profiles and tick cadences.
Each bot carried a per-day loss limit (for example, -$12,500 for the gold and crude bots, -$5,000 for BTC and copper) and a circuit-breaker flag that remained inactive all night — an early hint that no bot came close to a risk breach.
---
## 3. Reconstructing the Night: A Timeline
The logs allow a precise reconstruction of the session.
22:10 — The data starts flowing. The Gold Safe-Haven Momentum bot, subscribed to GCM6, logs its first `MARKET_DATA` event at 22:10:39 with a bid of 4047.6. Within a minute it is emitting `DIAGNOSTICS` snapshots showing a mid price of 4047.8 and a realized volatility reading (`iv`/`hv`) near 0.00068. This is the moment the Redis gateway came alive for gold.
22:10–23:30 — Steady, moderate cadence. Market-data lines arrive in bursts. The tick buckets tell the story: 3 batches in the 22:00 half-hour, 2 in 22:30, 1 in 23:00, 1 in 23:30. These are aggregated batches — each `MARKET_DATA` line carries a `tick_count` field that climbs rapidly (2840 by 22:47), so a small number of log lines represents a large number of underlying ticks. The gold price drifts gently lower: 4047.6 → 4037.4 → 4032.8 across the first seventy minutes.
00:30–01:40 — The overnight session peaks, then the gateway fades. Activity picks up after midnight (8 batches in the 00:30 bucket), with GCM6 trading between roughly 4026.5 and 4032.7. At 01:17 the `tick_count` field spikes to 18,260, then 20,980 by 01:20 — clear evidence of a heavy burst of underlying ticks being delivered in a short window. The final gold `MARKET_DATA` line is stamped 01:40:58 with an ask of 4028.1 and `tick_count` frozen at 6330 in the cumulative heartbeat view. After that, gold goes quiet on the market-data front even though heartbeats continue.
00:57 — A second wave of bots comes online. The Bitcoin regulatory-hedge bot and the crude-oil bot both emit `BOT_START` at 00:57, within thirteen seconds of each other, strongly suggesting a coordinated (likely scripted or manager-driven) relaunch. BTC begins receiving data almost immediately (first tick 00:57:46, ask 65350). Crude oil, by contrast, records zero `MARKET_DATA` events for the entire session — a stark and important anomaly we will return to.
01:34–01:36 — Symbol resolution kicks in. At 01:34:57 the BTC bot logs a second `BOT_START`, this time with `symbol:"BTC", resolved:"BTCN6"` — the front-month contract resolution layer had come online or refreshed. The copper bot starts at 01:36:48 already resolved to HGN6. This is a meaningful operational detail: the bots are not merely connecting to a raw symbol; a contract-resolution step maps the logical symbol to the tradable front month, and that step happened at different times for different instruments.
01:36 onward — Copper joins, the rest stay dark. The copper bot (HGN6) receives 338 market-data events from 01:36:51 through the following morning, with the bid rising from 6.2965 to an ask of 6.3295 — a modest overnight drift in copper. Meanwhile the natural-gas, yen, NQ, and ES bots log heartbeats and `NO_DATA` / `GATEWAY_WAIT` events but never a single tick. The ES bot is still emitting `GATEWAY_WAIT … gateway may be offline` as late as 11:04 the next morning.
---
## 4. What the Data Actually Shows
Beneath the timeline, several quantitative patterns deserve close attention, because each one maps to a concrete HFT lesson.
### 4.1 One-sided quotes are the norm overnight
Almost every `MARKET_DATA` payload shows exactly one side of the book populated and the other side at zero: `"bid":4047.6,"ask":0` or `"bid":0,"ask":65350`. The mid-price diagnostics, meanwhile, report `spread_pct:0` and `bid == ask == price`. This is characteristic of overnight, thin-liquidity sessions where the top of book is frequently one-sided, and it has two implications.
First, any strategy that requires a two-sided quote to compute a fair mid or a spread will see a degenerate (zero-width) book for much of the night. Second, the strategy code is clearly tolerant of this — it synthesizes a price from whichever side is present rather than rejecting the tick. That is the correct defensive behavior, but it also means the "price" the strategy reasons about overnight is really the last available single-sided quote, not a true two-sided market. For HFT, the lesson is that entry logic must explicitly account for quote completeness. A momentum signal computed on a one-sided, possibly stale bid is not the same signal computed on a tight two-sided market, and the system should tag such ticks with a quality flag rather than treating them as equivalent.
### 4.2 Realized volatility differs by an order of magnitude across instruments
The `DIAGNOSTICS` events embed a realized-volatility estimate (`iv`/`hv`) computed from the tick stream, plus an `iv_percentile` that situates the current reading within its recent history. The contrast between instruments is striking:
- Gold (GCM6): realized vol around 0.0006–0.0008 (i.e. 0.06–0.08%), with `iv_percentile` oscillating between 0.16 and 0.84 — a quiet, mid-range vol regime.
- Bitcoin (BTCN6): realized vol around 0.0063 (0.63%) — roughly nine to ten times the gold reading — sitting at the 75th percentile.
This is precisely the kind of cross-asset vol surface an HFT system should exploit. A fixed-threshold momentum strategy that works on gold will be systematically under-triggered on Bitcoin and over-triggered if the same threshold is applied naively in the other direction. The data argues strongly for vol-normalized entry signals: express entry thresholds in units of current realized volatility (e.g. "enter when price moves more than 1.5σ from its short-horizon mean") rather than in absolute price ticks. The bots are already computing the vol; the opportunity is to wire it directly into the signal.
### 4.3 The tick_count resets reveal silent reconnects
One of the most valuable forensic details is the behavior of the cumulative `tick_count`. In gold, the `MARKET_DATA`-level `tick_count` climbs to 20,980 by 01:20, yet the heartbeat-level `ticks` field reports 6,339, and the gold `DIAGNOSTICS` stream shows its `ticks` counter resetting to 1 at 22:37:59, 22:46:50, and again within the BTC log at 01:34:57. A counter that resets to 1 while the process keeps running is a fingerprint of an internal resubscription or Redis reconnect: the bot dropped its gateway subscription, re-established it, and restarted its per-connection tick counter.
For an HFT platform this is a critical observability win and a warning at once. The win: the reconnect is graceful and automatic — the bot never crashes, it simply resumes. The warning: every reconnect represents a data gap, a window in which ticks were published to Redis but not consumed by this subscriber. Because Redis pub/sub is fire-and-forget (no replay, no persistence by default), any tick emitted during a disconnect is permanently lost to that bot. A strategy holding an open position through such a gap would be flying blind on stale state. The logs make the case for two concrete improvements: sequence numbers on every tick so gaps are detectable, and a monotonic, never-resetting tick counter so operators can distinguish "few ticks because the market was quiet" from "few ticks because we were disconnected."
### 4.4 The crude-oil anomaly: subscribed, alive, and fed nothing
The crude-oil bot is the night's most instructive failure. It connects to Redis successfully (`REDIS_CONNECT status OK`), subscribes to its channel, logs `BOT_STARTED … listening for market data`, emits a healthy `DIAGNOSTICS` snapshot at 00:57:28 showing a price of 84.99 — and then records zero market-data events for the rest of the session, while logging only a single `GATEWAY_WAIT` and no `NO_DATA` flood.
This is the classic "silent starvation" failure mode, and it is more dangerous than an honest crash. The process is alive, its heartbeat is presumably ticking, and nothing in its own log screams "error" — yet it is receiving no data. The most likely causes are upstream: the Rithmic gateway was not publishing CL ticks at all (perhaps because CL was not subscribed at the gateway, the instrument wasn't permissioned, or the front-month resolution for CL failed silently at the gateway level), or the channel name the bot subscribed to did not match the channel the gateway published to. The gold and BTC bots prove the bus itself worked; the failure is instrument-specific. The HFT lesson is that liveness is not the same as data-flow health. Every bot needs a data-flow watchdog: if no tick has arrived within N seconds during a session when the instrument should be trading, raise an explicit alert rather than waiting for a human to notice a quiet log.
### 4.5 A fleet of healthy bots that simply never saw data
ES, NQ, natural gas, and yen never received a single tick. Their logs are dominated by `GATEWAY_WAIT ("gateway may be offline; will keep retrying subscribe")` and `NO_DATA` heartbeats. This is not a strategy problem and not a code bug — it is an infrastructure availability problem. The Rithmic gateway, the single point through which all market data flows, was either down, partially subscribed, or not publishing those instruments for much of the night. Four of nine bots — nearly half the fleet — were effectively deaf.
This is the single most important finding of the analysis, and it reframes the entire night. The binding constraint on this system's overnight performance was not signal quality, not latency, and not risk management. It was the uptime and coverage of one market-data gateway. In HFT terms, the system had a single point of failure on its most critical input. No amount of cache-line alignment or lock-free queue optimization inside the strategy process can compensate for a feed that is not delivering data.
---
## 5. The Overarching Lesson: Zero Trades Was the Right Outcome
It is worth stating plainly, because it is the most counterintuitive and most important conclusion: the system did the right thing by not trading.
Consider the conditions. Half the fleet received no data at all and correctly refused to trade blind. The instruments that did receive data traded in thin, one-sided, low-participation overnight conditions. The strategy bots maintain entry logic that evidently did not find a sufficiently strong, sufficiently confirmed signal to justify committing capital — and crucially, the risk infrastructure (daily loss limits, circuit breakers, per-bot simulated position tracking) was armed and monitoring throughout, even though it was never breached.
An HFT system that fires orders merely because it is running is not disciplined; it is dangerous. The absence of trades, in a session characterized by degraded data and thin liquidity, is evidence that the gating logic works. The right way to read `cumulative_pnl:0.000000` is not "the bots made nothing" but "the bots lost nothing, took no unforced risk, and stood ready." In a domain where a single uncontrolled order can erase weeks of gains, capital preservation during a low-quality session is a feature, not a bug. The system passed the most fundamental test of any risk-managed trading platform: it did no harm.
That said, "correctly idle" and "optimally idle" are different things, and the logs point to several ways the system could convert future quiet nights into actual edge.
---
## 6. Actionable Opportunities for the HFT System
The analysis converges on a prioritized set of improvements, ordered by expected impact on risk-adjusted return.
1. Eliminate the single point of failure on market data. This is the highest-leverage change. Run a redundant Rithmic gateway (active/standby or active/active with deduplication), add a second independent feed source for the most critical instruments, and implement automatic failover so that a gateway restart does not leave four bots deaf. Instrument the gateway itself with the same structured telemetry the bots already emit, so its health is as observable as the strategies that depend on it.
2. Add sequence numbers and gap detection to every tick. Stamp each published tick with a monotonically increasing per-instrument sequence number at the gateway. Let each bot detect gaps on receipt and log an explicit `DATA_GAP` event with the count of missed ticks. This converts silent data loss into a measurable, alertable quantity and is a prerequisite for trusting any signal computed on the stream.
3. Build a data-flow watchdog per instrument. For each subscribed symbol, track time-since-last-tick against an instrument-aware expectation (overnight gold trades; a completely silent CL during its session is anomalous). Fire a `DATA_STARVATION` alert when the threshold is exceeded. This directly addresses the crude-oil failure mode.
4. Move to vol-normalized, quality-aware entry signals. Consume the already-computed realized-volatility and `iv_percentile` fields inside the signal, expressing entry thresholds in σ units. Tag each tick with a completeness/quality flag (two-sided vs one-sided, fresh vs stale) and weight or gate signals accordingly. This aligns the trigger sensitivity with each instrument's actual overnight behavior.
5. Make tick counters monotonic and add reconnect metrics. Never reset a tick counter on resubscribe; instead, keep a cumulative counter plus a separate `reconnect_count` and `last_reconnect_ts`. This makes disconnect frequency and recency first-class observability data.
6. Persist and mine the diagnostic telemetry. The `DIAGNOSTICS` events are a ready-made dataset: vol regime, iv percentile, loss-limit headroom, circuit-breaker state, all timestamped. Persisting these to a time-series store and analyzing them across sessions would let the team auto-tune thresholds, detect regime shifts, and quantify how often each bot was "close to trading" — turning the current binary traded/didn't-trade view into a continuous measure of near-misses.
7. Record and analyze near-misses explicitly. When entry logic evaluates a signal and declines to trade, log the reason and the margin by which it declined (e.g. `SIGNAL_REJECTED, reason:"below_vol_threshold", value:1.2, required:1.5`). This is the single richest source of strategy-improvement signal available, and it costs almost nothing to emit.
---
## 7. Conclusion
The overnight run of July 23–24 produced no trades and no P&L, and it would be easy to file it away as uneventful. In reality it was one of the more informative sessions the system could have produced, precisely because it stress-tested the parts of the platform that matter most when nothing goes according to plan.
The logs demonstrate a C++20 stack that is architecturally sound: bots that reconnect gracefully, risk gates that hold, telemetry that is structured and rich, and a diversified strategy fleet behaving consistently across asset classes. They also expose the system's true bottleneck with unusual clarity — not compute, not latency, not strategy sophistication, but the reliability and coverage of a single market-data gateway that left half the fleet without a tick to chew on.
For a high-frequency trading operation, that is the most valuable kind of lesson, because it redirects engineering effort toward the constraint that actually binds. The fastest order path in the world is worthless if the data feeding it is absent, stale, or silently gapped. The fixes that emerge from this analysis — redundant feeds, sequence-number gap detection, per-instrument starvation watchdogs, vol-normalized quality-aware signals, and systematic mining of near-miss telemetry — are not exotic. They are the well-established foundations of every mature HFT platform. What this quiet night provided is the evidence, stamped in JSON and signed with real timestamps, that these are exactly the right places to invest next.
The bots did not trade last night. But they told us, in precise and actionable detail, how to make sure that when the right conditions arrive, the system will be ready to trade — and to do so faster, safer, and with more confidence than it could have before.