Tensor Field Trading System: Bridging Rithmic API and External Strategies via PostgreSQL

Abstract

In the domain of high-frequency trading (HFT) and algorithmic execution, the decoupling of market data ingestion from strategy execution is a critical architectural pattern. This article details the programming logic required to construct a robust trading gateway using C# and the Rithmic .NET API. The system described utilizes a "Tensor Field" approach to Limit Order Book (LOB) analysis, calculating higher-order derivatives of price and quantity to detect institutional quote stuffing. Furthermore, it explores the integration of a PostgreSQL-based Publish-Subscribe (Pub/Sub) pattern to facilitate asynchronous communication between the core Rithmic API gateway and external client strategy programs. This architecture ensures that the heavy lifting of market data processing is isolated from the variable latency of strategy logic.

1. Architectural Overview

The proposed system operates on a distributed microservices architecture, conceptually divided into three distinct layers: the Market Gateway, the Data Bus, and the Strategy Client.

1. The Market Gateway (The Core): This is the C# application hosting the Rithmic API connection. Its primary responsibility is deterministic state management. It connects to the exchange, reconstructs the Limit Order Book (LOB) in real-time, computes complex mathematical derivatives (the "Tensor Field"), and manages the lifecycle of order execution.

2. The Data Bus (PostgreSQL): Acting as the central nervous system, PostgreSQL is utilized not merely for storage, but as a low-latency message broker using its NOTIFY and LISTEN command set. This allows for the broadcasting of complex market tensors to subscribers and the receipt of trading signals from external clients.

3. The Strategy Client: This represents external C# programs containing proprietary trading logic. These clients subscribe to the tensor data, apply their specific alpha models, and push execution instructions back to the Gateway via the Data Bus.

2. The Rithmic Connection Logic

The foundation of the Market Gateway is the REngine, the primary interface provided by the Rithmic API. The logic governing this connection is state-dependent and event-driven.

2.1 Initialization and Asynchronous Login

The program begins by instantiating the REngine with specific parameters defining the environment (e.g., Rithmic UAT or Production). A critical logical step here is the separation of callbacks. The system employs an AdmCallbacks class for administrative alerts and a RCallbacks class for market and trading data.

The login process is non-blocking. The logic must handle a state machine that tracks two distinct connection statuses: Market Data (MD) and Trading System (TS). The program initiates the login and then enters a wait state, polling flags set by the Alert callback. Only when both MD and TS connections are confirmed does the logic proceed to account retrieval.

2.2 Account and Trade Route Resolution

Once connected, the system queries for available accounts. The logic iterates through the received list, selecting the primary Futures Commission Merchant (FCM) and Introducing Broker (IB) identifiers. Crucially, the system must resolve a "Trade Route." The logic filters available routes based on the specific exchange (e.g., CME) and the account status, ensuring that orders are routed through a valid, active gateway.

2.3 Subscription Management

The subscription logic is granular. Rather than a blanket subscription, the system requests specific flags: Quotes (for the order book), Best (for Top of Book), Prints (for time and sales), and OpenInterest. This ensures the callback stream is populated with the necessary data points to construct the Tensor Field without overwhelming the network stack with irrelevant instrument data.

3. The Mathematical Core: Order Book Tensor Field Logic

The heart of the system is the TensorFieldProcessor. This component transforms raw market data into a structured, multi-dimensional representation of the market state, referred to as a "Tensor."

3.1 Limit Order Book (LOB) Reconstruction

Rithmic provides updates via BidQuote and AskQuote callbacks. The logic within the TensorCallbacks class must maintain a local cache of the order book. Since updates are incremental, the system uses a synchronization lock (_bookLock) to prevent race conditions between the incoming Rithmic thread and the processing thread.

The reconstruction logic sorts incoming bids in descending order and asks in ascending order. It truncates the book to a configurable depth (e.g., 10 levels) to focus computational resources on the most relevant liquidity. This snapshot is then encapsulated into a LOBTensor object.

3.2 Derivative Calculations

The "Tensor Field" concept relies on treating the order book as a dynamic surface changing over time. The processor maintains a rolling history of price and quantity data.

• First Derivative (Velocity): The logic calculates the rate of change of the total book quantity (). This metric helps identify liquidity rushing into or fleeing from the book.

• Second Derivative (Acceleration): The logic calculates the acceleration of the micro-price (). This identifies the curvature of price movement, often preceding a directional impulse.

• Open Interest Velocity: By tracking the rate of change in Open Interest, the system attempts to distinguish between new money entering the market versus the closing of existing positions.

3.3 The HFT Signal and Sigmoid Activation

To synthesize these raw metrics into a usable signal, the system employs a non-linear activation function. The logic combines the quantity derivative, price acceleration, and open interest flow using weighted coefficients ().

The sum of these weighted inputs is passed through a Sigmoid function:

This logic normalizes the volatile inputs into a bounded range between 0 and 1, providing a clean "HFT Signal" score for each price level. This score represents the probability of that price level being sustained or broken.

4. Signal Detection: Quote Stuffing and Fading

The system includes specific logic to detect manipulative market behavior, specifically "Quote Stuffing"—the rapid entry and cancellation of orders to create false liquidity.

4.1 Covariance Analysis

The detection logic utilizes a statistical approach. It extracts the top three levels of the order book and analyzes the relationship between Order Age and Bid Quantity.

The logic calculates the covariance between the duration an order has rested in the book and its size. In a healthy market, these variables might be uncorrelated or positively correlated. However, in a quote stuffing scenario, high-frequency algorithms often flash large quantities for very short durations. A high variance in order age combined with specific covariance thresholds triggers a "Quote Stuffing Detected" flag.

4.2 The Quote Fade Algorithm

Upon detecting quote stuffing, the system logic triggers a "Quote Fade" strategy. The premise is that the flashed liquidity is fake and will vanish (a "liquidity vacuum").

The logic identifies the best bid price and calculates a target entry price a specific number of ticks below that bid. The reasoning is that once the fake support is pulled, the price will drop to fill the vacuum. A QuoteFadeSignal object is generated, containing the target price, side, and the calculated covariance metric.

5. The PostgreSQL Pub/Sub Intermediary Logic

To decouple the Rithmic Gateway from external client strategies, the system employs PostgreSQL as a message broker. This section details the logic required to implement this pattern, which is not present in the provided source code but is essential for the requested architecture.

5.1 The Publication Logic (Gateway -> DB)

When the TensorFieldProcessor completes a calculation cycle, it triggers an OnTensorUpdate event. The logic handling this event must serialize the LOBTensor object into a JSON format.

The system connects to the PostgreSQL database and executes a SQL command:NOTIFY market_data_stream, 'payload'

Where 'payload' is the JSON representation of the tensor. This approach is superior to polling a table because it pushes data to listeners immediately. The logic must handle the serialization of complex objects, ensuring that arrays (like the price vector) and timestamps are formatted correctly for JSONB parsing by the client.

5.2 The Subscription Logic (Client -> DB -> Gateway)

The Gateway must also listen for instructions. The logic involves spawning a dedicated thread that maintains an open connection to PostgreSQL and executes the LISTEN order_entry_stream command.

This thread blocks, waiting for notifications. When an external client publishes a trade request, the Postgres driver interrupts the block. The Gateway logic then deserializes the incoming JSON payload into a TradeRequest object.

Crucially, this logic must be thread-safe. The notification arrives on a database thread, but the execution must interact with the REngine, which may have thread affinity or require synchronization. The logic passes the request to the ExecutionManager via a thread-safe queue or a dispatcher.

6. Client Strategy Communication Logic

The external client program mirrors the Gateway's logic but in reverse.

6.1 Ingesting Tensors

The client establishes a connection to PostgreSQL and executes LISTEN market_data_stream. Upon receiving a notification, the client logic deserializes the JSON back into a local LOBTensor object.

This allows the client to "see" the market exactly as the Gateway sees it, including the derived HFT signals and covariance metrics, without needing to perform the heavy mathematical lifting or manage the Rithmic API connection itself.

6.2 Strategy Execution and Latency

The client runs its own trading loops. For example, it might analyze the ImbalanceRatio from the tensor. If the imbalance exceeds a threshold, the client generates an order.

Instead of calling an API, the client logic serializes its intent (Symbol, Side, Quantity, Price) into JSON and executes:

NOTIFY order_entry_stream, 'order_payload'

The logic here must account for the "hop" latency. While PostgreSQL NOTIFY is fast (sub-millisecond on local networks), it is not instantaneous. The strategy logic must be robust enough to handle the time delta between the tensor timestamp and the moment the order reaches the Gateway.

7. Execution Management and Risk Logic

The ExecutionManager class acts as the gatekeeper between the generated signals (internal or external) and the exchange.

7.1 Validation and Limits

Before any order is sent to Rithmic, the logic performs strict validation.

1. Readiness Check: Verifies that the REngine, Account, and Trade Route are fully initialized.

2. Signal Validity: Checks the IsValid flag on the signal object.

3. Position Limits: The logic calculates the projected position. It adds the current position to the requested quantity. If the absolute value exceeds MAX_POSITION_SIZE, the order is rejected immediately. This is a critical safety mechanism to prevent runaway algorithms.

7.2 Rithmic Order Entry

If validation passes, the logic constructs a LimitOrderParams object. It maps the internal signal data to Rithmic constants (e.g., BUY_SELL_TYPE_SELL, ORDER_DURATION_IOC).

The logic uses ORDER_ENTRY_TYPE_AUTO to allow Rithmic to handle the specifics of the order routing. It also attaches the signal context to the UserMsg field, allowing for traceability in the audit logs. The sendOrder method is wrapped in a try-catch block to handle OMException errors (e.g., exchange disconnects, invalid prices).

7.3 State Tracking

The ExecutionManager listens to LineUpdate and FillReport callbacks.

• Line Update: When an order changes state (Pending -> Open -> Filled/Cancelled), the logic updates an internal dictionary _activeOrders.

• Fill Report: This is the source of truth for position keeping. The logic increments or decrements the _currentPosition counter based on the side of the fill. This local tracking is faster than querying the API for position updates, though it requires periodic reconciliation (e.g., via the PnlUpdate callback).

8. Concurrency and Thread Safety

The application operates in a highly concurrent environment. There are at least three distinct threads of execution active simultaneously:

1. The Rithmic Callback Thread: Delivers market data and order updates.

2. The Processing Thread: Performs the tensor math and signal generation.

3. The PostgreSQL Thread: Listens for incoming external commands.

8.1 Locking Strategies

The logic employs explicit locking (lock (object)) to protect shared resources.

• bookLock: Protects the currentBids and _currentAsks lists. Without this, the processor might attempt to read the book while a new quote is being inserted, leading to collection modification exceptions or corrupted data.

• _lock (ExecutionManager): Protects the position counter and the active order dictionary. This ensures that if a fill arrives exactly when a new signal is generated, the position check remains atomic and accurate.

8.2 Data Consistency

To ensure the Tensor represents a valid snapshot, the logic buffers updates. The EndQuote callback is utilized as a delimiter. The system accumulates individual BidQuote and AskQuote updates but only triggers the heavy ProcessOrderBook logic when the EndQuote signal indicates that the exchange has finished sending the current batch of updates.

9. Conclusion

The programming logic detailed above represents a sophisticated approach to institutional trading infrastructure. By combining the raw speed and data richness of the Rithmic .NET API with the analytical power of Tensor Field mathematics, the system can detect subtle market microstructure anomalies like quote stuffing. Furthermore, by integrating PostgreSQL as a Pub/Sub layer, the architecture achieves a clean separation of concerns. This allows for the development of lightweight, agile client strategies that can leverage the heavy-duty processing of the Gateway without being tightly coupled to the complexities of the exchange connectivity. This modular design enhances system stability, scalability, and ease of testing, meeting the rigorous demands of modern algorithmic trading.