10 The End-to-End Pipeline: Data to Live Trading

Author: Todd B. Adams Reinforces: Proposal §3 — High-Level Architecture · Reading order: chapter 02 of this part; follows the platform as a research substrate, precedes the multiplex data substrate.

What this chapter establishes. The instrument is not a research notebook that stops at a backtest. It is a production pipeline that runs each night from raw vendor data to a reconciled paper-broker order, under one run identity, one decision window, and no look-ahead. We argue that this production discipline — not any bespoke model — is what qualifies the platform to host and honestly evaluate the proposal’s network research.

10.1 The nightly pipeline

flowchart LR
    A["Acquire<br/>raw end-of-day data"] --> B["Validate<br/>clean & conform"]
    B --> C["Model<br/>point-in-time"]
    C --> D["Factors<br/>rank the names"]
    D --> E["Strategy<br/>apply a mechanic"]
    E --> F["Portfolio<br/>weights + risk caps"]
    F --> G["Paper order<br/>submit & reconcile"]

One run identity end-to-end; one nightly decision window; no future data is ever visible on a historical date. Adapted from the platform whitepaper.

Each stage is a tested subsystem with its own immutable output and its own dashboard view; the chain is what a nightly run actually does, not an aspiration. The apparatus notes below name the responsible software only to make the pipeline reproducible — the point of each row is the transformation, not the package.

Stage	What it does	Point-in-time integrity	Implemented?
Acquire	Ingest exact vendor payloads, preserve untouched (append-only); every fetch and every failure is recorded	audit-first, so the information set on any past date is reconstructable	implemented
Validate	Clean, type, dedupe, conform; restrict to an investable universe	—	implemented
Model	Star-schema analytical layer with strict point-in-time integrity — on any date, only what was knowable then is visible	the no-look-ahead guarantee the network tensors would inherit	implemented
Factors	Rank every name point-in-time; one row per date × instrument × factor	edge weights are the knowable-on-the-date values	implemented
Strategy	Apply a mechanic (e.g. “hold top-ranked for a month”) to a validated factor	—	implemented
Portfolio	Turn rankings into position sizes, compliant by construction (position, sector, and leverage caps applied before any order)	—	implemented
Paper order	Whole-share orders to an Interactive Brokers paper account; real fills at the real open; next-day reconciliation	—	implemented

Orchestration across all seven stages is a single flow under one run identity, so the whole night is one reproducible experiment. Supporting detail lives in the factor pipeline, the study population, and portfolio construction appendices; the analytical-layer contract is described in the platform whitepaper.

10.2 The live end is real, and honestly gated

The final stage is the one most quant research never reaches. This pipeline reaches it, and we report its status precisely:

Real broker fills, not a simulator. A pre-open submit-and-settle watch submits the target book at roughly 09:10 ET, then polls until every order settles, capturing real fills and marking positions same-day. The first live paper fills have been recorded — a diagnostic run booked 7 fills totalling 951 shares.
A fail-closed safety rail. Between deciding and submitting, a guard re-checks the intended book against its risk caps; any breach rejects the entire submission — no partial fill, no override.
Paper-only by dated decision, not by scope. The loop runs paper-only for now; the paper-to-live gate permits real capital only after an attributed paper soak of at least 63 market days. Live money is therefore gated behind survived forward evidence rather than out of scope — the crossover is scheduled, not yet taken, so this capability is partially built rather than complete.

10.3 Why this matters for the research

Three properties of the pipeline are exactly what the proposal’s network model needs from a host, and what a standalone graph-model codebase would have to build from scratch:

Point-in-time integrity end-to-end. A network-based early-warning signal is meaningful only if it never peeks at the future. The proposal’s supra-adjacency tensors would be built from the point-in-time analytical layer, so the proposed network layers inherit that guarantee without new engineering.
A terminal executability test. The proposal’s phase-transition signal is useful only if it can inform a tradeable book. The pipeline already turns a signal into cost-aware, capacity-aware, risk-capped, broker-submitted orders — so a network signal can be evaluated on net, executed profit-and-loss, not just in-sample fit.
Every run is an immutable experiment. Each night emits a run identity, per-stage structured sidecars, and a dashboard bundle (chapter 09). A network producer added to the flow becomes a first-class, reproducible experiment with the same audit trail as every existing stage.

The proposal’s data-ingestion engine and the platform’s Acquire→Model stages are the same idea at different maturities. Where the proposal’s three network layers map onto this substrate — one built, two proposed but with adjacent machinery — is chapter 03; where the network engine would mount onto the flow is chapter 06.

Cross-links: platform whitepaper · platform capability map · proposal §3