9  The End-to-End Pipeline: Data to Live Trading

Author: Todd B. Adams Reinforces: Proposal §3 — High-Level Architecture · Reading order: doc 02 of the research-platform package

The claim of this doc. SBFoundation is not a research notebook that stops at a backtest. It is a production assembly line that runs every night from raw vendor data to a reconciled paper-broker order, under one run identity, one decision window, and no look-ahead. That production discipline — not a bespoke model — is what qualifies the platform to host and honestly evaluate the proposal’s network research.

9.1 The nightly assembly line — BUILT

flowchart LR
    A["Acquire<br/>raw end-of-day data"] --> B["Validate<br/>clean & conform"]
    B --> C["Model<br/>point-in-time"]
    C --> D["Factors<br/>rank the names"]
    D --> E["Strategy<br/>apply a mechanic"]
    E --> F["Portfolio<br/>weights + risk caps"]
    F --> G["Paper order<br/>submit & reconcile"]

One run identity end-to-end; one nightly decision window; no future data is ever visible on a historical date. Adapted from the platform whitepaper §4.

Each stage is a real, tested subsystem with its own package, its own immutable output, and its own dashboard view. The chain is not aspirational — it is what a NIGHTLY run does.

Stage What it does Package(s) Evidence / detail Status
Acquire Ingest exact vendor payloads, preserve untouched (append-only Bronze); every fetch and every failure recorded sbbronze Bronze contracts, dataset reference BUILT
Validate Clean, type, dedupe, conform; restrict to an investable universe sbsilver, sbuniverse pipeline steps, universe API BUILT
Model Star-schema analytics layer with strict point-in-time integrity — on any date you see only what was knowable then sbgold DuckDB storage contract BUILT
Factors Rank every name point-in-time; one row per date × instrument × factor sbfactors factors, factor lifecycle BUILT
Strategy Apply a mechanic (e.g. “hold top-ranked for a month”) to a validated factor sbcontracts, sbportfolio portfolio construction BUILT
Portfolio Turn rankings into position sizes, compliant by construction (position/sector/leverage caps baked in before any order) sbportfolio, sbriskmodel, sballocation portfolio construction BUILT
Paper order Whole-share orders to an Interactive Brokers paper account; real fills at the real open; next-day reconciliation sbexecution execution reference BUILT

Orchestration across all seven stages is a single Prefect flow with one run_id — see the run-type task reference and prefect task reference.

9.2 The “live” end is real, and honestly gated — BUILT / PARTIAL

The final stage is the one most quant research never reaches. SBFoundation reaches it:

  • Real broker fills, not a simulator. A pre-open submit-and-settle watch submits the target book to IB at ~09:10 ET, then polls until every order settles, capturing real fills and marking positions same-day (execution reference). The first live paper fills are recorded — a real run booked 7 fills / 951 shares.
  • A fail-closed safety rail. Between “decide” and “submit,” a guard re-checks the intended book against its risk caps; any breach rejects the entire submission — no partial, no override.
  • Paper-only, by dated decision — not “never.” The loop runs paper-only through the current phase; the lifecycle gate only permits live capital after a ≥ 63 market-day attributed paper soak. Live money is gated behind survived evidence, not out of scope. PARTIAL because the crossover to real capital is scheduled, not yet taken.

9.3 Why this matters for the research

Three properties of the pipeline are exactly what the proposal’s network model needs from a host — and what a standalone GNN codebase would have to build from scratch:

  1. Point-in-time integrity end-to-end. A network-based early-warning signal is only meaningful if it never peeks at the future. The proposal’s Supra-Adjacency tensors (factor loadings, dynamic edge weights) would be built from Gold, which is already engineered so a historical date sees only what was knowable then. PROPOSED network layers inherit this guarantee for free.
  2. A terminal executability test. The proposal’s phase-transition signal is only useful if it can inform a tradeable book. The platform already turns a signal into cost-aware, capacity-aware, risk-capped, broker-submitted orders — so a network signal can be evaluated on the metric that matters (net, executed P&L), not just in-sample fit.
  3. Every run is an immutable experiment. Each night emits a run_id, per-stage JSON sidecars, and a dashboard bundle (doc 09). A network producer added to the flow becomes a first-class, reproducible experiment with the same audit trail as every existing stage.

The proposal’s data-ingestion engine and the platform’s Acquire→Model stages are the same idea at different maturities. Where the proposal’s three network layers map onto this substrate — one built, two gaps with adjacent machinery — is the subject of doc 03; where the network engine mounts onto the flow is doc 06.

Cross-links: platform whitepaper · business capability map · architecture packages