16 Reproducibility & Experiment Infrastructure
Author: Todd B. Adams Reinforces: Proposal §3 — pipeline & evaluation · Reading order: doc 09 of the research-platform package
The claim of this doc. Doctoral research is judged partly on reproducibility. On SBFoundation every run is an immutable, identity-stamped, fully audited experiment — not a notebook re-run whose state is lost. A network producer added to the platform inherits this audit trail automatically, so its results are reproducible by construction.
16.1 Every run is an experiment record — BUILT
| Property | What it gives the research | Where |
|---|---|---|
One run_id per run (YYMMDD_XXXXXX) |
A single identity stamped end-to-end; every artifact traces to one run | run analysis |
| Immutable inputs/outputs | A run’s data and results do not change after the fact | architecture |
| Per-stage JSON sidecars | Each producer emits a structured, versioned result blob (schema-versioned) | run-type task reference |
| Harvested run bundle | All sidecars rolled into one run_<id>.json the dashboard reads |
run analysis |
| Reharvest | Re-derive a past run’s bundle against current Gold — reproduce a result on demand | run analysis |
| Operator dashboard (SPA) | Every run, stage, factor, and signal browsable | SPA components |
| Run modes | NIGHTLY (monitoring) vs NIGHTLY_FULL / RESEARCH_DAY (deep evidence) |
run-type task reference |
16.2 Provenance, not just results
The platform records what it received and when, not only what it concluded:
- Audit-first ingestion. Every vendor fetch — including failures — is recorded in Bronze, so “what did we actually have on date X?” is always answerable (Bronze contract). For a phase-transition study, being able to reconstruct the exact information set available on a historical date is essential and non-negotiable.
- Point-in-time knowledge dates. Fundamentals and slow-cadence factors are snapped to the market day their information became knowable, with the leak audited against SEC EDGAR ground truth (EDGAR PIT findings).
- Schema-versioned sidecars. Producers version their output so a downstream reader degrades gracefully across runs of different vintage — the network producer would do the same.
16.3 A disciplined change process behind every producer
The platform’s own SDLC (project-sdlc) is itself part of the reproducibility story: features are specified as design briefs, decomposed into tracked backlog tasks, built test-first, and validated through tiered gates before merge. A research increment (doc 07) enters through the same process, so the code that produces a published result is reviewed, tested, and traceable to a specification — not a one-off script.
16.4 What this means for the proposal
The proposal’s evaluation engine (§3, subsystem 4) needs to compare a network signal against realized stress across many historical dates and re-run cleanly as the model changes. On this platform that comparison is a producer whose every run is identity-stamped, sidecar-recorded, and reharvestable — so a reviewer (or a future you) can reproduce any reported number, and the model’s evolution is a legible history rather than a lost sequence of notebook states.
Cross-links: run analysis · project SDLC · SPA components · proposal §3