Factor Library

This chapter is generated from the canonical platform doc, which lives in docs/ (single source of truth).

Read all five sidecars for run 260526_6ddcd9. Below is what each page shows, what your data actually says, and how that maps to the experimental → validated → deprecated lifecycle in ops.factor.

1. Factor Diagnostics — `compute_factor_diagnostics_260526_6ddcd9.json`

What it shows (per ResearchFactorDiagnosticsView.js): - Quadrant scatter: x = -log10(ADF p-value), y = Shannon entropy (bits). Thresholds at p<0.05 and H≥3. - Three sortable tables split by cadence (daily / quarterly / annual).

Your 79 factors (all currently factor_status: experimental):

Quadrant	Count	Read
Healthy (stationary + H≥3)	7	`beta`, `market_beta_252d`, `ma_50d`, `residual_momentum_252d_eod`, `rsi_14d`, `amihud`, `bb_pct`
Thin (stationary, low entropy)	32	Mostly the quarterly fundamentals (`net_margin`, `gross_margin_q`, `momentum_12_1` variants, etc.) — they pass ADF but the values are nearly degenerate over the cross-section
Drift (non-stationary, high entropy)	4	`pct_from_52w_high`, `vwap_20d`, `adv_20d`, `momentum_6m` — trending series, not statistically stationary
Excluded (non-stationary + low entropy)	36	Most annual factors — 10–12 observations is too few for ADF to be reliable at all

Annual factors (n_obs=10–12) and quarterly factors (n_obs=36) are too short for ADF to be informative. The ADF column for those rows should be treated as decorative, not decisive.

Yes — and the right answer is mostly not “add data.” Here’s why, and what actually moves the needle.

Where the constraint actually is

Looked at factor_diagnostics_service.py: the diagnostic computes a daily cross-sectional mean of the factor across the allowlist, then runs ADF + entropy on that single time series. Per-cadence lookback already adapts:

Cadence	Lookback	min n_obs
daily	252 days	60
quarterly	3650 days (~10 yr)	8
annual	4380 days (~12 yr)	5

So the test is ingesting all the history that exists. The 10–12 obs you see for annual factors is the full Strawberry history — extending the lookback alone won’t help until more data is in Bronze.

Options, ranked by effort × payoff

1. (Best) Stop using ADF on cross-sectional means for slow-cadence factors

ADF on a 12-point series has essentially no power — the test was designed for T≥50. For stock-picking factors what actually matters is cross-sectional rank stability and dispersion, not whether the grand mean of ROIC across the universe is mean-reverting (it never is — fundamentals trend with the economy). Replace or supplement with:

Cross-sectional dispersion stability — for each period, compute IQR (or std) of the factor across instruments; then test whether that dispersion series is stationary. This measures “is the factor still discriminating?” which is the real question.
Rank persistence — Spearman ρ between instrument ranks at t and t+1. A factor that ranks the same names quarter-after-quarter is informative regardless of level non-stationarity.
Half-life from AR(1) — fit ϕ on the cross-sectional mean and report -log(2)/log(|ϕ|). Works with N=10 (won’t be precise, but won’t be meaningless either).

This is a code change in _diagnostics_math.py and the sidecar schema — no new ingestion.

2. Test the change, not the level

Half your fundamental factors are non-stationary in levels by construction but stationary in changes:

roic ADF p=0.97; ΔROIC almost certainly passes.
roa, roe, gross_margin_moat, incremental_roic — same pattern.

If the downstream signal uses YoY change anyway (revenue_growth_yoy, earnings_growth_yoy already do — and they pass ADF cleanly), the diagnostic should match. For each fundamental factor declare in config/factors/<id>.yaml whether the diagnostic should run on level or first_difference; default to first_difference for ratios and margins. Cheap, mechanical fix.

3. Switch to panel unit-root for cadences with low T

Since the service already pools across the universe, you already have what panel tests need: short-T, large-N. Im-Pesaran-Shin (IPS) or Pesaran’s CIPS are the standard tools for T=10, N=3000 panels. Statsmodels doesn’t ship them but arch or linearmodels does (or ~80 lines hand-rolled). Far more power than per-series ADF, no new data needed.

Implemented in F-134 (docs/backlog/feature-panel-unit-root-diagnostic-design-brief.md) — Pesaran (2007) CIPS, hand-rolled in src/sbdiag/services/_diagnostics_math.py, case II (intercept-only), p_lags=0. Runs on quarterly + annual factors with N ≥ 100 instruments and T ≥ 8/5 observations; daily factors are routed to panel_test_method='skipped_daily' by design. The v3 sidecar surfaces panel_test_method, panel_test_statistic, panel_test_pvalue, panel_test_n_instruments, panel_test_n_dates per item plus an n_panel_tested envelope tally; no third-party dependency added (neither linearmodels nor arch ships CIPS).

4. Emit `insufficient_history` instead of a low-power p-value

Today, an annual factor with n_obs=12 returns is_stationary: false (because p=0.4 is above 0.05) — but false here is meaningless; ADF just couldn’t reject. The honest output is a fourth state alongside ok/error/insufficient: low_power, gated on per-cadence n_obs thresholds (e.g. ADF needs ≥50 obs, KPSS ≥30, AR(1) half-life ≥10). The current status: ok quadrant scatter is overconfident — those 36 “excluded” annual factors aren’t necessarily bad, they’re just untestable.

5. (Only if you really want more data) Backfill fundamentals to 25–30 years

FMP’s annual/quarterly endpoints typically go back to ~1995 for liquid US large/mid caps, ~2000 for the broader Russell 3000. You’d gain:

Annual: 12 → ~30 obs (enough for ADF to have power)
Quarterly: 36 → ~120 obs (already healthy, would become very healthy)
Cost: ~3× the fundamental Bronze footprint, one-shot historical ingest via the CLI subcommands python -m sborchestration.cli backfill-annual <start> <end> / backfill-quarter <start> <end> (Bronze-only). There is no AnnualBronzeIngester.__main__ block — that mechanism was corrected by F-140; see CLAUDE.md §13.

Worth doing eventually, but it doesn’t fix the diagnostic — it just gets the bad diagnostic to a workable n. Doing 1–4 first is higher leverage and cheaper. Note the backfill alone is inert unless sbdiag.settings.ANNUAL_LOOKBACK_DAYS / QUARTERLY_LOOKBACK_DAYS are also widened (both 11000 ≈ 30 yr after F-140), or the diagnostic trailing window caps n_obs regardless of Gold depth. This is exactly what F-140 (milestone m-113) scopes.

What I’d actually do

Two-PR sequence, both small:

PR-1 (no data): add diagnostic_transform: level | first_difference to config/factors/*.yaml (default first_difference for ratio/margin factors); add cross-sectional dispersion + AR(1)-half-life + rank-persistence metrics to the sidecar; gate status='low_power' per cadence. This alone will reclassify ~30 of your 36 “excluded” annual factors into a usable diagnostic state.
PR-2 (only if PR-1 leaves gaps): implement IPS panel unit-root in _diagnostics_math.py, surface alongside the per-series ADF for quarterly/annual factors. Borrows strength from the cross-section instead of pretending T=12 is enough. Landed as F-134 with Pesaran CIPS rather than IPS — CIPS handles cross-sectional dependence (driven by common market shocks) which IPS does not, see F-134 brief §1 for the discussion.

Backfilling fundamentals is a separate, larger task — only worth it once you’ve exhausted the analytical fixes above.

2. Per-Factor IC — `compute_ic_260526_6ddcd9.json`

What it shows (per ResearchIcView.js): matrix of 79 rows × 5 horizon columns (1/5/10/21/63 day forward returns). Each cell colour-coded by |IC_IR|: green ≥0.5, amber 0.2–0.5, red <0.2. Right-edge bar = Fisher-z composite (single value per factor — same across the row by design).

Promotion horizon is 21d. Standouts at h=21d:

Group	Factor	mean_IC@21d	IC_IR@21d	Sign
Momentum (passes)	`residual_momentum_252d_eod`	+0.068	+0.724	+
Momentum (passes)	`momentum_12_1` / `momentum_12m_1m` (duplicates)	+0.063	+0.586	+
Momentum (passes)	`momentum_12_1_vol_scaled`	+0.061	+0.629	+
Value (passes)	`value_composite`	+0.090	+0.750	+
Quality / earnings (weak‑moderate)	`internal_forecast_eps`, `operating_margin_q`, `pct_from_52w_high`	0.07–0.13	0.30–0.52	+
Low‑vol / size (inverted)	`volatility_30/60/126/252d`, `market_cap`, `amihud`, `pe_ratio`	‑0.07 to ‑0.13	‑0.5 to ‑0.7	−
Sign‑flip with horizon	`piotroski`, `earnings_yield`, `roa`, `fcf_yield`	neg at h=1, pos at h≥21	n/a	depends on horizon

The sign‑flip group is the classic value‑investor pattern (wrong direction intraday, right direction at month+). The high‑magnitude negative IRs (vol, size) are real signal — the raw factors are anti‑predictive, so a strategy would trade them inverted.

Two things to flag: - beta and market_beta_252d are mathematically identical (ADF stat, IC, entropy all match to 14 digits). One is a stale alias — delete it. - momentum_12_1 and momentum_12m_1m are identical for the same reason. - market_beta_252d only has horizons (21, 63, 126) while everything else has (1, 5, 10, 21, 63). This is a config divergence in config/factors/market_beta_252d.yaml worth fixing.

3. Factor Contribution — `factor_contribution_260526_6ddcd9.json`

What it shows (per ResearchFactorContributionView.js): XGBoost gain / weight / cover importances of the source factors that feed each composite. Only meaningful for composites.

Your data on this run: 79 of 80 items are status: single_input (atomic — nothing to attribute). Exactly one composite produced contribution rows: residual_momentum_252d (2 inputs):

Input	Gain	Weight	Cover
`momentum_12m_1m`	4.99e+29	1516	2.1M
`market_beta_252d`	2.10e+29	1146	4.4M

Reframed in F-138 (fixed in feature-composite-factor-library-audit-design-brief.md, milestone m-111):

The composite library is structurally empty by design — not by a classification bug. The four named “composites” (value_composite, quality_composite, qmj_composite, momentum_12_1_vol_scaled) are correctly declared kind: atomic by design in config/factors/*.yaml. They are computed upstream in src/sbfactors/fundamental/value.py and siblings as single z-score blend columns and lifted via FactorValueLiftService as one factor_id each. They will always land as single_input in this sidecar until a future feature builds FactorCompositionService (promised in CLAUDE.md §1).
residual_momentum_252d is the only true composite — and it is deprecated per config/factors/residual_momentum_252d.yaml:23 (B-F-108.1 / TASK-907). F-138 routes deprecated factors to a new status='deprecated_skipped' sidecar item with zero rows in ops.research_factor_contribution; the SPA renders a third collapsible “Deprecated (skipped)” section. Post-F-138, this composite no longer pollutes the matrix.
The 4.99e+29 gain magnitudes are fixed at the booster, not at the data. F-138 z-scores fwd_return to unit variance, passes base_score=0.0, and pins reg_alpha=0.1 + reg_lambda=1.0 on the XGBRegressor. Gain magnitudes are now in standardised-return units (typically O(0.01)–O(1)); the relative ordering of source factors within a composite is preserved. Sidecar schema_version bumped 1 → 2 with a new top-level envelope key y_standardized: true so consumers can interpret the rescaled units.

4. Alphalens — `write_alphalens_tearsheets_260526_6ddcd9.json`

What it shows (per ResearchAlphalensView.js): mean return by quantile (typically 5 quintiles), top‑minus‑bottom spread, top/bottom quantile IC, and turnover per factor × horizon. Tearsheet HTMLs sit next to the sidecar.

Status: 79 × 5 = 1965 quantile rows, n_skipped=0.

Fixed in F-137 (sidecar schema_version 2). Forward returns are now winsorized at (0.01, 0.99) per‑date cross‑section before quantile aggregation, and the sidecar emits a non_monotonic_quintiles tri‑state flag per (factor, horizon). The pre‑fix sample from this run is retained on every summary[] row as raw_top_minus_bottom_spread for forensic comparison — e.g. adv_20d h=21 carried raw_top_minus_bottom_spread = -0.302 against the +44 / +37 / -36 absurdities listed below.

factor	h=21 spread (pre‑fix)
`ps_ratio`	+44.78
`pb_ratio`	+37.40
`market_cap`	+37.06
`pfcf_ratio`	-36.78
`fcf_yield`	-3.31
`ma_50d`	-0.99

Post‑fix top_minus_bottom_spread magnitudes are bounded — Tier 4 acceptance in F‑137 brief §15 requires |top_minus_bottom_spread| < 1.0 at every horizon and < 0.30 for pb_ratio / pfcf_ratio at h=21. Verification fills in here from the first post‑deploy nightly. The cumulative‑vs‑period suspect raised alongside this bug was audited and closed in docs/architecture/return-convention-audit.md; the root cause is exclusively un‑winsorized cross‑section outlier dominance.

Non‑monotonic quintiles is now a first‑class flag — the SPA renders a ⚠ non-monotonic chip whenever top‑ and bottom‑quintile IC carry the same sign. Pre‑fix examples included ps_ratio and bb_lower_20 at h=21d.

Turnover values (0.06–0.7) look sensible — momentum/MACD around 0.5–0.7 (high churn), fundamentals around 0.05–0.15 (slow).

5. Factor MCPT — `compute_factor_mcpt_260526_6ddcd9.json`

What it shows (per ResearchFactorMcptView.js): Masters MCPT — empirical p‑value of the factor’s IC_IR against B=100 bar‑permuted null realizations, plus the bias‑corrected unbiased_ic_ir.

This is the formal validation gate. Right now it’s barely usable, for three reasons:

Bucket	Count	Why
`p = 1.000` (ceiling)	37	Overlap‑corrected null exceeds the actual IC. Every high‑IC daily/momentum/vol factor falls here. Their `unbiased_ic_ir` is sharply negative (e.g. `beta` → ‑6.8, `volatility_252d` → ‑4.3, `momentum_1m` → ‑2.1)
`p = 0.010` (floor)	33	`null_mean_ic_ir ≈ 0.000` with zero variance — the permutation null collapses for slow‑moving fundamentals because there isn’t enough cross‑sectional dispersion at annual/quarterly cadence
Intermediate	5	These are the only interpretable rows
Real significance	4	`bb_lower_20`, `ma_50d`, `ma_200d`, `vwap_20d` — non‑collapsed null and positive unbiased IR (0.21 to 1.35)

B=100 puts the p‑value floor at 1/(B+1) ≈ 0.010. Half of your factors are pinned at that floor with a degenerate null and half are pinned at the ceiling with an overcorrected null. Without raising B to ≥1000 and verifying that the bar‑permutation overlap correction is sized right for each cadence, MCPT cannot decide promotion for the bulk of the list.

F-191 — Per-factor nightly MCPT escalation — RETIRED (O-088 + O-090)

This mechanism no longer exists. It let an individual promotion-candidate factor run an elevated B=500 MCPT during an ordinary RunMode.NIGHTLY (which otherwise ran at B=100, min p ≈ 0.0099 — structurally unable to clear the Harvey-Liu-Zhu floor HLZ_PVALUE_FLOOR = 0.0027), bounded by a global cap.

O-088 made compute_factor_mcpt deep-run-only — it now runs at B=500 for every factor on the Friday NIGHTLY_FULL deep run (and ad-hoc RESEARCH_DAY runs) and not on plain weekday nightlies at all. Since the deep-run B already equals the old escalated B, per-factor escalation only ever bought ≤4-day-earlier timing at identical rigor, so O-088 deleted the escalation machinery (_escalated_factor_ids, the DEFAULT_MCPT_PERMUTATIONS_ESCALATED / IC-IR-threshold settings, the sidecar escalation block). O-090 then removed the vestigial operator surface — the research.mcpt_escalate_nightly factor-config flag, the apply-escalation CLI, and the DEFAULT_/ENV_MCPT_NIGHTLY_ESCALATION_CAP constants. A strongest candidate now simply accrues its B=500 evidence on the next deep run (surfaced as an eligible — awaiting the weekly deep-run MCPT annotation by next-actions).

Brief: optimization-mcpt-permute-weekly-cadence-design-brief.md (O-088). Historical: feature-nightly-mcpt-escalation-design-brief.md (F-191).

What this tells you about your factor list

Cross‑page synthesis (and ignoring the data‑quality red flags above):

Duplicates to remove now: beta ↔︎ market_beta_252d (identical), momentum_12_1 ↔︎ momentum_12m_1m (identical). That’s 2 factors you can delete with zero information loss.
The “best by IC” cluster is small, coherent, and consistent with academic literature:
- Momentum: residual_momentum_252d_eod (IR=0.72), momentum_12_1_vol_scaled (IR=0.63), momentum_12_1 (IR=0.59) — same family, the residual version dominates
- Inverted low‑vol / size: volatility_* (IR ≈ ‑0.55), market_cap (IR ≈ ‑0.5) — strong negative predictive power, would be traded short
- Slow value/quality: value_composite (IR=0.75), internal_forecast_eps (IR=0.50), operating_margin_q (IR=0.31)
- Price‑level: pct_from_52w_high (IR=0.52), ma_200d (IR weak but MCPT‑clean), ma_50d (MCPT‑clean)
The “thin/excluded” quadrant on diagnostics maps perfectly to the “p=0.010 null‑collapse” cluster on MCPT. Annual fundamentals like ohlson_o, roic_spread_trend, rule_of_40, qmj_composite, etc. don’t have enough observations for either test to be meaningful. Don’t promote any of them on this run’s evidence — extend the lookback or downsample less aggressively first.
The drift quadrant (pct_from_52w_high, vwap_20d, adv_20d, momentum_6m) is non‑stationary by ADF but shows real IC — typical of trending price features. The “stationarity” check is a guardrail for these, not a kill switch; treat the diagnostics red ✗ as a known property, not a failure.
Your composite layer is one‑deep. residual_momentum_252d is the only composite that produced contribution rows, and the named composites (value_composite, quality_composite, etc.) didn’t. Either they’re being misclassified at load time, or they’re declared in YAML but RecipeMethod runs but doesn’t emit contribution rows — worth checking sbfactorcontrib and the F-114 task.

Lifecycle impact per factor group

All 79 factors are currently experimental. Per CLAUDE.md §1 (Quick Reference), the validation gate is MCPT p‑value + IC IR. Given the data quality issues above, here’s the honest read:

Can move toward validated once MCPT is fixed (B≥1000, overlap correction audited): - residual_momentum_252d_eod — strongest combined evidence (IC IR=0.72 at 21d, diagnostics healthy, related composite has clean contribution attribution) - momentum_12_1_vol_scaled — IR=0.63, MCPT currently ceiling‑pinned but vol‑scaling is what should let it survive overlap correction - value_composite — IR=0.75 at 21d, sign‑flips at h=1 (don’t trade intraday) - ma_50d, ma_200d, vwap_20d, bb_lower_20 — the 4 factors that already pass MCPT with non‑degenerate nulls today

Stay experimental, hold for fixes: - The volatility / size / amihud cluster — strong negative IR is real signal, but you need to confirm with unbiased_ic_ir after a proper MCPT run, then promote them with the inverted convention - All annual fundamentals with n_obs ≤ 12 — cannot be validated on current history. Extend lookback (or accept they won’t promote until you have ~5 years of fundamentals).

Candidates for deprecated immediately: - beta (use market_beta_252d — or vice versa, just pick one) - momentum_12m_1m (duplicate of momentum_12_1) - The most clearly degenerate annual factors with null_mean_ic_ir = 0.000 AND |actual_IC| < 0.01 (e.g. gross_margin_moat actual=0.010, wacc actual=‑0.010, operating_margin_moat actual=‑0.008) — they’re near‑constant in cross‑section

Before promoting anyone, fix these three things or the lifecycle gate is unreliable: 1. Raise MCPT n_permutations from 100 to ≥1000 — the 0.010 floor is masking real differences within the “passing” group. RESOLVED in F-136 (2026-05-26): nightly default bumped to mcpt_permutations_nightly = 1000 (p-floor → 0.001); methodology switched from per-symbol bar-permutation to per-date cross-sectional factor permutation (permute_factor_cross_sectional in sbpermtest.permutation_core). The ceiling-bug factors (beta null_mean=+6.86, etc.) and floor-bug factors (gross_margin_moat null_std=0) are both addressed — the latter is now routed to skip_reason='insufficient_cross_section' when fewer than mcpt_min_valid_dates=20 dates have cross-sectional dispersion. See docs/backlog/feature-F-136-mcpt-correctness-design-brief.md. §5 above is the pre-F-136 evidence; post-F-136 bucket counts land after the first nightly run with the new method. 2. Audit the Alphalens spread scaling — 44× quintile spreads aren’t physical; either winsorize, or fix the cumulative-vs-period return convention in sbalphalens. 3. Fix the market_beta_252d horizon list (currently 21/63/126 vs. everyone else’s 1/5/10/21/63) — config divergence in config/factors/market_beta_252d.yaml.

1. Factor Diagnostics — compute_factor_diagnostics_260526_6ddcd9.json