Universe Construction API

This chapter is generated from the canonical platform doc, which lives in docs/ (single source of truth).

Universe Construction API

Full spec for UniverseService. Referenced from CLAUDE.md §16.

UniverseService (src/sbuniverse/services/universe_service.py) provides three point-in-time universe methods. All filter gold.dim_instrument lifecycle columns (ipo_date, delisting_date) so only instruments active on as_of_date are returned.

Methods

Method Description Reliable From
get_index_tickers(index_code, as_of_date) Exact historical index membership via gold.fact_index_constituent SP500: Oct 2008; NASDAQ100: Jan 1995; DJIA: Jan 1994
get_broad_universe_tickers(as_of_date) All fmp_promotion_allowlist-eligible instruments alive on date ~Jan 1995 (~1,546 symbols); grows to ~6,885 by 2024
get_filtered_tickers(exchanges, sectors, industries, countries, as_of_date) Dimension-filtered universe via three-tier fallback Same as broad; market cap filter uses current snapshot only

Three-Tier Fallback (get_filtered_tickers)

  1. Tier 1 — silver.fmp_market_screener (preferred; not yet populated)
  2. Tier 2 — silver.fmp_company_profile_bulk (available; has exchange, sector, industry, country, market_cap snapshot)
  3. Tier 3 — gold.dim_instrument JOIN silver.fmp_promotion_allowlist (always available; no dimension filters, no market cap)

Package Boundary

  • sbuniverse.infra.universe_repo.UniverseRepo — canonical implementation of all query methods
  • sbuniverse.infra.universe_repo.UniverseRepo — equivalent standalone implementation for use within sbfoundation (avoids circular import via sbfoundation/__init__.py)
  • sbuniverse.services.universe_service.UniverseService — adds run-context utilities (run_id, now, today, from_date, last_quarter_end, next_market_day) on top of the repo

Research Universe (factor-research scope)

The factor-research universe is separate from the strategy/ingestion universes above. It is the cross-section every factor-research SQL surface (sbic, sbalphalens, sbpermtest, sbdiag, sbfactorcontrib) measures factors over, injected as a shared membership CTE rather than a hardcoded allowlist join.

Production universe (F-148 / RUS-1)

sbuniverse.research_universe_membership_cte(config) emits a CTE (instrument_sk, date_sk) for the single production research universe, driven by config/universes/research.yaml via sbcontracts.research_universe_config.ResearchUniverseConfigService:

  • mode: allowlist_only (default, current production) — allowlist ∩ lifecycle ∩ US-market-day.
  • mode: base — additionally a point-in-time price floor + trailing dollar-ADV floor (the tradeable Base universe).

Variants (F-168 / RUS-2)

sbuniverse.research_universe_variant_cte(base_config, variant) emits the membership CTE chain for a universe variant — Base ∩ axis-filter — for the deferred calibration sweep (RUS-3). The 9 variants are declared in config/universes/research-variants.yaml and loaded by sbcontracts.research_universe_variant.ResearchUniverseVariantService:

Variant(s) Axis Filter (∩ Base)
base base none (Base only)
cap_small / cap_mid / cap_large cap market-cap tier vs per-date percentile cutoffs
liq_top20 / liq_top50 liquidity top fraction by per-date dollar-ADV percentile rank
vol_high / vol_low volatility above/below per-date median trailing-vol
sector_neutral sector_neutral Base membership; scoring transform (see below)

Notes:

  • Base inheritance — every membership variant forces mode: base (dataclasses.replace), so the price/ADV floors always apply regardless of the production research.yaml mode. Base floors are inherited from research.yaml, not duplicated.
  • Cap breakpoints — config-selectable breakpoint_method (nyse default | cross_sectional): NYSE-listed (gold.dim_exchange.exchange_code) per-date percentiles applied to the full cross-section, over a config-selectable market_cap_source (computed_pit default = close × ASOF carry-forwardgold.fact_quarter.weighted_average_shs_out_diluted;profile_staticalso implemented;annual_market_cap_f` deferred).
  • Sector-neutral is not a membership filter — it z-scores the factor within (date_sk, sector_sk) before the cross-sectional IC, carried as the ResearchUniverseVariant.sector_neutralize flag and applied by sbic.services._ic_sql.filtered_factor_cte / threaded through ICService.factor_ic_ir(..., variant=...).

RUS-2 ships definitions + the scoring flag only — the sweep runner, ranking, sidecar, and SPA are RUS-3. See docs/backlog/feature-research-universe-variants-design-brief.md.

Known Gaps

  • Historical market cap — market_cap in Tier 1/2 reflects today’s snapshot, not the as_of_date value. Point-in-time market cap requires fact_eod backfill (see universe_screener_features_exec_plan.md).
  • Liquidity filters — ADV, ADTV unavailable until fact_eod is backfilled.
  • SP500 pre-2008 — constituent change log is incomplete before Oct 2008.
  • NULL ipo_date — ~162 allowlist symbols have no ipo_date; treated as always-eligible (conservative).