/ › expansion › 02-data-spine

Stream 02 — Data Spine

The substrate every public Futuros surface reads from. Phase 2 takes ingestion from ~12 RSS feeds to 80–120 typed, provenance-stamped sources, normalizes them into a regional knowledge graph, and extends truth-infrastructure discipline to every scraped fact.

Why this is load-bearing

Stream 3 (Atlas) reads typed indicators with geographic dimensions — no atlas without a populated facts_v1 table.
Stream 4 (Pulse) reads structured event streams + diff-able indicators — no diff layer without normalized history.
Stream 5 (Simulator) reads time-series with stable IDs — no Monte Carlo over CSVs we re-fetch ad-hoc.
All public surfaces inherit truth-infra discipline: every number ships with a citation_id.

Decisions taken

#	Decision	Rationale (short)
D1	Storage = DuckDB + retain SQLite	Columnar OLAP, file-only, WASM-portable; vector store stays for unstructured text
D2	Single `IngestAdapter` protocol	`discover → fetch → normalize → cite`. RSS, SDMX, X, scrape — same shape
D3	Truth-gate per fact	Every `Fact` row has `citation_id NOT NULL`; orphans go to `fact_quarantine`
D4	Disabled-by-default	Daemon never silently hits a new API; explicit human enable
D5	Paid sources gated by Phase 0 budget	X API + SemiAnalysis recommended (~USD 8.4K/yr); Bloomberg / Pitchbook explicitly out
D6	Bilingual native at fact layer	`lang` column on every fact; translation at query time, never at storage
D7	License + ToS check is hard gate	No source enters inventory without classification

Source inventory — 102 sources, 12 categories

Class	Count	Examples
Macro / dev indicators	14	World Bank, IMF SDMX, OECD SDMX, ECLAC CEPALSTAT, IDB Numbers, OWID, FAO, ILO
Energy / climate / minerals	19	IEA, IRENA, USGS MCS, Cochilco, MINEM-PE, Coordinador CL, ANDE-PY, Itaipú
Compute / AI / cable	18	Epoch AI, Metaculus, Manifold, Stanford AI Index, OECD.AI, Telegeography, arXiv cs.AI/cs.CL
Government press / gazettes	19	Brasil Planalto, México DOF (SIDOF), Chile gob.cl, Colombia MinTIC, Paraguay Presidencia, Argentina Boletín Oficial
Multilateral / regional	10	IDB News, CAF, ECLAC, MERCOSUR, CELAC, World Bank LAC, IMF LAC
Op-ed / editorial nodes	11	Folha, El País América, La Nación, Reforma, Mercopress, Infobae, Brecha, la diaria
Social (X / Bluesky)	7	Tier 1/2 named-individual lists + topic streams (Itaipú, frontier-labs, cobre/litio)
Forecast aggregators	6	Metaculus, Manifold, Kalshi, Polymarket, AI Impacts
Hyperscaler filings	6	SEC EDGAR 10-Q + 13F (incl. Situational Awareness LP), B3, hyperscaler PR feeds
Central banks	10	BCB, Banxico, BCRA, BCCh, BanRep CO, BCRP, BCP-PY, BCU, BIS
Legislative trackers	8	Senado/Câmara BR, CDM-MX, Congreso CL/CO/PY, IDEA elections
Geographic primitives	7	GeoBoundaries, Natural Earth, IBGE, INEGI, DANE-CO, INE-CL, DGEEC-PY

Broken feeds — diagnosis & fix plan

Feed	Symptom	Root cause	Fix
IDB Publications RSS	403 Forbidden	Edge bot block	Replace with IDB News RSS + add HTML scrape for publications
CEPAL noticias	404 Not Found	RSS path retired	Replace with `/es/comunicados/feed` + sitemap fallback
CAF noticias	SSL cert verify	Python on macOS truststore	Pin `certifi`, use `certifi.where()` as cafile
DOF México	SSL cert verify + no RSS	Same SSL + home page only	`certifi` fix + new `dof-mx` adapter against SIDOF JSON API
MinTIC Colombia	200 OK, empty body	JS-rendered home	Replace with Drupal RSS endpoint at `/portal/inicio/Sala-de-prensa/Noticias/RSS/`
gob.pe / pcm	Intermittent timeout	Slow upstream	Bump per-feed timeout to 45s + 1 retry

Knowledge graph — entity model

Stored in data/graph/knowledge.duckdb alongside the existing SQLite vector store. Both file-based, both rebuildable independently.

Entities: country, subregion, sector, individual, organization, instrument, pilot, indicator, fact, event, source, claim.
Relations (edge tables): org-in-country, individual-org, pilot-country, pilot-pillar, pilot-indicator, event-actor, event-country, instrument-party, resource-flow.
fact is the atomic measurement: (indicator_id, entity_id, period) → value_num | value_str with citation_id NOT NULL by schema constraint.
fact_quarantine holds facts without citation; never read by public surfaces.
facts_v1 is the public read API; Streams 3/4/5 select explicit columns by version.

Citations registry — extension

The existing research/sources/_citations.yml (human-curated) gains a sibling _citations_scraped.yml (auto-extended by adapters). Same schema; new optional fields:

auto_extended, adapter_id, provenance chain (discover → fetch → normalize)
last_verified, last_verified_status, verify_cadence, next_verify
value_at_first_fetch for drift detection
license_class, tos_check

New CLI: python -m tools.citations verify-scraped --due-only refetches scraped citations on cadence and supersedes drifted entries (never deletes; opens citation-drift issues for human review).

Reference adapters shipped

Adapter	Path	Tier	Feature flag
World Bank Indicators v2	`tools/ingestion/wb.py`	free	`FUTUROS_ADAPTER_WB=1`
X / Twitter API v2	`tools/ingestion/x_twitter.py`	paid (Phase-0 gated)	`FUTUROS_ADAPTER_X=1` + `X_BEARER_TOKEN`

Both implement the IngestAdapter protocol and ship with --dry-run. The X adapter's dry-run exercises rate-limit budget math without consuming tokens.

Implementation plan — 6 weeks

Week 1: cross-cutting fixes (certifi, per-feed timeout, YAML config), broken-feed URL replacements, DuckDB graph init, citations split.
Week 2: World Bank reference adapter, X / Twitter reference adapter (dry-run), adapter protocol formalization, feed-health resilience.
Week 3 (MVP cutoff): enable 30 RSS feeds, run WB across LATAM × top-30, draft Stream-3 handoff.
Week 4: IMF SDMX, OECD SDMX, ECLAC CEPALSTAT, generic SDMX shell.
Week 5: IRENA, IEA, USGS minerals, Coordinador CL + ANDE PY, forecast aggregators.
Week 6: verify-scraped daily cron, coherence + red-team, Stream-3/4/5 sign-off.

What downstream streams get

Stream	Reads from	Available after
Stream 3 — Atlas	`facts_v1` (economic, energy, minerals, compute), `rel_resource_flow`, geographic primitives	Week 3 partial → Week 6 full
Stream 4 — Pulse	`events_v1`, indicator `topic` tags, diff over `superseded_by` chain	Week 4
Stream 5 — Simulator	`facts_v1` historical series, hyperscaler capex, forecast priors	Week 3+

Documents

state/expansion/02-data-spine/00-charter.md — mission, scope, decisions
state/expansion/02-data-spine/01-source-inventory.md — 102 sources, classified
state/expansion/02-data-spine/02-broken-feeds-fix-plan.md — diagnosis + remedy per feed
state/expansion/02-data-spine/03-ingestion-adapter-specs.md — IngestAdapter protocol + 10 adapter classes
state/expansion/02-data-spine/04-knowledge-graph-schema.md — entity model, relations, DuckDB layout
state/expansion/02-data-spine/05-citations-registry-extension.md — schema, CLI, verify-scraped
state/expansion/02-data-spine/06-implementation-plan.md — 6-week phased rollout
state/expansion/02-data-spine/99-handoff.md — Stream 3/4/5 contract