Skip to content

Multi-Jurisdiction Regulatory Ingestion Pipeline

Questions or comments?

Post them as a comment on the tracking issue -- requires a GitHub account with access to this repo.

Scope

A pipeline that crawls and parses government regulatory sources into a structured corpus. Today covers U.S. federal CFR content only — an initial seed set of roughly 60 HR-relevant regulations, ingested from the sources tracked in the Nova Codex Compliance Map and USA Federal Govt Regulations tracker. California and New York source lists have been gathered (USA States HR Compliance Links) but not yet crawled or ingested; Canadian federal regulations are scoped but not started.

Per ../../../schema/nova-codex-curator.md, the crawler side of this already exists in code: per-country YAML CountryConfig → SourceConfig → PartSpec configuration, an us_ecfr handler plus a generic web_scraper handler, incremental sync via S3-persisted SyncManifest/PartState (content-hash and amendment-date based), and ingestion (OCR → chunk → embed → index) into a per-country OpenSearch index. Extending to a new jurisdiction (state or country) is primarily a configuration + new-handler exercise on top of this existing pipeline, not a rebuild.

Value

This is the raw material Nova Policy Advisor's answers are grounded in — coverage gaps here are coverage gaps for every customer conversation in that jurisdiction. A repeatable pipeline turns "add a jurisdiction" from ad hoc engineering effort into scoped, staffable work the team can plan a roadmap around.

Acceptance criteria

  • A new jurisdiction (state or country) can be onboarded by adding a CountryConfig/SourceConfig entry (and a new SourceHandler implementation only if the jurisdiction's source format isn't already covered by us_ecfr or the generic web_scraper), without changes to the core pipeline.
  • Crawled content lands in the jurisdiction's own OpenSearch index (gov_regulations_{country}), consistent with the existing per-country indexing convention.
  • Incremental re-crawls only re-fetch parts whose version/content hash has changed, per the existing SyncManifest mechanism — full re-crawls are not required to pick up amendments.
  • California, New York, and Canada federal sources can be configured and crawled using this pipeline once their source lists (already gathered per the PRD) are turned into SourceConfig entries.
  • Nothing crawled here reaches the vector database's reviewed state without passing the SME document-review gate — see ../four-state-curation-lifecycle/spec.md and the schema delta in ../schema-changes.md for the review-gate state this pipeline currently lacks.