Skip to content

Test plan: Multi-Jurisdiction Regulatory Ingestion Pipeline

  • Configure a new SourceConfig for California using the already-gathered USA States HR Compliance Links; run a crawl; confirm documents land in a gov_regulations_us (or a new California-specific) index with correct state/jurisdiction metadata.
  • Re-run a crawl with no source changes; confirm no parts are re-fetched (content hash unchanged) and no duplicate documents are created.
  • Amend a single regulation part upstream (or simulate via a changed content hash); re-run the crawl; confirm only that part is re-fetched and its manifest entry's content_hash/last_amendment_date update.
  • Configure a Canadian federal source using a handler not yet built (e.g. a bilingual or SPARQL-based source per crawl_method); confirm the pipeline fails clearly (missing handler) rather than silently skipping the source.
  • Crawl a source configured with an aggressive rate_limit; confirm the crawler respects max_calls/window_seconds rather than overwhelming the government source.
  • Confirm a freshly crawled document does not appear as "validated" / Tier-1-grounded anywhere downstream until it clears SME review (cross- check against ../four-state-curation-lifecycle test plan).