Document Ingestion

Document ingestion converts raw PDF statements into structured, auditable transaction records. The process is split into two phases: deterministic ingestion (fully implemented) and rule derivation from document content (partially implemented via stubs).

Two-Phase Ingestion Model

Phase 1 — Deterministic Ingestion takes a PDF, extracts transactions using Blake3 content-hash IDs for deduplication, and writes raw records to the workbook and journal. This phase is fully implemented and produces stable, reproducible output regardless of processing order.

Phase 2 — Rule Derivation uses the extracted document structure to derive applicable classification rules. This requires the reqif-opa-mcp Python sidecar to build a DocumentGraph from the PDF, run it through an OPA policy gate, and produce RequirementCandidate objects. The Rust side then deserializes these into ReqIfCandidate structs and uses them to seed the RuleRegistry. This phase is currently stubbed.

Ingestion Pipeline Diagram

fn pdf_statement() -> filename_routing
fn filename_routing() -> blake3_ingest
fn filename_routing() -> docling_extract
fn docling_extract() -> document_graph
fn document_graph() -> requirement_candidates
fn requirement_candidates() -> opa_gate
fn opa_gate() -> reqif_baseline
fn reqif_baseline() -> rule_registry
fn rule_registry() -> classify_waterfall
fn classify_waterfall() -> workbook_commit

flowchart TD
    pdf_statement["pdf_statement"]
    filename_routing["filename_routing"]
    blake3_ingest["blake3_ingest"]
    docling_extract["docling_extract"]
    document_graph["document_graph"]
    requirement_candidates["requirement_candidates"]
    opa_gate["opa_gate"]
    reqif_baseline["reqif_baseline"]
    rule_registry["rule_registry"]
    classify_waterfall["classify_waterfall"]
    workbook_commit["workbook_commit"]
    pdf_statement --> filename_routing
    filename_routing --> blake3_ingest
    filename_routing --> docling_extract
    docling_extract --> document_graph
    document_graph --> requirement_candidates
    requirement_candidates --> opa_gate
    opa_gate --> reqif_baseline
    reqif_baseline --> rule_registry
    rule_registry --> classify_waterfall
    classify_waterfall --> workbook_commit

Ingestion Flow (Rhai DSL)

The following DSL block describes the intended end-to-end ingestion flow. Each step corresponds to a node in the pipeline diagram above.

fn ingest_artifact() -> extract_document
fn extract_document() -> build_graph
fn build_graph() -> derive_candidates
fn derive_candidates() -> opa_gate
fn opa_gate() -> emit_reqif
fn emit_reqif() -> load_rules

flowchart TD
    ingest_artifact["ingest_artifact"]
    extract_document["extract_document"]
    build_graph["build_graph"]
    derive_candidates["derive_candidates"]
    opa_gate["opa_gate"]
    emit_reqif["emit_reqif"]
    load_rules["load_rules"]
    ingest_artifact --> extract_document
    extract_document --> build_graph
    build_graph --> derive_candidates
    derive_candidates --> opa_gate
    opa_gate --> emit_reqif
    emit_reqif --> load_rules

Capability Status Table

Capability	Status	Implementation	Notes
`ingest_artifact`	Implemented	`IngestedLedger::ingest`	Blake3 hash dedup, journal write
`extract_document`	Planned	Python sidecar: `extract_docling_document`	Docling 2.78 PDF parse
`build_graph`	Planned	Python sidecar: `DocumentGraph` assembly	`DocumentNode` tree construction
`derive_candidates`	Planned	Python sidecar: `RequirementCandidate`	Heuristic requirement extraction
`opa_gate`	Planned	Python sidecar: OPA policy evaluation	Policy: admit or reject candidates
`emit_reqif`	Planned	Python sidecar: `emit_reqif_xml`	ReqIF XML baseline output
`load_rules`	Implemented	`RuleRegistry::load_from_dir`	Loads transaction `.rhai` rules and optional `ReqIfCandidate` sidecars

ReqIF-OPA-MCP Integration

The Python sidecar at https://github.com/PromptExecution/reqif-opa-mcp implements the full ArtifactRecord → DocumentGraph → RequirementCandidate → OPA → ReqIF pipeline. It is invoked as a subprocess by the Rust core and communicates via NDJSON over stdout.

Key Python types and their Rust mirrors:

Python (sidecar)	Rust (ledger-core)	Notes
`ArtifactRecord`	—	Source document metadata; consumed by sidecar only
`DocumentNode`	`DocumentChunk`	Canonical graph node with text, parent, anchors
`DocumentGraph`	—	Intermediate; not materialized in Rust
`RequirementCandidate`	`ReqIfCandidate`	Deserialized from sidecar NDJSON output

The sidecar also exposes an MCP server interface for querying existing ReqIF baselines and evaluating new candidates against established policy. This allows the agent to ask “does this transaction match any known requirement?” before falling back to keyword-match rule selection.

The Rust bridge will call the sidecar via std::process::Command, read its NDJSON output, and deserialize each line into a ReqIfCandidate. These candidates are then stored in the RuleRegistry alongside their associated rule files.

Vector Search Stub

SemanticRuleSelector (defined in crates/ledger-core/src/rule_registry.rs) will eventually use vector embeddings to rank rule files by semantic similarity to each transaction’s description. The embedding space is anchored to ReqIfCandidate text so that rules are selected based on document-derived semantics rather than string keywords.

The interface is already defined:

#![allow(unused)]
fn main() {
pub trait SemanticRuleSelector {
    fn select_rules_semantic(&self, tx: &SampleTransaction, top_k: usize) -> Vec<PathBuf>;
    fn build_embedding_index(&mut self) -> Result<(), RuleRegistryError>;
}
}

Implementation is blocked on:

A local embedding model (ONNX via ort, or fastembed-rs)
A vector index (usearch, hnsw, or qdrant sidecar)
ReqIfCandidate objects being populated from the sidecar bridge

Until these are available, select_rules_deterministic provides a stable keyword-match fallback. See Rule Engine for the full classification pipeline.

Keyboard shortcuts