add Qodo review

docs(concepts): rewrite every code example against real API
docs(quickstart): read parsed full_text (#1415 )
2026-09-04 04:01:07 +00:00 · 2026-09-03 15:01:26 +05:00 · 2026-09-03 14:53:22 +05:00 · 2026-09-03 14:40:58 +05:00 · 2026-09-03 14:33:23 +05:00 · 2026-09-03 14:12:49 +05:00
4 changed files with 352 additions and 151 deletions
@@ -38,18 +38,19 @@ This structure makes knowledge **searchable**, **connectable**, **queryable**, a
 Scanning text to find and classify real-world entities:

 ```python
-# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
-{
-    "entities": [
-        {"text": "Apple Inc.",  "type": "ORGANIZATION", "confidence": 0.98},
-        {"text": "Steve Jobs",  "type": "PERSON",       "confidence": 0.99},
-        {"text": "1976",        "type": "DATE",         "confidence": 0.95},
-        {"text": "Cupertino",   "type": "LOCATION",     "confidence": 0.97}
-    ]
-}
+# "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
+[
+    Entity(text="Apple Inc.", label="ORG",    start_char=0,  end_char=10, confidence=0.98),
+    Entity(text="Steve Jobs", label="PERSON", start_char=25, end_char=35, confidence=0.99),
+    Entity(text="1976",       label="DATE",   start_char=39, end_char=43, confidence=0.95),
+    Entity(text="Cupertino",  label="GPE",    start_char=47, end_char=56, confidence=0.97),
+]
 ```

-Each entity gets a type, confidence score, and a link to its source document. Three extraction methods are available:
+`NERExtractor(method=...).extract(text)` returns a list of `Entity` objects, each
+with a `label`, character offsets (`start_char` / `end_char`), a `confidence`
+score, and a `metadata` dict recording the extraction method. Three methods are
+available:

 | Method | Speed | Accuracy | Requirements |
 | :------ | :----- | :-------- | :------------ |
@@ -62,15 +63,19 @@ Each entity gets a type, confidence score, and a link to its source document. Th
 Finding how entities connect to each other:

 ```python
-{
-    "relationships": [
-        {"subject": "Steve Jobs", "predicate": "founded",    "object": "Apple Inc.", "confidence": 0.92},
-        {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino",  "confidence": 0.89}
-    ]
-}
+jobs  = Entity(text="Steve Jobs", label="PERSON", start_char=25, end_char=35)
+apple = Entity(text="Apple Inc.", label="ORG",    start_char=0,  end_char=10)
+
+[
+    Relation(subject=jobs,  predicate="founded",     object=apple, confidence=0.92),
+    Relation(subject=apple, predicate="located_in",  object=Entity(text="Cupertino", label="GPE", start_char=47, end_char=56), confidence=0.89),
+]
 ```

-Relationships can be extracted via rule-based methods, ML models, or LLMs: each producing typed triplets with confidence scores and source attribution.
+`RelationExtractor(method=...).extract(text, entities=entities)` returns a list of
+`Relation` objects: typed subject-predicate-object triples (the endpoints are
+`Entity` objects) with confidence scores and source attribution. Extraction runs
+via pattern rules, ML models, or LLMs.


 ## Knowledge Graph vs. Vector Store
@@ -94,9 +99,10 @@ Both store information for AI retrieval: but they're built for different jobs.
    ```python
    from semantica.kg import GraphBuilder, PathFinder

-    graph   = GraphBuilder(merge_entities=True).build(entities=entities, relationships=rels)
-    finder  = PathFinder()
-    path    = finder.dijkstra_shortest_path(graph, "Steve Jobs", "Tim Cook")
+    graph = GraphBuilder(merge_entities=True).build(
+        {"entities": entities, "relationships": rels}
+    )
+    path  = PathFinder().dijkstra_shortest_path(graph, "Steve Jobs", "Tim Cook")
    ```
  </Tab>

@@ -140,8 +146,16 @@ Both store information for AI retrieval: but they're built for different jobs.
    context = AgentContext(
        vector_store=VectorStore(backend="faiss", dimension=768),
        knowledge_graph=ContextGraph(advanced_analytics=True),
+        graph_expansion=True,
    )
-    result = context.query("Who founded Apple?", mode="graphrag")
+
+    # store() extracts entities and populates the graph + vector index
+    context.store([{"content": "Steve Jobs co-founded Apple Inc. in 1976."}])
+
+    # retrieve() blends vector similarity with graph traversal
+    results = context.retrieve("Who founded Apple?", use_graph=True, expand_graph=True)
+    for r in results:
+        print(r["score"], r["content"], r["source"])
    ```
  </Tab>
 </Tabs>
@@ -221,70 +235,80 @@ Inferred: Steve Jobs has a connection to Cupertino
    Applies IF/THEN rules repeatedly until no new facts can be derived. Best for alert systems, compliance checks, and trigger-based workflows.

    ```python
-    from semantica.reasoning import Reasoner, Rule, Fact, RuleType
+    from semantica.reasoning import Reasoner

    engine = Reasoner()
-    engine.add_fact(Fact(subject="Alice", predicate="is_a", obj="Manager"))
-    engine.add_rule(Rule(
-        rule_type=RuleType.FORWARD_CHAIN,
-        conditions=[{"subject": "?x", "predicate": "is_a", "object": "Manager"}],
-        conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"}
-    ))
-    result = engine.infer()
+    engine.add_fact("Manager(Alice)")
+    engine.add_rule("IF Manager(?x) THEN HasAuthority(?x)")
+
+    results = engine.forward_chain()   # list of InferenceResult
+    for r in results:
+        print(r.conclusion)           # "HasAuthority(Alice)"
    ```
  </Tab>
  <Tab title="Rete Network">
    Efficient pattern matching for large rule sets: the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts.

    ```python
-    from semantica.reasoning import ReteEngine
+    from semantica.reasoning import ReteEngine, Rule, Fact

    engine = ReteEngine()
-    engine.load_rules("rules/domain_rules.json")
-    results = engine.run(kg)
+    engine.build_network([
+        Rule(rule_id="r1", name="manager_authority",
+             conditions=["Manager(?x)"], conclusion="HasAuthority(?x)"),
+    ])
+    engine.add_fact(Fact(fact_id="f1", predicate="Manager", arguments=["Alice"]))
+
+    matches = engine.match_patterns()
+    results = engine.execute_matches(matches)   # ["HasAuthority(?x)"]
    ```
  </Tab>
-  <Tab title="Deductive & Abductive">
-    **Deductive**: classical syllogistic reasoning from premises to guaranteed conclusions.
-
-    **Abductive**: infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases.
+  <Tab title="LLM Reasoning">
+    `GraphReasoner` answers open-ended questions over a knowledge graph with an
+    LLM, returning a natural-language answer grounded in the graph's facts. Best
+    for exploratory and investigative questions that fixed rules can't anticipate.

    ```python
    from semantica.reasoning import GraphReasoner

-    graph_reasoner = GraphReasoner(kg)
-    graph_reasoner.add_rule({"if": [{"subject": "?a", "predicate": "parent_of", "object": "?b"}], "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?b"}})
-    inferences = graph_reasoner.infer(kg)
+    reasoner = GraphReasoner(provider="openai", model="gpt-4o-mini")
+    answer = reasoner.reason(kg, "Which suppliers are indirectly exposed to the Acme outage?")
    ```
  </Tab>
  <Tab title="Datalog (v0.4.0)">
    Recursive Horn clause rules with fixpoint semantics: handles transitive closure and recursive relationships that forward chaining cannot express.

    ```python
-    from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule
+    from semantica.reasoning import DatalogReasoner

    reasoner = DatalogReasoner()
-    reasoner.add_fact(DatalogFact("parent", ("alice", "bob")))
-    reasoner.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y)."))
-    reasoner.evaluate()
-    results = reasoner.query("ancestor(alice, ?Z)")
+    reasoner.add_fact("parent(alice, bob)")
+    reasoner.add_fact("parent(bob, charlie)")
+    reasoner.add_rule("ancestor(X, Y) :- parent(X, Y).")
+    reasoner.add_rule("ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).")
+
+    reasoner.derive_all()
+    results = reasoner.query("ancestor(alice, ?Z)")   # {"Z": "bob"} and {"Z": "charlie"}, order not guaranteed
    ```
  </Tab>
  <Tab title="Engine Comparison">

-    | Engine | Description | Best For |
-    | :------ | :----------- | :-------- |
-    | Forward chaining | Applies rules until fixpoint | Alert systems, compliance checks |
-    | Rete network | Efficient pattern matching | Large rule sets, high fact throughput |
-    | Deductive | Classical syllogistic reasoning | Mathematical and logical inference |
-    | Abductive | Most likely explanation | Diagnostics, investigation |
-    | SPARQL | Query-based inference over RDF | Semantic web, ontology reasoning |
-    | Datalog (v0.4.0) | Recursive Horn clause rules | Transitive closure, graph reachability |
+    | Engine | Class | Best For |
+    | :------ | :----- | :-------- |
+    | Forward chaining | `Reasoner` | Alert systems, compliance checks |
+    | Rete network | `ReteEngine` | Large rule sets, high fact throughput |
+    | SPARQL expansion | `SPARQLReasoner` | Semantic web, ontology reasoning over RDF |
+    | Datalog (v0.4.0) | `DatalogReasoner` | Transitive closure, graph reachability |
+    | Temporal | `TemporalReasoningEngine` | Allen interval algebra, time-aware inference |
+    | LLM over the graph | `GraphReasoner` | Open-ended, investigative questions |

  </Tab>
 </Tabs>

-All engines produce **explainable inference paths**: not black-box conclusions. Every derived fact includes the rules and premises that produced it.
+`Reasoner.forward_chain()` returns `InferenceResult` objects that carry the rule
+applied (`rule_used`) and the premises it fired on, and `ExplanationGenerator`
+turns one into a step-by-step natural-language justification: reasoning here is
+**not** a black box.


 ## Temporal Intelligence
@@ -313,11 +337,16 @@ Explore the semantic neighborhood of any entity in your graph: useful for unders
 ```python
 from semantica.kg import SimilarityCalculator

-calc   = SimilarityCalculator()
-scores = calc.calculate_similarity(entity_a, entity_b)
+calc = SimilarityCalculator(method="cosine")   # "cosine" | "euclidean" | "manhattan" | "correlation"
+
+# Similarity for every unique pair of node embeddings: {(node_a, node_b): score}
+pairs = calc.pairwise_similarity({"apple": vec_apple, "google": vec_google, "nest": vec_nest})
+
+# Or rank a set of embeddings by closeness to one query vector
+nearest = calc.find_most_similar(embeddings, query_embedding, top_k=10)
 ```

-**Features:** N×N semantic distance matrices, ego-mode visualization, distance band classification (`near` / `mid` / `far`), embedding cache optimization for large graphs.
+**Features:** N×N semantic distance matrices, ego-mode visualization, distance band classification (`direct` / `near` / `mid-range` / `distant`), embedding cache optimization for large graphs.

 The [Visualization module](/reference/visualization) renders distance matrices as interactive heatmaps and ego-mode neighborhood graphs. The [Explorer](/reference/explorer) embeds distance intelligence directly in the browser dashboard.

@@ -341,11 +370,11 @@ Real-world data contains the same entity under many names: "Apple", "Apple Inc."
    ```python
    from semantica.deduplication import DuplicateDetector, EntityMerger

-    detector = DuplicateDetector(similarity_threshold=0.85)
-    duplicates = detector.detect_duplicates(entities)
+    detector   = DuplicateDetector(similarity_threshold=0.85)
+    candidates = detector.detect_duplicates(entities)

-    merger = EntityMerger()
-    deduplicated_entities = merger.merge_duplicates(entities)
+    merger     = EntityMerger()
+    operations = merger.merge_duplicates(entities, strategy="keep_most_complete")
    ```
  </Tab>
 </Tabs>
@@ -361,19 +390,21 @@ Every fact in Semantica links back to:
 - The **reasoning steps** that produced any inferred fact

 <Note>
-  This is W3C PROV-O compliant lineage: suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export.
+  This is W3C PROV-O compliant lineage: suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). `ProvenanceManager.export_prov(format="turtle")` serialises the recorded lineage as PROV-O RDF.
 </Note>

 ```python
 from semantica.provenance import ProvenanceManager

-prov    = ProvenanceManager()
-lineage = prov.get_entity_lineage("apple_inc")
+prov = ProvenanceManager()
+prov.track_entity("apple_inc", source="report.pdf",
+                  metadata={"extractor": "NamedEntityRecognizer", "confidence": 0.98})

-print(f"Source:    {lineage.source_document}")
-print(f"Method:    {lineage.extraction_method}")
-print(f"Extracted: {lineage.timestamp}")
-print(f"Checksum:  {lineage.checksum}")
+record = prov.get_provenance("apple_inc")   # dict; use get_lineage() for the full chain
+print(record["source_document"])
+print(record["timestamp"])
+print(record["checksum"])
+print(record["metadata"])          # extractor, confidence, and any custom keys
 ```


@@ -456,27 +487,27 @@ Semantica is designed for extension. Any component: ingestor, extractor, graph b
    **Extension points available:** ingestors, parsers, normalizers, extractors, reasoning engines, export formats, vector store backends, graph store backends, visualization renderers.

  </Accordion>
-  <Accordion title="MethodRegistry: add domain-specific graph operations">
+  <Accordion title="MethodRegistry: swap a built-in graph operation for your own">

-    `MethodRegistry` lets you register custom methods on knowledge graph objects by name: useful for adding domain-specific graph operations without subclassing.
+    `method_registry` lets you register an alternative implementation for a
+    knowledge-graph task (`build`, `analyze`, `centrality`, `resolve`, …) under a
+    name, then select it wherever that task runs.

    ```python
-    from semantica.kg import MethodRegistry
+    from semantica.kg import method_registry
+    from semantica.kg.methods import calculate_centrality

-    registry = MethodRegistry()
-
-    def find_supply_chain_hops(graph, source_node, max_hops=3):
-        """Custom BFS traversal for supply chain graphs."""
+    def fast_centrality(graph, **kwargs):
+        """Custom centrality implementation."""
        ...

-    # Register under a string key
-    registry.register("supply_chain_hops", find_supply_chain_hops)
+    # register(task, name, func)
+    method_registry.register("centrality", "fast_centrality", fast_centrality)

-    # Call by name on any graph object
-    result = registry.call("supply_chain_hops", kg, source_node="Supplier_A", max_hops=5)
+    # The task wrappers consult method_registry, so the name is now selectable:
+    scores = calculate_centrality(kg, method="fast_centrality")

-    # List all registered methods
-    print(registry.list_methods())   # ["supply_chain_hops", ...]
+    print(method_registry.list_all("centrality"))   # {"centrality": ["fast_centrality", ...]}
    ```

  </Accordion>
@@ -84,13 +84,13 @@ icon: "rocket"
    # 1. Ingest
    sources = FileIngestor().ingest("data/report.pdf")

-    # 2. Parse
-    parsed = DocumentParser().parse(sources[0])
+    # 2. Parse (extract_text returns a plain string for any supported format)
+    text = DocumentParser().extract_text(sources[0].path)

-    # 3. Extract
+    # 3. Extract (extractors take text, return Entity / Relation objects)
    ner           = NERExtractor(method="pattern")  # no API key needed
-    entities      = ner.extract(parsed)
-    relationships = RelationExtractor().extract(parsed, entities=entities)
+    entities      = ner.extract(text)
+    relationships = RelationExtractor(method="pattern").extract(text, entities=entities)

    # 4. Build
    graph = GraphBuilder(merge_entities=True).build(
@@ -144,23 +144,31 @@ icon: "rocket"
    context = AgentContext(
        vector_store=VectorStore(backend="faiss", dimension=768),
        knowledge_graph=ContextGraph(advanced_analytics=True),
+        graph_expansion=True,       # blend graph traversal into retrieval
+        max_expansion_hops=3,       # how far to walk from the seed nodes
    )

-    # Load your knowledge graph
-    context.load_graph("company_kg.json")
+    # store() runs extraction and populates both the vector index and the graph
+    context.store([
+        {"content": "Steve Wozniak co-founded Apple with Steve Jobs in 1976."},
+        {"content": "Tony Fadell led the iPod team at Apple, then founded Nest."},
+    ])

-    # Multi-hop GraphRAG query
-    result = context.query(
+    # GraphRAG retrieval: seed from vector matches, expand along graph edges
+    results = context.retrieve(
        "What companies were founded by people who worked at Apple?",
-        mode="graphrag",
-        reasoning=True,
+        use_graph=True,
+        expand_graph=True,
    )
-
-    # Every claim links back to a source node
-    for claim in result.claims:
-        print(f"{claim.text}  →  source: {claim.source_node}")
+    for r in results:
+        print(f"[{r['score']:.3f}]  {r['content'][:70]}  (source: {r['source']})")
    ```

+    Each result carries `content`, `score`, `source`, and `metadata`. For a
+    grounded natural-language answer plus an auditable traversal, use
+    `context.query_with_reasoning(query, llm_provider=...)` — it returns
+    `response`, `reasoning_path`, `sources`, and `confidence`.
+
    **Next:** [GraphRAG concepts →](/concepts#graphrag)
  </Tab>

@@ -78,11 +78,13 @@ from semantica.parse import DocumentParser
 parser = DocumentParser()
 parsed = parser.parse(sources[0].path)   # parse() takes a path string

-print(parsed["text"][:200])   # extracted text
-print(parsed["metadata"])     # file_path, encoding, size, and format-specific keys
+print(parsed["full_text"][:200])   # extracted text
+print(parsed["metadata"])          # document properties (fields vary by format)
 ```

-`parse()` returns a `dict` with `text`, `full_text`, and `metadata` keys.
+`parse()` returns a `dict`. `full_text` and `metadata` are present for every
+format; other keys depend on the parser (`pages` for PDF, `tables` and
+`paragraphs` for DOCX, `tables` for `DoclingParser`).

 <Tip>
  For PDFs with tables, charts, or multi-column layouts, use `DoclingParser` (`pip install semantica[parse-docling]`): it applies advanced layout analysis and returns structured table data alongside text.
@@ -107,7 +109,7 @@ Identify named entities and extract typed relationships between them.
 ```python Pattern-based (fast, no API key)
 from semantica.semantic_extract import NERExtractor, RelationExtractor

-text = parsed["text"]
+text = parsed["full_text"]

 ner      = NERExtractor(method="pattern")
 entities = ner.extract(text)
@@ -122,7 +124,7 @@ relationships = rel.extract(text, entities=entities)
 from semantica.semantic_extract import NERExtractor, RelationExtractor

 # Reads GROQ_API_KEY from the environment; provider/llm_model select the backend
-text = parsed["text"]
+text = parsed["full_text"]

 ner           = NERExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
 entities      = ner.extract(text)
@@ -277,7 +279,7 @@ builder = GraphBuilder(merge_entities=True)

 all_entities, all_rels = [], []
 for source in FileIngestor().ingest("data/reports/"):
-    text     = parser.parse(source.path)["text"]
+    text     = parser.parse(source.path)["full_text"]
    entities = ner.extract(text)
    rels     = rel.extract(text, entities=entities)
    all_entities.extend(entities)
@@ -413,7 +415,7 @@ store    = GraphStore(backend="neo4j", uri="bolt://localhost:7687",
 builder  = GraphBuilder(merge_entities=True, graph_store=store)

 for info in ingestor.scan_directory("data/reports/", recursive=True):
-    text     = parser.parse(info["path"])["text"]   # one document loaded at a time
+    text     = parser.parse(info["path"])["full_text"]   # one document loaded at a time
    entities = ner.extract(text)
    rels     = rel.extract(text, entities=entities)
    builder.build({"entities": entities, "relationships": rels})
@@ -1,64 +1,224 @@
 ---
 title: "Evals Module"
-description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance: coming soon."
+description: "Score decision records, audit trails, and reasoning output with deterministic and model-backed evaluators plus a small run harness."
 icon: "chart-line"
 ---

-**`semantica.evals`** is planned as a comprehensive evaluation framework for measuring **extraction accuracy, graph quality, and pipeline performance**.
+`semantica.evals` measures the quality of decision intelligence outputs. It takes
+the decisions, audit trails, and reasoning text your pipeline produces and scores
+them against expectations you define, returning a structured summary you can log,
+assert on in tests, or track across runs.

-<Warning>
-  **`semantica.evals` is not yet implemented.** The module is a placeholder with `__all__ = []`. No classes or functions are available for import. This page describes the planned API only.
-</Warning>
+- A registry of named evaluators, from exact string matching to ROUGE overlap and
+  LLM-as-judge
+- `decision_scores`, a composite evaluator for `Decision` objects that checks
+  outcome, confidence bounds, required fields, provenance, and (optionally)
+  policy compliance
+- A `evaluate()` runner that applies several evaluators to a list of cases and
+  aggregates pass / fail / error counts
+- Per-evaluator **objectives** that let you override an evaluator's built-in
+  verdict at the run level

-## Planned Features
+<Note>
+  The module is versioned separately from the package: `semantica.evals.__version__`
+  is `"0.1.0"`. The public surface described here is stable, but expect additive
+  changes (new evaluators, new objective options) before it reaches 1.0.
+</Note>

-When released, `semantica.evals` will provide:
+## Public API

-| Planned Class | Role |
-| :--- | :--- |
-| `KGEvaluator` | Completeness, consistency, schema compliance, coverage, and orphan node detection |
-| `ExtractionEvaluator` | NER precision / recall / F1 and relation extraction metrics against gold datasets |
-| `PipelineBenchmark` | Throughput (docs/sec), per-step latency, peak memory, and error rate |
-| `RegressionTracker` | Record runs and compare metrics across commits or config changes |
-| `EvalReport` | Structured report: `{scores, regressions, recommendations}` |
-| `DeduplicationEvaluator` | Merge precision, false positive / false negative rates |
-| `ReasoningEvaluator` | Inference accuracy, rule coverage, and derivation depth |
-
-## Current Workaround
-
-Until `semantica.evals` ships, use `semantica.ontology.OntologyEvaluator` for ontology quality metrics:
+| Name | Kind | Role |
+| :--- | :--- | :--- |
+| `evaluate(cases, evaluators, config=None, target_fn=None)` | function | Run named evaluators over each case, return an `EvalSummary` |
+| `list_evaluators()` | function | Sorted names of every registered evaluator |
+| `get_evaluator(name)` | function | Look up a single evaluator function by name |
+| `EvalMetric` | dataclass (frozen) | One evaluator's result: `score`, `passed`, `meta` |
+| `CaseResult` | namedtuple | One case's result: `case_id`, `status`, `metrics`, `details` |
+| `EvalSummary` | dataclass | Aggregate across cases: `total`, `passed`, `failed`, `errors`, `pass_rate`, `cases` |

 ```python
-from semantica.ontology import OntologyEvaluator
-
-evaluator = OntologyEvaluator()
-
-# evaluate_ontology takes the ontology dict only
-result = evaluator.evaluate_ontology(ontology)
-
-print("Coverage:    ", result.coverage_score)
-print("Completeness:", result.completeness_score)
-print("Gaps:        ", result.gaps)
-print("Suggestions: ", result.suggestions)
-
-# Full report with class granularity and relation completeness
-report = evaluator.generate_report(ontology)
-print("Coverage score:    ", report["evaluation"]["coverage_score"])
-print("Completeness score:", report["evaluation"]["completeness_score"])
-print("Relation coverage: ", report["relation_completeness"]["relation_coverage"])
+import semantica.evals as evals
+from semantica.evals import evaluate, list_evaluators, get_evaluator
 ```

-`EvaluationResult` fields returned by `evaluate_ontology()`:
+## Built-in evaluators

-| Field | Type | Description |
-| :----- | :---- | :----------- |
-| `coverage_score` | `float` | Fraction of competency questions answerable by the ontology |
-| `completeness_score` | `float` | Average of class and property completeness scores |
-| `gaps` | `List[str]` | Identified gaps in coverage |
-| `suggestions` | `List[str]` | Improvement suggestions |
-| `metrics` | `dict` | Detailed sub-metrics |
+Every evaluator is a plain function `fn(actual, expected, config=None) -> EvalMetric`
+registered under a stable name. `list_evaluators()` returns the current set:

- [Semantic Extract](/reference/semantic_extract) — Extraction module.
- [Knowledge Graph](/reference/kg) — Graph quality assessment.
- [Pipeline](pipeline) — Pipeline performance metrics.
- [Ontology Evaluator](ontology) — Available now for ontology quality metrics.
+```python
+>>> list_evaluators()
+['decision_scores', 'exact_match', 'keyword_check', 'length_range',
+ 'levenshtein', 'llm_as_judge', 'numeric_range', 'regex_match', 'rouge',
+ 'temporal_range']
+```
+
+| Name | Passes when | Relevant `config` keys |
+| :--- | :--- | :--- |
+| `exact_match` | `actual == expected` | none |
+| `regex_match` | `re.search(expected, actual)` matches | none |
+| `keyword_check` | every required term appears in `actual` (word-boundary) | `required` (falls back to `expected`) |
+| `numeric_range` | `min <= actual <= max` | `min`, `max` (both required) |
+| `temporal_range` | ISO datetime `actual` falls in `[min, max]` | `min`, `max` as ISO strings (both required) |
+| `length_range` | `min <= len(actual) <= max` | `min` (default 0), `max` (required) |
+| `levenshtein` | normalized similarity `>= threshold` | `threshold` (default 0.8) |
+| `rouge` | ROUGE-1 F1 `> 0` and `>= threshold` | `threshold` (default 0.0) |
+| `llm_as_judge` | caller-supplied `judge_fn(actual, expected)` returns truthy | `judge_fn` (required callable) |
+| `decision_scores` | all configured sub-checks on a `Decision` pass | see below |
+
+An evaluator that cannot run (bad regex, unparseable datetime, no `judge_fn`) returns an
+`EvalMetric` with an `"error"` key in `meta` rather than raising. Evaluators that
+require numeric bounds (`numeric_range`, `length_range`) instead return a failing
+metric with a `"reason"` key when the bound is missing — they do not raise and do
+not set `"error"`.
+
+### `decision_scores`
+
+`decision_scores` accepts a `Decision` (from `semantica.context.decision_models`)
+or its dict form and runs a set of field-level and governance checks. The score is
+the fraction of checks that passed; `passed` is `True` only when all of them did.
+
+| Sub-check | Controlled by |
+| :--- | :--- |
+| Outcome matches | `expected_outcome` in config, or the case's `expected`; **skipped** when neither is set |
+| Confidence in range | `min_confidence` (default 0.0), `max_confidence` (default 1.0); always run |
+| `decision_maker`, `reasoning`, `scenario` non-empty | always run |
+| Provenance present in metadata | `provenance_key` (default `"provenance"`); always run |
+| Policy compliance | `policy_engine` and `policy_id` both set; skipped otherwise |
+
+Passing `causal_chain_exists` in config raises `NotImplementedError`. That key is a
+reserved slot for a future release.
+
+## Running an evaluation
+
+`evaluate()` takes a list of cases and a list of evaluator names. A case is either
+a `(expected, actual)` tuple or a dict:
+
+```python
+{
+    "id": "loan-001",          # optional, generated if absent
+    "expected": ...,           # optional; some evaluators read it, some don't
+    "actual": ...,             # the value under test
+    "config": {...},           # optional, per-evaluator settings for this case
+    "target_fn": callable,     # optional, called with the case to produce `actual`
+}
+```
+
+If `actual` is missing, the runner calls the case's `target_fn` (or the
+`target_fn` passed to `evaluate()`) to produce it. Per-case `config` is deep-merged
+over the top-level `config`, so a case can override one evaluator's settings
+without discarding the rest.
+
+```python
+from datetime import datetime
+
+from semantica.context.decision_models import Decision
+from semantica.evals import evaluate
+
+decision = Decision(
+    decision_id="d-1",
+    category="loan",
+    scenario="loan-request",
+    reasoning="vetted against lending policy v3",
+    outcome="approve",
+    confidence=0.87,
+    timestamp=datetime.now(),
+    decision_maker="approver-a",
+    metadata={"provenance": "workflow:loan/v3"},
+)
+
+cases = [
+    {
+        "id": "loan-001",
+        "actual": decision,
+        "config": {
+            "decision_scores": {
+                "expected_outcome": "approve",
+                "min_confidence": 0.7,
+            }
+        },
+    },
+]
+
+summary = evaluate(cases, ["decision_scores"])
+print(summary.pass_rate)   # 1.0
+```
+
+Evaluators run independently per case. If one raises, that case's `status` becomes
+`"error"` and the exception text is captured in the metric's `meta`; the rest of
+the run continues.
+
+## Objectives
+
+By default each evaluator decides its own pass / fail. An **objective** overrides
+that verdict at the run level, keyed by evaluator name under `config`:
+
+```python
+# Raise levenshtein's bar from its default 0.8 to 0.9
+evaluate(
+    [("apple", "aple")],
+    evaluators=["levenshtein"],
+    config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.9}}},
+)
+
+# Lower is better
+evaluate(
+    [("night", "nacht")],
+    evaluators=["levenshtein"],
+    config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}},
+)
+
+# Expect the metric NOT to match
+evaluate(
+    [("ok", "ok")],
+    evaluators=["exact_match"],
+    config={"exact_match": {"objective": {"expect": False}}},
+)
+```
+
+Rules:
+
+- `maximize` with `threshold`: pass iff `score >= threshold`. `maximize` with no
+  threshold is a no-op and the evaluator's own verdict stands.
+- `minimize` with `threshold`: pass iff `score <= threshold`. `minimize`
+  **requires** a threshold; omitting it raises `ValueError`.
+- `expect` (`True` / `False`): pass iff `bool(score)` equals it. Cannot be combined
+  with `direction` or `threshold`, and must be a real boolean.
+- A metric that already carries an `"error"` in its `meta` is unaffected by any
+  objective.
+- Invalid objective config is validated for every case before any evaluator runs,
+  so a bad objective fails the whole run up front rather than partway through.
+
+## Reading the summary
+
+```python
+summary = evaluate(cases, ["decision_scores"])
+
+summary.total, summary.passed, summary.failed, summary.errors
+summary.pass_rate    # passed / total, or 1.0 for an empty case list
+
+for case in summary.cases:
+    print(case.case_id, case.status)          # status: "pass" | "fail" | "error"
+    for name, metric in case.metrics.items():
+        print(name, metric.score, metric.passed)
+        print(metric.meta.get("reasons", {}))  # per-sub-check failure reasons
+```
+
+`EvalMetric` is frozen (`score: float`, `passed: bool`, `meta: dict`). `CaseResult`
+is a namedtuple, and `EvalSummary` is a plain dataclass, so all three are
+straightforward to serialize for logging or regression tracking.
+
+## Notes
+
+- `llm_as_judge` needs `config["judge_fn"]`, a callable
+  `judge_fn(actual, expected) -> bool` you supply. No LLM backend is imported
+  unless you pass one in.
+- `decision_scores` governance checks are opt-in: policy compliance is only
+  evaluated when both `policy_engine` and `policy_id` are present.
+
+## See also
+
+- [Decision Intelligence](/guides/decision-intelligence) — producing the `Decision` records this module scores
+- [Reasoning](/reference/reasoning) — inference output that reasoning-text evaluators can measure
+- [Policy Engine](/guides/policy-engine) — the `policy_engine` used by `decision_scores`
+- [Ontology Evaluator](/reference/ontology) — separate tooling for ontology quality metrics
Author	SHA1	Message	Date
Zohaib Hassnain	071f78c77c	add Qodo review	2026-09-03 15:01:26 +05:00
Zohaib Hassnain	301c4636d7	docs(concepts): rewrite every code example against real API	2026-09-03 14:53:22 +05:00
Zohaib Hassnain	9e8db764d1	docs(quickstart): read parsed full_text (#1415 ) * docs(quickstart): read parsed full_text * correct schema	2026-09-03 14:40:58 +05:00
Zohaib Hassnain	865aad54df	docs(getting-started): fix broken APIs in the Knowledge Graph and GraphRAG tabs (#1414 ) * docs(getting-started): fix broken APIs in KG and GraphRAG tabs * docs: tighten GraphRAG example * docs: use extract_text() so the PDF example doesn't keyError	2026-09-03 14:33:23 +05:00
Zohaib HassnainandSameer Kadam	2d776b7370	docs(evals): update docs for the current evals API (#1398 ) * document evals API * docs(evals): fix evaluator behavior details --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>	2026-09-03 14:12:49 +05:00