mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
Mohd Kaif
5d70d0c10d
* docs: replace Exported Classes import blocks with summary tables across all 25 modules * docs: add method/parameter tables to parse, ingest, ontology, normalize, triplet_store, change_management, conflicts, export, graph_store, provenance, and semantic_extract modules
54 lines
2.1 KiB
Markdown
54 lines
2.1 KiB
Markdown
---
|
|
title: "Evals Module"
|
|
description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance — coming soon."
|
|
icon: "chart-line"
|
|
---
|
|
|
|
`semantica.evals` is planned as a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance.
|
|
|
|
<Warning>
|
|
**`semantica.evals` is not yet implemented.** The module exists as a placeholder (`__all__ = []`). No classes or functions are available for import. This page describes the planned API.
|
|
</Warning>
|
|
|
|
## Planned Features
|
|
|
|
When released, `semantica.evals` will provide:
|
|
|
|
| Planned Class | Role |
|
|
| --- | --- |
|
|
| `KGEvaluator` | Completeness, consistency, schema compliance, coverage, and orphan node detection |
|
|
| `ExtractionEvaluator` | NER precision / recall / F1 and relation extraction metrics against gold datasets |
|
|
| `PipelineBenchmark` | Throughput (docs/sec), per-step latency, peak memory, and error rate |
|
|
| `RegressionTracker` | Record runs and compare metrics across commits or config changes |
|
|
| `EvalReport` | Structured report: `{scores, regressions, recommendations}` |
|
|
| `DeduplicationEvaluator` | Merge precision, false positive / false negative rates |
|
|
| `ReasoningEvaluator` | Inference accuracy, rule coverage, and derivation depth |
|
|
|
|
## Current Workaround
|
|
|
|
Until `semantica.evals` ships, use `semantica.ontology.OntologyEvaluator` for ontology quality metrics:
|
|
|
|
```python
|
|
from semantica.ontology import OntologyEvaluator
|
|
|
|
evaluator = OntologyEvaluator()
|
|
report = evaluator.evaluate(ontology, kg)
|
|
print(f"Coverage: {report.coverage:.2%}")
|
|
print(f"Completeness: {report.completeness:.2%}")
|
|
```
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Semantic Extract" icon="magnifying-glass" href="semantic_extract">
|
|
Extraction module.
|
|
</Card>
|
|
<Card title="Knowledge Graph" icon="diagram-project" href="kg">
|
|
Graph quality assessment.
|
|
</Card>
|
|
<Card title="Pipeline" icon="gear" href="pipeline">
|
|
Pipeline performance metrics.
|
|
</Card>
|
|
<Card title="Ontology Evaluator" icon="sitemap" href="ontology">
|
|
Available now for ontology quality metrics.
|
|
</Card>
|
|
</CardGroup>
|