semantica/docs/reference/evals.md

title, description, icon

title	description	icon
Evals Module	Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance — coming soon.	chart-line

semantica.evals is planned as a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance.

**`semantica.evals` is not yet implemented.** The module exists as a placeholder (`__all__ = []`). No classes or functions are available for import. This page describes the planned API.

Planned Features

When released, semantica.evals will provide:

Planned Class	Role
KGEvaluator	Completeness, consistency, schema compliance, coverage, and orphan node detection
ExtractionEvaluator	NER precision / recall / F1 and relation extraction metrics against gold datasets
PipelineBenchmark	Throughput (docs/sec), per-step latency, peak memory, and error rate
RegressionTracker	Record runs and compare metrics across commits or config changes
EvalReport	Structured report: {scores, regressions, recommendations}
DeduplicationEvaluator	Merge precision, false positive / false negative rates
ReasoningEvaluator	Inference accuracy, rule coverage, and derivation depth

Current Workaround

Until semantica.evals ships, use semantica.ontology.OntologyEvaluator for ontology quality metrics:

from semantica.ontology import OntologyEvaluator

evaluator = OntologyEvaluator()
report    = evaluator.evaluate(ontology, kg)
print(f"Coverage:     {report.coverage:.2%}")
print(f"Completeness: {report.completeness:.2%}")

Extraction module. Graph quality assessment. Pipeline performance metrics. Available now for ontology quality metrics.