semantica/docs/reference/evals.md

title, description, icon

title	description	icon
Evals Module	Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance: coming soon.	chart-line

semantica.evals is planned as a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance.

**`semantica.evals` is not yet implemented.** The module is a placeholder with `__all__ = []`. No classes or functions are available for import. This page describes the planned API only.

Planned Features

When released, semantica.evals will provide:

Planned Class	Role
KGEvaluator	Completeness, consistency, schema compliance, coverage, and orphan node detection
ExtractionEvaluator	NER precision / recall / F1 and relation extraction metrics against gold datasets
PipelineBenchmark	Throughput (docs/sec), per-step latency, peak memory, and error rate
RegressionTracker	Record runs and compare metrics across commits or config changes
EvalReport	Structured report: {scores, regressions, recommendations}
DeduplicationEvaluator	Merge precision, false positive / false negative rates
ReasoningEvaluator	Inference accuracy, rule coverage, and derivation depth

Current Workaround

Until semantica.evals ships, use semantica.ontology.OntologyEvaluator for ontology quality metrics:

from semantica.ontology import OntologyEvaluator

evaluator = OntologyEvaluator()

# evaluate_ontology takes the ontology dict only
result = evaluator.evaluate_ontology(ontology)

print("Coverage:    ", result.coverage_score)
print("Completeness:", result.completeness_score)
print("Gaps:        ", result.gaps)
print("Suggestions: ", result.suggestions)

# Full report with class granularity and relation completeness
report = evaluator.generate_report(ontology)
print("Coverage score:    ", report["evaluation"]["coverage_score"])
print("Completeness score:", report["evaluation"]["completeness_score"])
print("Relation coverage: ", report["relation_completeness"]["relation_coverage"])

EvaluationResult fields returned by evaluate_ontology():

Field	Type	Description
coverage_score	float	Fraction of competency questions answerable by the ontology
completeness_score	float	Average of class and property completeness scores
gaps	List[str]	Identified gaps in coverage
suggestions	List[str]	Improvement suggestions
metrics	dict	Detailed sub-metrics

• Semantic Extract — Extraction module.
• Knowledge Graph — Graph quality assessment.
• Pipeline — Pipeline performance metrics.
• Ontology Evaluator — Available now for ontology quality metrics.