mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-13 04:04:09 +00:00
KaifAhmad1
9113ef3428
- Rewrote all 26 reference module pages: removed blockquote taglines and horizontal rule separators, added "What You Get" bullet summaries, added constructor/method parameter tables, expanded thin files (graph_store, triplet_store, visualization, provenance) with full API coverage, added backend comparison tables and real-world usage patterns - Renamed Modules tab from "API Reference" and group from "Context & Knowledge" to "Context & Intelligence" in docs.json - Fixed logo: copied "Semantica Logo.png" to web-safe semantica-logo.png and updated all 4 references in docs.json - Improved core docs (index, modules, concepts, quickstart, installation, getting-started) with better fonts, bullet points, and complete module listings (mcp_server, evals, core, utils previously missing) - Rewrote community pages (community, community-projects, contributing-guide, use-cases, architecture, faq, learning-more, glossary) with heading hierarchy fixes, expanded definitions, and better structure - Fixed markdown linter warnings: MD036 bold-as-heading, MD001 heading skips, MD040 missing code fence language, MD032 blank lines around lists
46 lines
1.9 KiB
Markdown
46 lines
1.9 KiB
Markdown
---
|
|
title: "Evals Module"
|
|
description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance."
|
|
icon: "chart-line"
|
|
---
|
|
|
|
`semantica.evals` provides a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance. Use it to benchmark extractors, validate pipeline output, and track quality regressions across runs.
|
|
|
|
<Warning>
|
|
**Coming Soon** — This module is currently in active development. Documentation will be expanded in the next release.
|
|
</Warning>
|
|
|
|
## Planned Capabilities
|
|
|
|
The Evals module will cover five evaluation areas:
|
|
|
|
| Area | What It Measures |
|
|
| ---- | ---------------- |
|
|
| **KG Quality** | Completeness, consistency, schema compliance, coverage metrics |
|
|
| **Extraction Accuracy** | NER precision / recall / F1, relation extraction metrics |
|
|
| **Pipeline Performance** | Throughput (docs/sec), latency per step, error rates |
|
|
| **Deduplication** | Merge accuracy, false positive / negative rates |
|
|
| **Reasoning** | Inference correctness, rule coverage, derivation depth |
|
|
|
|
## Scope
|
|
|
|
- **Offline evaluation** — compare against gold-standard annotated datasets
|
|
- **Regression tracking** — compare pipeline runs across commits or config changes
|
|
- **Live monitoring** — record quality metrics during production pipeline runs
|
|
- **Benchmark suites** — standard NER, RE, and KG construction benchmarks
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Semantic Extract" icon="magnifying-glass" href="semantic_extract">
|
|
Extraction module to evaluate.
|
|
</Card>
|
|
<Card title="Knowledge Graph" icon="diagram-project" href="kg">
|
|
Graph quality assessment.
|
|
</Card>
|
|
<Card title="Pipeline" icon="gear" href="pipeline">
|
|
Pipeline performance metrics.
|
|
</Card>
|
|
<Card title="Deduplication" icon="copy" href="deduplication">
|
|
Deduplication accuracy evaluation.
|
|
</Card>
|
|
</CardGroup>
|