mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
# Semantica Performance Benchmark Suite
|
|
|
|
This document outlines the architecture, directory structure, and usage of the performance benchmarking suite for the Semantica Agentic RAG framework.
|
|
|
|
## Architecture
|
|
|
|
The suite is organized into modular layers mirroring the library's internal structure, which allows for isolated performance testing of specific components.
|
|
|
|
### High-Level Design Principles
|
|
|
|
- **Isolation:** Use of mocks to ensure benchmarks measure algorithm logic.
|
|
|
|
- **Virtualization:** A custom `conftest.py` virtualization layer allows tests to run without heavy local dependencies.
|
|
|
|
- **Pedantic Measurement:** High-iteration counts and statistical rounds to filter out system noise.
|
|
|
|
## Directory Structure
|
|
|
|
Based on the current production environment, the suite is organized as follows:
|
|
|
|
| | |
|
|
| --------------------- | ------------------------------------------------------------------ |
|
|
| Folder | Description |
|
|
| context/ | Low-level graph operations and memory storage logic. |
|
|
| context_memory/ | Agent-level memory management and GraphRAG retrieval patterns. |
|
|
| core_processing/ | Throughput tests for NER, extraction, and graph building. |
|
|
| export/ | Serialization benchmarks for JSON, CSV, RDF, and GraphML. |
|
|
| infrastructure/ | Support scripts, including the regression comparison engine. |
|
|
| input_layer/ | Ingestion, parsing, and splitting performance. |
|
|
| normalize/ | Text cleaning, encoding handling, and date normalization. |
|
|
| ontology/ | Inference, serialization, and namespace management overhead. |
|
|
| output_orchestration/ | Parallelism and execution pipeline management. |
|
|
| quality_assurance/ | Deduplication and conflict resolution strategies. |
|
|
| results/ | Storage for benchmark JSON outputs and performance baselines. |
|
|
| storage/ | Latency tests for Vector stores (FAISS) and Triplet stores (Jena). |
|
|
| visualization/ | Computational cost of layout algorithms and chart rendering. |
|
|
|
|
## Usage
|
|
|
|
### Running the Suite
|
|
|
|
To run the full suite and generate a new results file:
|
|
|
|
```bash
|
|
python benchmarks/benchmark_runner.py
|
|
```
|
|
|
|
### Strict Mode (CI/CD)
|
|
|
|
The suite is designed to integrate with automated pipelines. Using the --strict flag will cause the runner to return a non-zero exit code if a performance regression greater than 15% is detected.
|
|
|
|
```bash
|
|
python benchmarks/benchmark_runner.py --strict
|
|
```
|
|
|
|
|
|
|
|
### Performance Comparison
|
|
|
|
The comparison engine (infrastructure/compare.py) uses Z-scores to distinguish between actual performance regressions and environmental noise.
|
|
|
|
- Regression: Change > 15% AND Z-score > 2.0.
|
|
|
|
- Noise: Change > 15% but Z-score < 2.0.
|
|
|
|
### Updating Baseline
|
|
|
|
When a performance change is intentional (e.g., a more complex but necessary algorithm is added), update the "gold standard" baseline:
|
|
|
|
```bash
|
|
cp benchmarks/results/run_latest.json benchmarks/results/baseline.json
|
|
```
|