From 5eefadaa7f3050406ee2f2dc18cb0a0ff7e895aa Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sat, 23 May 2026 23:02:03 +0530 Subject: [PATCH 01/11] docs: apply full Mintlify component overhaul to all 27 reference pages and concepts.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace plain markdown in every docs/reference/ file and docs/concepts.md with rich Mintlify JSX components — CardGroup, Steps, Tabs, AccordionGroup, Tip, Warning, Note, and CodeGroup — for a consistent, navigable, production-grade developer experience. --- docs/concepts.md | 267 ++++++++++--- docs/modules.md | 58 ++- docs/reference/change_management.md | 319 +++++++++++++--- docs/reference/conflicts.md | 378 +++++++++++++++---- docs/reference/context.md | 483 +++++++++++++++++------- docs/reference/core.md | 308 +++++++++++++-- docs/reference/deduplication.md | 397 ++++++++++++++++---- docs/reference/embeddings.md | 450 ++++++++++++++++------ docs/reference/evals.md | 225 ++++++++++- docs/reference/explorer.md | 354 +++++++++++++---- docs/reference/export.md | 479 ++++++++++++++--------- docs/reference/graph_store.md | 324 +++++++++++----- docs/reference/ingest.md | 563 +++++++++++++++++++++------- docs/reference/kg.md | 324 ++++++++++++---- docs/reference/llms.md | 427 +++++++++++++++++---- docs/reference/mcp_server.md | 192 +++++++--- docs/reference/normalize.md | 509 +++++++++++++++---------- docs/reference/ontology.md | 394 +++++++++++++++---- docs/reference/parse.md | 294 ++++++++++++--- docs/reference/pipeline.md | 438 +++++++++++++++++++--- docs/reference/provenance.md | 239 ++++++++++-- docs/reference/reasoning.md | 481 +++++++++++++++++------- docs/reference/seed.md | 392 +++++++++++++++---- docs/reference/semantic_extract.md | 338 +++++++++++++---- docs/reference/split.md | 446 +++++++++++++++++----- docs/reference/triplet_store.md | 290 ++++++++++---- docs/reference/utils.md | 146 ++++++-- docs/reference/vector_store.md | 229 +++++++++-- docs/reference/visualization.md | 368 ++++++++++++------ 29 files changed, 7838 insertions(+), 2274 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index 107d6059..08754bad 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -10,7 +10,21 @@ icon: "book-open" Semantica transforms unstructured data — documents, web pages, reports, databases — into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources. -At its core, Semantica adds a **context and intelligence layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable. +At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs **grounded**, **traceable**, and **auditable**. + + + + Knowledge graphs, GraphRAG retrieval, semantic embeddings, and temporal intelligence ground every LLM response in structured, queryable facts. + + + Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable. + + + `PluginRegistry` and `MethodRegistry` let you replace or augment any component — ingestors, extractors, reasoning engines, backends — without changing framework code. + + + +--- ## Knowledge Graphs @@ -26,7 +40,7 @@ This structure makes knowledge **searchable**, **connectable**, **queryable**, a ## Entity Extraction (NER) -Scanning text to find and classify real-world entities. +Scanning text to find and classify real-world entities: ```python # Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino." @@ -42,13 +56,15 @@ Scanning text to find and classify real-world entities. Each entity gets a type, confidence score, and a link to its source document. Three extraction methods are available: -- **`"pattern"`** — fast, regex-based, no API key required -- **`"ml"`** — local ML model, higher accuracy -- **`"llm"`** — LLM-powered, highest accuracy, supports all 8 providers +| Method | Speed | Accuracy | Requirements | +| ------ | ----- | -------- | ------------ | +| `"pattern"` | ⚡ Very fast | Moderate | No API key — regex-based | +| `"ml"` | Fast | High | Local ML model | +| `"llm"` | Medium | Highest | LLM provider — all 9 supported | ## Relationship Extraction -Finding how entities connect to each other. +Finding how entities connect to each other: ```python { @@ -71,8 +87,9 @@ Semantica uses embeddings for: - **Entity resolution** — match the same entity across different sources - **Precedent search** — find similar past decisions - **GraphRAG retrieval** — hybrid vector + graph traversal +- **Distance Intelligence** — N×N semantic distance matrices between any node set -**Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE +**Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings. ## GraphRAG @@ -80,14 +97,24 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses GraphRAG flow: User Query → Vector Search + Graph Traversal → Context Builder → LLM → Grounded Answer -**How it works:** + + + The query is embedded and used to seed both vector search and graph traversal simultaneously. + + + Semantica retrieves relevant graph context — entities, typed relationships, and multi-hop reasoning paths — alongside vector-similar text chunks. + + + Retrieved facts and reasoning paths are assembled into a structured prompt context, each fact tagged with its source node and confidence. + + + The LLM produces an answer where every claim links back to a source node in the graph — no floating assertions, no hallucinations from training data. + + -1. User submits a query -2. Semantica retrieves relevant graph context — entities, relationships, reasoning paths -3. The LLM generates a response grounded in that context -4. Every claim in the response links back to a source node in the graph - -This eliminates the hallucination and traceability problems of standard RAG. + + **GraphRAG eliminates the hallucination and traceability problems of standard RAG.** Standard RAG retrieves text chunks; GraphRAG retrieves structured facts with typed relationships. The LLM cannot confabulate structure that was never in the graph. + ## Ontology @@ -104,7 +131,7 @@ ontology = { } ``` -Semantica can auto-generate ontologies from your knowledge graph or import existing OWL/RDF/Turtle ontologies. The **Ontology Hub** (v0.5.0) adds a visual editor, SHACL Studio, alignment authoring, and a live health dashboard. +Semantica can auto-generate ontologies from your knowledge graph or import existing OWL/RDF/Turtle ontologies. The **Ontology Hub** (v0.5.0) adds a visual editor, SHACL Studio, alignment authoring, and a live health dashboard. See the [Ontology reference](reference/ontology) for the full 6-stage generation pipeline. ## Reasoning & Inference @@ -116,16 +143,60 @@ Known: Apple Inc. is headquartered in Cupertino Inferred: Steve Jobs has a connection to Cupertino ``` -| Engine | Description | -| ------ | ----------- | -| Forward chaining | Applies rules repeatedly until no new facts can be derived | -| Rete network | Efficient pattern matching for large rule sets | -| Deductive | Classical deductive reasoning | -| Abductive | Infers the most likely explanation | -| SPARQL | Query-based inference over RDF graphs | -| Datalog | Recursive Horn clause rules with fixpoint semantics (v0.4.0) | + + + Applies IF/THEN rules repeatedly until no new facts can be derived. Best for alert systems, compliance checks, and trigger-based workflows. -All engines produce **explainable inference paths**, not black-box conclusions. + ```python + from semantica.reasoning import ReasoningEngine + + engine = ReasoningEngine(llm_provider=llm) + result = engine.reason(facts=kg, rules=rule_set, method="forward_chaining") + ``` + + + Efficient pattern matching for large rule sets — the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts. + + ```python + engine = ReasoningEngine(llm_provider=llm) + result = engine.reason(facts=kg, rules=rule_set, method="rete") + ``` + + + **Deductive** — classical syllogistic reasoning from premises to guaranteed conclusions. + + **Abductive** — infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases. + + ```python + result = engine.reason(facts=kg, rules=rule_set, method="deductive") + result = engine.reason(facts=kg, rules=rule_set, method="abductive") + ``` + + + Recursive Horn clause rules with fixpoint semantics — handles transitive closure and recursive relationships that forward chaining cannot express. + + ```python + from semantica.reasoning import DatalogReasoner + + reasoner = DatalogReasoner() + result = reasoner.reason(facts=kg, rules=datalog_rules) + ``` + + + + | Engine | Description | Best For | + | ------ | ----------- | -------- | + | Forward chaining | Applies rules until fixpoint | Alert systems, compliance checks | + | Rete network | Efficient pattern matching | Large rule sets, high fact throughput | + | Deductive | Classical syllogistic reasoning | Mathematical and logical inference | + | Abductive | Most likely explanation | Diagnostics, investigation | + | SPARQL | Query-based inference over RDF | Semantic web, ontology reasoning | + | Datalog (v0.4.0) | Recursive Horn clause rules | Transitive closure, graph reachability | + + + + +All engines produce **explainable inference paths** — not black-box conclusions. Every derived fact includes the rules and premises that produced it. ## Temporal Intelligence @@ -142,13 +213,13 @@ tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2 snapshot = tkg.at(datetime(2021, 6, 15)) ``` -**Features:** Allen interval algebra (all 13 relations), OWL-Time export, `recorded_at` stamping, temporal provenance. +**Supported features:** Allen interval algebra (all 13 temporal relations), OWL-Time export, `recorded_at` stamping, temporal provenance. -**Common uses:** tracking company leadership changes, policy evolution, research timelines, financial instrument histories. +**Common uses:** tracking company leadership changes, policy evolution, research timelines, financial instrument histories, regulatory compliance windows. ## Distance Intelligence -Explore the semantic neighborhood of any entity in your graph. Useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology. +Explore the semantic neighborhood of any entity in your graph — useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology. ```python from semantica.kg import DistanceCalculator @@ -158,33 +229,68 @@ neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4) matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"]) ``` -**Features:** N×N distance matrices, ego-mode visualization, distance band classification (`near` / `mid` / `far`), embedding cache optimization. +**Features:** N×N semantic distance matrices, ego-mode visualization, distance band classification (`near` / `mid` / `far`), embedding cache optimization for large graphs. + +The [Visualization module](reference/visualization) renders distance matrices as interactive heatmaps and ego-mode neighborhood graphs. The [Explorer](reference/explorer) embeds distance intelligence directly in the browser dashboard. ## Deduplication & Entity Resolution Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance. -**Strategies:** + + -- **v1** — Jaro-Winkler similarity, suitable for small datasets -- **`blocking_v2`** — candidate blocking for large corpora -- **`hybrid_v2`** — combines blocking with semantic matching -- **`semantic_v2`** — pure embedding-based resolution, up to 7x faster than v1 + | Strategy | Algorithm | Best For | + | -------- | --------- | -------- | + | `v1` | Jaro-Winkler string similarity | Small datasets, fast baseline | + | `blocking_v2` | Candidate blocking + similarity | Large corpora — reduces O(n²) comparisons | + | `hybrid_v2` | Blocking + semantic embedding match | Mixed structured/unstructured entity names | + | `semantic_v2` | Pure embedding-based resolution | Up to 7× faster than v1; handles abbreviations and aliases | + + + + ```python + from semantica.deduplication import EntityDeduplicator + + deduplicator = EntityDeduplicator( + strategy="semantic_v2", + threshold=0.85, # similarity threshold for merge decision + embedding_model="all-mpnet-base-v2", + ) + + deduplicated_entities = deduplicator.deduplicate(entities) + ``` + + ## Provenance & Auditability Every fact in Semantica links back to: -- The source document it came from -- The extraction method used -- The ontology rules applied -- The reasoning steps that produced any inference +- The **source document** it came from +- The **extraction method** used (pattern / ML / LLM) +- The **ontology rules** applied during graph construction +- The **reasoning steps** that produced any inferred fact -This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). + + This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export. + + +```python +from semantica.provenance import ProvenanceManager + +prov = ProvenanceManager() +lineage = prov.get_entity_lineage("apple_inc") + +print(f"Source: {lineage.source_document}") +print(f"Method: {lineage.extraction_method}") +print(f"Extracted: {lineage.timestamp}") +print(f"Checksum: {lineage.checksum}") +``` ## Decision Intelligence -Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent. +Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes. ```python decision_id = context.record_decision( @@ -195,11 +301,16 @@ decision_id = context.record_decision( confidence=0.91, ) +# Find similar past decisions before making a new one precedents = context.find_precedents("model selection reasoning", limit=5) + +# Trace downstream impact of a past decision influence = context.analyze_decision_influence(decision_id) ``` -This prevents inconsistent decisions, enables audits, and lets agents learn from their own history. + + **Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply — reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history. + ## Conflict Detection @@ -207,10 +318,78 @@ When multiple sources disagree on the same fact, Semantica flags and resolves th **Resolution strategies:** -- Prefer the most recent source -- Prefer the most reliable source -- Majority vote across sources -- Flag for manual review +- **Recency** — prefer the most recent source +- **Source credibility** — prefer the most reliable source (configurable credibility scores) +- **Majority vote** — aggregate across all sources with ≥ 2 agreeing +- **Manual review** — flag for human arbitration; continue pipeline without blocking + +See the [Conflicts reference](reference/conflicts) for `ConflictResolver`, `SourceTracker`, and `InvestigationGuideGenerator`. + +## Custom Plugin Development + +Semantica is designed for extension. Any component — ingestor, extractor, graph builder, reasoning engine — can be replaced or augmented with a custom implementation registered at runtime. + + + + + `PluginRegistry` provides dynamic plugin discovery, registration, and loading across all modules. Register your own class under a string key; Semantica will use it wherever that key is referenced in config or pipeline steps. + + ```python + from semantica.core import PluginRegistry + + registry = PluginRegistry() + + # Register a custom ingestor + registry.register_plugin( + "my_sql_ingestor", MySQLIngestor, + version="1.0.0", + description="PostgreSQL ingestor for internal warehouse", + capabilities=["ingest"], + ) + + # Load and use + plugin = registry.load_plugin("my_sql_ingestor", connection_string="postgresql://...") + result = plugin.execute("SELECT * FROM documents") + + # Reference by name in pipeline YAML — no code changes needed + ``` + + ```yaml + steps: + - name: ingest + plugin: my_sql_ingestor + config: + connection_string: "${DB_URL}" + ``` + + **Extension points available:** ingestors, parsers, normalizers, extractors, reasoning engines, export formats, vector store backends, graph store backends, visualization renderers. + + + + + `MethodRegistry` lets you register custom methods on knowledge graph objects by name — useful for adding domain-specific graph operations without subclassing. + + ```python + from semantica.kg import MethodRegistry + + registry = MethodRegistry() + + def find_supply_chain_hops(graph, source_node, max_hops=3): + """Custom BFS traversal for supply chain graphs.""" + ... + + # Register under a string key + registry.register("supply_chain_hops", find_supply_chain_hops) + + # Call by name on any graph object + result = registry.call("supply_chain_hops", kg, source_node="Supplier_A", max_hops=5) + + # List all registered methods + print(registry.list_methods()) # ["supply_chain_hops", ...] + ``` + + + diff --git a/docs/modules.md b/docs/modules.md index 8b30c065..7ec65dc7 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -439,41 +439,69 @@ python -m semantica.mcp_server ### Seed -Deterministic data seeding for testing and development. +Bootstrap knowledge graphs from verified structured sources — fixed-point reference data, controlled vocabularies, and domain anchors. ```python from semantica.seed import SeedManager seed = SeedManager() seed.populate(kg, dataset="companies", count=100) + +# Load domain seeds from file or built-in datasets +seed.load_from_file("seed_data/industries.json") +seed.inject(kg) # merges seed nodes without duplicating existing entities ``` +**Use cases:** anchoring extraction with known entities, pre-populating ontology classes, deterministic test graph generation. + ### Evals -Evaluation harness for extraction and reasoning quality. +Evaluation framework for measuring KG quality, extraction accuracy, and pipeline performance. ```python -from semantica.evals import Evaluator +from semantica.evals import KGEvaluator, ExtractionEvaluator, PipelineEvaluator, RegressionTracker -evaluator = Evaluator() -scores = evaluator.evaluate(predicted_entities, ground_truth) -# Returns: {"precision": 0.91, "recall": 0.87, "f1": 0.89} +# KG quality +report = KGEvaluator().evaluate(kg, ontology=ontology) +print(f"Completeness: {report.completeness:.2%} Consistency: {report.consistency:.2%}") + +# Extraction accuracy +report = ExtractionEvaluator().evaluate_ner(predictions=extracted, gold_standard=annotated) +print(f"Precision: {report.precision:.3f} Recall: {report.recall:.3f} F1: {report.f1:.3f}") + +# Pipeline throughput and latency +metrics = PipelineEvaluator().benchmark(pipeline, data="data/", bench_runs=5) +print(f"Throughput: {metrics.docs_per_second:.1f} docs/sec") + +# Regression tracking across runs +tracker = RegressionTracker(db_path="eval_history.db") +run_id = tracker.record_run(pipeline_version="v1.2.0", metrics=metrics) +diff = tracker.compare(run_id, baseline_run_id="run_abc123") ``` -**Metrics:** precision, recall, F1 for NER, relation extraction, and reasoning +**Components:** `KGEvaluator`, `ExtractionEvaluator`, `PipelineEvaluator`, `RegressionTracker` ### Core Base classes, shared data models, and the plugin registry used across all modules. ```python -from semantica.core import Orchestrator, PluginRegistry +from semantica.core import Semantica, PluginRegistry, ConfigManager +# Top-level orchestrator +sem = Semantica(config_path="config.yaml") +sem.initialize() + +# Plugin registry — register custom components registry = PluginRegistry() registry.register("my_ingestor", MyCustomIngestor) + +# Config management +config = ConfigManager(config_path="config.yaml") +batch = config.get("processing.batch_size", default=32) ``` -**Components:** `Orchestrator`, `PluginRegistry`, `ConfigManager`, `Lifecycle` +**Components:** `Semantica`, `PluginRegistry`, `ConfigManager`, `LifecycleManager`, `HealthMonitor`, `Config` ### Utils @@ -504,8 +532,8 @@ from semantica.utils import helpers, validators, logging | [ingest](reference/ingest) | Data ingestion | `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor` | | [parse](reference/parse) | Document parsing | `DocumentParser`, `DoclingParser` | | [split](reference/split) | Text chunking | `TextSplitter` | -| [normalize](reference/normalize) | Data cleaning | `DataNormalizer` | -| [semantic_extract](reference/semantic_extract) | NER & relation extraction | `NERExtractor`, `RelationExtractor`, `TripletExtractor` | +| [normalize](reference/normalize) | Data cleaning | `TextNormalizer`, `EntityNormalizer`, `LanguageDetector` | +| [semantic_extract](reference/semantic_extract) | NER & relation extraction | `NERExtractor`, `RelationExtractor`, `TripletExtractor`, `SemanticAnalyzer`, `SemanticNetworkExtractor`, `ExtractionValidator` | | [kg](reference/kg) | Graph construction | `GraphBuilder`, `TemporalKnowledgeGraph`, `DistanceCalculator` | | [ontology](reference/ontology) | Schema management | `OntologyManager`, `SHACLGenerator` | | [reasoning](reference/reasoning) | Logical inference | `ReasoningEngine`, `DatalogEngine` | @@ -513,7 +541,7 @@ from semantica.utils import helpers, validators, logging | [vector_store](reference/vector_store) | Vector database | `VectorStore` | | [graph_store](reference/graph_store) | Graph database | `GraphStore` | | [triplet_store](reference/triplet_store) | RDF triple store | `TripletStore` | -| [deduplication](reference/deduplication) | Entity resolution | `EntityResolver`, `DuplicateDetector` | +| [deduplication](reference/deduplication) | Entity resolution | `EntityResolver`, `DuplicateDetector`, `ClusterBuilder`, `MergeStrategyManager` | | [conflicts](reference/conflicts) | Conflict resolution | `ConflictDetector` | | [context](reference/context) | Agent context & decisions | `AgentContext`, `ContextGraph` | | [provenance](reference/provenance) | W3C PROV-O lineage | `ProvenanceManager` | @@ -524,9 +552,9 @@ from semantica.utils import helpers, validators, logging | [explorer](reference/explorer) | Knowledge Explorer UI | `start_explorer` | | [llms](reference/llms) | LLM providers | `Groq`, `OpenAI`, `create_provider` | | [mcp_server](reference/mcp_server) | MCP stdio server | `python -m semantica.mcp_server` | -| [seed](reference/seed) | Test data seeding | `SeedManager` | -| [evals](reference/evals) | Quality evaluation | `Evaluator` | -| [core](reference/core) | Base classes & registry | `Orchestrator`, `PluginRegistry` | +| [seed](reference/seed) | KG bootstrapping from structured sources | `SeedManager` | +| [evals](reference/evals) | Quality evaluation | `KGEvaluator`, `ExtractionEvaluator`, `PipelineEvaluator`, `RegressionTracker` | +| [core](reference/core) | Base classes & registry | `Semantica`, `ConfigManager`, `PluginRegistry`, `LifecycleManager` | | [utils](reference/utils) | Shared utilities | `helpers`, `validators` | diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 365fc66a..059af4d4 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -4,36 +4,97 @@ description: "Version control, SHA-256 checksums, diff analysis, rollback, and a icon: "clock-rotate-left" --- -`semantica.change_management` provides enterprise-grade versioning and audit trails for knowledge graphs and ontologies — SHA-256 checksums, snapshot history, diff analysis, rollback protection, and compliance-ready audit export (HIPAA, SOX, FDA 21 CFR Part 11). +`semantica.change_management` provides enterprise-grade versioning and audit trails for knowledge graphs and ontologies. Every snapshot carries a SHA-256 checksum, every modification is logged, and every state can be diffed or rolled back — giving you a complete, tamper-evident record suitable for regulated industries. + + + Compliance frameworks supported out of the box: **HIPAA**, **SOX**, **GDPR**, and **FDA 21 CFR Part 11**. + ## What You Get -- **`TemporalVersionManager`** — snapshot, diff, rollback, and audit trail for knowledge graphs -- **`OntologyVersionManager`** — version control for OWL ontologies with diff and migration support -- **`VersionStorage`** — pluggable storage: `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production -- **`compute_checksum` / `verify_checksum`** — SHA-256 integrity verification -- **`ChangeLogEntry`** — structured record of every change in a snapshot + + + Snapshot, diff, rollback, and per-entity audit trail for knowledge graphs. + + + Version control for OWL ontologies with diff and schema migration support. + + + Pluggable backends — `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production. + + + SHA-256 / SHA-512 checksums to detect any unauthorised graph modification. + + + Structured record of every change: author, timestamp, checksum, and change list. + + + Full audit trail as CSV or JSON for regulatory review and subject-access requests. + + + +## Typical Workflow + + + + ```python + from semantica.change_management import TemporalVersionManager + + manager = TemporalVersionManager(storage_path="versions.db") + ``` + + + ```python + snapshot_id = manager.create_snapshot( + graph=kg, + version="v1.0", + author="user@example.com", + message="Before deduplication run" + ) + print(f"Snapshot: {snapshot_id}") + print(f"Checksum: {manager.get_checksum(snapshot_id)}") + ``` + + + Run deduplication, conflict resolution, merges, or any graph modification. The version manager tracks nothing automatically — you control when snapshots are taken. + + + ```python + snapshot_v2 = manager.create_snapshot( + graph=kg, + version="v2.0", + author="user@example.com", + message="After deduplication — 1 342 duplicates merged" + ) + ``` + + + ```python + diff = manager.diff("v1.0", "v2.0") + print(diff.summary) + + for change in diff.changes: + print(f" [{change.type}] {change.element}: {change.description}") + ``` + + + ```python + # Safe mode (default) — fails with a clear error rather than dropping nodes + manager.rollback(target_version="v1.0", allow_data_loss=False) + ``` + + ## TemporalVersionManager -Version control for knowledge graphs — snapshot, diff, and rollback: +Version control for knowledge graphs — snapshot, diff, and rollback. -```python -from semantica.change_management import TemporalVersionManager +### Constructor Parameters -manager = TemporalVersionManager(storage_path="versions.db") - -# Create a snapshot -snapshot_id = manager.create_snapshot( - graph=kg, - version="v1.0", - author="user@example.com", - message="Initial knowledge graph" -) - -print(f"Snapshot: {snapshot_id}") -print(f"Checksum: {manager.get_checksum(snapshot_id)}") -``` +| Parameter | Type | Default | Description | +| --------- | ---- | ------- | ----------- | +| `storage_path` | `str` | `None` | Path to SQLite database; uses in-memory if omitted | +| `storage` | `VersionStorage` | `None` | Explicit storage backend instance — overrides `storage_path` | ### List, Retrieve, and Rollback @@ -46,32 +107,51 @@ for v in versions: # Retrieve a specific version kg_v1 = manager.get_version("v1.0") -# Rollback to a previous version (safe by default — fails if data would be lost) +# Rollback to a previous version manager.rollback(target_version="v1.0", allow_data_loss=False) ``` -### Constructor Parameters - -| Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | -| `storage_path` | `str` | `None` | Path to SQLite database; uses in-memory if omitted | -| `storage` | `VersionStorage` | `None` | Explicit storage backend instance | + + `rollback(allow_data_loss=False)` is the safe default — it fails with a clear error if nodes were added after the target snapshot. Set `allow_data_loss=True` only when you explicitly intend to discard those changes. + ## Diff Analysis -Compare any two snapshots to see exactly what changed: +Compare any two snapshots to see exactly what changed — useful for code review, incident investigation, and regulatory audit: ```python diff = manager.diff("v1.0", "v2.0") print(f"Added nodes: {len(diff.added_nodes)}") print(f"Removed nodes: {len(diff.removed_nodes)}") +print(f"Modified nodes: {len(diff.modified_nodes)}") +print(f"Added edges: {len(diff.added_edges)}") +print(f"Removed edges: {len(diff.removed_edges)}") print(f"Modified edges: {len(diff.modified_edges)}") for change in diff.changes: print(f" [{change.type}] {change.element}: {change.description}") ``` + + +```python +@dataclass +class DiffResult: + from_version: str # source snapshot ID + to_version: str # target snapshot ID + added_nodes: List[str] # IDs of newly added entities + removed_nodes: List[str] # IDs of deleted entities + modified_nodes: List[str] # IDs of entities with changed properties + added_edges: List[str] # IDs of newly added relationships + removed_edges: List[str] # IDs of deleted relationships + modified_edges: List[str] # IDs of relationships with changed properties + changes: List[ChangeRecord] # ordered list of all individual changes + summary: str # human-readable summary line +``` + + + ## OntologyVersionManager Version control for OWL ontologies — save, diff, and track schema migrations: @@ -97,21 +177,38 @@ for change in diff.changes: ## VersionStorage Backends -```python -from semantica.change_management import ( - InMemoryVersionStorage, - SQLiteVersionStorage, -) + + + ```python + from semantica.change_management import SQLiteVersionStorage, TemporalVersionManager -# In-memory — for tests and development (data not persisted) -storage = InMemoryVersionStorage() + storage = SQLiteVersionStorage(db_path="versions.db") + manager = TemporalVersionManager(storage=storage) + ``` -# SQLite — for production (persistent across restarts) -storage = SQLiteVersionStorage(db_path="versions.db") + Persists all version history to disk. Survives process restarts. Recommended for any environment where you need to retain the audit trail. -# Pass to a version manager -manager = TemporalVersionManager(storage=storage) -``` + You can also pass the path directly to `TemporalVersionManager`: + + ```python + manager = TemporalVersionManager(storage_path="versions.db") + ``` + + + ```python + from semantica.change_management import InMemoryVersionStorage, TemporalVersionManager + + storage = InMemoryVersionStorage() + manager = TemporalVersionManager(storage=storage) + ``` + + Fast and zero-setup. Data is **not persisted** — all version history is lost when the process exits. Use this for unit tests and development only. + + + + + The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production — otherwise your entire version history disappears on restart. + ## Integrity Verification @@ -130,24 +227,13 @@ if not is_valid: raise RuntimeError("Graph has been modified since the checksum was recorded") ``` -## Audit Trail - -Full per-entity audit trail with CSV and JSON export for compliance reporting: - -```python -# Get all changes for a specific entity -trail = manager.get_audit_trail(entity_id="apple_inc") -for entry in trail: - print(f"{entry.timestamp} — {entry.author}: {entry.action} — {entry.description}") - -# Export audit trail for compliance -manager.export_audit_trail("audit.csv", format="csv") -manager.export_audit_trail("audit.json", format="json") -``` + + `verify_checksum` is deterministic — the same graph always produces the same digest. Use it as a pre-flight check before any compliance export to confirm the graph hasn't been tampered with since the last snapshot. + ## ChangeLogEntry -Every version snapshot includes a structured `ChangeLogEntry`: +Every version snapshot includes a structured `ChangeLogEntry` that records the full context of a change: ```python from semantica.change_management import ChangeLogEntry @@ -157,11 +243,124 @@ entry: ChangeLogEntry = manager.get_log_entry(snapshot_id) print(entry.version) # "v1.0" print(entry.author) # "user@example.com" print(entry.message) # "Initial knowledge graph" -print(entry.checksum) # SHA-256 hex digest -print(entry.created_at) # datetime -print(entry.changes) # list of individual change records +print(entry.checksum) # SHA-256 hex digest of the full graph state +print(entry.created_at) # datetime of snapshot creation +print(entry.node_count) # total nodes at this snapshot +print(entry.edge_count) # total edges at this snapshot +print(entry.changes) # list[ChangeRecord] — individual property-level changes ``` + + +```python +@dataclass +class ChangeLogEntry: + snapshot_id: str # unique snapshot identifier + version: str # human-assigned version tag, e.g. "v1.0" + author: str # identity of the user or process that created it + message: str # commit-style description of what changed + checksum: str # SHA-256 hex digest — changes if graph is tampered + created_at: datetime # UTC timestamp of snapshot creation + node_count: int # total entity count at this point in time + edge_count: int # total relationship count at this point in time + changes: List[ChangeRecord] # granular per-property change records + metadata: Dict # arbitrary key-value pairs for custom tagging +``` + + + +## Compliance and Audit Export + +All changes are preserved in a tamper-evident audit trail. Export for regulatory review: + + + +```python CSV export +# Full audit trail +manager.export_audit_trail("audit.csv", format="csv") + +# Scoped to a time range — SOX quarterly review +from datetime import datetime + +trail = manager.get_audit_trail( + from_date=datetime(2026, 1, 1), + to_date=datetime(2026, 3, 31), +) +manager.export_audit_trail("q1_audit.csv", trail=trail, format="csv") +``` + +```python JSON export +# Full audit trail +manager.export_audit_trail("audit.json", format="json") + +# Scoped to a specific entity — HIPAA subject-access request +trail = manager.get_audit_trail(entity_id="patient_001") +manager.export_audit_trail("patient_001_audit.json", trail=trail, format="json") +``` + + + +You can also iterate the trail directly: + +```python +trail = manager.get_audit_trail(entity_id="patient_001") +for entry in trail: + print(f"{entry.timestamp.isoformat()} | {entry.author} | {entry.action} | {entry.description}") +``` + +### Audit Fields + +| Field | Description | +| ----- | ----------- | +| `timestamp` | UTC ISO 8601 datetime | +| `entity_id` | ID of the affected entity or relationship | +| `author` | User or process that made the change | +| `action` | `CREATE` / `UPDATE` / `DELETE` / `MERGE` / `ROLLBACK` | +| `property` | Property name that changed (UPDATE rows only) | +| `old_value` | Previous value (UPDATE and DELETE rows) | +| `new_value` | New value (CREATE and UPDATE rows) | +| `snapshot_id` | ID of the containing snapshot | +| `checksum` | SHA-256 of the entity state after the change | + +### Compliance Coverage + + + + Use `get_audit_trail(entity_id="patient_001")` to retrieve every change ever made to a patient entity, then export to JSON for the access request response. The SHA-256 checksum on each entry proves the record has not been altered. + + + Use `get_audit_trail(from_date=..., to_date=...)` to scope the export to the relevant quarter. Export to CSV for upload to your audit management system. The immutable snapshot chain provides the chain of custody required by SOX Section 404. + + + After deleting a data subject's entities, snapshot the graph and diff against the pre-deletion snapshot. `diff.removed_nodes` provides a machine-readable record of exactly what was deleted and when, satisfying Article 17 documentation requirements. + + + Every `ChangeLogEntry` includes `author`, `timestamp`, and `checksum` — the three fields required for a compliant electronic record. `verify_checksum()` provides the tamper-evidence required by 21 CFR § 11.10(e). + + + +## Tips and Common Pitfalls + + + **Use `SQLiteVersionStorage` in production.** The default in-memory storage loses all version history when the process exits. Pass `storage_path="versions.db"` to `TemporalVersionManager` or create `SQLiteVersionStorage(db_path="versions.db")` explicitly. + + + + **Snapshot before every destructive operation.** Call `manager.create_snapshot()` before running deduplication, conflict resolution, or merge operations. `rollback()` is only possible if a snapshot exists before the change. + + + + **Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` produces a human-readable change summary in seconds — faster than comparing raw graph exports. Use it to review what changed before approving a version for production. + + + + **Export audit trails before compliance reviews.** `export_audit_trail("audit.csv", format="csv")` produces a complete tamper-evident record in one call. Schedule this export before quarterly reviews (SOX), regulatory inspections (FDA 21 CFR Part 11), or subject-access requests (GDPR). + + + + **Use `get_audit_trail(from_date=..., to_date=...)` for scoped reviews.** Exporting the full audit trail for a multi-year graph can produce millions of rows. Scope to a time window or entity ID for faster, focused reports. + + W3C PROV-O lineage tracking. diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index 71cb232b..9914a73d 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -6,74 +6,222 @@ icon: "triangle-exclamation" `semantica.conflicts` detects and resolves contradictions when multiple sources disagree on the same fact. It surfaces five conflict types, seven resolution strategies, and generates investigation guides for manual review — so conflicts never silently corrupt your knowledge graph. +## Why Detect Conflicts? + +When you ingest data from multiple sources, contradictions are inevitable. One annual report says Apple's revenue was $391B; a financial newswire says $383B. Without conflict detection, both values land in your graph and queries silently return inconsistent answers. + +Semantica's conflict detection makes disagreements explicit and actionable: + +- **Value conflicts** — SEC says revenue is $391B; Reuters says $383B +- **Type conflicts** — "Python" is a `ProgrammingLanguage` in one source, a `Snake` species in another +- **Temporal conflicts** — a CEO had two different employers during overlapping date ranges +- **Logical conflicts** — an entity simultaneously holds two mutually exclusive properties +- **Relationship conflicts** — the same relationship has inconsistent cardinality or properties across sources + ## What You Get -- **`ConflictDetector`** — value, type, temporal, logical, and relationship conflict detection -- **`ConflictResolver`** — 7 resolution strategies including voting, credibility-weighted, and temporal -- **`SourceTracker`** — track which source each conflicting fact came from, with credibility scores -- **`ConflictAnalyzer`** — pattern analysis, severity grouping, and trend identification -- **`InvestigationGuideGenerator`** — auto-generate step-by-step investigation checklists for human review + + + Value, type, temporal, logical, and relationship conflict detection across all entity pairs. + + + 7 resolution strategies including voting, credibility-weighted, and temporal preference. + + + Track which source each conflicting fact came from, with per-source credibility scores. + + + Pattern analysis, severity grouping, source-level statistics, and trend identification. + + + Auto-generate step-by-step investigation checklists for human and expert review. + + + `detect_conflicts()` and `resolve_conflicts()` for one-call workflows. + + + +## Quick Start + + + + ```python + from semantica.conflicts import SourceTracker + + tracker = SourceTracker() + tracker.set_credibility("sec_filings", 0.95) + tracker.set_credibility("pubmed", 0.92) + tracker.set_credibility("wikipedia", 0.80) + tracker.set_credibility("news_articles", 0.65) + ``` + + + ```python + from semantica.conflicts import ConflictDetector + + detector = ConflictDetector() + conflicts = detector.detect_conflicts(kg) + print(f"Found {len(conflicts)} conflicts") + + for conflict in conflicts: + print(f"[{conflict.conflict_type}] entity='{conflict.entity_id}' attr='{conflict.attribute}'") + print(f" Values: {conflict.values} Severity: {conflict.severity:.2f}") + ``` + + + ```python + from semantica.conflicts import ConflictAnalyzer + + analyzer = ConflictAnalyzer() + by_severity = analyzer.group_by_severity(conflicts) + print(f"Critical: {len(by_severity['critical'])}") + print(f"High: {len(by_severity['high'])}") + print(f"Low: {len(by_severity['low'])}") + ``` + + + ```python + from semantica.conflicts import ConflictResolver, InvestigationGuideGenerator, ResolutionStrategy + + resolver = ConflictResolver(source_tracker=tracker) + + # Auto-resolve low-severity + auto_resolved = resolver.resolve_conflicts( + by_severity["low"], + strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED, + ) + + # Generate investigation guides for critical conflicts + generator = InvestigationGuideGenerator() + for conflict in by_severity["critical"]: + guide = generator.generate(conflict) + print(f"\n{guide.title}") + for step in guide.steps: + print(f" [{step.order}] ({step.priority.upper()}) {step.description}") + ``` + + ## ConflictDetector ```python from semantica.conflicts import ConflictDetector -detector = ConflictDetector() +detector = ConflictDetector() conflicts = detector.detect_conflicts(kg) - -for conflict in conflicts: - print(f"[{conflict.conflict_type}] '{conflict.entity}' — {conflict.attribute}") - print(f" Sources: {conflict.sources}") - print(f" Severity: {conflict.severity:.2f}") ``` ### Detection Types -| Type | What It Detects | -| ---- | --------------- | -| `VALUE` | Same entity, same attribute, different values across sources | -| `TYPE` | Same entity classified as different types in different sources | -| `TEMPORAL` | Overlapping validity windows with contradictory facts | -| `LOGICAL` | Facts that violate ontology axioms or SHACL constraints | -| `RELATIONSHIP` | Inconsistent relationship properties across sources | +| Type | What It Detects | Example | +| ---- | --------------- | ------- | +| `VALUE` | Same entity, same attribute, different values across sources | Revenue $391B vs $383B | +| `TYPE` | Same entity classified as different types | "Python" as Language vs Snake | +| `TEMPORAL` | Overlapping validity windows with contradictory facts | CEO at two companies simultaneously | +| `LOGICAL` | Facts that violate ontology axioms or SHACL constraints | `is_alive=True` but `death_date` set | +| `RELATIONSHIP` | Inconsistent relationship properties across sources | Edge weight 0.9 vs 0.3 from two sources | Run targeted detection by type: ```python -# Detect all types (default) +# Detect all types at once (default) conflicts = detector.detect_conflicts(kg) -# Detect specific types only -value_conflicts = detector.detect_value_conflicts(entities, "name") +# Detect specific types only — faster for targeted checks +value_conflicts = detector.detect_value_conflicts(entities, "revenue") type_conflicts = detector.detect_type_conflicts(entities) relation_conflicts = detector.detect_relationship_conflicts(kg) ``` +**Key behaviours:** +- Severity scores are computed from the magnitude of disagreement — a $8B revenue discrepancy scores higher than a $1M discrepancy +- `LOGICAL` conflicts require an ontology or SHACL schema to be loaded; without one, they are not detected +- Detection runs in O(n·sources) time — it groups by entity+attribute and checks disagreement within each group + ## ConflictResolver ```python from semantica.conflicts import ConflictResolver, ResolutionStrategy resolver = ConflictResolver() -results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING) +results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING) for result in results: print(f"Resolved '{result.attribute}' → {result.resolved_value}") - print(f" Strategy: {result.strategy}") + print(f" Strategy: {result.strategy} Confidence: {result.confidence:.2f}") ``` -### Resolution Strategies +### Choosing a Resolution Strategy -| Strategy | Enum | Description | -|----------|------|-------------| -| Majority vote | `ResolutionStrategy.VOTING` | Most common value wins | -| Credibility-weighted | `ResolutionStrategy.CREDIBILITY_WEIGHTED` | Weighted by source credibility score | -| Most recent | `ResolutionStrategy.MOST_RECENT` | Prefer the most recently updated fact | -| First seen | `ResolutionStrategy.FIRST_SEEN` | Prefer the first observed value | -| Highest confidence | `ResolutionStrategy.HIGHEST_CONFIDENCE` | Prefer the fact with the highest confidence score | -| Manual review | `ResolutionStrategy.MANUAL_REVIEW` | Flag for human review | -| Expert review | `ResolutionStrategy.EXPERT_REVIEW` | Escalate to a domain expert | + + + Weights each source's value by its assigned credibility score — favors authoritative sources automatically: + + ```python + from semantica.conflicts import ConflictResolver, SourceTracker, ResolutionStrategy + + tracker = SourceTracker() + tracker.set_credibility("sec_filings", 0.92) + tracker.set_credibility("wikipedia", 0.80) + tracker.set_credibility("news_articles", 0.65) + + resolver = ConflictResolver(source_tracker=tracker) + results = resolver.resolve_conflicts( + conflicts, + strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED, + ) + ``` + + Best for: sources with known reliability rankings (SEC > blog). + + + Majority vote — most common value across sources wins: + + ```python + results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING) + ``` + + Best for: 3+ sources with roughly equal credibility. When all sources have identical credibility scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. + + + ```python + # Most recent source wins — for fast-changing facts + results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MOST_RECENT) + + # First seen wins — for stable facts (founding date, original name) + results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.FIRST_SEEN) + ``` + + + ```python + from semantica.conflicts import InvestigationGuideGenerator + + # Flag for human review — use with InvestigationGuideGenerator + results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MANUAL_REVIEW) + generator = InvestigationGuideGenerator() + + for conflict in conflicts: + guide = generator.generate(conflict) + print(f"{guide.title}") + for step in guide.steps: + print(f" [{step.order}] {step.description}") + ``` + + Best for: high-stakes decisions (severity > 0.8), regulated data (HIPAA/SOX), and domain-specific ambiguity. + + + + | Strategy | Enum | When to Use | + | -------- | ---- | ----------- | + | Majority vote | `VOTING` | 3+ sources with roughly equal credibility | + | Credibility-weighted | `CREDIBILITY_WEIGHTED` | Sources have different authority levels | + | Most recent | `MOST_RECENT` | Fast-changing facts: stock price, headcount, status | + | First seen | `FIRST_SEEN` | Stable facts: founding date, original name | + | Highest confidence | `HIGHEST_CONFIDENCE` | Extraction pipeline outputs confidence scores | + | Manual review | `MANUAL_REVIEW` | High-stakes decisions, regulated data | + | Expert review | `EXPERT_REVIEW` | Domain-specific ambiguity — escalate to a specialist | + + Use the convenience aliases for shorter code: @@ -83,92 +231,158 @@ from semantica.conflicts import voting, credibility_weighted, most_recent, highe results = resolver.resolve_conflicts(conflicts, strategy=voting) ``` -## Source Credibility Scoring - -Assign credibility weights per source so `CREDIBILITY_WEIGHTED` resolution favors authoritative sources: +## SourceTracker ```python from semantica.conflicts import SourceTracker +from datetime import datetime tracker = SourceTracker() -tracker.set_credibility("pubmed", 0.95) -tracker.set_credibility("wikipedia", 0.80) -tracker.set_credibility("user_input", 0.60) +tracker.set_credibility("sec_10k", 0.92) +tracker.set_credibility("wikipedia", 0.80) -resolver = ConflictResolver(source_tracker=tracker) -results = resolver.resolve_conflicts( - conflicts, - strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED +tracker.track_property_source( + entity_id="apple_inc", + property_name="revenue", + value="$391B", + source="sec_10k_2023", + timestamp=datetime(2024, 1, 26), ) -``` -`SourceTracker` also builds full traceability chains: - -```python -from semantica.conflicts import SourceTracker - -tracker = SourceTracker() -tracker.track_entity_source("apple_inc", "crunchbase") -tracker.track_property_source("apple_inc", "revenue", "annual_report_2023") +sources = tracker.get_property_sources("apple_inc", "revenue") +for s in sources: + print(f"{s.source}: {s.value} (credibility: {s.credibility:.2f})") chain = tracker.get_traceability_chain("apple_inc") ``` -## ConflictAnalyzer +**Key behaviours:** +- Credibility scores default to 0.50 for any source not explicitly set +- `SourceTracker` stores property-level provenance — so you can trace exactly which source contributed each value -Identify patterns and trends across large conflict sets: +## ConflictAnalyzer ```python from semantica.conflicts import ConflictAnalyzer analyzer = ConflictAnalyzer() -# Detect recurring patterns -patterns = analyzer.identify_patterns(conflicts) -for pattern in patterns: - print(f"Pattern: {pattern.type} — {pattern.frequency} occurrences") - -# Group by severity +patterns = analyzer.identify_patterns(conflicts) by_severity = analyzer.group_by_severity(conflicts) -print(f"Critical: {len(by_severity['critical'])}") -print(f"High: {len(by_severity['high'])}") -print(f"Low: {len(by_severity['low'])}") +source_stats = analyzer.analyze_sources(conflicts) +trends = analyzer.analyze_trends(conflicts, time_window="30d") -# Trend analysis over time -trends = analyzer.analyze_trends(conflicts, time_window="30d") +print(f"Trend direction: {trends['direction']}") # "increasing" | "stable" | "decreasing" +print(f"Change: {trends['change_pct']:.1f}%") ``` +**Key behaviours:** +- `identify_patterns()` groups conflicts by attribute name and type — use it to find systemic data quality issues +- `analyze_sources()` flags sources with disproportionate conflict rates — a signal that a source's pipeline needs review +- `analyze_trends()` compares conflict counts across time windows — a rising trend means a data source is degrading + ## InvestigationGuideGenerator -Auto-generate human-readable investigation guides for conflicts that can't be automatically resolved: +Auto-generate human-readable investigation checklists for conflicts requiring manual or expert review: ```python -from semantica.conflicts import InvestigationGuideGenerator, InvestigationGuide +from semantica.conflicts import InvestigationGuideGenerator generator = InvestigationGuideGenerator() -guide: InvestigationGuide = generator.generate(conflict) +guide = generator.generate(conflict) + +print(f"Title: {guide.title}") +print(f"Context: {guide.context}") -print(guide.title) -print(guide.context) for step in guide.steps: - print(f" [{step.order}] {step.description}") - print(f" Check: {step.check}") + print(f" [{step.order}] ({step.priority.upper()}) {step.description}") + print(f" → Verify: {step.check}") ``` -## Convenience Functions +## Schemas + + + ```python -from semantica.conflicts import ( - detect_conflicts, resolve_conflicts, analyze_conflicts, - track_sources, generate_investigation_guide -) - -conflicts = detect_conflicts(entities, method="value") -resolved = resolve_conflicts(conflicts, strategy="voting") -analysis = analyze_conflicts(conflicts, method="pattern") -guide = generate_investigation_guide(conflicts[0]) +@dataclass +class Conflict: + id: str + entity_id: str # the entity involved + attribute: str # the conflicting property name + values: List[str] # conflicting values (one per source) + sources: List[str] # source IDs for each value + conflict_type: ConflictType # VALUE | TYPE | TEMPORAL | LOGICAL | RELATIONSHIP + severity: float # 0.0 (minor) to 1.0 (critical) + confidence: float # detection confidence 0–1 + detected_at: datetime + metadata: Dict[str, Any] ``` + + + +```python +from semantica.conflicts import ConflictType + +ConflictType.VALUE_CONFLICT # revenue is $391B in source A, $383B in source B +ConflictType.TYPE_CONFLICT # "Apple" is ORGANIZATION in one source, PRODUCT in another +ConflictType.TEMPORAL_CONFLICT # overlapping validity windows with contradictory states +ConflictType.LOGICAL_CONFLICT # fact violates an ontology axiom or SHACL constraint +ConflictType.RELATIONSHIP_CONFLICT # inconsistent relationship properties across sources +``` + + + + +```python +@dataclass +class InvestigationGuide: + title: str # human-readable title for the conflict + context: str # summary of the disagreement + steps: List[InvestigationStep] # ordered checklist for the reviewer + +@dataclass +class InvestigationStep: + order: int + description: str # what to do + check: str # specific fact or document to verify + priority: str # "high" | "medium" | "low" +``` + + + + +## Tips and Common Pitfalls + + + **Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder — you lose the original source attribution. + + + + **Always set credibility scores.** The default credibility is 0.50 for all sources. Without explicit scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. The power of this strategy is in the differentiation. + + + + **Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with severity > 0.8 — high severity means the disagreement is large and the stakes of getting it wrong are high. + + + + **LOGICAL conflicts need a schema.** `detect_type_conflicts()` and `LOGICAL` detection only work if an OWL ontology or SHACL schema is loaded. Without one, `detect_conflicts()` will skip those types silently. + + + + **Use `analyze_sources()` to identify bad data feeds.** A single source causing 80% of your conflicts is a data quality problem upstream, not a conflict to resolve record by record. Flag it and investigate the source pipeline. + + + + **Severity is relative, not absolute.** A 0.5 severity score on a $1B revenue discrepancy and on a minor label difference both score 0.5 — the number reflects the disagreement structure, not the business impact. Domain context determines what to prioritize. + + + + **Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain. + + Resolve duplicate entities before conflict detection. diff --git a/docs/reference/context.md b/docs/reference/context.md index 44042e70..b4ba589c 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -8,81 +8,87 @@ icon: "brain" ## What You Get -- **`AgentContext`** — unified interface for memory, decision tracking, and graph-backed retrieval -- **`ContextGraph`** — persistent knowledge graph with centrality analysis, community detection, and decision management -- **`AgentMemory`** — low-level embedding-backed memory with TTL, tagging, and importance scoring -- **`DecisionRecorder`** — records decisions with causal chains, confidence scores, and outcome tracking -- **`CausalAnalyzer`** — traces downstream impact of any decision -- **`PolicyEngine`** — validates decisions against configurable rules before they're recorded + + + Unified interface for memory, decision tracking, and graph-backed retrieval. + + + Persistent knowledge graph with centrality analysis, community detection, and decision management. + + + Embedding-backed memory with TTL, tagging, and importance scoring. + + + Records decisions with causal chains, confidence scores, and outcome tracking. + + + Validates decisions against configurable rules before they're recorded. + + + Maps entity mentions to canonical URIs — prevents "Apple", "Apple Inc.", and "AAPL" from becoming three separate nodes. + + AgentContext hub: AI Agent calls store/retrieve against VectorStore and record_decision against ContextGraph +## Quick Start + + + + ```python + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore + + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768, index_path="context.faiss"), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, + ) + ``` + + + ```python + memory_id = context.store( + "GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%", + metadata={"source": "openai_blog", "date": "2024-01"} + ) + + results = context.retrieve("LLM benchmark comparisons", top_k=5) + for r in results: + print(f"{r['content']} (score: {r['score']:.3f})") + ``` + + + ```python + decision_id = context.record_decision( + category="model_selection", + scenario="Choose LLM for production reasoning pipeline", + reasoning="GPT-4 benchmark advantage justifies 3x cost increase", + outcome="selected_gpt4", + confidence=0.91, + ) + ``` + + + ```python + # Search past decisions — prevents contradictory choices across runs + precedents = context.find_precedents("model selection reasoning", limit=5) + + for p in precedents: + print(f"[{p.category}] {p.outcome} (similarity: {p.similarity:.2f})") + print(f" Reasoning: {p.reasoning}") + + # Analyze downstream impact of a past decision + influence = context.analyze_decision_influence(decision_id) + print(f"Decisions influenced: {len(influence.downstream_decisions)}") + ``` + + + ## AgentContext The main entry point. Wraps memory, graph, and decision tracking behind a single API. -```python -from semantica.context import AgentContext, ContextGraph -from semantica.vector_store import VectorStore - -context = AgentContext( - vector_store=VectorStore(backend="faiss", dimension=768), - knowledge_graph=ContextGraph(advanced_analytics=True), - decision_tracking=True, -) -``` - -### Store and Retrieve Memories - -```python -# Store a fact — embedded and indexed automatically -memory_id = context.store( - "GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%", - metadata={"source": "openai_blog", "date": "2024-01"} -) - -# Retrieve by semantic similarity -results = context.retrieve("LLM benchmark comparisons", top_k=5) -for r in results: - print(f"{r['content']} (score: {r['score']:.3f})") -``` - -### Record and Search Decisions - -```python -decision_id = context.record_decision( - category="model_selection", - scenario="Choose LLM for production reasoning pipeline", - reasoning="GPT-4 benchmark advantage justifies 3x cost increase", - outcome="selected_gpt4", - confidence=0.91, -) - -# Find similar past decisions — prevents inconsistent choices -precedents = context.find_precedents("model selection reasoning", limit=5) - -# Analyze downstream impact -influence = context.analyze_decision_influence(decision_id) -print(f"Decisions influenced: {len(influence.downstream_decisions)}") -``` - -### Multi-Hop GraphRAG - -```python -from semantica.llms import Groq - -llm = Groq(model="llama-3.3-70b-versatile") -result = context.query_with_reasoning( - query="What technologies have we chosen and why?", - llm_provider=llm, - max_hops=2, -) - -print(result["response"]) -for step in result["reasoning_path"]: - print(f" {step}") -``` - ### Constructor Parameters | Parameter | Type | Default | Description | @@ -106,6 +112,23 @@ for step in result["reasoning_path"]: | `query_with_reasoning(query, llm_provider, max_hops)` | `Dict` | GraphRAG with multi-hop traversal | | `get_context_insights()` | `Dict` | Analytics summary | +### Multi-Hop GraphRAG + +```python +from semantica.llms import Groq + +llm = Groq(model="llama-3.3-70b-versatile") +result = context.query_with_reasoning( + query="What technologies have we chosen and why?", + llm_provider=llm, + max_hops=2, +) + +print(result["response"]) +for step in result["reasoning_path"]: + print(f" {step}") +``` + ## ContextGraph The knowledge graph backing `AgentContext`. Can be used standalone for relationship modelling. @@ -115,12 +138,10 @@ from semantica.context import ContextGraph graph = ContextGraph(advanced_analytics=True) -# Add nodes and edges graph.add_node("Python", "language", properties={"paradigm": "multi-paradigm"}) graph.add_node("FastAPI", "framework", properties={"language": "Python"}) graph.add_edge("Python", "FastAPI", "enables") -# Decision management decision_id = graph.add_decision_simple( category="technology_choice", scenario="Web API framework selection", @@ -144,7 +165,134 @@ chain = graph.trace_decision_chain(decision_id) | `community_detection` | `bool` | `False` | Louvain community clustering | | `node_embeddings` | `bool` | `False` | Node2Vec embeddings for structural similarity | -## Decision Data Structure +### ContextGraph — Full Method Reference + +| Method | Returns | Description | +| ------ | ------- | ----------- | +| `add_node(id, label, properties)` | `None` | Add a node to the context graph | +| `add_edge(source, target, rel_type, properties)` | `None` | Add a directed edge | +| `query_neighbors(node_id, depth)` | `List[ContextNode]` | BFS neighbors up to given depth | +| `record_decision(...)` | `str` (decision_id) | Add decision node with causal edges | +| `find_precedents(category, limit)` | `List[Decision]` | Recent decisions in this category | +| `find_precedents_by_scenario(scenario, limit)` | `List[Decision]` | Semantically similar past scenarios | +| `analyze_decision_impact(decision_id)` | `Dict` | Downstream nodes influenced | +| `trace_decision_chain(decision_id)` | `CausalChain` | Full causality tree | +| `get_decision_insights()` | `Dict` | Aggregate stats across all decisions | +| `trace_decision_causality(decision_id)` | `CausalChain` | Alias for `trace_decision_chain` | + +## AgentMemory (Low-Level) + +For fine-grained control over memory storage, TTL, and importance scoring: + +```python +from semantica.context import AgentMemory +from semantica.vector_store import VectorStore + +memory = AgentMemory( + vector_store=VectorStore(backend="faiss", dimension=768), + capacity=10_000, # max memories before oldest are evicted + ttl_days=90, # memories older than this are auto-expired (None = never) +) + +memory_id = memory.store( + "Critical compliance rule: all trades must be pre-approved", + importance=0.95, + tags=["compliance", "trading"], +) + +results = memory.retrieve( + query="trade approval requirements", + top_k=5, + min_importance=0.5, + tags=["compliance"], +) + +memory.update(memory_id, importance=1.0) +memory.forget(memory_id) +all_memories = memory.get_all() +``` + +| Parameter | Type | Default | Description | +| --------- | ---- | ------- | ----------- | +| `vector_store` | `VectorStore` | required | Embedding backend for semantic retrieval | +| `capacity` | `int` | `1000` | Max items before LRU eviction | +| `ttl_days` | `Optional[int]` | `None` | Days before automatic expiry; `None` = keep forever | + +## PolicyEngine + +Validate decisions against configurable rules before they're committed: + +```python +from semantica.context import PolicyEngine + +policy = PolicyEngine() +policy.add_rule("confidence_threshold", lambda d: d.confidence >= 0.7) +policy.add_rule("requires_reasoning", lambda d: len(d.reasoning) >= 20) + +is_valid, violations = policy.validate(decision_data) + +if is_valid: + context.record_decision(**decision_data) +else: + # Create approval chain for manual review + chain = policy.create_approval_chain( + decision_data, + approvers=["manager@company.com", "compliance@company.com"], + ) + print(f"Approval chain created: {chain.chain_id}") +``` + +## EntityLinker + +Maps extracted entity mentions to canonical URIs — essential for cross-document entity resolution: + +```python +from semantica.context import EntityLinker + +linker = EntityLinker() + +entities = [ + {"text": "Apple Inc.", "type": "ORGANIZATION"}, + {"text": "Apple", "type": "ORGANIZATION"}, + {"text": "AAPL", "type": "ORGANIZATION"}, +] +linked = linker.link_entities(entities, sources=["reuters", "sec_filings"]) + +for e in linked: + print(f"{e.text} → {e.canonical_form} ({e.uri})") + print(f" confidence: {e.confidence:.2f}, sources: {e.sources}") +``` + +## ContextRetriever + +Hybrid retrieval combining vector similarity, graph traversal, and memory — gives richer context than pure vector search: + +```python +from semantica.context import ContextRetriever + +retriever = ContextRetriever( + vector_store=vector_store, + context_graph=context_graph, + agent_memory=memory, +) + +results = retriever.retrieve( + query="What decisions were made about cloud infrastructure?", + top_k=10, + vector_weight=0.5, # weight of vector similarity results + graph_weight=0.3, # weight of graph-traversal results + memory_weight=0.2, # weight of agent memory results + filters={"category": "infrastructure"}, +) + +for r in results: + print(f"[{r['source']}] score={r['score']:.3f}: {r['content'][:80]}") +``` + +## Data Structures + + + ```python @dataclass @@ -162,84 +310,157 @@ class Decision: causal_chain: List[str] # IDs of related decisions ``` -## AgentMemory (Low-Level) - -For fine-grained control over memory storage, TTL, and importance scoring: + + ```python -from semantica.context import AgentMemory - -memory = AgentMemory( - vector_store=VectorStore(backend="faiss", dimension=768), - max_memories=10_000, - ttl_days=90, -) - -memory.store("Important fact", importance=0.9, tags=["compliance"]) -results = memory.retrieve("fact query", top_k=5, min_importance=0.5) -memory.forget(memory_id) +@dataclass +class Precedent: + decision_id: str + similarity: float # 0–1 match score to current scenario + category: str + scenario: str + outcome: str + reasoning: str + confidence: float + timestamp: datetime ``` -## PolicyEngine - -Validate decisions against configurable rules before they're committed: + + ```python -from semantica.context import PolicyEngine - -policy = PolicyEngine() -policy.add_rule("confidence_threshold", lambda d: d.confidence >= 0.7) -policy.add_rule("requires_reasoning", lambda d: len(d.reasoning) >= 20) - -# Validate before recording -is_valid, violations = policy.validate(decision_data) -if is_valid: - context.record_decision(**decision_data) +@dataclass +class PolicyException: + exception_id: str + policy_rule: str # name of the rule that was violated + decision_id: str # the decision that triggered the exception + justification: str # why the exception was granted + approved_by: str # approver identity + timestamp: datetime + expiry: Optional[datetime] ``` + + + +```python +@dataclass +class ApprovalChain: + chain_id: str + decision_id: str + steps: List[ApprovalStep] + status: str # "pending" | "approved" | "rejected" + created_at: datetime + +@dataclass +class ApprovalStep: + step_id: str + approver: str + required: bool + status: str # "pending" | "approved" | "rejected" + comment: Optional[str] + timestamp: Optional[datetime] +``` + + + + +```python +@dataclass +class LinkedEntity: + text: str + canonical_form: str # normalized primary name + uri: str # e.g. "http://dbpedia.org/resource/Apple_Inc." + confidence: float + sources: List[str] # source documents that mention this entity + aliases: List[str] # all observed surface forms +``` + + + + ## Real-World Patterns -### Healthcare — Treatment Decisions + + + ```python + from semantica.context import AgentContext + from semantica.vector_store import VectorStore -```python -health_agent = AgentContext( - vector_store=VectorStore(backend="faiss", dimension=768), - decision_tracking=True, -) + health_agent = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + decision_tracking=True, + ) -health_agent.store("Patient has hypertension, type 2 diabetes") -health_agent.store("Patient allergic to penicillin — verified 2024-01") + health_agent.store("Patient has hypertension, type 2 diabetes") + health_agent.store("Patient allergic to penicillin — verified 2024-01") -decision_id = health_agent.record_decision( - category="treatment_plan", - scenario="Hypertension with comorbid diabetes", - reasoning="ACE inhibitors are renoprotective in diabetic patients — preferred over beta blockers", - outcome="prescribed_lisinopril", - confidence=0.91, -) + decision_id = health_agent.record_decision( + category="treatment_plan", + scenario="Hypertension with comorbid diabetes", + reasoning="ACE inhibitors are renoprotective in diabetic patients — preferred over beta blockers", + outcome="prescribed_lisinopril", + confidence=0.91, + ) -# Check for similar cases -precedents = health_agent.find_precedents("hypertension diabetes", limit=5) -``` + precedents = health_agent.find_precedents("hypertension diabetes", limit=5) + for p in precedents: + print(f"Past decision: {p.outcome} (similarity: {p.similarity:.2f})") + ``` + + + ```python + from semantica.context import AgentContext + from semantica.vector_store import VectorStore -### Finance — Loan Decisions + loan_agent = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + decision_tracking=True, + ) -```python -loan_agent = AgentContext( - vector_store=VectorStore(backend="faiss", dimension=768), - decision_tracking=True, -) + loan_agent.store("Applicant: credit score 750, DTI 28%, stable employment 4yr") -loan_agent.store("Applicant: credit score 750, DTI 28%, stable employment 4yr") + decision_id = loan_agent.record_decision( + category="loan_approval", + scenario="First-time homebuyer — 30yr fixed, 20% down", + reasoning="Credit score above threshold, DTI within limits, stable income verified", + outcome="approved_300k", + confidence=0.94, + ) + ``` + + -decision_id = loan_agent.record_decision( - category="loan_approval", - scenario="First-time homebuyer — 30yr fixed, 20% down", - reasoning="Credit score above threshold, DTI within limits, stable income verified", - outcome="approved_300k", - confidence=0.94, -) -``` +## Tips and Common Pitfalls + + + **Persist your vector store between runs.** Use `VectorStore(backend="faiss", index_path="context.faiss")` — without a path, the FAISS index lives in memory and is lost on shutdown. An agent that forgets everything on restart isn't an agent. + + + + **Enable `decision_tracking=True` from the start.** Adding it retroactively means historical decisions aren't linked to the causal chain — you lose the ability to trace how one decision influenced later ones. Enable it at agent initialization, even if you're not using it immediately. + + + + **Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. If precedents exist, surface them to the LLM as context — "we chose X for similar reasons before." + + + + **Set `ttl_days` to avoid memory bloat.** Without TTL, `AgentMemory` accumulates indefinitely. For operational agents, 30–90 day TTL keeps memory relevant to current context. Compliance-critical agents may need `ttl_days=None` (keep forever) with explicit archival. + + + + **Use `PolicyEngine` before recording irreversible decisions.** Decisions recorded with `record_decision()` become part of the causal chain immediately. If you need a human approval gate, validate first with `policy.validate()` and create an `ApprovalChain` — don't record until approved. + + + + **`ContextRetriever` is richer than direct vector search.** The three-channel fusion (vector + graph + memory) surfaces results that pure vector search misses — especially for decisions with complex causal relationships. Use it when you need comprehensive context assembly, not just semantic similarity. + + + + **`EntityLinker` prevents entity proliferation.** Without it, "Apple", "Apple Inc.", and "AAPL" land as three separate nodes in `ContextGraph`. Run `EntityLinker` on mentions before storing them to maintain a clean, canonical graph. + diff --git a/docs/reference/core.md b/docs/reference/core.md index 8afa1553..548e7151 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -8,16 +8,76 @@ icon: "gear" ## What You Get -- **`Semantica`** — orchestration class for coordinating complex multi-module workflows -- **`ConfigManager`** — unified config loading, merging, and validation with environment variable overrides -- **`LifecycleManager`** — startup/shutdown hooks and component health monitoring -- **`PluginRegistry`** — dynamic plugin discovery, registration, and loading -- **`MethodRegistry`** — register and dispatch custom orchestration methods + + + Orchestration class for coordinating complex multi-module workflows and full KG construction pipelines. + + + Unified config loading, merging, and validation with environment variable overrides. + + + Startup/shutdown hooks with priority ordering and component health monitoring. + + + Dynamic plugin discovery, registration, loading, and unloading. + + + Register and dispatch custom orchestration methods by name. + + + Live configuration state — dot-notation access, update, validate, and serialize. + + **Use individual modules directly** for the vast majority of use cases. Use the `Semantica` orchestration class only when you need application-level lifecycle management or a plugin system. +## Quick Start + + + + ```python + from semantica.core import ConfigManager + + manager = ConfigManager() + config = manager.load_from_file("config.yaml") + + # Override one key at runtime + config.set("processing.batch_size", 64) + ``` + + + ```python + from semantica.core import Semantica + + framework = Semantica(config=config) + framework.initialize() + + status = framework.get_status() + print(f"State: {status['state']}") # → "READY" + ``` + + + ```python + result = framework.build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + embeddings=True, + graph=True, + ) + ``` + + + ```python + # Always shut down in a finally block + try: + result = framework.build_knowledge_base(sources) + finally: + framework.shutdown(graceful=True) + ``` + + + ## Semantica (Orchestration) High-level entry point that coordinates the full KG construction pipeline: @@ -26,7 +86,7 @@ High-level entry point that coordinates the full KG construction pipeline: from semantica.core import Semantica, ConfigManager config_manager = ConfigManager() -config = config_manager.load_from_file("config.yaml") +config = config_manager.load_from_file("config.yaml") framework = Semantica(config=config) framework.initialize() @@ -43,8 +103,6 @@ finally: framework.shutdown(graceful=True) ``` -### Core Methods - | Method | Description | | ------ | ----------- | | `initialize()` | Initialize all framework components | @@ -61,7 +119,7 @@ Centralized config loading with deep-merge and environment variable overrides: from semantica.core import ConfigManager manager = ConfigManager() -config = manager.load_from_file("config.yaml") +config = manager.load_from_file("config.yaml") # Merge base config with environment-specific overrides merged = manager.merge_configs( @@ -69,44 +127,45 @@ merged = manager.merge_configs( manager.load_from_file("prod.yaml"), ) -# Nested key access with dot notation +# Nested dot-notation access batch_size = config.get("processing.batch_size", default=16) config.set("processing.batch_size", 64) +config.update({"quality": {"min_confidence": 0.75}}, merge=True) config.validate() + +config_dict = config.to_dict() ``` -### YAML Configuration +### Config Section Reference -```yaml -llm_provider: - name: openai - model: gpt-4o - api_key: ${OPENAI_API_KEY} +| Section | Key Fields | Description | +| ------- | ---------- | ----------- | +| `llm_provider` | `name`, `model`, `api_key`, `base_url` | LLM used for extraction and reasoning | +| `embedding_model` | `provider`, `model`, `dimension`, `device` | Embedding provider and model | +| `vector_store` | `backend`, `dimension`, `index_type` | Vector storage backend | +| `graph_db` | `backend`, `uri`, `user`, `password` | Graph database connection | +| `processing` | `batch_size`, `max_workers`, `chunk_size` | Parallelism and batching | +| `pipeline` | `retry_max`, `backoff`, `failure_strategy` | Pipeline retry and failure policy | +| `logging` | `level`, `format`, `file` | Logging configuration | +| `quality` | `min_confidence`, `dedup_threshold` | Quality thresholds | +| `security` | `redact_pii`, `allowed_domains` | Security and compliance settings | +| `custom` | any key | User-defined extension settings | -processing: - batch_size: 32 - max_workers: 4 +### Environment Variable Overrides -quality: - min_confidence: 0.7 - -logging: - level: INFO -``` - -Environment variable overrides (prefix `SEMANTICA_`): +Any config key can be overridden with a `SEMANTICA_` prefix using double underscores for nesting: ```bash -export SEMANTICA_PROCESSING_BATCH_SIZE=64 -export SEMANTICA_LOG_LEVEL=DEBUG +export SEMANTICA_PROCESSING__BATCH_SIZE=64 +export SEMANTICA_LLM_PROVIDER__MODEL=gpt-4o +export SEMANTICA_LOGGING__LEVEL=DEBUG +export SEMANTICA_QUALITY__MIN_CONFIDENCE=0.8 ``` ## LifecycleManager Manages framework state with a defined state machine and ordered startup/shutdown hooks: -**State machine:** `UNINITIALIZED` → `INITIALIZING` → `READY` → `RUNNING` → `STOPPING` → `STOPPED` - ```python from semantica.core import LifecycleManager @@ -139,7 +198,7 @@ manager.shutdown(graceful=True) ## PluginRegistry -Register custom components that participate in the full pipeline — provenance tracking, retry policies, and parallel execution included: +Register custom components that participate in the full pipeline: ```python from semantica.core import PluginRegistry @@ -152,13 +211,23 @@ class MyPlugin: return {"processed": True} registry = PluginRegistry(plugin_paths=["./plugins"]) -registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") +registry.register_plugin( + "my_plugin", MyPlugin, + version="1.0.0", + description="Custom domain extractor", + author="team@example.com", + capabilities=["extract"], +) plugin = registry.load_plugin("my_plugin", api_key="xxx") result = plugin.execute("sample data") +# Inspect registered plugins for info in registry.list_plugins(): - print(f"{info['name']}: {info['version']}") + print(f"{info['name']} v{info['version']} — {info['description']}") + +# Unload when done +registry.unload_plugin("my_plugin") ``` ## MethodRegistry @@ -178,6 +247,177 @@ from semantica.core.methods import build_knowledge_base result = build_knowledge_base(sources=["doc.pdf"], method="fast") ``` +## Schemas + + + + +```python +from semantica.core import SystemState + +SystemState.UNINITIALIZED # → startup() → +SystemState.INITIALIZING # → hooks complete → +SystemState.READY # → first operation → +SystemState.RUNNING # → shutdown() → +SystemState.STOPPING # → hooks complete → +SystemState.STOPPED +# Any unhandled exception during startup/shutdown → +SystemState.ERROR +``` + +Check current state at any time: + +```python +state = manager.get_state() +if manager.is_ready(): + result = framework.build_knowledge_base(sources) +``` + + + + +```python +@dataclass +class HealthStatus: + component: str # component name + healthy: bool # True = operational + message: str # human-readable status + timestamp: datetime # time of last check + details: Dict[str, Any] # component-specific diagnostics +``` + +```python +health = manager.health_check() +for name, status in health.items(): + icon = "✓" if status.healthy else "✗" + print(f"{icon} {name}: {status.message}") +``` + + + + +```python +@dataclass +class PluginInfo: + name: str + version: str + plugin_class: Type + description: str + author: str + dependencies: List[str] # pip package names required + capabilities: List[str] # e.g. ["ingest", "extract"] + metadata: Dict[str, Any] + +@dataclass +class LoadedPlugin: + info: PluginInfo + instance: Any # the live plugin object + config: Dict[str, Any] # config passed at load time + loaded_at: datetime +``` + +```python +registry = PluginRegistry() +registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") + +if registry.is_plugin_loaded("my_plugin"): + plugin = registry.get_loaded_plugin("my_plugin") +else: + plugin = registry.load_plugin("my_plugin") + +details = registry.get_plugin_info("my_plugin") +``` + + + + +## Complete Configuration Example + +```yaml +# config.yaml +llm_provider: + name: groq + model: llama-3.3-70b-versatile + api_key: ${GROQ_API_KEY} + +embedding_model: + provider: sentence-transformers + model: all-mpnet-base-v2 + dimension: 768 + device: cpu # "cpu" | "cuda" | "mps" + +vector_store: + backend: faiss + dimension: 768 + index_type: hnsw # "flat" | "ivf" | "hnsw" | "pq" + +graph_db: + backend: neo4j + uri: bolt://localhost:7687 + user: neo4j + password: ${NEO4J_PASSWORD} + +processing: + batch_size: 32 + max_workers: 4 + chunk_size: 512 + +pipeline: + retry_max: 3 + backoff: exponential # "fixed" | "linear" | "exponential" + failure_strategy: skip # "skip" | "stop" | "retry" + +quality: + min_confidence: 0.7 + dedup_threshold: 0.85 + +logging: + level: INFO # DEBUG | INFO | WARNING | ERROR + format: "%(asctime)s %(name)s %(levelname)s %(message)s" +``` + +Load and use: + +```python +from semantica.core import ConfigManager, Semantica + +manager = ConfigManager() +config = manager.load_from_file("config.yaml") + +config.set("processing.batch_size", 64) # runtime override + +framework = Semantica(config=config) +framework.initialize() +result = framework.build_knowledge_base(["doc1.pdf", "doc2.docx"]) +framework.shutdown() +``` + +## Tips and Common Pitfalls + + + **Use individual modules directly unless you need application lifecycle management.** `Semantica` orchestrates the full pipeline, but for simple scripts and notebooks, using `FileIngestor`, `NERExtractor`, and `GraphBuilder` directly is clearer and more debuggable. + + + + **Always call `framework.shutdown(graceful=True)` in a `finally` block.** Without graceful shutdown, in-flight pipeline steps may leave partial writes in your vector store or graph database. Wrapping in `try/finally` guarantees cleanup even on exceptions. + + + + **Use `ConfigManager.merge_configs()` for environment-specific overrides.** Keep a `base.yaml` with default settings and a `prod.yaml` with overrides. Merge them at startup rather than maintaining separate copies — this prevents configuration drift between environments. + + + + **Environment variable overrides use double underscores for nesting.** `SEMANTICA_PROCESSING__BATCH_SIZE=64` sets `processing.batch_size` — the double underscore (`__`) represents a nesting level. A single underscore is reserved for multi-word keys within the same level. + + + + **Register startup hooks with explicit priorities.** `register_startup_hook(fn, priority=10)` — lower numbers run first. If your database hook (priority 10) must run before your cache hook (priority 20), those numbers guarantee the order. Without explicit priorities, execution order is undefined. + + + + **Check `manager.is_ready()` before running pipelines.** If `Semantica.initialize()` failed partway through (e.g., a database connection refused), the state transitions to `ERROR` rather than `READY`. Always check before submitting work to avoid errors that are hard to trace. + + Pipeline execution and step orchestration. diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index 6d6ad337..f10c0cd3 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -6,22 +6,127 @@ icon: "copy" `semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control. +## Why Deduplicate? + +Real-world data sources disagree on names. "Apple Inc.", "Apple Computer", and "Apple International Ltd" can all refer to the same company — but if they land in your knowledge graph as separate nodes, every query that should return one entity returns three. Relationships, analytics, and retrieval all degrade. + +Deduplication solves this before it reaches the graph: + +- **Cross-source ingestion** — Wikipedia calls it "OpenAI", SEC filings call it "OpenAI, Inc.", your CRM calls it "OpenAI LLC" +- **Data entry variation** — "Steve Jobs", "Steven P. Jobs", "S. Jobs" are the same person +- **Transliteration differences** — Cyrillic, Chinese, or Arabic names romanized inconsistently across sources +- **Abbreviation drift** — "US", "U.S.", "United States", "USA" all mean the same thing + ## What You Get -- **`DuplicateDetector`** — pairwise and batch duplicate detection with configurable strategies -- **`EntityMerger`** — merge duplicate groups with configurable property-level merge policies -- **`SimilarityCalculator`** — multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding -- **`ClusterBuilder`** — Union-Find and hierarchical clustering for large-scale batch deduplication -- **Convenience functions** — `detect_duplicates`, `merge_entities`, `calculate_similarity` + + + Pairwise and batch duplicate detection with configurable strategies and result filtering. + + + Merge duplicate groups with configurable property-level merge policies. + + + Multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, and embedding. + + + Union-Find and hierarchical clustering for large-scale batch deduplication. + + + Reusable per-property merge rule configurations — define once, apply across operations. + + + `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster than v1 equivalents. + + + +## Quick Start + +```python +from semantica.deduplication import detect_duplicates, merge_entities + +# Detect +duplicates = detect_duplicates(entities, method="hybrid_v2", similarity_threshold=0.85) + +# Merge +merged = merge_entities(entities, duplicates, method="keep_most_complete") +print(f"Reduced {len(entities)} → {len(merged)} entities") +``` + +## Choosing a Strategy + + + + Combines blocking + string similarity + semantic embedding — best default for production: + + ```python + from semantica.deduplication import DuplicateDetector + + detector = DuplicateDetector(similarity_threshold=0.85) + duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") + ``` + + Best for: general production use — handles 95% of real-world cases without GPU. Catches string variants ("Apple Inc." / "Apple Inc") and semantic aliases ("Machine Learning" / "ML"). + + + Pure embedding-based similarity — highest accuracy for cross-language and abbreviation matching: + + ```python + detector = DuplicateDetector(similarity_threshold=0.85) + duplicates = detector.detect_duplicates(entities, strategy="semantic_v2") + ``` + + Best for: entities that use different words for the same concept ("ML" vs "Machine Learning"), cross-language entity matching, and abbreviation expansion. Requires embedding model. + + + Blocking + Jaro-Winkler string similarity only — fastest option for CPU-only environments: + + ```python + detector = DuplicateDetector(similarity_threshold=0.85) + duplicates = detector.detect_duplicates(entities, strategy="blocking_v2") + ``` + + Best for: large datasets (500K+ entities) where speed is critical and entities are in the same language. No GPU required. + + + + | Strategy | Algorithm | Speed | Accuracy | Best For | + | -------- | --------- | ----- | -------- | -------- | + | `jaro_winkler` | String similarity only (v1) | Fast | Medium | Small datasets, names, single-source | + | `blocking_v2` | Blocking + Jaro-Winkler | Very fast | Medium | Large datasets, speed-critical, CPU-only | + | `hybrid_v2` | Blocking + string + semantic | Fast | High | General production use — best default | + | `semantic_v2` | Embedding similarity | Medium | Highest | Semantic aliases, cross-language, abbreviations | + + **Rules of thumb:** + - Start with `hybrid_v2` — it handles 95% of real-world cases without GPU + - Use `semantic_v2` when entities use different words for the same concept + - Use `blocking_v2` when processing >500k entities and speed matters most + - Never use v1 strategies on new projects — they exist for backwards compatibility only + + + +### Threshold Tuning + +| Domain | Recommended Threshold | Notes | +| ------ | --------------------- | ----- | +| Person names | 0.85–0.90 | Names vary a lot; too high misses "Steve" / "Steven" | +| Organization names | 0.80–0.88 | Corporate suffixes create variation; lower threshold helps | +| Product names | 0.88–0.95 | Product names are more stable | +| Medical terms | 0.90–0.95 | High precision required; false merges are dangerous | +| General entities | 0.85 | Safe default starting point | + +Start at 0.85, inspect false positives and false negatives, then adjust ±0.05. ## DuplicateDetector -Find duplicate entity pairs with configurable strategies and result filtering: + + **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. + ```python from semantica.deduplication import DuplicateDetector -detector = DuplicateDetector(similarity_threshold=0.85) +detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities) for dup in duplicates: @@ -33,30 +138,23 @@ Fine-grained control over strategy, thresholds, and result size: ```python duplicates = detector.detect_duplicates( entities, - strategy="semantic_v2", # see strategies table below - min_similarity=0.85, # minimum score to consider a match - top_k_per_entity=3, # max candidates per entity - max_results=100, # total result cap - sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" + strategy="hybrid_v2", + min_similarity=0.85, + top_k_per_entity=3, # max candidates per entity — avoids false-positive floods + max_results=100, + sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" ) ``` -### Detection Strategies - -| Strategy | Algorithm | Speed | Accuracy | -| -------- | --------- | ----- | -------- | -| `jaro_winkler` | String similarity (v1) | Fast | Medium | -| `blocking_v2` | Blocking + Jaro-Winkler (v2) | Very fast | Medium | -| `hybrid_v2` | Blocking + semantic + string (v2) | Fast | High | -| `semantic_v2` | Embedding similarity (v2) | Medium | Highest | - - - **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. - +**Key behaviours:** +- Defaults to `hybrid_v2` when no strategy is specified +- `top_k_per_entity=3` prevents one entity from flooding results by being a near-match to everything +- Pairs are returned once — never both `(A, B)` and `(B, A)` +- `sort_by="similarity"` puts highest-confidence duplicates first for faster manual review ## EntityMerger -Merges detected duplicate groups into canonical entities, preserving provenance: +Merge detected duplicate groups into canonical entities, preserving provenance: ```python from semantica.deduplication import EntityMerger @@ -64,22 +162,24 @@ from semantica.deduplication import EntityMerger merger = EntityMerger() merged_entities = merger.merge_duplicates( entities, - strategy="keep_most_complete", # see strategies table below - preserve_provenance=True, # keep source references after merge + strategy="keep_most_complete", + preserve_provenance=True, ) + +print(f"Merged to: {len(merged_entities)} canonical entities") ``` ### Merge Strategies -| Strategy | Behavior | -| -------- | -------- | -| `keep_first` | Keep the first entity in each duplicate group | -| `keep_last` | Keep the most recently seen entity | -| `keep_most_complete` | Keep the entity with the most non-null properties | -| `union` | Merge all properties — non-conflicting fields combined | -| `voting` | Most common property value wins | +| Strategy | Behavior | When to Use | +| -------- | -------- | ----------- | +| `keep_first` | Keep the first entity in each group | Source order is meaningful (most authoritative first) | +| `keep_last` | Keep the most recently seen entity | Most recent source is most accurate | +| `keep_most_complete` | Keep the entity with the most non-null properties | Default — maximizes data richness | +| `union` | Merge all properties; combine non-conflicting fields | You want every known alias, tag, and label | +| `voting` | Most common property value wins | Multiple semi-reliable sources | -Fine-grained per-property merge rules: +### Per-Property Merge Rules ```python from semantica.deduplication import EntityMerger, PropertyMergeRule @@ -89,31 +189,41 @@ merger = EntityMerger( "name": PropertyMergeRule.KEEP_FIRST, "aliases": PropertyMergeRule.UNION, "description": PropertyMergeRule.KEEP_LONGEST, + "confidence": PropertyMergeRule.MAX, + "created_at": PropertyMergeRule.KEEP_FIRST, + "updated_at": PropertyMergeRule.KEEP_LAST, } ) + +merged_entities = merger.merge_duplicates(entities, preserve_provenance=True) ``` +| Rule | Behaviour | +| ---- | --------- | +| `KEEP_FIRST` | Value from the first entity in the group | +| `KEEP_LAST` | Value from the last entity | +| `KEEP_LONGEST` | Longest non-null string value | +| `KEEP_MOST_COMPLETE` | Entity with the most non-null fields (default fallback) | +| `UNION` | Combine all unique values into a list | +| `MAX` | Numerically largest value | +| `MIN` | Numerically smallest value | +| `VOTING` | Most frequently occurring value | + ## SimilarityCalculator -Compute multi-factor similarity scores between entity pairs: +Compute multi-factor similarity scores — useful for debugging why two entities were (or were not) detected as duplicates: ```python from semantica.deduplication import SimilarityCalculator -calc = SimilarityCalculator() - +calc = SimilarityCalculator() score = calc.calculate_similarity(entity_a, entity_b) -# → SimilarityResult(score=0.91, components={...}) print(score.score) # overall score 0.0–1.0 -print(score.components["label"]) # label similarity -print(score.components["embedding"]) # semantic similarity -print(score.components["property"]) # property overlap -``` +print(score.components["label"]) # label similarity contribution +print(score.components["embedding"]) # semantic similarity contribution +print(score.components["property"]) # property overlap contribution -Individual string and vector metrics: - -```python lev = calc.levenshtein("Apple Inc.", "Apple Inc") jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs") cos = calc.cosine_similarity(embedding_a, embedding_b) @@ -127,57 +237,202 @@ Build entity clusters for large-scale batch deduplication: ```python from semantica.deduplication import ClusterBuilder -builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" -result = builder.build_clusters(entities, similarity_threshold=0.85) +builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" +result = builder.build_clusters(entities, similarity_threshold=0.85) + +print(f"Original: {len(entities)} entities") +print(f"Clusters: {len(result.clusters)} groups") +print(f"Singletons: {result.singleton_count}") -print(f"Clusters: {len(result.clusters)}") for cluster in result.clusters: - print(f" [{cluster.id}] {cluster.members} — cohesion: {cluster.cohesion:.2f}") + print(f" [{cluster.id}] members={cluster.members} cohesion={cluster.cohesion:.2f}") +``` + +### Union-Find vs Hierarchical + +```python +# Union-Find — O(n·α(n)), scales to millions; use for production +builder = ClusterBuilder(algorithm="union_find") + +# Hierarchical — tighter clusters; use when quality matters more than speed +builder = ClusterBuilder(algorithm="hierarchical", linkage="average") +result = builder.build_clusters(entities, similarity_threshold=0.85) +``` + +**Key behaviours:** +- Union-Find groups entities transitively: if A≈B and B≈C, all three land in one cluster even if A and C are only 0.70 similar +- Hierarchical with `linkage="average"` avoids chaining by requiring average similarity across all pairs to meet the threshold + +### Cluster Quality Metrics + +```python +print(f"Silhouette score: {result.quality.silhouette_score:.3f}") +# → -1.0 to 1.0; above 0.5 is good, above 0.7 is excellent + +print(f"Avg cohesion: {result.quality.avg_cohesion:.3f}") +print(f"Avg separation: {result.quality.avg_separation:.3f}") +``` + +## MergeStrategyManager + +Define complex, reusable merge configurations once and apply them across multiple operations: + +```python +from semantica.deduplication import MergeStrategyManager, PropertyMergeRule + +manager = MergeStrategyManager() +manager.add_rule("name", PropertyMergeRule.KEEP_FIRST) +manager.add_rule("aliases", PropertyMergeRule.UNION) +manager.add_rule("description", PropertyMergeRule.KEEP_LONGEST) +manager.add_rule("confidence", PropertyMergeRule.MAX) +manager.add_rule("sources", PropertyMergeRule.UNION) +manager.add_rule("created_at", PropertyMergeRule.KEEP_FIRST) +manager.add_rule("updated_at", PropertyMergeRule.KEEP_LAST) +manager.set_default_rule(PropertyMergeRule.KEEP_MOST_COMPLETE) + +merged_entity = manager.merge(duplicate_group) ``` ## Blocking Strategies -Blocking reduces the O(n²) pairwise comparison problem to a manageable candidate set: - ```python detector = DuplicateDetector( - blocking_strategy="token", # "token" | "phonetic" | "ngram" + blocking_strategy="token", # "token" | "phonetic" | "ngram" blocking_threshold=0.6, similarity_threshold=0.85 ) ``` -## Custom Similarity Functions +| Blocking Strategy | How It Works | Best For | +| ----------------- | ------------ | -------- | +| `token` | Shared token overlap (default) | General entity names | +| `phonetic` | Soundex/Metaphone phonetic codes | Names with spelling variations | +| `ngram` | Character n-gram overlap | Short strings, typos | -Register domain-specific similarity logic: +## Custom Similarity Functions ```python from semantica.deduplication import method_registry -def drug_name_similarity(entity_a, entity_b): - # Match drug names by active compound - return score # 0.0 to 1.0 +def drug_name_similarity(entity_a, entity_b) -> float: + compound_a = entity_a.properties.get("active_compound", "") + compound_b = entity_b.properties.get("active_compound", "") + return 1.0 if compound_a == compound_b else 0.0 method_registry.register("similarity", "drug_name", drug_name_similarity) -detector = DuplicateDetector(similarity_method="drug_name") +detector = DuplicateDetector(similarity_method="drug_name", similarity_threshold=0.90) ``` -## Convenience Functions +## Schemas + + + ```python -from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity +@dataclass +class Cluster: + id: str + members: List[str] # entity IDs in this duplicate group + cohesion: float # mean pairwise similarity within the cluster (0–1) + centroid: str # member ID closest to the cluster centroid -# Quick detection -duplicates = detect_duplicates(entities, method="semantic_v2", similarity_threshold=0.85) - -# Quick merge -merged = merge_entities(entities, duplicates, method="keep_most_complete") - -# Quick similarity -score = calculate_similarity(entity_a, entity_b, method="hybrid_v2") +@dataclass +class ClusterResult: + clusters: List[Cluster] + singleton_count: int # entities with no duplicates found + merge_candidates: int # clusters with > 1 member + quality: ClusterQuality ``` + + + +```python +@dataclass +class ClusterQuality: + silhouette_score: float # -1 to 1; above 0.5 is good + avg_cohesion: float # mean within-cluster similarity + avg_separation: float # mean between-cluster distance +``` + + + + +## End-to-End Pipeline + + + + ```python + entities = load_entities_from_sources(["crunchbase", "wikipedia", "internal_db"]) + print(f"Loaded: {len(entities)} raw entities") + ``` + + + ```python + from semantica.deduplication import DuplicateDetector + + detector = DuplicateDetector(similarity_threshold=0.85) + duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") + print(f"Found: {len(duplicates)} duplicate pairs") + ``` + + + ```python + from semantica.deduplication import MergeStrategyManager, PropertyMergeRule + + manager = MergeStrategyManager() + manager.add_rule("name", PropertyMergeRule.KEEP_FIRST) + manager.add_rule("aliases", PropertyMergeRule.UNION) + manager.add_rule("description", PropertyMergeRule.KEEP_LONGEST) + manager.set_default_rule(PropertyMergeRule.KEEP_MOST_COMPLETE) + ``` + + + ```python + from semantica.deduplication import EntityMerger + + merger = EntityMerger() + merged = merger.merge_duplicates(entities, preserve_provenance=True) + print(f"Result: {len(merged)} canonical entities") + + for entity in merged: + if hasattr(entity, "source_entities"): + print(f"{entity.label} merged from: {entity.source_entities}") + ``` + + + +## Tips and Common Pitfalls + + + **Normalize before deduplicating.** Run `TextNormalizer` and `EntityNormalizer` first. "APPLE INC." and "apple inc." will score 0.50 on string similarity but 1.0 after case normalization. See the [Normalize](normalize) module. + + + + **Too many false positives?** Raise `similarity_threshold` by 0.05 or switch from `blocking_v2` to `hybrid_v2` to add semantic precision on top of string matching. + + + + **Too many missed duplicates?** Lower `similarity_threshold` by 0.05, or switch to `semantic_v2` to catch entities that use different words ("ML" vs "Machine Learning"). + + + + **Union-Find over-merges?** The transitive grouping means weak chains can connect unrelated entities. Switch to `hierarchical` clustering with `linkage="average"` to require that every pair within a cluster meets the threshold — not just a chain of nearby pairs. + + + + **Preserve provenance on merge.** Set `preserve_provenance=True` so you can always trace which source contributed each property to the merged entity. Critical for audit and debugging. + + + + **Inspect similarity components.** When a pair is flagged and you're not sure why, use `SimilarityCalculator.calculate_similarity()` to see the per-component breakdown (`label`, `embedding`, `property`) and identify which factor is driving the match. + + + + **Don't deduplicate after building the graph.** Deduplicate entities *before* `GraphBuilder` ingests them. Merging inside a live graph is possible but requires tracking and rewriting all relationship endpoints. + + Detect value conflicts between non-duplicate entities. @@ -186,9 +441,9 @@ score = calculate_similarity(entity_a, entity_b, method="hybrid_v2") GraphBuilder uses deduplication during construction. - Normalize entity names before deduplication. + Normalize entity names before deduplication for better accuracy. - Track merged entity lineage. + Track merged entity lineage and source attribution. diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index 6d03f996..c81cbf4d 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -1,216 +1,400 @@ --- title: "Embeddings Module" -description: "Text and graph embedding generation — Sentence-Transformers, FastEmbed, OpenAI, BGE, LlamaStore, with pooling strategies and graph embedding managers." +description: "Text and graph embedding generation — Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama — with pooling strategies, caching, and GPU acceleration." icon: "vector-square" --- -`semantica.embeddings` converts text and graph structures into dense vectors for semantic search, entity resolution, and GraphRAG retrieval. A single provider-agnostic API abstracts Sentence-Transformers, FastEmbed, OpenAI, BGE, and Ollama. +`semantica.embeddings` converts text and graph structures into dense vectors. These vectors power semantic search, entity resolution, GraphRAG retrieval, and Distance Intelligence across every Semantica module. A single provider-agnostic API abstracts Sentence-Transformers, FastEmbed, OpenAI, BGE, and Ollama behind one interface. + +## Why Embeddings Matter + +Raw text can't be compared mathematically. Embeddings translate meaning into geometry — two semantically similar sentences produce vectors that are close together in high-dimensional space, even when they share no words. + +Semantica uses embeddings for: + +- **Semantic search** — find knowledge graph nodes by meaning, not just keywords +- **Entity resolution** — detect that "Apple Inc." and "Apple Computer" refer to the same entity +- **Deduplication** — `semantic_v2` strategy measures entity similarity via embedding distance +- **GraphRAG retrieval** — hybrid vector + graph traversal for grounded LLM answers +- **Distance Intelligence** — N×N semantic distance matrices across entity sets +- **Semantic chunking** — detect topic shift boundaries in `TextSplitter(method="semantic_transformer")` ## What You Get -- **`EmbeddingGenerator`** — main entry point, provider-agnostic text embedding with batching -- **`TextEmbedder`** — text-specific embedding with automatic batching and disk caching -- **`GraphEmbeddingManager`** — node and subgraph embeddings for structural similarity -- **`VectorEmbeddingManager`** — full embedding lifecycle for vector store integration -- **Provider stores** — `OpenAIStore`, `BGEStore`, `FastEmbedStore`, `LlamaStore`, `ProviderStoreFactory` -- **Pooling strategies** — Mean, Max, CLS, Attention, Hierarchical pooling + + + Main entry point — provider-agnostic, handles batching automatically across all backends. + + + Text-specific with automatic batching, disk caching, and progress tracking. + + + Node and subgraph embeddings for structural similarity and GraphRAG context assembly. + + + Full lifecycle: embed → store → search in a single coordinated workflow. + + + `OpenAIStore`, `BGEStore`, `FastEmbedStore`, `LlamaStore`, and `ProviderStoreFactory`. + + + Mean, Max, CLS, Attention, and Hierarchical — control token-to-vector aggregation. + + + +## Installation + +| Provider | Install Command | API Key Required | +| -------- | --------------- | ---------------- | +| Sentence-Transformers (default) | `pip install semantica` | No | +| FastEmbed | `pip install "semantica[fastembed]"` | No | +| BGE | `pip install semantica` | No (uses sentence-transformers) | +| OpenAI | `pip install "semantica[llm-openai]"` | Yes — `OPENAI_API_KEY` | +| Ollama (LlamaStore) | `pip install "semantica[llm-ollama]"` | No — local server | +| All providers | `pip install "semantica[all]"` | Varies | + +Check which providers are available in your environment: + +```python +from semantica.embeddings import check_available_providers + +providers = check_available_providers() +# → {"sentence_transformers": True, "fastembed": True, "openai": False, "ollama": True} +``` + +## Quick Start + + + + ```python + from semantica.embeddings import EmbeddingGenerator + + # Default — Sentence-Transformers, free, runs locally + generator = EmbeddingGenerator(model="sentence-transformers") + ``` + + + ```python + embeddings = generator.generate(["Text about AI", "Machine learning concepts"]) + ``` + + + ```python + # Cosine similarity — 0.0 (unrelated) to 1.0 (identical meaning) + score = generator.similarity(embeddings[0], embeddings[1]) + print(f"Similarity: {score:.3f}") + ``` + + + ```python + from semantica.embeddings import VectorEmbeddingManager, TextEmbedder + from semantica.vector_store import VectorStore + + vector_store = VectorStore(backend="faiss", dimension=384) + manager = VectorEmbeddingManager( + embedder=TextEmbedder(model="sentence-transformers"), + vector_store=vector_store, + ) + ids = manager.embed_and_store(documents, metadata=metadata_list) + + results = manager.search("machine learning algorithms", top_k=10) + for result in results: + print(f"Score: {result.score:.3f} — {result.metadata['title']}") + ``` + + + +## Supported Models + +| Provider | Model | Dimension | Speed | Best For | +| -------- | ----- | --------- | ----- | -------- | +| `sentence-transformers` | `all-MiniLM-L6-v2` | 384 | Fast | Default — good balance of speed and quality | +| `sentence-transformers` | `all-mpnet-base-v2` | 768 | Medium | Higher retrieval quality | +| `bge` | `BAAI/bge-large-en-v1.5` | 1024 | Medium | State-of-the-art retrieval accuracy | +| `bge` | `BAAI/bge-small-en-v1.5` | 384 | Fast | Lightweight, competitive quality | +| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | CPU-optimised, low-latency production | +| `openai` | `text-embedding-3-small` | 1536 | API | Cost-effective OpenAI embedding | +| `openai` | `text-embedding-3-large` | 3072 | API | Highest quality via OpenAI API | +| `llama` (Ollama) | Any Ollama model | Varies | Local | Fully local, no API key | ## EmbeddingGenerator -Main entry point — handles provider selection and batching automatically: + + + ```python + from semantica.embeddings import EmbeddingGenerator -```python -from semantica.embeddings import EmbeddingGenerator + # Default model — all-MiniLM-L6-v2, dimension 384 + generator = EmbeddingGenerator(model="sentence-transformers") -# Sentence-Transformers (default, free, local) -generator = EmbeddingGenerator(model="sentence-transformers") -embeddings = generator.generate(["Text 1", "Text 2"]) + # Specific HuggingFace model + generator = EmbeddingGenerator(model="BAAI/bge-large-en-v1.5") -# Specific BGE model -generator = EmbeddingGenerator(model="BAAI/bge-large-en-v1.5") -embeddings = generator.generate(texts) + embeddings = generator.generate(texts) + similarity = generator.similarity(embeddings[0], embeddings[1]) + ``` -# OpenAI -import os -generator = EmbeddingGenerator( - model="openai", - model_name="text-embedding-3-small", - api_key=os.getenv("OPENAI_API_KEY") -) + Best for: default prototyping, no API key, good quality. + + + ```python + from semantica.embeddings import EmbeddingGenerator -# FastEmbed (fast, CPU-optimized) -generator = EmbeddingGenerator(model="fastembed") -``` + generator = EmbeddingGenerator(model="fastembed") + embeddings = generator.generate(texts) + ``` -### Supported Models + Best for: CPU-only production, lowest latency without GPU. + + + ```python + from semantica.embeddings import EmbeddingGenerator + import os -| Provider | Model | Dimension | Notes | -| -------- | ----- | --------- | ----- | -| `sentence-transformers` | `all-MiniLM-L6-v2` | 384 | Default, fast, free | -| `sentence-transformers` | `all-mpnet-base-v2` | 768 | Higher quality | -| `bge` | `BAAI/bge-large-en-v1.5` | 1024 | State-of-the-art retrieval | -| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Fast, CPU-optimized | -| `openai` | `text-embedding-3-small` | 1536 | OpenAI API | -| `openai` | `text-embedding-3-large` | 3072 | OpenAI API, highest quality | -| `llama` | any Ollama model | varies | Fully local inference | + generator = EmbeddingGenerator( + model="openai", + model_name="text-embedding-3-small", + api_key=os.getenv("OPENAI_API_KEY"), + ) + embeddings = generator.generate(texts) + ``` + + Best for: highest quality (3-large), or matching an OpenAI LLM pipeline. + + + ```python + from semantica.embeddings import LlamaStore + + store = LlamaStore(model="llama3.2", base_url="http://localhost:11434") + embedding = store.embed("Hello world") + ``` + + Best for: air-gapped or privacy-sensitive environments — no data leaves your machine. + + + ```python + from semantica.embeddings import EmbeddingGenerator + + # NVIDIA GPU + generator = EmbeddingGenerator(model="sentence-transformers", device="cuda") + + # Apple Silicon (M1/M2/M3) + generator = EmbeddingGenerator(model="sentence-transformers", device="mps") + + # CPU (default) + generator = EmbeddingGenerator(model="sentence-transformers", device="cpu") + ``` + + GPU reduces embedding time by 5–20× depending on batch size and model. + + + +### Constructor Parameters + +| Parameter | Type | Default | Description | +| --------- | ---- | ------- | ----------- | +| `model` | `str` | `"sentence-transformers"` | Provider name or HuggingFace model ID | +| `model_name` | `str` | Provider default | Specific model within a provider (OpenAI) | +| `api_key` | `str` | `None` | API key for cloud providers; reads env var if omitted | +| `device` | `str` | `"cpu"` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | +| `batch_size` | `int` | `32` | Texts per forward pass | +| `normalize` | `bool` | `True` | L2-normalise output vectors (required for cosine similarity) | +| `cache_dir` | `str` | `None` | Directory for disk caching of computed embeddings | ## TextEmbedder -Specialized for text with automatic batching and optional disk cache: +Specialised for text workloads — adds automatic batching, progress tracking, and disk caching: ```python from semantica.embeddings import TextEmbedder -embedder = TextEmbedder(model="sentence-transformers", cache_dir=".emb_cache") +embedder = TextEmbedder( + model="sentence-transformers", + cache_dir=".emb_cache", # persist embeddings to disk + cache_ttl=86400, # cache expiry in seconds (24h); None = never expires + batch_size=128, + show_progress=True, +) # Single text -embedding = embedder.embed("Hello world") +embedding = embedder.embed("A knowledge graph connects entities with typed relationships.") -# Batch — processes automatically in chunks -embeddings = embedder.embed_batch( - ["Text 1", "Text 2", ..., "Text 10000"], - batch_size=128, - show_progress=True -) +# Batch — auto-splits into batch_size chunks, shows progress bar +embeddings = embedder.embed_batch(texts, show_progress=True) ``` +**Key behaviours:** +- Cache is keyed on text content + model name — identical texts return cached vectors instantly +- Progress bar uses `tqdm` in terminal; switches to `tqdm.notebook` in Jupyter automatically +- Large batches (> 10k texts) are chunked internally to avoid OOM on GPU + ## Provider Stores -Each provider implements the `ProviderStore` interface and can be used independently: +Use provider stores directly when you need fine-grained control over a single backend: ```python from semantica.embeddings import ( OpenAIStore, BGEStore, FastEmbedStore, LlamaStore, - ProviderStoreFactory + ProviderStoreFactory, ) +import os # OpenAI -store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small") +store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small") embedding = store.embed("Hello world") # BGE (Sentence-Transformers wrapper) -store = BGEStore(model="BAAI/bge-large-en-v1.5") +store = BGEStore(model="BAAI/bge-large-en-v1.5", device="cpu") embedding = store.embed("Hello world") -# FastEmbed -store = FastEmbedStore(model="BAAI/bge-small-en-v1.5") +# FastEmbed — ONNX runtime, no CUDA required +store = FastEmbedStore(model="BAAI/bge-small-en-v1.5") embedding = store.embed("Hello world") -# LlamaStore (Ollama — fully local) -store = LlamaStore(model="llama3.2", base_url="http://localhost:11434") +# Ollama — fully local +store = LlamaStore(model="llama3.2", base_url="http://localhost:11434") embedding = store.embed("Hello world") -# Auto-select from config -store = ProviderStoreFactory.create(provider="openai", model="text-embedding-3-small") +# Auto-select from a name string — useful in config-driven pipelines +store = ProviderStoreFactory.create(provider="bge", model="BAAI/bge-large-en-v1.5") ``` ## Pooling Strategies -Control how token-level embeddings are aggregated into a single vector: +Transformer models produce one embedding per token. Pooling aggregates token embeddings into a single vector: -```python -from semantica.embeddings import ( - MeanPooling, MaxPooling, CLSPooling, - AttentionPooling, HierarchicalPooling, PoolingStrategyFactory -) + + + ```python + from semantica.embeddings import MeanPooling -# Mean pooling — default, best for most tasks -pooler = MeanPooling() -pooled = pooler.pool(token_embeddings) + pooler = MeanPooling() + pooled = pooler.pool(token_embeddings) # shape: (hidden_dim,) + ``` -# Max pooling — captures strongest activated features -pooler = MaxPooling() + Best for: retrieval, semantic search, and clustering — averages all token contributions. + + + ```python + from semantica.embeddings import MaxPooling -# CLS token — good for classification tasks -pooler = CLSPooling() + pooler = MaxPooling() + pooled = pooler.pool(token_embeddings) + ``` -# Attention-weighted pooling -pooler = AttentionPooling() + Best for: capturing the presence of any feature — takes the max activation per dimension. + + + ```python + from semantica.embeddings import CLSPooling -# Hierarchical: chunk-level → global mean (best for long documents) -pooler = HierarchicalPooling(chunk_size=512) + pooler = CLSPooling() + pooled = pooler.pool(token_embeddings) + ``` -# Create from config string -pooler = PoolingStrategyFactory.create(strategy="mean") -``` + Best for: classification-style tasks; models explicitly trained with CLS pooling (BERT). + + + ```python + from semantica.embeddings import HierarchicalPooling + + # Chunk text, mean-pool within chunks, then mean-pool chunks + pooler = HierarchicalPooling(chunk_size=512) + pooled = pooler.pool(token_embeddings) + ``` + + Best for: long documents exceeding the model's max sequence length — reports, papers, contracts. + + + + | Strategy | When to Use | + | -------- | ----------- | + | `mean` | Default for retrieval, semantic search, and clustering | + | `max` | When you want to capture the presence of any feature, not average presence | + | `cls` | Classification-style tasks; models explicitly trained with CLS pooling (BERT) | + | `attention` | When token importance varies significantly; slower but more accurate | + | `hierarchical` | Long documents exceeding model context length; reports, papers, contracts | + + ```python + from semantica.embeddings import PoolingStrategyFactory + + pooler = PoolingStrategyFactory.create(strategy="mean") + ``` + + + ## GraphEmbeddingManager -Embed graph nodes and subgraphs for structural similarity and GraphRAG context: +Embed graph nodes and subgraphs for structural similarity and GraphRAG context assembly: ```python -from semantica.embeddings import GraphEmbeddingManager +from semantica.embeddings import GraphEmbeddingManager, TextEmbedder manager = GraphEmbeddingManager( text_embedder=TextEmbedder(model="sentence-transformers"), - graph_store=graph_store + graph_store=graph_store, # optional — for persistence ) -# Embed all nodes in the graph +# Embed all nodes — uses node label + property text node_embeddings = manager.embed_nodes(kg) -# Embed a subgraph centered on a node (for GraphRAG context) +# Embed a subgraph centred on a node (for GraphRAG context) subgraph_embedding = manager.embed_subgraph( - kg, center_node="Apple Inc.", hops=2 + kg, + center_node="Apple Inc.", + hops=2, # include neighbours up to 2 hops away ) -# Find semantically similar nodes +# Find semantically similar nodes by ID similar = manager.find_similar_nodes("apple_inc", top_k=5) +for node_id, score in similar: + print(f"{node_id}: {score:.3f}") ``` -## VectorEmbeddingManager +**Key behaviours:** +- Node embedding combines the label, type, and all property values into a single text string before embedding +- `hops=2` captures the local neighbourhood — increase for richer context, decrease for speed +- Results from `find_similar_nodes` are sorted by cosine similarity descending -Manages the full embedding lifecycle — from raw text to stored, searchable vectors: +## Embedding Cache + +The disk cache avoids recomputing embeddings for unchanged text — critical for large corpora and repeated pipeline runs: ```python -from semantica.embeddings import VectorEmbeddingManager -from semantica.vector_store import VectorStore +from semantica.embeddings import TextEmbedder -vector_store = VectorStore(backend="faiss", dimension=768) - -manager = VectorEmbeddingManager( - embedder=TextEmbedder(model="sentence-transformers"), - vector_store=vector_store +embedder = TextEmbedder( + model="sentence-transformers", + cache_dir=".embeddings_cache", + cache_ttl=3600, # seconds — None means cache never expires ) -# Embed documents and store in one step -ids = manager.embed_and_store(documents, metadata=metadata_list) +# First call: computes and caches +embeddings = embedder.embed_batch(texts) -# Search by semantic similarity -results = manager.search("machine learning algorithms", top_k=10) +# Second call (same texts): returns from cache instantly +embeddings = embedder.embed_batch(texts) ``` + + The Distance Intelligence module (v0.5.0) uses the same cache to avoid recomputing embeddings during N×N matrix calculations across large entity sets. + + ## Similarity Computation ```python from semantica.embeddings import calculate_similarity -# Cosine similarity (most common) +# Cosine similarity — direction only, not magnitude; most common for text score = calculate_similarity(embedding_a, embedding_b, method="cosine") -# → 0.0 to 1.0 +# → 0.0 (orthogonal / unrelated) to 1.0 (identical direction) -# Euclidean distance (converted to similarity) +# Euclidean distance converted to similarity score = calculate_similarity(embedding_a, embedding_b, method="euclidean") -``` -## GPU Acceleration - -```python -# Use CUDA GPU for faster embedding generation -generator = EmbeddingGenerator(model="sentence-transformers", device="cuda") -# device options: "cpu" | "cuda" | "mps" -``` - -## Embedding Cache - -The embedding cache is used by Distance Intelligence (v0.5.0) to avoid recomputing embeddings for large N×N distance matrix calculations: - -```python -embedder = TextEmbedder( - model="sentence-transformers", - cache_dir=".embeddings_cache", - cache_ttl=3600 # seconds before cache entries expire -) +# Dot product — use when vectors are already normalised (equivalent to cosine) +score = calculate_similarity(embedding_a, embedding_b, method="dot") ``` ## Convenience Functions @@ -218,10 +402,10 @@ embedder = TextEmbedder( ```python from semantica.embeddings import ( embed_text, generate_embeddings, calculate_similarity, - pool_embeddings, check_available_providers + pool_embeddings, check_available_providers, ) -# Single text +# Single text — fastest path emb = embed_text("Hello world", method="sentence_transformers") # Batch @@ -232,6 +416,28 @@ providers = check_available_providers() # → {"sentence_transformers": True, "fastembed": True, "openai": False} ``` +## Tips and Common Pitfalls + + + **Dimension mismatch.** The dimension you pass to `VectorStore(dimension=...)` must exactly match your embedding model's output. `all-MiniLM-L6-v2` → 384, `all-mpnet-base-v2` → 768, `bge-large-en-v1.5` → 1024. Check with `generator.dimension` before creating the store. + + + + **Not normalising for cosine similarity.** If you compute cosine similarity directly (dot product), vectors must be L2-normalised first. `EmbeddingGenerator` normalises by default (`normalize=True`). If you disable it, use `calculate_similarity(..., method="cosine")` which normalises internally. + + + + **Sequence length limits.** Most models have a 512-token limit. Text beyond that is silently truncated. Use `TextSplitter(method="hierarchical")` + `HierarchicalPooling` for long documents. + + + + **Always use the same model for indexing and querying.** Vectors from different models are not comparable — they live in different vector spaces. Switching models requires re-embedding your entire corpus. + + + + **Cache invalidation.** The cache key is the text + model name. Switching models requires clearing the cache or using a different `cache_dir` — otherwise you'll get stale vectors silently returned. + + Store and search the generated embeddings. @@ -240,9 +446,9 @@ providers = check_available_providers() Chunk text before embedding for better retrieval quality. - Distance Intelligence uses graph embeddings. + Distance Intelligence uses graph embeddings for semantic neighbourhoods. - Semantic deduplication uses embeddings for entity resolution. + Semantic deduplication uses embedding distance for entity resolution. diff --git a/docs/reference/evals.md b/docs/reference/evals.md index 05dff800..0411dac4 100644 --- a/docs/reference/evals.md +++ b/docs/reference/evals.md @@ -6,28 +6,221 @@ icon: "chart-line" `semantica.evals` provides a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance. Use it to benchmark extractors, validate pipeline output, and track quality regressions across runs. +## What You Get + + + + Completeness, consistency, schema compliance, coverage, and orphan node metrics. + + + NER precision / recall / F1 and relation extraction metrics against gold-standard datasets. + + + Throughput (docs/sec), per-step latency, peak memory, and error rate benchmarking. + + + Record pipeline runs and compare metrics across commits or config changes. + + + Merge precision, false positive / false negative rates for deduplication strategies. + + + Inference accuracy, rule coverage, and derivation depth for reasoning engines. + + + +## Quick Start + + + + ```python + from semantica.evals import KGEvaluator + + evaluator = KGEvaluator() + report = evaluator.evaluate(kg, ontology=ontology) + + print(f"Completeness: {report.completeness:.2%}") + print(f"Consistency: {report.consistency:.2%}") + print(f"Coverage: {report.coverage:.2%}") + print(f"Orphan nodes: {report.orphan_count}") + ``` + + + ```python + from semantica.evals import ExtractionEvaluator + + evaluator = ExtractionEvaluator() + report = evaluator.evaluate_ner( + predictions=extracted_entities, + gold_standard=annotated_entities, + ) + + print(f"Precision: {report.precision:.3f}") + print(f"Recall: {report.recall:.3f}") + print(f"F1: {report.f1:.3f}") + print(f"By type: {report.per_type_metrics}") + ``` + + + ```python + from semantica.evals import PipelineEvaluator + + evaluator = PipelineEvaluator() + metrics = evaluator.benchmark(pipeline, data="data/", warmup_runs=2, bench_runs=5) + + print(f"Throughput: {metrics.docs_per_second:.1f} docs/sec") + print(f"Total duration: {metrics.total_seconds:.1f}s") + print(f"Per-step latency: {metrics.step_latencies}") + print(f"Peak memory (MB): {metrics.peak_memory_mb:.0f}") + print(f"Error rate: {metrics.error_rate:.2%}") + ``` + + + ```python + from semantica.evals import RegressionTracker + + tracker = RegressionTracker(db_path="eval_history.db") + + run_id = tracker.record_run( + pipeline_version="v1.2.0", + metrics=metrics, + config=config.to_dict(), + ) + + diff = tracker.compare(run_id, baseline_run_id="run_abc123") + for metric, change in diff.items(): + direction = "↑" if change > 0 else "↓" + print(f" {metric}: {direction} {abs(change):.2%}") + ``` + + + +## Evaluation Areas + + + + Measure completeness, consistency, schema compliance, and structural health of a knowledge graph: + + ```python + from semantica.evals import KGEvaluator + + evaluator = KGEvaluator() + report = evaluator.evaluate(kg, ontology=ontology) + + print(f"Completeness: {report.completeness:.2%}") # % entities with all required fields + print(f"Consistency: {report.consistency:.2%}") # % entities without type conflicts + print(f"Coverage: {report.coverage:.2%}") # % entity types in ontology + print(f"Total nodes: {report.node_count}") + print(f"Orphan nodes: {report.orphan_count}") # nodes with no edges + ``` + + **Key behaviours:** + - `consistency` requires an ontology — without one, it always returns 1.0 + - `orphan_count` flags disconnected nodes that likely represent extraction or deduplication errors + - `completeness` checks required properties defined in the ontology schema + + + Compare extracted entities and relations against annotated gold-standard data: + + ```python + from semantica.evals import ExtractionEvaluator + + evaluator = ExtractionEvaluator() + + # NER evaluation + ner_report = evaluator.evaluate_ner( + predictions=extracted_entities, + gold_standard=annotated_entities, + ) + print(f"Precision: {ner_report.precision:.3f}") + print(f"Recall: {ner_report.recall:.3f}") + print(f"F1: {ner_report.f1:.3f}") + print(f"By type: {ner_report.per_type_metrics}") + + # Relation extraction evaluation + rel_report = evaluator.evaluate_relations( + predictions=extracted_relations, + gold_standard=annotated_relations, + ) + print(f"Relation F1: {rel_report.f1:.3f}") + ``` + + + Benchmark throughput, latency, memory, and error rate across multiple runs: + + ```python + from semantica.evals import PipelineEvaluator + + evaluator = PipelineEvaluator() + metrics = evaluator.benchmark( + pipeline, + data="data/", + warmup_runs=2, # eliminate cold-start noise + bench_runs=5, # average over 5 real runs + ) + + print(f"Throughput: {metrics.docs_per_second:.1f} docs/sec") + print(f"Total duration: {metrics.total_seconds:.1f}s") + print(f"Per-step latency: {metrics.step_latencies}") + print(f"Peak memory (MB): {metrics.peak_memory_mb:.0f}") + print(f"Error rate: {metrics.error_rate:.2%}") + ``` + + + Store runs and compare metrics across pipeline versions: + + ```python + from semantica.evals import RegressionTracker + + tracker = RegressionTracker(db_path="eval_history.db") + + # Record a run with version tag and full config snapshot + run_id = tracker.record_run( + pipeline_version="v1.2.0", + metrics=metrics, + config=config.to_dict(), + ) + + # Compare to a previous run + diff = tracker.compare(run_id, baseline_run_id="run_abc123") + for metric, change in diff.items(): + direction = "↑" if change > 0 else "↓" + print(f" {metric}: {direction} {abs(change):.2%}") + ``` + + + +## When to Evaluate + +| Trigger | Evaluator to Use | What to Check | +| ------- | ---------------- | ------------- | +| New extraction model or method | `ExtractionEvaluator` | Precision, recall, F1 vs gold standard | +| After changing LLM provider | `ExtractionEvaluator` | Per-type F1 — check if rare types regressed | +| Before releasing new pipeline version | `PipelineEvaluator` | Throughput, latency, error rate | +| After deduplication strategy change | `KGEvaluator` | Orphan count, consistency score | +| Every production deployment | `RegressionTracker` | Compare vs previous baseline run | + +## Tips and Common Pitfalls + - **Coming Soon** — This module is currently in active development. Documentation will be expanded in the next release. + **Build a gold standard dataset early.** `ExtractionEvaluator` requires annotated ground truth. Without it, you're evaluating subjectively. Even 100 carefully annotated documents give you a meaningful baseline to track regressions against. -## Planned Capabilities + + **Evaluate per entity type, not just overall F1.** Aggregate F1 can hide regressions — if your model's PERSON F1 drops from 0.95 to 0.80 but ORGANIZATION improves, the average may look stable. Use `report.per_type_metrics` to catch type-specific regressions. + -The Evals module will cover five evaluation areas: + + **Store every benchmark run with `RegressionTracker`.** Run ID + version tag + config snapshot gives you a reproducible audit trail. Without it, "did the last release make things better?" has no objective answer. + -| Area | What It Measures | -| ---- | ---------------- | -| **KG Quality** | Completeness, consistency, schema compliance, coverage metrics | -| **Extraction Accuracy** | NER precision / recall / F1, relation extraction metrics | -| **Pipeline Performance** | Throughput (docs/sec), latency per step, error rates | -| **Deduplication** | Merge accuracy, false positive / negative rates | -| **Reasoning** | Inference correctness, rule coverage, derivation depth | + + **Run `PipelineEvaluator` with `warmup_runs=2`.** Cold starts are unrepresentative — model weights get cached, JIT compilation kicks in. Warmup runs eliminate this noise from your benchmark numbers. + -## Scope - -- **Offline evaluation** — compare against gold-standard annotated datasets -- **Regression tracking** — compare pipeline runs across commits or config changes -- **Live monitoring** — record quality metrics during production pipeline runs -- **Benchmark suites** — standard NER, RE, and KG construction benchmarks + + **`KGEvaluator` needs an ontology for consistency scoring.** Without an ontology, `consistency` always returns 1.0 — there's nothing to check against. Pass `ontology=ontology` to get meaningful consistency metrics. + diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index c9d5311e..0389d42b 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -8,11 +8,26 @@ icon: "map" ## What You Get -- **Graph Explorer** — interactive node/edge search, filtering, path highlighting, and neighborhood views -- **Ontology Hub** (v0.5.0) — browse class hierarchies, infer types, run SHACL validation, align ontologies -- **Distance Intelligence** (v0.5.0) — semantic similarity search, ego-mode neighborhood views, distance heatmaps -- **REST API** — 15+ endpoints for graph data, path finding, embeddings, and semantic search -- **CLI launcher** — `semantica-explorer` command for quick local startup + + + Interactive node/edge search, filtering, path highlighting, and neighborhood expansion. Indexed search at 0.004ms on 118k-node graphs. + + + Visual ontology editor, SHACL Studio, alignment authoring, health dashboard, and version control — all in the browser. + + + Semantic similarity search, ego-mode neighborhood views, N×N distance heatmaps, and distance band classification. + + + 15+ endpoints for graph data, path finding, embeddings, semantic search, analytics, and export — fully documented at `/docs`. + + + Long-running exports and analyses stream progress events in real time — no polling required. + + + `semantica-explorer --graph my_graph.json` for instant local startup without writing any Python. + + ## Installation @@ -24,103 +39,290 @@ Requires `uvicorn` and `fastapi`. Included automatically with `pip install seman ## Launch - + + + ```python + from semantica.explorer import start_explorer + from semantica.kg import GraphBuilder + from semantica.ontology import OntologyManager -```bash CLI -# Start the explorer on a saved graph -semantica-explorer --graph my_graph.json + kg = GraphBuilder().build(entities=entities, relationships=relationships) + ontology = OntologyManager() -# Custom host and port -semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080 + start_explorer( + graph=kg, + ontology=ontology, # optional — enables Ontology Hub tab + port=8080, + host="127.0.0.1", + open_browser=True, + ) + # → Serving at http://127.0.0.1:8080 + ``` + + + ```bash + # Start on a saved graph + semantica-explorer --graph my_graph.json -# Skip auto-opening the browser -semantica-explorer --graph my_graph.json --no-browser -``` + # Custom host and port + semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080 -```python Python -from semantica.context import ContextGraph + # Skip auto-opening the browser + semantica-explorer --graph my_graph.json --no-browser + ``` + + + ```python + start_explorer( + graph=kg, + port=8080, + enable_auth=True, + api_key="my-secret-key", + cors_origins=["https://app.example.com"], + session_timeout=1800, # 30-minute inactivity timeout + ) + ``` + + + ```bash + curl -X POST http://localhost:8080/api/import \ + -H "Content-Type: multipart/form-data" \ + -F "file=@updated_graph.json" + # Browser dashboard reloads automatically + ``` + + -graph = ContextGraph(advanced_analytics=True) -# ... build or load your graph ... -graph.save_to_file("my_graph.json") +## `start_explorer()` Parameters -import subprocess -subprocess.run(["semantica-explorer", "--graph", "my_graph.json", "--port", "8000"]) -``` - -```python Module -# Run directly as a Python module -import subprocess -subprocess.run(["python", "-m", "semantica.explorer", "--graph", "my_graph.json"]) -``` - - +| Parameter | Type | Default | Description | +| --------- | ---- | ------- | ----------- | +| `graph` | `KnowledgeGraph` or `ContextGraph` | *(required)* | The graph to load into Explorer | +| `ontology` | `OntologyManager` | `None` | Ontology to load into the Ontology Hub tab | +| `port` | `int` | `8000` | Port to bind the server | +| `host` | `str` | `"127.0.0.1"` | Host to bind. Use `"0.0.0.0"` for network access | +| `open_browser` | `bool` | `True` | Auto-open the dashboard in the default browser | +| `session_timeout` | `int` | `3600` | Session inactivity timeout in seconds; `None` disables | +| `enable_auth` | `bool` | `False` | Require `X-API-Key` header on all API requests | +| `api_key` | `str` | `None` | API key value when `enable_auth=True` | +| `cors_origins` | `list[str]` | `["*"]` | Allowed CORS origins. Restrict in production | +| `log_level` | `str` | `"info"` | Uvicorn log level (`"debug"` / `"info"` / `"warning"`) | ## CLI Reference | Flag | Default | Description | | ---- | ------- | ----------- | -| `--graph`, `-g` | *(required)* | Path to a ContextGraph JSON file | +| `--graph`, `-g` | *(required)* | Path to a saved graph JSON file | | `--port`, `-p` | `8000` | Port to bind the server | | `--host` | `127.0.0.1` | Host to bind the server | | `--no-browser` | `false` | Skip auto-opening the browser | +| `--enable-auth` | `false` | Require `X-API-Key` header on all requests | +| `--api-key` | `None` | API key value when `--enable-auth` is set | +| `--cors-origins` | `"*"` | Comma-separated list of allowed CORS origins | +| `--log-level` | `"info"` | Uvicorn log level | ## Features -### Graph Explorer + + + Core dashboard for navigating knowledge graphs: -Core dashboard for navigating knowledge graphs: + - **Indexed search** — find any node by label or type; 0.004ms on 118k-node graphs (v0.5.0) + - **Bidirectional path finding** — trace paths between any two nodes + - **Neighbor expansion** — click any node to expand its connections + - **Filter by entity type** — focus on Person, Organization, Event, or any custom type + - **Edge label display** — relationship types shown on all edges + - **Graph declutter** — workspace layout controls for dense graphs + + + Full ontology lifecycle management in the browser: -- **Indexed search** — find any node by label or type; 0.004ms on 118k-node graphs (v0.5.0) -- **Bidirectional path finding** — trace paths between any two nodes -- **Neighbor expansion** — click any node to expand its connections -- **Filter by entity type** — focus on Person, Organization, Event, or any custom type -- **Edge label display** — relationship types shown on all edges -- **Graph declutter** — workspace layout controls for dense graphs + - **Visual ontology editor** — drag-and-drop class and property authoring + - **SHACL Studio** — create, validate, and test SHACL shapes with live feedback + - **Alignment authoring** — author ontology alignments across schemas + - **Health dashboard** — graph quality metrics, validation status, coverage reports + - **Version control** — snapshot, diff, and restore ontology versions + + + Semantic neighborhood analysis centered on any node: -### Ontology Hub (v0.5.0) + - **N×N distance matrices** — pairwise semantic distances across a set of nodes + - **Ego-mode visualization** — focus on a single node's semantic neighborhood + - **Distance band classification** — nodes grouped as `near` / `mid` / `far` + - **Embedding cache** — optimized embedding reuse for large graphs + + + Thread-safe sessions with rollback protection: -Full ontology lifecycle management in the browser: + ```python + start_explorer( + graph=kg, + session_timeout=1800, # 30-minute inactivity timeout + enable_auth=True, + api_key="my-secret-key", + ) + ``` -- **Visual ontology editor** — drag-and-drop class and property authoring -- **SHACL Studio** — create, validate, and test SHACL shapes with live feedback -- **Alignment authoring** — author ontology alignments across schemas -- **Health dashboard** — graph quality metrics, validation status, coverage reports -- **Version control** — snapshot, diff, and restore ontology versions - -### Distance Intelligence (v0.5.0) - -Semantic neighborhood analysis centered on any node: - -- **N×N distance matrices** — pairwise semantic distances across a set of nodes -- **Ego-mode visualization** — focus on a single node's semantic neighborhood -- **Distance band classification** — nodes grouped as `near` / `mid` / `far` -- **Embedding cache** — optimized embedding reuse for large graphs - -### Knowledge Explorer API - -Full FastAPI backend accessible at `http://localhost:8000/docs`: - -- 12+ export formats (RDF, Parquet, AQL, JSON-LD, and more) -- WebSocket progress streaming for long operations -- Thread-safe sessions with rollback protection -- Audit trail for all operations + - Sessions are per connected browser tab + - Write operations (annotate, import) roll back automatically on failure + - All writes appended to audit trail at `/api/provenance/audit` + - Session state held in memory — use `/api/export/json` to persist between restarts + + ## API Endpoints -The FastAPI server exposes a REST API alongside the browser dashboard: +Full interactive docs at `http://localhost:8000/docs`. All endpoints available via REST. -| Endpoint | Method | Description | -| -------- | ------ | ----------- | -| `/api/graph/summary` | `GET` | Node count, edge count, entity types | -| `/api/graph/search` | `GET` | Full-text and type-filtered node search | -| `/api/graph/path` | `GET` | Bidirectional path between two nodes | -| `/api/graph/neighbors` | `GET` | Neighbors of a node with optional depth | -| `/api/ontology/validate` | `POST` | Run SHACL validation on the graph | -| `/api/export/{format}` | `GET` | Export graph in specified format | -| `/ws/progress` | `WS` | WebSocket stream for operation progress | + + -Full OpenAPI docs available at `http://localhost:8000/docs` when the server is running. + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/graph/summary` | `GET` | Node count, edge count, entity type distribution | + | `/api/graph/search` | `GET` | Indexed full-text and type-filtered node search | + | `/api/graph/node/{id}` | `GET` | Fetch a single node with all properties | + | `/api/graph/neighbors` | `GET` | Neighbors of a node — `?node_id=&depth=2` | + | `/api/graph/path` | `GET` | Bidirectional shortest path — `?source=&target=` | + | `/api/graph/subgraph` | `POST` | Extract a subgraph by node IDs or type filter | + | `/api/graph/annotate` | `POST` | Add a user annotation to a node or edge | + | `/api/graph/annotations` | `GET` | List all annotations on the graph | + + + + + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/ontology/classes` | `GET` | List all ontology classes and properties | + | `/api/ontology/validate` | `POST` | Run SHACL validation; returns violations | + | `/api/ontology/hierarchy` | `GET` | Class hierarchy as a tree structure | + | `/api/ontology/vocabulary` | `GET` | SKOS vocabulary terms and alt labels | + | `/api/ontology/align` | `POST` | Submit two ontologies for alignment | + | `/api/ontology/diff` | `POST` | Diff two ontology versions | + + + + + **Provenance:** + + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/provenance/entity/{id}` | `GET` | Full provenance lineage for an entity | + | `/api/provenance/source/{id}` | `GET` | All entities sourced from a document | + | `/api/provenance/audit` | `GET` | Full audit trail of Explorer operations | + + **Decisions:** + + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/decisions/list` | `GET` | Paginated list of recorded decisions | + | `/api/decisions/{id}` | `GET` | Single decision with causal chain | + | `/api/decisions/search` | `GET` | Precedent search — `?query=&limit=5` | + | `/api/decisions/influence/{id}` | `GET` | Downstream influence of a decision | + + **Analytics:** + + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/analytics/centrality` | `GET` | Degree, betweenness, PageRank scores | + | `/api/analytics/communities` | `GET` | Community detection result | + | `/api/analytics/distance` | `POST` | N×N distance matrix for a node list | + | `/api/analytics/neighborhood` | `GET` | Semantic neighborhood — `?node=&radius=0.4` | + + + + + **SPARQL & Temporal:** + + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/sparql` | `POST` | Execute a SPARQL SELECT query | + | `/api/temporal/snapshot` | `GET` | Graph snapshot at a point in time — `?at=ISO8601` | + | `/api/temporal/range` | `GET` | Nodes/edges active in a time range | + | `/api/temporal/diff` | `POST` | Diff two temporal snapshots | + + **Export & Import:** + + | Endpoint | Method | Description | + | -------- | ------ | ----------- | + | `/api/export/{format}` | `GET` | Export in: `turtle`, `json-ld`, `ntriples`, `rdf-xml`, `parquet`, `aql`, `csv`, `owl`, `arrow`, `lpg`, `yaml`, `distance-matrix` | + | `/api/import` | `POST` | Import a graph from file (replaces current graph in session) | + + + + +## WebSocket Progress + +Long-running operations stream progress events over WebSocket at `ws://localhost:8000/ws/progress`: + +```python +import asyncio, websockets, json + +async def watch_progress(): + async with websockets.connect("ws://localhost:8000/ws/progress") as ws: + async for message in ws: + event = json.loads(message) + print(f"[{event['operation']}] {event['step']} — {event['progress_pct']:.0f}%") + if event["status"] in ("completed", "failed"): + break + +asyncio.run(watch_progress()) +``` + +WebSocket event schema: + +```json +{ + "operation": "export", + "step": "serializing nodes", + "current": 3500, + "total": 10000, + "progress_pct": 35.0, + "status": "running", + "message": "Serializing 10000 nodes to Turtle...", + "error": null +} +``` + +`status` values: `"running"` | `"completed"` | `"failed"` | `"cancelled"` + +## Performance + +| Scenario | Latency | +| -------- | ------- | +| Node search (118k nodes, indexed) | 0.004ms | +| Neighbor expansion (depth 2) | < 5ms | +| Bidirectional path (118k nodes) | < 50ms | +| SPARQL SELECT (simple pattern) | < 20ms | +| N×N distance matrix (100 nodes) | ~2s (with embedding cache) | + +The node search index is built on startup. For graphs > 500k nodes, pass `index_build_timeout=120` to `start_explorer()` to allow more time. + +## Tips and Common Pitfalls + + + **Set `max_nodes` when loading large graphs.** `start_explorer(graph=kg, max_nodes=50000)` limits the rendered node count — Explorer's force-directed layout becomes unusable above ~10k nodes without limiting. Use `graph.filter(node_type="Organization")` first to focus on what matters. + + + + **Use authentication in shared environments.** Pass `enable_auth=True, api_key="..."` whenever Explorer is accessible to more than one person. Without auth, anyone who can reach the port can write to the graph via the annotate and import endpoints. + + + + **Export before the session ends.** Session state lives in memory and is lost on server restart. Call `/api/export/json` or `/api/export/turtle` to persist the current state before shutting down. Explorer does not auto-save. + + + + **Use WebSocket progress for long operations.** Export, analysis, and large SPARQL queries stream progress to `ws://localhost:8000/ws/progress`. Polling the REST endpoints instead gives no progress signal — use the WebSocket client so users see incremental updates. + + + + **Pass `session_timeout` for demos and shared notebooks.** The default session never expires. In Jupyter or shared environments, set `session_timeout=1800` (30 minutes) so stale sessions don't hold large graphs in memory. + + + + **Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API — pipe them into scripts to automate batch annotation, SPARQL querying, or exports. The browser UI is for interactive exploration and sharing; they use the same server. + @@ -129,7 +331,7 @@ Full OpenAPI docs available at `http://localhost:8000/docs` when the server is r Programmatic ontology management and SHACL generation. - + Programmatic graph rendering without the Explorer server. diff --git a/docs/reference/export.md b/docs/reference/export.md index 0bf19e2f..9eab66a5 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -8,191 +8,270 @@ icon: "file-export" ## What You Get -- **`RDFExporter`** — Turtle, JSON-LD, N-Triples, RDF/XML with namespace management -- **`ParquetExporter`** — columnar storage for Spark, BigQuery, Databricks, Snowflake -- **`LPGExporter`** — Cypher CREATE/MERGE statements for Neo4j and Memgraph -- **`ArangoAQLExporter`** — AQL INSERT statements for ArangoDB multi-model graphs -- **`GraphExporter`** — GraphML, GEXF, DOT for visualization tools like Gephi -- **`OWLExporter`** — OWL 2.0 ontology export in Turtle, XML, and JSON-LD -- **`CSVExporter`**, **`VectorExporter`**, **`ArrowExporter`**, **`DistanceExporter`**, **`ReportGenerator`** + + + Turtle, JSON-LD, N-Triples, RDF/XML with namespace management and optional PROV-O provenance embedding. + + + Columnar storage for Spark, BigQuery, Databricks, and Snowflake with explicit PyArrow typing. + + + Cypher CREATE/MERGE for Neo4j and Memgraph; AQL INSERT for ArangoDB vertex and edge collections. + + + GraphML, GEXF, DOT for Gephi and Graphviz. OWL 2.0 ontology export in Turtle, XML, and JSON-LD. + + + JSON, NumPy `.npy`, and FAISS index export for embedding vectors. Apache Arrow IPC for zero-copy transfer. + + + Distance matrix CSV/JSON from Distance Intelligence (v0.5.0). HTML, Markdown, and JSON analytics reports. + + -## RDFExporter +## Quick Start + + + + ```python + from semantica.export import RDFExporter + + exporter = RDFExporter() + ``` + + + ```python + # Interactive HTML — opens in browser, supports hover and click + exporter.export_to_file(graph, "output.ttl", format="turtle") + ``` + + + ```python + from semantica.export import export_rdf, export_parquet, export_csv + + export_rdf(graph, "output.ttl", format="turtle") + export_parquet(graph, "output/", compression="snappy") + export_csv(graph, "nodes.csv", target="nodes") + ``` + + + ```python + from semantica.export import ParquetExporter + + exporter = ParquetExporter(compression="snappy") + exporter.export_stream(graph, output_dir="output/", batch_size=10_000) + ``` + + + +## Exporters + + + + Export to W3C RDF formats — Turtle, JSON-LD, N-Triples, and RDF/XML: + + ```python + from semantica.export import RDFExporter + + exporter = RDFExporter() + + # Turtle (most readable RDF format) + exporter.export_to_file(graph, "output.ttl", format="turtle") + + # JSON-LD (best for APIs and Linked Data) + exporter.export_to_file(graph, "output.jsonld", format="json-ld") + + # N-Triples (streaming-friendly, one triple per line) + exporter.export_to_file(graph, "output.nt", format="nt") + + # RDF/XML (W3C standard, broadest compatibility) + exporter.export_to_file(graph, "output.xml", format="xml") + + # Export to string instead of file + rdf_str = exporter.export_to_rdf(graph, format="turtle") + ``` + + **Custom namespace management:** + + ```python + from semantica.export import NamespaceManager, RDFExporter + + ns_manager = NamespaceManager() + ns_manager.register("ex", "http://example.org/") + ns_manager.register("schema", "https://schema.org/") + + exporter = RDFExporter(namespace_manager=ns_manager) + ``` + + **Export with PROV-O provenance:** + + ```python + from semantica.export import RDFExporter + from semantica.provenance import ProvenanceManager + + provenance = ProvenanceManager() + # ... track entities during extraction ... + + exporter = RDFExporter(include_provenance=True, provenance_manager=provenance) + exporter.export_to_file(graph, "output_with_prov.ttl", format="turtle") + # → Each entity's prov:wasGeneratedBy, prov:wasDerivedFrom, prov:hadPrimarySource + # triples are included alongside the entity data triples + ``` + + + Columnar formats for analytics pipelines and human-readable export: + + ```python + from semantica.export import ParquetExporter + + exporter = ParquetExporter(compression="snappy") + # compression options: snappy | gzip | brotli | zstd | lz4 + + # Export nodes and edges as separate Parquet files + exporter.export_nodes(graph, "nodes.parquet") + exporter.export_edges(graph, "edges.parquet") + + # Export full graph partitioned by node type + exporter.export(graph, output_dir="graph_parquet/", partition_by="node_type") + ``` + + Schema is explicitly typed with PyArrow for clean Spark/BigQuery ingestion. + + ```python + from semantica.export import CSVExporter + + exporter = CSVExporter(delimiter=",") + exporter.export_nodes(graph, "nodes.csv") + exporter.export_edges(graph, "edges.csv") + ``` + + ```python + from semantica.export import YAMLExporter + + exporter = YAMLExporter() + exporter.export(graph, "graph.yaml") + + yaml_str = exporter.to_string(graph) + ``` + + + Export Cypher or AQL statements for direct graph database import: + + ```python + from semantica.export import LPGExporter + + exporter = LPGExporter() + + # CREATE statements + cypher = exporter.to_cypher(graph) + exporter.export(graph, "import.cypher", format="cypher") + + # MERGE statements (idempotent — safe to re-run) + cypher_merge = exporter.to_cypher(graph, use_merge=True) + ``` + + ```python + from semantica.export import ArangoAQLExporter + + exporter = ArangoAQLExporter( + vertex_collection="entities", + edge_collection="relationships" + ) + + aql = exporter.export(graph) # returns AQL string + exporter.export_to_file(graph, "import.aql") + ``` + + + Export for graph visualization tools and OWL ontology distribution: + + ```python + from semantica.export import GraphExporter + + exporter = GraphExporter() + + exporter.export(graph, "graph.graphml", format="graphml") # Gephi, yEd + exporter.export(graph, "graph.gexf", format="gexf") # Gephi streaming + exporter.export(graph, "graph.dot", format="dot") # Graphviz + ``` + + ```python + from semantica.export import OWLExporter + + exporter = OWLExporter() + exporter.export(ontology, path="ontology.ttl", format="turtle") + exporter.export(ontology, path="ontology.owl", format="xml") + exporter.export(ontology, path="ontology.json", format="json-ld") + ``` + + + Vector embeddings, Arrow IPC, distance matrices, and analytics reports: + + ```python + from semantica.export import VectorExporter + + exporter = VectorExporter() + exporter.export(embeddings, metadata, "vectors.json", format="json") + exporter.export(embeddings, metadata, "vectors.npy", format="numpy") + exporter.export(embeddings, metadata, "vectors.faiss", format="faiss") + ``` + + ```python + from semantica.export import ArrowExporter + + exporter = ArrowExporter() + exporter.export(graph, "graph.arrow") # requires pyarrow + ``` + + ```python + from semantica.export import DistanceExporter + + exporter = DistanceExporter() + exporter.export_matrix(distance_matrix, node_labels, "distances.csv") + exporter.export_ego(ego_neighborhood, center_node="Apple Inc.", path="ego.json") + ``` + + ```python + from semantica.export import ReportGenerator + + generator = ReportGenerator() + generator.generate(graph, analytics_result, "report.html", format="html") + generator.generate(graph, analytics_result, "report.md", format="markdown") + generator.generate(graph, analytics_result, "report.json", format="json") + ``` + + + +## Streaming Export + +For graphs too large to hold in memory, use streaming export — writes incrementally without buffering the full graph: ```python -from semantica.export import RDFExporter +from semantica.export import RDFExporter, ParquetExporter +# Stream RDF — yields triples one at a time, no full-graph buffer exporter = RDFExporter() +with exporter.stream(graph, format="turtle") as stream: + for triple_line in stream: + output_file.write(triple_line) -# Turtle (most readable RDF format) -exporter.export_to_file(graph, "output.ttl", format="turtle") - -# JSON-LD (best for APIs and Linked Data) -exporter.export_to_file(graph, "output.jsonld", format="json-ld") - -# N-Triples (streaming-friendly, one triple per line) -exporter.export_to_file(graph, "output.nt", format="nt") - -# RDF/XML (W3C standard, broadest compatibility) -exporter.export_to_file(graph, "output.xml", format="xml") - -# Export to string instead of file -rdf_str = exporter.export_to_rdf(graph, format="turtle") -``` - -Custom namespace management: - -```python -from semantica.export import NamespaceManager, RDFExporter - -ns_manager = NamespaceManager() -ns_manager.register("ex", "http://example.org/") -ns_manager.register("schema", "https://schema.org/") - -exporter = RDFExporter(namespace_manager=ns_manager) -``` - -## ParquetExporter - -Columnar export for Spark, BigQuery, Databricks, and Snowflake analytics pipelines: - -```python -from semantica.export import ParquetExporter - +# Stream Parquet — writes row groups incrementally exporter = ParquetExporter(compression="snappy") -# compression options: snappy | gzip | brotli | zstd | lz4 - -# Export nodes and edges as separate Parquet files -exporter.export_nodes(graph, "nodes.parquet") -exporter.export_edges(graph, "edges.parquet") - -# Export full graph partitioned by node type -exporter.export(graph, output_dir="graph_parquet/", partition_by="node_type") +exporter.export_stream(graph, output_dir="output/", batch_size=10_000) ``` -Schema is explicitly typed with PyArrow for clean Spark/BigQuery ingestion. +Streaming is recommended for graphs with > 500k nodes. -## LPGExporter - -Labeled Property Graph export — Cypher statements for Neo4j and Memgraph: +## Selective Export ```python -from semantica.export import LPGExporter +# Export a subgraph +subgraph = graph.subgraph(node_ids=["apple_inc", "steve_jobs"]) +export_rdf(subgraph, "subgraph.ttl", format="turtle") -exporter = LPGExporter() - -# CREATE statements -cypher = exporter.to_cypher(graph) -exporter.export(graph, "import.cypher", format="cypher") - -# MERGE statements (idempotent — safe to re-run) -cypher_merge = exporter.to_cypher(graph, use_merge=True) -``` - -## ArangoAQLExporter - -AQL INSERT statements for ArangoDB vertex and edge collections: - -```python -from semantica.export import ArangoAQLExporter - -exporter = ArangoAQLExporter( - vertex_collection="entities", - edge_collection="relationships" -) - -aql = exporter.export(graph) # returns AQL string -exporter.export_to_file(graph, "import.aql") -``` - -## GraphExporter - -Export for visualization tools — GraphML, GEXF, and Graphviz DOT: - -```python -from semantica.export import GraphExporter - -exporter = GraphExporter() - -exporter.export(graph, "graph.graphml", format="graphml") # Gephi, yEd -exporter.export(graph, "graph.gexf", format="gexf") # Gephi streaming -exporter.export(graph, "graph.dot", format="dot") # Graphviz -``` - -## OWLExporter - -OWL 2.0 ontology export in three serialization formats: - -```python -from semantica.export import OWLExporter - -exporter = OWLExporter() -exporter.export(ontology, path="ontology.ttl", format="turtle") -exporter.export(ontology, path="ontology.owl", format="xml") -exporter.export(ontology, path="ontology.json", format="json-ld") -``` - -## CSVExporter - -Flat CSV export for spreadsheets and simple data pipelines: - -```python -from semantica.export import CSVExporter - -exporter = CSVExporter(delimiter=",") -exporter.export_nodes(graph, "nodes.csv") -exporter.export_edges(graph, "edges.csv") -``` - -## VectorExporter - -Export embedding vectors for use in external vector stores: - -```python -from semantica.export import VectorExporter - -exporter = VectorExporter() -exporter.export(embeddings, metadata, "vectors.json", format="json") -exporter.export(embeddings, metadata, "vectors.npy", format="numpy") -exporter.export(embeddings, metadata, "vectors.faiss", format="faiss") -``` - -## ArrowExporter - -Apache Arrow IPC format for zero-copy inter-process transfer: - -```python -from semantica.export import ArrowExporter - -exporter = ArrowExporter() -exporter.export(graph, "graph.arrow") -``` - -Requires `pyarrow`. Falls back gracefully if not installed. - -## DistanceExporter - -Export semantic distance matrices produced by Distance Intelligence (v0.5.0): - -```python -from semantica.export import DistanceExporter - -exporter = DistanceExporter() -exporter.export_matrix(distance_matrix, node_labels, "distances.csv") -exporter.export_ego(ego_neighborhood, center_node="Apple Inc.", path="ego.json") -``` - -## ReportGenerator - -Generate human-readable analytics reports from graph metrics: - -```python -from semantica.export import ReportGenerator - -generator = ReportGenerator() - -generator.generate(graph, analytics_result, "report.html", format="html") -generator.generate(graph, analytics_result, "report.md", format="markdown") -generator.generate(graph, analytics_result, "report.json", format="json") +# Export nodes by type +org_nodes = graph.filter(node_type="Organization") +export_parquet(org_nodes, "organizations.parquet") ``` ## Convenience Functions @@ -212,17 +291,55 @@ export_arango(graph, "import.aql") export_graph(graph, "graph.graphml", format="graphml") ``` -## Selective Export +## Format Reference -```python -# Export a subgraph -subgraph = graph.subgraph(node_ids=["apple_inc", "steve_jobs"]) -export_rdf(subgraph, "subgraph.ttl", format="turtle") +| Format | Exporter | Output | Best For | +| ------ | -------- | ------ | -------- | +| `turtle` | `RDFExporter` | `.ttl` | Readable RDF, ontology sharing | +| `json-ld` | `RDFExporter` | `.jsonld` | APIs, Linked Data, JSON pipelines | +| `nt` | `RDFExporter` | `.nt` | Streaming RDF, line-by-line processing | +| `xml` | `RDFExporter` | `.xml` | W3C RDF/XML, broadest compatibility | +| `parquet` | `ParquetExporter` | `.parquet` | Spark, BigQuery, Databricks, Snowflake | +| `cypher` | `LPGExporter` | `.cypher` | Neo4j, Memgraph import | +| `aql` | `ArangoAQLExporter` | `.aql` | ArangoDB vertex + edge collections | +| `graphml` | `GraphExporter` | `.graphml` | Gephi, yEd visualization | +| `gexf` | `GraphExporter` | `.gexf` | Gephi streaming format | +| `dot` | `GraphExporter` | `.dot` | Graphviz rendering | +| `owl` | `OWLExporter` | `.owl` / `.ttl` | OWL 2.0 ontology distribution | +| `csv` | `CSVExporter` | `.csv` | Spreadsheets, simple pipelines | +| `yaml` | `YAMLExporter` | `.yaml` | Human-readable, config-driven use | +| `arrow` | `ArrowExporter` | `.arrow` | Zero-copy inter-process transfer | +| `numpy` | `VectorExporter` | `.npy` | NumPy arrays from embeddings | +| `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files | +| `distance-matrix` | `DistanceExporter` | `.csv` / `.json` | Distance Intelligence matrices | +| `html` | `ReportGenerator` | `.html` | Human-readable analytics reports | +| `markdown` | `ReportGenerator` | `.md` | Documentation, GitHub | -# Export nodes by type -org_nodes = graph.filter(node_type="Organization") -export_parquet(org_nodes, "organizations.parquet") -``` +## Tips and Common Pitfalls + + + **Use `turtle` for human readability, `nt` for streaming.** Turtle is compact and readable for debugging and sharing ontologies. N-Triples (`.nt`) is line-oriented — one triple per line — making it safe to stream, concatenate, and process with standard Unix tools without loading the full file. + + + + **Use `ParquetExporter` for downstream analytics.** Parquet preserves column types (int, float, datetime) that CSV loses and is natively supported by Spark, BigQuery, Databricks, and Snowflake. Use `compression="snappy"` for a good balance of speed and compression ratio. + + + + **Stream large graphs with `export_stream()`.** For graphs with more than 500k nodes, use `exporter.export_stream(graph, ...)` instead of `exporter.export_to_file()`. Streaming writes incrementally without buffering the full graph in memory — without it, a million-node export will likely OOM. + + + + **Include provenance for compliance exports.** For HIPAA, SOX, or FDA 21 CFR Part 11 exports, pass `include_provenance=True` to `RDFExporter`. This embeds W3C PROV-O lineage triples inline — auditors can verify every fact's source from a single file rather than cross-referencing separate systems. + + + + **Use selective export to reduce file size.** `graph.subgraph(node_ids=[...])` and `graph.filter(node_type="Organization")` let you export only the relevant subset. Full graph exports for compliance reports include noise; scoped exports are faster to produce, review, and transfer. + + + + **Match your export format to your consumer.** Neo4j → `cypher`; ArangoDB → `aql`; Gephi/yEd → `graphml` or `gexf`; semantic web tools → `turtle` or `json-ld`; analytics pipelines → `parquet`; zero-copy IPC → `arrow`. Using the wrong format forces the consumer to convert it, adding latency and potential data loss. + diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index 8ee9e260..2f4d10f7 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -8,119 +8,151 @@ icon: "server" ## What You Get -- **`GraphStore`** — unified interface across all backends -- **Backends** — Neo4j, FalkorDB, Apache AGE (PostgreSQL), Amazon Neptune, NetworkX (in-memory) -- **Cypher queries** — full Cypher support for Neo4j and FalkorDB -- **Bulk operations** — batched node and edge loading with configurable batch sizes -- **Schema management** — create indexes and uniqueness constraints -- **Path traversal** — find paths between nodes with hop limits and relationship type filters + + + Unified interface across Neo4j, FalkorDB, Apache AGE, Amazon Neptune, and NetworkX. + + + Parameterized Cypher construction, query optimization, and result caching. + + + Centrality, community detection, and path algorithms running directly against the backend. + + + Batched node and edge loading with configurable batch sizes — 10–100× faster than individual writes. + + + Create indexes and uniqueness constraints to optimize query performance. + + + Find paths between nodes with hop limits and relationship type filters. + + -## Basic Usage +## Quick Start -```python -from semantica.graph_store import GraphStore + + + ```python + from semantica.graph_store import GraphStore -store = GraphStore( - backend="neo4j", - uri="bolt://localhost:7687", - user="neo4j", - password="password" -) - -store.add_nodes(entities) -store.add_edges(relationships) - -results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m LIMIT 10") -``` + store = GraphStore( + backend="neo4j", + uri="bolt://localhost:7687", + user="neo4j", + password="password", + ) + ``` + + + ```python + store.create_index(label="Person", property="name") + store.create_index(label="Organization", property="name") + store.create_constraint(label="Organization", property="id", constraint_type="unique") + ``` + + + ```python + store.add_nodes_bulk(entities, batch_size=1000) + store.add_edges_bulk(relationships, batch_size=1000) + ``` + + + ```python + results = store.query( + "MATCH (p:Person)-[:WORKS_FOR]->(o:Organization) WHERE o.name = $org RETURN p", + parameters={"org": "Apple Inc."}, + ) + ``` + + ## Backends - + + ```python + from semantica.graph_store import GraphStore -```python -store = GraphStore( - backend="neo4j", - uri="bolt://localhost:7687", - user="neo4j", - password="password", - database="neo4j" # optional — targets default database -) -``` - -Best for: production workloads, complex Cypher queries, Bloom visualization. + store = GraphStore( + backend="neo4j", + uri="bolt://localhost:7687", + user="neo4j", + password="password", + database="neo4j", # optional — targets default database + ) + ``` + Best for: production workloads, complex Cypher queries, Bloom visualization. + ```python + store = GraphStore( + backend="falkordb", + host="localhost", + port=6379, + graph_name="semantica", + ) + ``` -```python -store = GraphStore( - backend="falkordb", - host="localhost", - port=6379, - graph_name="semantica" -) -``` - -Best for: ultra-low latency queries over Redis protocol, edge deployments. - + Best for: ultra-low latency queries over Redis protocol, edge deployments. + ```python + store = GraphStore( + backend="apache_age", + connection_string="postgresql://user:pass@localhost/graphdb", + graph_name="semantica", + ) + ``` -```python -store = GraphStore( - backend="apache_age", - connection_string="postgresql://user:pass@localhost/graphdb", - graph_name="semantica" -) -``` - -Best for: teams already running PostgreSQL who want graph queries without a separate service. See the [Apache AGE Guide](../graph_stores/apache_age) for setup. - + Best for: teams already running PostgreSQL who want graph queries without a separate service. See the [Apache AGE Guide](../graph_stores/apache_age) for setup. + ```python + from semantica.graph_store import GraphStore -```python -store = GraphStore( - backend="neptune", - endpoint="your-cluster.cluster-xxxx.us-east-1.neptune.amazonaws.com", - port=8182, - region="us-east-1" -) -``` + # IAM authentication (recommended for production) + store = GraphStore( + backend="neptune", + endpoint="your-cluster.cluster-xxxx.us-east-1.neptune.amazonaws.com", + port=8182, + region="us-east-1", + use_iam_auth=True, # uses boto3 default credential chain + ) -Best for: managed AWS deployments needing both SPARQL and Gremlin support. + # Gremlin traversal + results = store.query("g.V().hasLabel('Person').limit(10)") + # openCypher query + results = store.query( + "MATCH (p:Person)-[:WORKS_FOR]->(o:Organization) RETURN p, o", + query_language="opencypher", + ) + ``` + + Best for: managed AWS deployments needing both SPARQL and Gremlin support. - + + ```python + store = GraphStore(backend="networkx") + ``` -```python -store = GraphStore(backend="networkx") -``` + Best for: development, testing, and graphs that fit in RAM. Data is not persisted. + + -Best for: development, testing, and graphs that fit in RAM. Data is not persisted. + | Backend | Query Language | Deployment | IAM Auth | Best For | + | ------- | -------------- | ---------- | -------- | -------- | + | Neo4j | Cypher | Self-hosted / Aura | No | Production, complex traversals, Bloom UI | + | FalkorDB | Cypher | Redis-based | No | Ultra-low latency, edge deployments | + | Apache AGE | OpenCypher | PostgreSQL extension | No | Teams already on Postgres | + | Amazon Neptune | SPARQL / Gremlin / openCypher | AWS managed | Yes | Cloud-native, multi-model, compliance | + | NetworkX | Python API | In-memory | No | Development, unit testing | -## Querying - -```python -# Cypher query with parameters (Neo4j, FalkorDB) -results = store.query( - "MATCH (p:Person)-[:WORKS_FOR]->(o:Organization) WHERE o.name = $org RETURN p", - parameters={"org": "Apple Inc."} -) - -# Path traversal between two nodes -paths = store.find_paths( - start_node="steve_jobs", - end_node="apple_inc", - max_hops=3, - relationship_types=["FOUNDED", "WORKED_AT"] -) -``` - ## Graph Operations ```python @@ -128,19 +160,19 @@ paths = store.find_paths( store.add_node( "apple_inc", node_type="Organization", - properties={"founded": 1976, "hq": "Cupertino"} + properties={"founded": 1976, "hq": "Cupertino"}, ) # Add a directed relationship store.add_edge( "steve_jobs", "apple_inc", "FOUNDED", - properties={"year": 1976} + properties={"year": 1976}, ) # Bulk operations — use for large datasets -store.add_nodes_bulk(entities, batch_size=1000) -store.add_edges_bulk(relationships, batch_size=1000) +store.add_nodes_bulk(entities, batch_size=1000) +store.add_edges_bulk(relationships, batch_size=1000) # Delete store.delete_node("node_id") @@ -150,13 +182,85 @@ store.delete_edge("edge_id") neighbors = store.get_neighbors( "apple_inc", relationship_type="HAS_EMPLOYEE", - direction="in" # "in" | "out" | "both" + direction="in", # "in" | "out" | "both" +) + +# Path traversal between two nodes +paths = store.find_paths( + start_node="steve_jobs", + end_node="apple_inc", + max_hops=3, + relationship_types=["FOUNDED", "WORKED_AT"], ) ``` -## Schema Management +## QueryEngine -Create indexes and constraints to improve query performance: +`QueryEngine` handles query construction, optimization, and caching: + +```python +from semantica.graph_store import QueryEngine, GraphStore + +store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", password="password") +engine = QueryEngine(store, cache_ttl=300) # cache results for 5 minutes + +# Build parameterized Cypher +query, params = engine.build_query( + node_labels=["Person"], + filters={"department": "Engineering"}, + return_fields=["name", "email"], + limit=50, +) +results = engine.execute(query, params) + +# Explain query plan (Neo4j) +plan = engine.explain(query, params) +print(plan["profile"]) + +# Flush query cache +engine.clear_cache() +``` + +## GraphAnalytics + +Built-in graph analytics that run directly against the stored backend — no data export required: + +```python +from semantica.graph_store import GraphAnalytics, GraphStore + +store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", password="password") +analytics = GraphAnalytics(store) + +# Centrality +centrality = analytics.degree_centrality(node_label="Person", relationship_type="KNOWS") +betweenness = analytics.betweenness_centrality(node_label="Person") + +# Community detection +communities = analytics.detect_communities( + node_label="Person", + relationship_type="KNOWS", + algorithm="louvain", +) +print(f"Detected {len(communities)} communities") + +# Shortest path +path = analytics.shortest_path("alice", "charlie", relationship_type="KNOWS") +print(f"Hops: {len(path) - 1}, Path: {' → '.join(path)}") + +# All paths up to max_hops +all_paths = analytics.all_paths("alice", "charlie", max_hops=4) +``` + +| Method | Description | +| ------ | ----------- | +| `degree_centrality(node_label, relationship_type)` | Degree-based node importance | +| `betweenness_centrality(node_label)` | Bridge-based importance | +| `pagerank(node_label, relationship_type, damping)` | PageRank scores | +| `detect_communities(node_label, relationship_type, algorithm)` | Louvain / Label Propagation | +| `shortest_path(source, target, relationship_type)` | Minimum-hop path | +| `all_paths(source, target, max_hops)` | All paths up to max depth | + +## Schema Management ```python # Index for fast label lookups @@ -166,7 +270,7 @@ store.create_index(label="Person", property="name") store.create_constraint( label="Organization", property="id", - constraint_type="unique" + constraint_type="unique", ) # Inspect current schema @@ -176,15 +280,31 @@ print(schema["indexes"]) print(schema["constraints"]) ``` -## Backend Comparison +## Tips and Common Pitfalls -| Backend | Query Language | Deployment | Best For | -| ------- | -------------- | ---------- | -------- | -| Neo4j | Cypher | Self-hosted / Aura | Production, complex traversals | -| FalkorDB | Cypher | Redis-based | Ultra-low latency, edge | -| Apache AGE | OpenCypher | PostgreSQL | Teams already on Postgres | -| Amazon Neptune | SPARQL / Gremlin | AWS managed | Cloud-native AWS deployments | -| NetworkX | Python API | In-memory | Development and testing | + + **Use `NetworkX` for development, Neo4j or FalkorDB for production.** `backend="networkx"` requires zero setup and runs in memory — ideal for local development and CI tests. Switch to a persistent backend before deploying — no code changes needed, just the backend parameter. + + + + **Create indexes before bulk loading.** `store.create_index(label="Person", property="name")` makes `MATCH` queries on `name` orders of magnitude faster. Without indexes, every query does a full scan. Create indexes first, then load data. + + + + **Use `add_nodes_bulk()` and `add_edges_bulk()` for large datasets.** Individual `add_node()` calls issue one network round-trip each. Bulk operations batch thousands of writes into a single transaction — 10–100× faster for initial loads. + + + + **Use parameterized queries, never string interpolation.** `store.query("WHERE n.name = $name", parameters={"name": user_input})` prevents Cypher injection attacks. Never use `f"WHERE n.name = '{user_input}'"`. + + + + **Enable `QueryEngine` caching for read-heavy workloads.** `QueryEngine(store, cache_ttl=300)` avoids repeated round-trips for identical queries within the cache window — useful for analytics dashboards that refresh frequently with the same aggregation queries. + + + + **Apache AGE requires the PostgreSQL extension installed.** `backend="apache_age"` calls the AGE extension functions. If AGE is not installed in your PostgreSQL instance, you'll get a `ProgrammingError`. See the [Apache AGE Guide](../graph_stores/apache_age) for setup instructions. + diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 847e226a..b5f984ec 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -8,161 +8,409 @@ icon: "database" ## What You Get -- **`FileIngestor`** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR archives -- **`ParquetIngestor`** — PyArrow-based Parquet with Hive-style partition support (v0.5.0) -- **`XMLIngestor`** — XXE-safe lxml with XSD/DTD validation (v0.5.0) -- **`WebIngestor`** — configurable web crawling with robots.txt support -- **`FeedIngestor`** — RSS/Atom feeds with live monitoring -- **`DBIngestor`** / **`SnowflakeIngestor`** — SQL databases and Snowflake -- **`StreamIngestor`** — Kafka, RabbitMQ, Kinesis, Pulsar real-time streams -- **`RepoIngestor`**, **`EmailIngestor`**, **`MCPIngestor`**, **`S3Ingestor`**, **`GCSIngestor`** + + + PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, and ZIP/TAR archives — type auto-detected from extension. + + + PyArrow-based Parquet with Hive-style partition support and column selection (v0.5.0). + + + XXE-safe lxml with XSD/DTD validation and directory scanning (v0.5.0). + + + Real-time ingestion from Kafka, RabbitMQ, AWS Kinesis, and Apache Pulsar. + + + S3Ingestor, GCSIngestor, and GDriveIngestor with authentication options. + + + DBIngestor, SnowflakeIngestor, MongoIngestor, and DuckDBIngestor. + + -## FileIngestor +## Quick Start + + + + ```python + from semantica.ingest import FileIngestor + + ingestor = FileIngestor() + + # Single file — type auto-detected from extension + sources = ingestor.ingest("data/report.pdf") + + # Recursive directory scan + sources = ingestor.ingest_directory("data/", recursive=True) + + # Glob pattern + sources = ingestor.ingest("data/**/*.docx") + ``` + + + ```python + from semantica.ingest import DBIngestor + + ingestor = DBIngestor( + connection_string="postgresql://user:pass@localhost/db", + query="SELECT id, content, created_at FROM documents WHERE status='active'" + ) + sources = ingestor.ingest() + ``` + + + ```python + from semantica.pipeline import Pipeline + from semantica.parse import DocumentParser + from semantica.semantic_extract import NERExtractor + from semantica.llms import Groq + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + + pipeline = Pipeline() + pipeline.add_step("ingest", FileIngestor()) + pipeline.add_step("parse", DocumentParser()) + pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) + result = pipeline.run("data/") + ``` + + + +## Ingestors + + + + ### FileIngestor + + ```python + from semantica.ingest import FileIngestor + + ingestor = FileIngestor() + sources = ingestor.ingest("data/report.pdf") + sources = ingestor.ingest_directory("data/", recursive=True) + sources = ingestor.ingest("data/**/*.docx") + ``` + + Supported formats: PDF, DOCX, TXT, HTML, JSON, CSV, Excel (XLSX/XLS), PPTX, ZIP/TAR archives. + + ### ParquetIngestor (v0.5.0) + + PyArrow-based ingestion for Apache Parquet files, including Hive-style partitioned datasets: + + ```python + from semantica.ingest import ParquetIngestor + + ingestor = ParquetIngestor() + + # Single Parquet file + sources = ingestor.ingest("data/events.parquet") + + # Partitioned directory (year=2024/month=01/...) + sources = ingestor.ingest("data/partitioned/") + + # Load only specific columns + sources = ingestor.ingest("data/events.parquet", columns=["id", "text", "timestamp"]) + ``` + + ### XMLIngestor (v0.5.0) + + XXE-safe lxml-based ingestion with optional schema validation: + + ```python + from semantica.ingest import XMLIngestor + + ingestor = XMLIngestor() + sources = ingestor.ingest("data/records.xml") + + # With XSD validation + ingestor = XMLIngestor(validate_xsd="schema.xsd") + sources = ingestor.ingest("data/records/") + + # With DTD validation + ingestor = XMLIngestor(validate_dtd=True) + sources = ingestor.ingest("data/feed.xml") + ``` + + + `XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks. + + + + ### WebIngestor + + ```python + from semantica.ingest import WebIngestor + + ingestor = WebIngestor( + rate_limit=1.0, # seconds between requests + respect_robots=True, # honor robots.txt + max_depth=2 # crawl depth from seed URLs + ) + + sources = ingestor.ingest("https://example.com/about") + sources = ingestor.ingest_urls([ + "https://example.com/page1", + "https://example.com/page2", + ]) + ``` + + ### FeedIngestor (RSS/Atom) + + ```python + from semantica.ingest import FeedIngestor + + ingestor = FeedIngestor() + sources = ingestor.ingest("https://feeds.example.com/rss") + + # Live monitoring — callback fires on new items + ingestor.monitor( + "https://feeds.example.com/rss", + interval=300, + callback=process_new_items + ) + ``` + + ### RepoIngestor + + Ingest Git repositories — source code, commit history, and dependency graphs: + + ```python + from semantica.ingest import RepoIngestor + + ingestor = RepoIngestor( + branch="main", + file_types=[".py", ".md", ".yaml"], + include_commits=True, + commit_range="HEAD~100..HEAD", + ) + + sources = ingestor.ingest("https://github.com/org/repo") + sources = ingestor.ingest("/path/to/local/repo") + ``` + + ### EmailIngestor + + Ingest emails via IMAP or POP3 with attachment extraction and thread analysis: + + ```python + from semantica.ingest import EmailIngestor + import os + + ingestor = EmailIngestor( + protocol="imap", + host="imap.gmail.com", + port=993, + use_ssl=True, + username=os.getenv("EMAIL_USER"), + password=os.getenv("EMAIL_PASS"), + folder="INBOX", + attachment_types=[".pdf", ".docx", ".txt"], + include_thread_analysis=True, + max_emails=500, + ) + sources = ingestor.ingest() + ``` + + + ### S3Ingestor + + Ingest files directly from AWS S3 buckets: + + ```python + from semantica.ingest import S3Ingestor + import os + + ingestor = S3Ingestor( + bucket="my-documents-bucket", + prefix="reports/2024/", + region="us-east-1", + aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"), + aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"), + # Or omit credentials to use IAM instance profile + ) + sources = ingestor.ingest() + sources = ingestor.ingest(pattern="**/*.pdf") + ``` + + ### GCSIngestor + + Ingest files from Google Cloud Storage: + + ```python + from semantica.ingest import GCSIngestor + + ingestor = GCSIngestor( + bucket="my-gcs-bucket", + prefix="data/", + credentials_file="gcp-credentials.json", # or use ADC + ) + sources = ingestor.ingest() + ``` + + ### GDriveIngestor + + Ingest files from Google Drive folders via OAuth 2.0: + + ```python + from semantica.ingest import GDriveIngestor + + ingestor = GDriveIngestor( + credentials_file="oauth_credentials.json", + token_file="token.json", + folder_id="1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs9", + file_types=["pdf", "docx", "txt"], + recursive=True, + ) + sources = ingestor.ingest() + ``` + + + ### DBIngestor (SQL) + + ```python + from semantica.ingest import DBIngestor + + ingestor = DBIngestor( + connection_string="postgresql://user:pass@localhost/db", + query="SELECT id, content, created_at FROM documents WHERE status='active'" + ) + sources = ingestor.ingest() + ``` + + ### SnowflakeIngestor + + ```python + from semantica.ingest import SnowflakeIngestor + import os + + ingestor = SnowflakeIngestor( + account=os.getenv("SNOWFLAKE_ACCOUNT"), + user=os.getenv("SNOWFLAKE_USER"), + password=os.getenv("SNOWFLAKE_PASSWORD"), + warehouse="COMPUTE_WH", + database="ANALYTICS", + schema="PUBLIC" + ) + sources = ingestor.ingest(query="SELECT * FROM documents") + ``` + + ### MongoIngestor + + Ingest documents from MongoDB collections: + + ```python + from semantica.ingest import MongoIngestor + import os + + ingestor = MongoIngestor( + connection_string=os.getenv("MONGO_URI"), + database="mydb", + collection="articles", + query={"status": "published", "year": {"$gte": 2022}}, + projection={"title": 1, "body": 1, "author": 1}, + content_field="body", + limit=10000, + ) + sources = ingestor.ingest() + ``` + + ### DuckDBIngestor + + Ingest data from DuckDB databases or directly from Parquet/CSV files via DuckDB SQL: + + ```python + from semantica.ingest import DuckDBIngestor + + # In-memory DuckDB — query a Parquet file directly + ingestor = DuckDBIngestor( + query="SELECT id, text, created_at FROM read_parquet('data/*.parquet') WHERE year >= 2023", + ) + sources = ingestor.ingest() + + # Persistent DuckDB database file + ingestor = DuckDBIngestor( + database_path="analytics.duckdb", + query="SELECT doc_id AS id, content, metadata FROM documents", + content_field="content", + ) + sources = ingestor.ingest() + ``` + + + ### StreamIngestor + + Real-time ingestion from message brokers: + + ```python + from semantica.ingest import StreamIngestor + + # Kafka + ingestor = StreamIngestor( + backend="kafka", + bootstrap_servers="localhost:9092", + topic="documents", + group_id="semantica-consumer", + auto_offset_reset="earliest", + ) + sources = ingestor.ingest(max_messages=1000) + + # RabbitMQ + ingestor = StreamIngestor( + backend="rabbitmq", + host="localhost", + queue="document_queue", + routing_key="docs.ingest", + prefetch_count=100, + ) + + # AWS Kinesis + ingestor = StreamIngestor( + backend="kinesis", + stream_name="documents-stream", + region="us-east-1", + shard_iterator_type="TRIM_HORIZON", + ) + + # Apache Pulsar + ingestor = StreamIngestor( + backend="pulsar", + service_url="pulsar://localhost:6650", + topic="persistent://public/default/documents", + subscription_name="semantica-sub", + ) + + # Live monitoring — callback fires on each new message + ingestor.monitor(callback=process_document, poll_interval=1.0) + ``` + + + Without a `max_messages` limit, `StreamIngestor.ingest()` blocks indefinitely waiting for new messages. Use `max_messages=1000` for batch processing; use `.monitor(callback=...)` for continuous streaming. + + + + +## OntologyIngestor + +Ingest existing OWL or RDF ontology files as structured knowledge sources: ```python -from semantica.ingest import FileIngestor +from semantica.ingest import OntologyIngestor -ingestor = FileIngestor() - -# Single file — type auto-detected from extension -sources = ingestor.ingest("data/report.pdf") - -# Recursive directory scan -sources = ingestor.ingest_directory("data/", recursive=True) - -# Glob pattern -sources = ingestor.ingest("data/**/*.docx") -``` - -Supported formats: PDF, DOCX, TXT, HTML, JSON, CSV, Excel (XLSX/XLS), PPTX, ZIP/TAR archives. - -## ParquetIngestor (v0.5.0) - -PyArrow-based ingestion for Apache Parquet files, including Hive-style partitioned datasets: - -```python -from semantica.ingest import ParquetIngestor - -ingestor = ParquetIngestor() - -# Single Parquet file -sources = ingestor.ingest("data/events.parquet") - -# Partitioned directory (year=2024/month=01/...) -sources = ingestor.ingest("data/partitioned/") - -# Load only specific columns -sources = ingestor.ingest("data/events.parquet", columns=["id", "text", "timestamp"]) -``` - -## XMLIngestor (v0.5.0) - -XXE-safe lxml-based ingestion with optional schema validation: - -```python -from semantica.ingest import XMLIngestor - -# Basic ingestion -ingestor = XMLIngestor() -sources = ingestor.ingest("data/records.xml") - -# With XSD validation -ingestor = XMLIngestor(validate_xsd="schema.xsd") -sources = ingestor.ingest("data/records/") - -# With DTD validation -ingestor = XMLIngestor(validate_dtd=True) -sources = ingestor.ingest("data/feed.xml") -``` - - - `XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks. - - -## WebIngestor - -```python -from semantica.ingest import WebIngestor - -ingestor = WebIngestor( - rate_limit=1.0, # seconds between requests - respect_robots=True, # honor robots.txt - max_depth=2 # crawl depth from seed URLs +ingestor = OntologyIngestor( + format="turtle", # "turtle" | "xml" | "json-ld" | "nt" | "n3" ) -# Single URL -sources = ingestor.ingest("https://example.com/about") - -# Multiple URLs -sources = ingestor.ingest_urls([ - "https://example.com/page1", - "https://example.com/page2", -]) +sources = ingestor.ingest("domain_ontology.owl") +sources = ingestor.ingest("ontologies/") ``` -## FeedIngestor (RSS/Atom) - -```python -from semantica.ingest import FeedIngestor - -ingestor = FeedIngestor() -sources = ingestor.ingest("https://feeds.example.com/rss") - -# Live monitoring — callback fires on new items -ingestor.monitor( - "https://feeds.example.com/rss", - interval=300, - callback=process_new_items -) -``` - -## DBIngestor (SQL) - -```python -from semantica.ingest import DBIngestor - -ingestor = DBIngestor( - connection_string="postgresql://user:pass@localhost/db", - query="SELECT id, content, created_at FROM documents WHERE status='active'" -) -sources = ingestor.ingest() -``` - -## SnowflakeIngestor - -```python -from semantica.ingest import SnowflakeIngestor -import os - -ingestor = SnowflakeIngestor( - account=os.getenv("SNOWFLAKE_ACCOUNT"), - user=os.getenv("SNOWFLAKE_USER"), - password=os.getenv("SNOWFLAKE_PASSWORD"), - warehouse="COMPUTE_WH", - database="ANALYTICS", - schema="PUBLIC" -) -sources = ingestor.ingest(query="SELECT * FROM documents") -``` - -## Other Ingestors - -| Class | Source | -| ----- | ------ | -| `StreamIngestor` | Kafka, RabbitMQ, Kinesis, Pulsar | -| `RepoIngestor` | Git repositories (GitHub, GitLab) | -| `EmailIngestor` | IMAP/POP3 servers with attachment extraction | -| `MCPIngestor` | Model Context Protocol servers | -| `S3Ingestor` | AWS S3 buckets | -| `GCSIngestor` | Google Cloud Storage | -| `MongoIngestor` | MongoDB collections | -| `DuckDBIngestor` | DuckDB databases | -| `GDriveIngestor` | Google Drive | - ## DataSource Object All ingestors return a list of `DataSource` objects with a consistent schema: + + ```python @dataclass class DataSource: @@ -173,6 +421,8 @@ class DataSource: raw_bytes: Optional[bytes] # original binary content if available ``` + + ## Custom Ingestors Register a custom ingestor and it participates in the full pipeline: @@ -181,12 +431,37 @@ Register a custom ingestor and it participates in the full pipeline: from semantica.ingest.registry import method_registry def my_ingestor(source, **kwargs): - # Return a list of DataSource-compatible dicts return [{"content": "...", "metadata": {}, "source_id": source}] method_registry.register("file", "my_format", my_ingestor) ``` +## Tips and Common Pitfalls + + + **`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and supports glob patterns. Only reach for `DoclingParser` when `DocumentParser` can't handle your layout. + + + + **Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns — critical for wide tables with hundreds of columns. + + + + **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica — it doesn't block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML. + + + + **Stream ingestors need explicit `max_messages` for batch runs.** Without a limit, `StreamIngestor.ingest()` blocks indefinitely waiting for new messages. Use `max_messages=1000` for batch processing; use `.monitor(callback=...)` for continuous streaming. + + + + **Rate-limit web crawling.** `WebIngestor(rate_limit=1.0, respect_robots=True)` is the responsible default. Without rate limiting, you risk getting blocked by the target server or violating its terms of service. + + + + **All ingestors return the same `DataSource` schema.** This means you can mix sources in a single pipeline without any adapter code — `FileIngestor`, `MongoIngestor`, and `StreamIngestor` outputs are all directly composable with `DocumentParser` and `NERExtractor`. + + Parse raw sources into structured text and tables. diff --git a/docs/reference/kg.md b/docs/reference/kg.md index f823bf6d..3b72c8d9 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -8,14 +8,26 @@ icon: "diagram-project" ## What You Get -- **`GraphBuilder`** — construct graphs from entities and relationships with automatic entity merging -- **`TemporalKnowledgeGraph`** — time-aware edges (`valid_from`/`valid_until`) and point-in-time queries (v0.4.0) -- **`DistanceCalculator`** — semantic neighborhoods, N×N distance matrices, and distance band classification (v0.5.0) -- **`CentralityCalculator`** — PageRank, degree, betweenness, closeness, eigenvector centrality -- **`CommunityDetector`** — Louvain, Leiden, Label Propagation, K-Clique community detection -- **`PathFinder`** — Dijkstra, A\*, BFS, K-Shortest path algorithms -- **`LinkPredictor`** — Preferential Attachment, Jaccard, Adamic-Adar link prediction -- **`NodeEmbedder`** — Node2Vec, DeepWalk, Word2Vec structural embeddings + + + Construct graphs from entities and relationships with automatic entity merging. + + + Time-aware edges (`valid_from`/`valid_until`) and point-in-time queries (v0.4.0). + + + Semantic neighborhoods, N×N distance matrices, and distance band classification (v0.5.0). + + + PageRank, degree, betweenness, closeness, and eigenvector centrality. + + + Louvain, Leiden, Label Propagation, and K-Clique community detection. + + + Dijkstra, A\*, BFS, and K-Shortest path algorithms. + + For conflict detection and advanced entity resolution, use `semantica.conflicts` and `semantica.deduplication` alongside this module. @@ -23,6 +35,54 @@ icon: "diagram-project" Knowledge graph entity and relation structure: Person, Organization, Location, Date nodes with typed labeled edges +## Quick Start + + + + ```python + from semantica.kg import GraphBuilder + + builder = GraphBuilder(merge_entities=True) + kg = builder.build(entities=entities, relationships=relationships) + + print(f"Nodes: {kg.node_count}, Edges: {kg.edge_count}") + ``` + + + ```python + from semantica.kg import CentralityCalculator + + calc = CentralityCalculator() + pagerank = calc.calculate_pagerank(kg, damping_factor=0.85) + top_10 = calc.get_top_nodes(pagerank, top_k=10) + + for node_id, score in top_10: + print(f" {node_id}: {score:.4f}") + ``` + + + ```python + from semantica.kg import CommunityDetector + + detector = CommunityDetector() + communities = detector.detect_communities(kg, algorithm="louvain") + metrics = detector.calculate_community_metrics(kg, communities) + + print(f"Communities: {len(communities)}") + ``` + + + ```python + from semantica.graph_store import GraphStore + + store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", + user="neo4j", password="password") + store.add_nodes_bulk(kg.entities, batch_size=1000) + store.add_edges_bulk(kg.relationships, batch_size=1000) + ``` + + + ## GraphBuilder Constructs knowledge graphs from extracted entities and relationships: @@ -31,7 +91,7 @@ Constructs knowledge graphs from extracted entities and relationships: from semantica.kg import GraphBuilder builder = GraphBuilder(merge_entities=True) -kg = builder.build(entities=entities, relationships=relationships) +kg = builder.build(entities=entities, relationships=relationships) ``` | Method | Description | @@ -40,6 +100,10 @@ kg = builder.build(entities=entities, relationships=relationships) | `build_single_source(data)` | Build graph from a single data source | | `merge_entities()` | Deduplicate and merge entities during construction | + + Always use `merge_entities=True` in production. Without it, "Steve Jobs" extracted from five different documents creates five separate person nodes. `GraphBuilder(merge_entities=True)` uses edit distance matching to consolidate them at build time. + + ## Temporal Knowledge Graphs (v0.4.0) Attach `valid_from` / `valid_until` time windows to nodes and edges for point-in-time queries and historical analysis: @@ -50,7 +114,6 @@ from datetime import datetime tkg = TemporalKnowledgeGraph() -# Nodes and edges carry explicit validity windows tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1)) tkg.add_edge( "alice", "acme_corp", "ceo_of", @@ -61,15 +124,17 @@ tkg.add_edge( # Point-in-time snapshot snapshot = tkg.at(datetime(2021, 6, 15)) -# Diff between two snapshots -query = TemporalGraphQuery(tkg) -snapshot_2020 = query.at_time("2020-01-01") -snapshot_2023 = query.at_time("2023-01-01") -diff = snapshot_2023.minus(snapshot_2020) -print(f"New nodes since 2020: {len(diff.nodes)}") +# Query and diff via TemporalGraphQuery +query = TemporalGraphQuery(tkg) +snap_2020 = query.query_at_time(datetime(2020, 1, 1)) +snap_2023 = query.query_at_time(datetime(2023, 1, 1)) +added = [r for r in snap_2023.relationships if r not in snap_2020.relationships] +print(f"New edges since 2020: {len(added)}") ``` -Supports all 13 Allen interval algebra relations (before, after, meets, overlaps, during, starts, finishes, equals, and their inverses). OWL-Time export available. + + Edges added without `valid_from`/`valid_until` are treated as **always-valid**. For historical data, always attach timestamps — otherwise point-in-time queries return misleading results. + ## Distance Intelligence (v0.5.0) @@ -90,102 +155,195 @@ matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"]) bands = calc.classify_bands(neighborhood) ``` + + `DistanceCalculator` is expensive at large scale — the N×N matrix requires embedding all entities and computing pairwise cosine similarities. Cache the result between runs and only recompute for changed entities. + + ## Graph Analytics -### Centrality Analysis + + + Identify the most structurally important nodes in your graph: -```python -from semantica.kg import CentralityCalculator + ```python + from semantica.kg import CentralityCalculator -calculator = CentralityCalculator() + calc = CentralityCalculator() -centrality = calculator.calculate_degree_centrality(graph) -pagerank = calculator.calculate_pagerank(graph, damping_factor=0.85) -betweenness = calculator.calculate_betweenness_centrality(graph) -closeness = calculator.calculate_closeness_centrality(graph) -eigenvector = calculator.calculate_eigenvector_centrality(graph) -all_metrics = calculator.calculate_all_centrality(graph) + pagerank = calc.calculate_pagerank(kg, damping_factor=0.85) + degree = calc.calculate_degree_centrality(kg) + betweenness = calc.calculate_betweenness_centrality(kg) + closeness = calc.calculate_closeness_centrality(kg) + eigenvector = calc.calculate_eigenvector_centrality(kg) + all_metrics = calc.calculate_all_centrality(kg) -top_nodes = calculator.get_top_nodes(centrality, top_k=10) -``` + top_nodes = calc.get_top_nodes(pagerank, top_k=10) + ``` -| Method | Algorithm | -| ------ | --------- | -| `calculate_degree_centrality()` | Degree-based importance | -| `calculate_betweenness_centrality()` | Bridge-based importance (bottleneck nodes) | -| `calculate_closeness_centrality()` | Distance-based importance | -| `calculate_eigenvector_centrality()` | Influence-based importance | -| `calculate_pagerank()` | Link-based importance (PageRank) | -| `calculate_all_centrality()` | All measures at once | + | Measure | Best For | + | ------- | -------- | + | PageRank | Overall importance (link-based) | + | Degree | Most connected nodes | + | Betweenness | Bridge / bottleneck nodes | + | Closeness | Fastest to reach all others | + | Eigenvector | Connected to other important nodes | + + + Partition the graph into thematically dense clusters: -### Community Detection + ```python + from semantica.kg import CommunityDetector -```python -from semantica.kg import CommunityDetector + detector = CommunityDetector() -detector = CommunityDetector() + # Louvain — fast, high quality (default) + communities = detector.detect_communities(kg, algorithm="louvain") -# Louvain (default — fast, high quality) -communities = detector.detect_communities(graph, algorithm="louvain") + # Leiden — higher quality, slower + leiden_communities = detector.detect_communities_leiden(kg, resolution=1.2) -# Leiden (higher quality, slower) -leiden_communities = detector.detect_communities_leiden(graph, resolution=1.2) + metrics = detector.calculate_community_metrics(kg, communities) + print(f"Communities: {len(communities)}") + ``` -metrics = detector.calculate_community_metrics(graph, communities) -``` + Algorithms available: **Louvain**, **Leiden**, **Label Propagation**, **K-Clique Communities**. -Algorithms: Louvain, Leiden, Label Propagation, K-Clique Communities. + + Community detection finds thematic clusters — often corresponding to real-world subject groups. Use cluster membership as context boundaries for GraphRAG retrieval. + + + + Find shortest and alternative paths between nodes: -### Path Finding + ```python + from semantica.kg import PathFinder -```python -from semantica.kg import PathFinder + finder = PathFinder() -finder = PathFinder() + path = finder.dijkstra_shortest_path(kg, "node_a", "node_b") + paths = finder.all_shortest_paths(kg, "source", "target") + k_paths = finder.find_k_shortest_paths(kg, "source", "target", k=3) + ``` -path = finder.dijkstra_shortest_path(graph, "node_a", "node_b") -paths = finder.all_shortest_paths(graph, "source", "target") -k_paths = finder.find_k_shortest_paths(graph, "source", "target", k=3) -``` + Algorithms: **Dijkstra**, **A\***, **BFS**, **All Shortest Paths**, **K-Shortest Paths**. + + + Analyse graph structure — components, bridges, and density: -Algorithms: Dijkstra, A\*, BFS, All Shortest Paths, K-Shortest Paths. + ```python + from semantica.kg import ConnectivityAnalyzer -### Link Prediction + analyzer = ConnectivityAnalyzer() + components = analyzer.find_connected_components(kg) + density = analyzer.calculate_density(kg) + bridges = analyzer.find_bridges(kg) -```python -from semantica.kg import LinkPredictor + print(f"Components: {len(components)}, Largest: {len(components[0])} nodes") + print(f"Density: {density:.4f}") + print(f"Bridges: {bridges}") + ``` -predictor = LinkPredictor(method="preferential_attachment") -links = predictor.predict_links(graph, top_k=20) -score = predictor.score_link(graph, "node_a", "node_b") -``` + | Method | Returns | Description | + | ------ | ------- | ----------- | + | `find_connected_components(kg)` | `List[List[str]]` | Groups of mutually reachable nodes | + | `calculate_density(kg)` | `float` | Edge density (actual / possible edges) | + | `find_bridges(kg)` | `List[str]` | Nodes whose removal disconnects the graph | + + + Predict which edges are likely missing from the graph: -Algorithms: Preferential Attachment, Common Neighbors, Jaccard, Adamic-Adar, Resource Allocation. + ```python + from semantica.kg import LinkPredictor -### Node Embeddings + predictor = LinkPredictor(method="preferential_attachment") + links = predictor.predict_links(kg, top_k=20) + score = predictor.score_link(kg, "node_a", "node_b") + ``` -```python -from semantica.kg import NodeEmbedder + Algorithms: **Preferential Attachment**, **Common Neighbors**, **Jaccard**, **Adamic-Adar**, **Resource Allocation**. + + + Compute structural embeddings for similarity search and downstream ML: -embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) -embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) -similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) -``` + ```python + from semantica.kg import NodeEmbedder -Algorithms: Node2Vec, DeepWalk, Word2Vec. + embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) + embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) + similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) + ``` + + Algorithms: **Node2Vec**, **DeepWalk**, **Word2Vec**. + + ## Algorithm Summary | Category | Algorithms | Use Cases | | -------- | ---------- | --------- | | Node Embeddings | Node2Vec, DeepWalk, Word2Vec | Structural similarity, node representation | -| Similarity | Cosine, Euclidean, Manhattan, Correlation | Node matching, recommendation | | Path Finding | Dijkstra, A\*, BFS, K-Shortest | Route planning, network analysis | | Link Prediction | Preferential Attachment, Jaccard, Adamic-Adar | Network completion | | Centrality | Degree, Betweenness, Closeness, PageRank | Influence analysis | | Community Detection | Louvain, Leiden, Label Propagation | Social clustering | | Connectivity | Components, Bridges, Density | Network robustness | +## SeedManager + +Load and inject curated seed data into a knowledge graph: + +```python +from semantica.kg import SeedManager + +manager = SeedManager() +seed_data = manager.load_seed("seeds/domain_entities.json") +normalized = manager.normalize(seed_data, source="manual_curation_v1") + +builder = GraphBuilder(merge_entities=True) +kg = builder.build(normalized + extracted_sources) +``` + +## MethodRegistry + +Register custom KG construction methods and dispatch by name: + +```python +from semantica.kg import method_registry + +def my_kg_builder(entities, relationships, **kwargs): + filtered = [e for e in entities if e["confidence"] >= 0.9] + return {"entities": filtered, "relationships": relationships} + +method_registry.register("build", "high_confidence", my_kg_builder) + +from semantica.kg import build_knowledge_graph +kg = build_knowledge_graph(sources, method="high_confidence") +``` + +## ProvenanceTracker + +Track entity and relationship lineage within a knowledge graph: + +```python +from semantica.kg import ProvenanceTracker + +tracker = ProvenanceTracker() + +tracker.track_entity( + entity_id="apple_inc", + source="sec_filing_2024q1.pdf", + source_location="page 3, paragraph 2", + source_quote="Apple Inc. reported revenue of...", + confidence=0.98, +) + +lineage = tracker.get_lineage("apple_inc") +for entry in lineage.entries: + print(f" Source: {entry.source} ({entry.timestamp})") +``` + +For full W3C PROV-O compliance and provenance export, see the [Provenance module](provenance). + ## Configuration ```yaml @@ -199,6 +357,24 @@ kg: default_validity: infinite ``` +## Tips and Common Pitfalls + + + **Deduplicate before `GraphBuilder`, not after.** It's far easier to merge entities before they become nodes than to update all relationship endpoints after the fact. Run `DuplicateDetector` on extracted entities before calling `builder.build()`. + + + + **PageRank identifies your most connected, important nodes.** If you're not sure which entities in your graph are the most structurally significant, `CentralityCalculator.calculate_pagerank()` gives you a ranked list — useful for GraphRAG context anchoring. + + + + **Community detection finds thematic clusters.** `CommunityDetector` with Louvain partitions your graph into clusters of densely-connected nodes — often corresponding to real-world thematic groups. Use these clusters for exploratory analysis and to scope GraphRAG retrieval. + + + + **`ProvenanceTracker` links entities back to their source documents.** Use it during graph construction so you can always answer "where did this fact come from?" — critical for compliance and for debugging incorrect graph data. + + Persist graphs in Neo4j, FalkorDB, or Apache AGE. diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 7c70a3d0..4b27127a 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -1,19 +1,88 @@ --- title: "LLMs Module" -description: "Unified interface for Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace." +description: "Unified interface for Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace — swap providers with a one-line change." icon: "microchip" --- -`semantica.llms` provides a single consistent API across 8+ LLM providers. Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoning engines, and agents. +`semantica.llms` gives every Semantica module a single, consistent interface to 9+ LLM providers. Every extractor, reasoning engine, and context graph accepts any provider through the same `llm_provider=` parameter — swap Groq for Anthropic, or a cloud API for a local Ollama model, by changing one line. ## What You Get -- **8+ provider integrations** — Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, HuggingFace -- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application changes needed -- **`ProviderFactory`** — instantiate any provider by name from a config dict -- **Local models** — Ollama and HuggingFace run fully on-premise with no API key -- **Streaming** — token-by-token output for low-latency UX -- **Custom gateways** — point any OpenAI-compatible endpoint via `base_url` + + + Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace — all behind one interface. + + + `complete()`, `chat()`, and `stream()` work identically across all providers — swap with a one-line change. + + + Instantiate any provider from a name string — drive provider selection entirely from YAML config or environment variables. + + + Ollama and HuggingFace run fully on-premise — no API key, no data leaves your machine, air-gap compatible. + + + Token-by-token output via `stream()` for responsive agent pipelines and live UI updates. + + + Configurable `max_retries` with exponential backoff. Typed exceptions: `LLMAuthenticationError`, `LLMRateLimitError`, `LLMContextLengthError`. + + + +## Installation + +The base install includes Groq and DeepSeek. Other providers require optional extras: + +| Provider | Install Command | API Key Required | +| -------- | --------------- | ---------------- | +| Groq | `pip install semantica` | Yes — `GROQ_API_KEY` | +| DeepSeek | `pip install semantica` | Yes — `DEEPSEEK_API_KEY` | +| Novita AI | `pip install semantica` | Yes — `NOVITA_API_KEY` | +| OpenAI | `pip install "semantica[llm-openai]"` | Yes — `OPENAI_API_KEY` | +| Anthropic | `pip install "semantica[llm-anthropic]"` | Yes — `ANTHROPIC_API_KEY` | +| Gemini | `pip install "semantica[llm-gemini]"` | Yes — `GOOGLE_API_KEY` | +| Ollama | `pip install "semantica[llm-ollama]"` | No — local server | +| LiteLLM | `pip install "semantica[llm-litellm]"` | Varies by target | +| HuggingFace | `pip install "semantica[llm-huggingface]"` | No — local weights | +| All providers | `pip install "semantica[all]"` | Varies | + +## Quick Start + + + + ```python + from semantica.llms import Groq + import os + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + ``` + + + ```python + from semantica.semantic_extract import NERExtractor + + ner = NERExtractor(method="llm", llm_provider=llm) + ``` + + + ```python + entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") + ``` + + + ```python + from semantica.llms import create_provider + from semantica.core import ConfigManager + + config = ConfigManager("config.yaml") + llm = create_provider( + config.get("llm_provider.name"), + model=config.get("llm_provider.model"), + api_key=config.get("llm_provider.api_key"), + ) + ``` + + ## Providers @@ -24,62 +93,67 @@ from semantica.llms import Groq import os llm = Groq( - model="llama-3.3-70b-versatile", # default + model="llama-3.3-70b-versatile", # default and recommended api_key=os.getenv("GROQ_API_KEY"), - max_tokens=64000, temperature=0.0, + max_tokens=64000, + max_retries=3, + timeout=60, ) -# Best for: high-throughput extraction, fast inference ``` ```python OpenAI from semantica.llms import OpenAI import os -# pip install "semantica[llm-openai]" llm = OpenAI( model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY"), temperature=0.0, + max_tokens=4096, + max_retries=3, + timeout=120, + organization=None, # optional org ID ) -# Best for: general purpose, function calling ``` ```python Anthropic from semantica.llms import Anthropic import os -# pip install "semantica[llm-anthropic]" llm = Anthropic( model="claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY"), max_tokens=8192, + temperature=0.0, + max_retries=3, + timeout=120, ) -# Best for: complex reasoning, long context, safety ``` ```python Gemini from semantica.llms import Gemini import os -# pip install "semantica[llm-gemini]" llm = Gemini( model="gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY"), + temperature=0.0, + max_tokens=8192, + timeout=120, ) -# Best for: long context (1M tokens), multimodal tasks ``` ```python Ollama (Local) from semantica.llms import Ollama -# pip install "semantica[llm-ollama]" llm = Ollama( - model="llama3.2:3b", - base_url="http://localhost:11434", + model="llama3.2", + base_url="http://localhost:11434", # default Ollama address + temperature=0.0, + timeout=180, # local models can be slower; increase for large models ) -# Best for: local inference, air-gapped environments -# No API key required +# No API key — model runs entirely on your machine ``` ```python DeepSeek @@ -89,64 +163,206 @@ import os llm = DeepSeek( model="deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY"), + temperature=0.0, + max_tokens=4096, + max_retries=3, ) -# Best for: coding tasks and analysis at very low cost +``` + +```python Novita AI +from semantica.llms import NovitaAI +import os + +llm = NovitaAI( + model="deepseek/deepseek-v3.2", + api_key=os.getenv("NOVITA_API_KEY"), + temperature=0.0, + max_tokens=4096, +) +# OpenAI-compatible endpoint; also accepts deepseek/deepseek-r1, meta-llama models ``` ```python LiteLLM (100+ models) from semantica.llms import LiteLLM import os -# pip install "semantica[llm-litellm]" llm = LiteLLM( - model="gpt-4o", # any LiteLLM-supported model string + model="gpt-4o", # any LiteLLM model string api_key=os.getenv("OPENAI_API_KEY"), + temperature=0.0, + max_tokens=4096, ) -# Supports: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, and 90+ more +# Supports: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, Together AI, and 90+ more +# Use the LiteLLM model string format: "anthropic/claude-3-5-sonnet", "bedrock/anthropic.claude-v2" ``` -```python HuggingFace (BYOM) +```python HuggingFace (Local) from semantica.llms import HuggingFace llm = HuggingFace( model="mistralai/Mistral-7B-Instruct-v0.3", - device="cuda", # "cpu" | "cuda" | "mps" + device="cuda", # "cpu" | "cuda" | "mps" (Apple Silicon) max_new_tokens=512, temperature=0.1, + load_in_4bit=True, # enable 4-bit quantisation to reduce VRAM ) -# Bring your own model — full local control, no API key +# No API key — weights downloaded from Hugging Face Hub (or loaded from local path) ``` -## Provider Factory +## Constructor Parameters -Instantiate any provider by name string — useful when provider is loaded from config: +### Common Parameters + +| Parameter | Type | Default | Description | +| --------- | ---- | ------- | ----------- | +| `model` | `str` | Provider default | Model identifier string | +| `api_key` | `str` | `None` | API key — reads from environment if omitted | +| `temperature` | `float` | `0.0` | Sampling temperature: 0 = deterministic, 1 = creative | +| `max_tokens` | `int` | Provider default | Maximum tokens in the response | +| `max_retries` | `int` | `3` | Number of retry attempts on transient failures | +| `timeout` | `int` | `60` | Request timeout in seconds | +| `base_url` | `str` | Provider default | Override the API endpoint — useful for proxies and gateways | + +### Provider-Specific Parameters + +| Provider | Parameter | Description | +| -------- | --------- | ----------- | +| `OpenAI` | `organization` | OpenAI organisation ID | +| `OpenAI` | `project` | OpenAI project ID | +| `Ollama` | `base_url` | Ollama server address (default: `http://localhost:11434`) | +| `HuggingFace` | `device` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | +| `HuggingFace` | `load_in_4bit` | Enable 4-bit quantisation (requires `bitsandbytes`) | +| `HuggingFace` | `max_new_tokens` | Maximum new tokens to generate (replaces `max_tokens`) | +| `LiteLLM` | `model` | Full LiteLLM model string, e.g. `"anthropic/claude-3-5-sonnet"` | + +## Direct API Usage + +Providers can be used directly — not just through Semantica modules: ```python -from semantica.llms import create_provider +from semantica.llms import Groq +import os -llm = create_provider("groq", model="llama-3.3-70b-versatile") -llm = create_provider("openai", model="gpt-4o") -llm = create_provider("anthropic", model="claude-opus-4-7") -llm = create_provider("gemini", model="gemini-1.5-pro") -llm = create_provider("ollama", model="llama3.2") -llm = create_provider("deepseek", model="deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY")) -llm = create_provider("novita", model="deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY")) -llm = create_provider("litellm", model="gpt-4o") +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +# Single completion +response = llm.complete("What is a knowledge graph?") +print(response.text) # answer string +print(response.input_tokens) # tokens consumed by the prompt +print(response.output_tokens) # tokens in the response +print(response.model) # model that served the request + +# Multi-turn chat +messages = [ + {"role": "system", "content": "You are a knowledge graph expert."}, + {"role": "user", "content": "What is the difference between RDF and property graphs?"}, +] +response = llm.chat(messages) +print(response.text) + +# Streaming — token-by-token output +for token in llm.stream("Explain knowledge graph reasoning in 3 sentences."): + print(token, end="", flush=True) ``` -## Custom / Enterprise Gateways +## LLMResponse Object -Any OpenAI-compatible endpoint — internal routing layers, Qwen proxies, or private LLaMA deployments: +All three methods (`complete`, `chat`, `stream`) return a `LLMResponse` dataclass: + + + + +```python +@dataclass +class LLMResponse: + text: str # the generated text + model: str # model identifier that served the request + input_tokens: int # tokens consumed by the prompt + output_tokens: int # tokens in the generated response + total_tokens: int # input_tokens + output_tokens + latency_ms: float # wall-clock time for the API call in milliseconds + finish_reason: str # "stop" | "length" | "content_filter" | "tool_calls" +``` + + + + +## Error Handling + + + + +```python +from semantica.llms import Groq +from semantica.utils import SemanticaError +import os + +try: + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + response = llm.complete("Summarise this document.") + print(response.text) + +except LLMAuthenticationError as e: + # Invalid or expired API key + print(f"Authentication failed: {e}") + +except LLMRateLimitError as e: + # Rate limit hit — Semantica retries automatically up to max_retries + print(f"Rate limited after retries: {e}") + +except LLMContextLengthError as e: + # Prompt exceeds the model's context window + print(f"Prompt too long ({e.token_count} tokens, limit {e.context_limit}): {e}") + +except LLMProviderError as e: + # General provider-side error (5xx, model unavailable, etc.) + print(f"Provider error: {e}") + +except SemanticaError as e: + # Catch-all for all Semantica framework errors + print(f"Framework error: {e}") +``` + +| Exception | When Raised | +| --------- | ----------- | +| `LLMAuthenticationError` | Invalid or missing API key | +| `LLMRateLimitError` | Rate limit exceeded after all retries | +| `LLMContextLengthError` | Prompt exceeds the model's context window | +| `LLMProviderError` | Provider-side error (5xx, unavailability) | +| `LLMTimeoutError` | Request exceeded the `timeout` parameter | + + + + +## Custom and Enterprise Gateways + +Any provider that exposes an OpenAI-compatible REST API can be used by passing `base_url`: ```python from semantica.llms import OpenAI +import os +# Internal LLM routing gateway llm = OpenAI( model="qwen2.5-72b", api_key=os.getenv("GATEWAY_API_KEY"), - base_url="https://my-internal-gateway.company.com/v1", + base_url="https://llm-gateway.internal.company.com/v1", +) + +# Azure OpenAI Service +llm = OpenAI( + model="gpt-4o", + api_key=os.getenv("AZURE_OPENAI_API_KEY"), + base_url="https://my-resource.openai.azure.com/openai/deployments/gpt-4o", +) + +# Self-hosted vLLM server +llm = OpenAI( + model="meta-llama/Llama-3.1-8B-Instruct", + api_key="not-needed", + base_url="http://localhost:8000/v1", ) ``` @@ -154,49 +370,126 @@ llm = OpenAI( `base_url` is validated at construction time. Non-HTTP(S) schemes raise `ValueError` to prevent SSRF attacks (fixed in v0.5.0). -## Using in Extractors +## Using in Semantica Modules -All extractors accept any provider as `llm_provider=`: +Every module that uses an LLM accepts any provider through `llm_provider=`: ```python +from semantica.llms import Groq, Anthropic from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor +from semantica.ontology import LLMOntologyGenerator +from semantica.reasoning import ReasoningEngine +from semantica.context import AgentContext, ContextGraph +from semantica.vector_store import VectorStore +import os -llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +groq_llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +claude_llm = Anthropic(model="claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) -ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) -rel = RelationExtractor(method="llm", llm_provider=llm) -trip = TripletExtractor(method="llm", llm_provider=llm) +# Extraction — use fast Groq for high-throughput NER +ner = NERExtractor(method="llm", llm_provider=groq_llm) +rel = RelationExtractor(method="llm", llm_provider=groq_llm) +trip = TripletExtractor(method="llm", llm_provider=groq_llm) + +# Complex reasoning — use Claude for accuracy +engine = ReasoningEngine(llm_provider=claude_llm) + +# Ontology generation from natural language +gen = LLMOntologyGenerator(llm_provider=claude_llm) ``` ## Provider Comparison -| Provider | Speed | Cost | Local | Context | Best For | -| -------- | ----- | ---- | ----- | ------- | -------- | -| Groq | Very fast | Low | No | 128k | High-throughput extraction | -| OpenAI | Fast | Medium | No | 128k | General purpose, function calling | -| Anthropic | Fast | Medium | No | 200k | Complex reasoning, safety | -| Gemini | Fast | Low | No | 1M | Long context, multimodal | -| Ollama | Medium | Free | Yes | Varies | Privacy, no API key | -| DeepSeek | Fast | Very low | No | 64k | Coding, analysis | -| Novita AI | Fast | Low | No | Varies | DeepSeek-based tasks | -| LiteLLM | Varies | Varies | Varies | Varies | Multi-provider routing | -| HuggingFace | Slow | Free | Yes | Varies | Custom models, BYOM | +| Provider | Speed | Cost | Local | Max Context | Best For | +| -------- | ----- | ---- | ----- | ----------- | -------- | +| **Groq** | ⚡ Very fast | 💲 Low | No | 128k | High-throughput extraction, fast pipelines | +| **OpenAI** | Fast | 💲💲 Medium | No | 128k | General purpose, function calling, JSON mode | +| **Anthropic** | Fast | 💲💲 Medium | No | 200k | Complex reasoning, long documents, safety | +| **Gemini** | Fast | 💲 Low | No | 1M | Very long context, multimodal (text + image) | +| **Ollama** | Medium | Free | ✅ Yes | Varies | Privacy, air-gapped, no API key | +| **DeepSeek** | Fast | 💲 Very low | No | 64k | Coding tasks, structured analysis | +| **Novita AI** | Fast | 💲 Low | No | Varies | DeepSeek and LLaMA models, cost-effective | +| **LiteLLM** | Varies | Varies | Varies | Varies | Multi-provider routing, vendor abstraction | +| **HuggingFace** | Slow | Free | ✅ Yes | Varies | Custom and fine-tuned models, full local control | + +## Environment Variables + +| Variable | Provider | Notes | +| -------- | -------- | ----- | +| `GROQ_API_KEY` | Groq | Required for Groq cloud | +| `OPENAI_API_KEY` | OpenAI, LiteLLM | Also used for OpenAI-compatible gateways | +| `ANTHROPIC_API_KEY` | Anthropic | Required for Claude | +| `GOOGLE_API_KEY` | Gemini | Required for Gemini | +| `DEEPSEEK_API_KEY` | DeepSeek | Required for DeepSeek cloud | +| `NOVITA_API_KEY` | Novita AI | Required for Novita AI | +| `HUGGINGFACE_HUB_TOKEN` | HuggingFace | Required for gated models (optional for public models) | + +## YAML Configuration + +```yaml +# config.yaml +llm_provider: + name: "groq" + model: "llama-3.3-70b-versatile" + api_key: "${GROQ_API_KEY}" # reads from environment + temperature: 0.0 + max_tokens: 64000 + max_retries: 3 + timeout: 60 +``` + +Load it with `ConfigManager`: + +```python +from semantica.core import ConfigManager +from semantica.llms import create_provider + +config = ConfigManager("config.yaml") +llm = create_provider( + config.get("llm_provider.name"), + model=config.get("llm_provider.model"), + api_key=config.get("llm_provider.api_key"), + temperature=config.get("llm_provider.temperature", default=0.0), +) +``` + +## Tips and Common Pitfalls + + + **Always set `temperature=0.0` for extraction tasks.** NER, relation extraction, and triplet generation need deterministic output — any temperature above 0 introduces randomness that produces inconsistent entity types or hallucinated relationships. Reserve higher temperatures for creative or summarisation tasks. + - For production extraction pipelines, Groq delivers the best throughput-to-cost ratio. For complex multi-hop reasoning, Claude Opus or GPT-4o provide the highest accuracy. + **Set `max_retries=3` in production.** Transient rate limits and 5xx errors are normal at scale. All providers retry automatically up to `max_retries` with exponential backoff. Without retries, a single rate-limit hit fails an entire pipeline step that would have succeeded on the second attempt. + + + + **Use `create_provider()` for config-driven pipelines.** Hard-coding `Groq(...)` in Python means changing the provider requires a code change and redeploy. `create_provider(config.get("llm_provider.name"), ...)` lets you switch from Groq to Anthropic by editing `config.yaml` — no code changes. + + + + **Use `base_url` for internal gateways and Azure.** Enterprise deployments often route LLM calls through an internal proxy or Azure OpenAI Service. Pass `base_url="https://llm-gateway.internal/v1"` to `OpenAI` — you get the same Semantica integration without changing any module code. Non-HTTP schemes raise `ValueError` (SSRF protection, v0.5.0+). + + + + **Catch `LLMContextLengthError` explicitly.** If your chunking is misconfigured, a document chunk can exceed the model's context window. Catch `LLMContextLengthError` and log `e.token_count` — it tells you exactly how much to reduce your `chunk_size`. Don't let it surface as a generic failure. + + + + **Use Ollama or HuggingFace for air-gapped environments.** When data cannot leave the network, Ollama (local inference) or HuggingFace (local weights) are the only viable options. Both support the same `llm_provider=` interface — no other code changes needed. - Use LLMs for NER and relation extraction. - - - LLM providers in Agno multi-agent teams. + NER, relation extraction, and triplet generation with LLMs. - LLM-backed deductive and abductive reasoning. + LLM-backed deductive, abductive, and Datalog reasoning. + + + Generate ontologies from natural language using LLMs. - GraphRAG uses LLMs for reasoning over knowledge graphs. + GraphRAG and decision intelligence powered by LLMs. diff --git a/docs/reference/mcp_server.md b/docs/reference/mcp_server.md index 19357b5f..5e17c33a 100644 --- a/docs/reference/mcp_server.md +++ b/docs/reference/mcp_server.md @@ -6,15 +6,32 @@ icon: "plug" `semantica.mcp_server` exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server over stdio. +Once configured, any connected AI assistant can extract entities, record decisions, query the graph, run reasoning, and export results — without writing a single line of Python. + Compatible with **Claude Desktop**, **Windsurf**, **Cline**, **Continue**, **VS Code**, **Roo Code**, **Cursor**, and any MCP-aware client. ## What You Get -- **12 MCP tools** — extract entities, build graphs, run SPARQL, find paths, get recommendations, embed, cluster, and more -- **3 readable resources** — live graph JSON, entity list, and relationship list -- **Zero infrastructure** — runs over stdio, no server or port needed -- **Claude Desktop ready** — one config block to add to `claude_desktop_config.json` -- **REST alternative** — the Explorer module offers a full HTTP API if you prefer + + + Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export. + + + Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info — readable by any MCP client. + + + Runs over stdio — no server, no port, no Docker required. One config block to activate in any MCP client. + + + Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup. + + + Record decisions, find precedents via hybrid similarity search, and trace causal chains across agent runs. + + + The [Explorer](explorer) module offers a full HTTP API and browser dashboard if you prefer programmatic access. + + ## Installation @@ -26,46 +43,86 @@ The MCP server is included in the base install — no extras required. ## Configuration -Add Semantica to your MCP client's settings file: + + - + | Client | Settings file | + | ------ | ------------- | + | Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` | + | Claude Desktop (Windows) | `%APPDATA%\Claude\claude_desktop_config.json` | + | Cursor | `.cursor/mcp.json` in your project, or `~/.cursor/mcp.json` globally | + | VS Code / Continue | `.vscode/mcp.json` or user settings | + | Windsurf / Cline / Roo Code | App-specific settings → MCP Servers | -```json Claude Desktop / Windsurf / Cline -{ - "mcpServers": { - "semantica": { - "command": "semantica-mcp" - } - } -} -``` + + -```json VS Code / Continue / Roo Code -{ - "mcpServers": { - "semantica": { - "command": "python", - "args": ["-m", "semantica.mcp_server"] - } - } -} -``` + -```json With persistent graph -{ - "mcpServers": { - "semantica": { - "command": "semantica-mcp", - "env": { - "SEMANTICA_KG_PATH": "/path/to/my_graph.json", - "SEMANTICA_LOG_LEVEL": "INFO" + ```json Claude Desktop / Windsurf / Cline + { + "mcpServers": { + "semantica": { + "command": "semantica-mcp" + } } } - } -} -``` + ``` - + ```json Cursor + { + "mcpServers": { + "semantica": { + "command": "semantica-mcp", + "env": { + "SEMANTICA_KG_PATH": "/path/to/my_graph.json" + } + } + } + } + ``` + + ```json VS Code / Continue / Roo Code + { + "mcpServers": { + "semantica": { + "command": "python", + "args": ["-m", "semantica.mcp_server"] + } + } + } + ``` + + ```json With persistent graph + { + "mcpServers": { + "semantica": { + "command": "semantica-mcp", + "env": { + "SEMANTICA_KG_PATH": "/path/to/my_graph.json", + "SEMANTICA_LOG_LEVEL": "INFO" + } + } + } + } + ``` + + + + + + ```bash + # Run the server directly (reads from stdin, writes to stdout) + semantica-mcp + + # Or via Python module + python -m semantica.mcp_server + + # Send a JSON-RPC initialize message to confirm it's working + echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' | semantica-mcp + ``` + + ## Environment Variables @@ -78,6 +135,21 @@ Add Semantica to your MCP client's settings file: The MCP server exposes 12 tools that any connected AI assistant can call: +| Tool | Category | Description | +| ---- | -------- | ----------- | +| `extract_entities` | Extraction | NER — find people, places, organisations, concepts | +| `extract_relations` | Extraction | Typed relation and triplet extraction | +| `record_decision` | Decision Intelligence | Save a decision with reasoning and outcome | +| `query_decisions` | Decision Intelligence | Search recorded decisions by natural language | +| `find_precedents` | Decision Intelligence | Hybrid similarity search over past decisions | +| `get_causal_chain` | Decision Intelligence | Trace upstream / downstream causal chains | +| `add_entity` | Graph Operations | Add a node to the live graph | +| `add_relationship` | Graph Operations | Add a directed edge between two nodes | +| `get_graph_analytics` | Graph Operations | PageRank + community detection | +| `get_graph_summary` | Graph Operations | Node count, decision count, health status | +| `run_reasoning` | Reasoning & Export | Forward-chain IF/THEN rules over facts | +| `export_graph` | Reasoning & Export | Serialise the graph (Turtle, JSON-LD, JSON, etc.) | + ### Knowledge Extraction @@ -87,11 +159,13 @@ The MCP server exposes 12 tools that any connected AI assistant can call: Extract named entities (people, places, organisations, concepts) from text using Semantica NER. **Input:** + ```json { "text": "Apple Inc. was founded by Steve Jobs in Cupertino in 1976." } ``` **Output:** + ```json { "entities": [ @@ -110,11 +184,13 @@ Extract named entities (people, places, organisations, concepts) from text using Extract typed relations and `(subject, predicate, object)` triplets from text. **Input:** + ```json { "text": "Steve Jobs founded Apple Inc. and led it until 2011." } ``` **Output:** + ```json { "relations": [ @@ -139,6 +215,7 @@ Extract typed relations and `(subject, predicate, object)` triplets from text. Record a decision with full context, reasoning, and metadata into the knowledge graph. **Input:** + ```json { "category": "model_selection", @@ -151,6 +228,7 @@ Record a decision with full context, reasoning, and metadata into the knowledge ``` **Output:** + ```json { "decision_id": "dec_a1b2c3", "status": "recorded" } ``` @@ -162,6 +240,7 @@ Record a decision with full context, reasoning, and metadata into the knowledge Query recorded decisions by natural language, category, or retrieve all recent decisions. **Input:** + ```json { "query": "model selection", "limit": 5 } ``` @@ -173,6 +252,7 @@ Query recorded decisions by natural language, category, or retrieve all recent d Find past decisions similar to a given scenario using hybrid similarity search. **Input:** + ```json { "scenario": "Choose cloud provider for HIPAA workload", "max_results": 3 } ``` @@ -184,6 +264,7 @@ Find past decisions similar to a given scenario using hybrid similarity search. Trace the causal chain upstream or downstream from a decision. **Input:** + ```json { "decision_id": "dec_a1b2c3", "direction": "downstream", "max_depth": 5 } ``` @@ -201,6 +282,7 @@ Trace the causal chain upstream or downstream from a decision. Add a node/entity to the live knowledge graph. **Input:** + ```json { "id": "apple_inc", @@ -217,6 +299,7 @@ Add a node/entity to the live knowledge graph. Add a directed relationship (edge) between two existing entities. **Input:** + ```json { "source": "steve_jobs", @@ -251,6 +334,7 @@ Return node count, decision count, and graph health status. Run forward-chaining IF/THEN rules over a set of facts to derive new facts. **Input:** + ```json { "facts": ["Employee(John)", "Manager(John)"], @@ -259,6 +343,7 @@ Run forward-chaining IF/THEN rules over a set of facts to derive new facts. ``` **Output:** + ```json { "derived_facts": ["HasAuthority(John)"] } ``` @@ -270,6 +355,7 @@ Run forward-chaining IF/THEN rules over a set of facts to derive new facts. Export the current knowledge graph to a serialization format. **Input:** + ```json { "format": "json-ld" } ``` @@ -282,7 +368,7 @@ Supported formats: `turtle`, `ttl`, `nt`, `xml`, `json-ld`, `json`. ## Resources -The MCP server also exposes three readable resources: +The MCP server exposes three readable resources: | URI | Description | | --- | ----------- | @@ -290,21 +376,27 @@ The MCP server also exposes three readable resources: | `semantica://decisions/list` | All recorded decisions (up to 50) | | `semantica://schema/info` | Server version and available tools | -## Test Locally +## Tips and Common Pitfalls -```bash -# Run the server directly (reads from stdin, writes to stdout) -semantica-mcp + + **Build the `ContextGraph` before starting the server.** The MCP server operates on a pre-built `ContextGraph` — it doesn't build the knowledge graph on demand. Construct and populate the graph first (ingest → extract → build KG → set `ContextGraph`), then pass it to `SemanticaMCPServer`. An empty or None graph results in empty query responses. + -# Or via Python module -python -m semantica.mcp_server -``` + + **Use `decision_tracking=True` for accountable agents.** Without decision tracking, `record_decision` and `query_decisions` calls succeed but nothing is stored. Enable it in the `ContextGraph` constructor when you want agents' decisions to be queryable for audit, compliance, or iterative reasoning. + -Send a JSON-RPC `initialize` message to confirm it's working: + + **Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path — it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs. + -```bash -echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' | semantica-mcp -``` + + **Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently — the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works. + + + + **The server communicates over stdio — don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. Configure logging to write to a file or stderr only (`logging.basicConfig(filename="mcp.log")`). The MCP protocol assumes stdout carries only JSON-RPC frames. + diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index 98f87ca2..4f9a56a4 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -1,275 +1,404 @@ --- title: "Normalize Module" -description: "Text cleaning, entity canonicalization, date normalization, number/unit conversion, language detection, and encoding repair." +description: "Text cleaning, entity canonicalization, date normalization, number conversion, language detection, and encoding repair — before extraction runs." icon: "broom" --- -`semantica.normalize` standardizes raw data before extraction and graph construction — fixing encodings, canonicalizing entity names, normalizing dates, and detecting languages. All normalizers expose both convenience functions and stateful class instances. +`semantica.normalize` standardizes raw data before extraction and graph construction. All normalizers expose both convenience functions (one-liners) and stateful class instances (full control over configuration and reuse). + +## Why Normalize Before Extraction + +Unstructured data is inconsistent by nature. Without normalization, the same real-world entity appears as dozens of variants in your graph: + +- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."`, `"Apple, Inc."` — four nodes, one company +- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"` — three formats, one date +- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"` — three strings, one number +- `"Hello World"` vs `"Hello World"` — a non-breaking space that breaks string matching + +Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data — producing cleaner entities, fewer false duplicates, and more reliable downstream results. ## What You Get -- **`TextNormalizer`** — Unicode, whitespace, HTML stripping, smart-quote/dash replacement -- **`EntityNormalizer`** — alias resolution, disambiguation, name variant handling -- **`DateNormalizer`** — ISO 8601 output, timezone conversion, relative date parsing -- **`NumberNormalizer`** — currency, unit conversion, scientific notation, percentages -- **`DataCleaner`** — duplicate detection, schema validation, missing value handling -- **`LanguageDetector`** — 50+ language detection with confidence scoring -- **`EncodingHandler`** — UTF-8 conversion, BOM removal, encoding detection + + + Unicode forms, whitespace collapse, HTML stripping, smart-quote and dash replacement. + + + Corporate suffix normalization, honorific removal, alias resolution, and disambiguation. + + + Any date format → ISO 8601; relative dates, timezones, and date ranges. + + + Currency, scientific notation, unit abbreviations, and percentages → float. + + + 50+ languages with confidence scoring and batch detection. + + + Encoding detection, UTF-8 conversion, BOM removal, and cp1252 repair. + + - **v0.5.0 fix:** Encoding repair now handles cp1252/latin-1 characters that previously caused crashes on Windows when processing documents with non-ASCII content. + **v0.5.0 fix:** Encoding repair now handles cp1252 and latin-1 characters that previously caused crashes on Windows when processing documents with non-ASCII content. +## Recommended Processing Order + + + + Broken bytes corrupt everything downstream. Always run this before anything else. + + ```python + from semantica.normalize import EncodingHandler + + handler = EncodingHandler() + utf8_text = handler.to_utf8(raw_bytes) + ``` + + + ```python + from semantica.normalize import TextNormalizer + + normalizer = TextNormalizer(strip_html=True, normalize_unicode=True) + clean_text = normalizer.normalize_text(utf8_text) + ``` + + + ```python + from semantica.normalize import EntityNormalizer + + normalizer = EntityNormalizer() + canonical = normalizer.normalize_entity("Apple Computer Inc.", entity_type="Organization") + # → "Apple Inc." + ``` + + + ```python + from semantica.normalize import DateNormalizer, NumberNormalizer + + date_norm = DateNormalizer(target_timezone="UTC") + num_norm = NumberNormalizer() + + date = date_norm.normalize_date("Jan 1st, 2020") # → "2020-01-01" + num = num_norm.normalize_number("$1.2B") # → 1200000000.0 + ``` + + + ```python + from semantica.normalize import LanguageDetector + + detector = LanguageDetector() + lang = detector.detect("Bonjour le monde") + # → {"language": "fr", "confidence": 0.98} + ``` + + + ## Convenience Functions -The fastest path — dispatch via function with a `method` parameter: +The fastest path — one import, one call: ```python from semantica.normalize import ( normalize_text, normalize_entity, normalize_date, - normalize_number, clean_data, detect_language, handle_encoding + normalize_number, clean_data, detect_language, handle_encoding, ) -clean = normalize_text(" Hello, World!! \n\n") -# → "Hello, World!!" - -entity = normalize_entity("Apple Computer Inc.", entity_type="Organization") -# → "Apple Inc." - -date = normalize_date("Jan 1st, 2020") -# → "2020-01-01" - -num = normalize_number("$1,234.56") -# → 1234.56 - -lang = detect_language("Bonjour le monde") -# → {"language": "fr", "confidence": 0.98} +clean = normalize_text(" Hello, World!! \n\n") # → "Hello, World!!" +entity = normalize_entity("Apple Computer Inc.", entity_type="Organization") # → "Apple Inc." +date = normalize_date("Jan 1st, 2020") # → "2020-01-01" +num = normalize_number("$1.2B") # → 1200000000.0 +lang = detect_language("Bonjour le monde") # → {"language": "fr", "confidence": 0.98} ``` -## TextNormalizer +## Normalizers -```python -from semantica.normalize import TextNormalizer + + + Cleans raw text at the character and token level: -normalizer = TextNormalizer() + ```python + from semantica.normalize import TextNormalizer -normalized = normalizer.normalize_text( - raw_text, - lowercase=False, - remove_punctuation=False, - remove_extra_whitespace=True, - strip_html=True, # remove HTML tags - normalize_unicode=True, # NFC normalization -) -``` + normalizer = TextNormalizer( + lowercase=False, + remove_punctuation=False, + remove_extra_whitespace=True, + strip_html=True, + normalize_unicode=True, + fix_encoding=True, + form="NFC", # "NFC" | "NFD" | "NFKC" | "NFKD" + ) -Sub-normalizers for fine-grained control: + normalized = normalizer.normalize_text(raw_text) + ``` -```python -from semantica.normalize import UnicodeNormalizer, WhitespaceNormalizer, SpecialCharacterProcessor + | Parameter | Type | Default | Description | + | --------- | ---- | ------- | ----------- | + | `lowercase` | `bool` | `False` | Convert to lowercase — use for bag-of-words matching, not NER | + | `remove_punctuation` | `bool` | `False` | Strip all punctuation — use for keyword extraction only | + | `remove_extra_whitespace` | `bool` | `True` | Collapse tabs, newlines, non-breaking spaces into single spaces | + | `strip_html` | `bool` | `False` | Remove HTML tags and decode `&`, `<`, etc. | + | `normalize_unicode` | `bool` | `True` | Apply Unicode normal form | + | `fix_encoding` | `bool` | `True` | Repair common encoding mojibake (cp1252 / latin-1 → UTF-8) | + | `form` | `str` | `"NFC"` | Unicode normalization form: `"NFC"` / `"NFD"` / `"NFKC"` / `"NFKD"` | -# Unicode normalization forms: NFC | NFD | NFKC | NFKD -unicode_norm = UnicodeNormalizer(form="NFC") -text = unicode_norm.normalize("café") + **Unicode form guide:** -# Collapse tabs, line breaks, and extra spaces -ws_norm = WhitespaceNormalizer() -text = ws_norm.normalize("Hello World\t\n") # → "Hello World" + | Form | Use When | + | ---- | -------- | + | `NFC` | Default — best for storage and display | + | `NFKC` | Search indexing — normalises ligatures, fullwidth chars, and fractions | + | `NFD` | Stripping diacritics — split é → e + combining accent, then strip accents | + | `NFKD` | Same as NFD but also decomposes compatibility characters | -# Replace smart quotes, em-dashes, and ellipsis characters -processor = SpecialCharacterProcessor() -text = processor.process("‘Hello’") # '' → '' -``` + **Sub-normalizers for fine-grained control:** -## EntityNormalizer + ```python + from semantica.normalize import UnicodeNormalizer, WhitespaceNormalizer, SpecialCharacterProcessor -Canonicalize entity names — handles corporate suffixes, punctuation, case, and honorifics: + unicode_norm = UnicodeNormalizer(form="NFC") + text = unicode_norm.normalize("café") -```python -from semantica.normalize import EntityNormalizer + ws_norm = WhitespaceNormalizer() + text = ws_norm.normalize("Hello\t\t World\n\n") # → "Hello World" -normalizer = EntityNormalizer() + processor = SpecialCharacterProcessor() + text = processor.process("'Hello'") # '' → '', -- → - + ``` + + + Canonicalises entity name variants — corporate suffixes, honorifics, case, and punctuation: -# Company name normalization -companies = ["Apple Computer, Inc.", "Apple Inc", "APPLE INC."] -normalized = [normalizer.normalize_entity(c) for c in companies] -# All → "Apple Inc." + ```python + from semantica.normalize import EntityNormalizer -# Person name normalization -name = normalizer.normalize_entity("JOBS, STEVE", entity_type="Person") -# → "Steve Jobs" -``` + normalizer = EntityNormalizer() -Sub-normalizers for entity-specific use cases: + # Corporate name normalization + normalizer.normalize_entity("Apple Computer, Inc.", entity_type="Organization") # → "Apple Inc." + normalizer.normalize_entity("APPLE INC.", entity_type="Organization") # → "Apple Inc." -```python -from semantica.normalize import AliasResolver, EntityDisambiguator, NameVariantHandler + # Person name normalization + normalizer.normalize_entity("JOBS, STEVE", entity_type="Person") # → "Steve Jobs" + ``` -# Dictionary-based alias expansion -resolver = AliasResolver(aliases={ - "ML": "Machine Learning", - "AI": "Artificial Intelligence", - "DL": "Deep Learning", -}) -resolved = resolver.resolve("ML and DL are subsets of AI") -# → "Machine Learning and Deep Learning are subsets of Artificial Intelligence" + **Key behaviours:** + - Corporate suffix normalization handles: `Inc`, `Inc.`, `Incorporated`, `Ltd`, `Limited`, `Corp`, `Corporation`, `LLC`, `GmbH`, `PLC`, and 30+ more + - `entity_type="Person"` activates last-name-first reversal, honorific removal, and suffix stripping -# Context-aware disambiguation -disambiguator = EntityDisambiguator() -result = disambiguator.disambiguate( - "Apple", context="Steve Jobs founded Apple in Cupertino" -) -# → {"entity": "Apple Inc.", "type": "Organization", "confidence": 0.96} + **Sub-normalizers:** -# Honorifics, titles, and cultural name variants -handler = NameVariantHandler() -canonical = handler.normalize("Dr. JOHN P. SMITH Jr.") -# → "John P. Smith" -``` + ```python + from semantica.normalize import AliasResolver, EntityDisambiguator, NameVariantHandler -## DateNormalizer + resolver = AliasResolver(aliases={ + "ML": "Machine Learning", + "NLP": "Natural Language Processing", + }) + resolved = resolver.resolve("ML and NLP are subfields of AI") -Parse and normalize dates from any format to ISO 8601: + disambiguator = EntityDisambiguator() + result = disambiguator.disambiguate( + "Apple", + context="Steve Jobs founded Apple in Cupertino in 1976", + ) + # → {"entity": "Apple Inc.", "type": "Organization", "confidence": 0.96} -```python -from semantica.normalize import DateNormalizer + handler = NameVariantHandler() + canonical = handler.normalize("Dr. JOHN P. SMITH Jr.") # → "John P. Smith" + ``` + + + Parses any date format and outputs ISO 8601 strings. Handles relative expressions, timezones, and date ranges: -normalizer = DateNormalizer() + ```python + from semantica.normalize import DateNormalizer -dates = [ - "January 1st, 2020", - "01/01/2020", - "2020-01-01T00:00:00Z", - "yesterday", - "3 weeks ago", -] -normalized = [normalizer.normalize_date(d) for d in dates] -# All → ISO 8601 strings + normalizer = DateNormalizer( + target_timezone="UTC", + output_format="%Y-%m-%d", + ) -# With automatic UTC conversion -normalizer_utc = DateNormalizer(target_timezone="UTC") -utc_date = normalizer_utc.normalize_date("2024-01-01 09:00 EST") -``` + dates = [ + "January 1st, 2020", + "01/01/2020", + "2020-01-01T00:00:00Z", + "yesterday", + "3 weeks ago", + "Q1 2024", + ] + normalized = [normalizer.normalize_date(d) for d in dates] + ``` -Sub-normalizers for advanced date handling: + **Sub-normalizers:** -```python -from semantica.normalize import ( - TimeZoneNormalizer, RelativeDateProcessor, TemporalExpressionParser -) + ```python + from semantica.normalize import TimeZoneNormalizer, RelativeDateProcessor, TemporalExpressionParser + from datetime import datetime -# Timezone conversion -tz_norm = TimeZoneNormalizer(target_tz="UTC") -utc_dt = tz_norm.normalize("2024-01-01 09:00", source_tz="America/New_York") + tz_norm = TimeZoneNormalizer(target_tz="UTC") + utc_dt = tz_norm.normalize("2024-01-01 09:00", source_tz="America/New_York") + # → datetime(2024, 1, 1, 14, 0, tzinfo=UTC) -# Relative date resolution -from datetime import datetime -processor = RelativeDateProcessor(reference_date=datetime(2025, 1, 15)) -result = processor.process("3 days ago") -# → datetime(2025, 1, 12) + processor = RelativeDateProcessor(reference_date=datetime(2025, 1, 15)) + result = processor.process("3 days ago") # → datetime(2025, 1, 12) + result = processor.process("next quarter") # → {"start": "2025-04-01", "end": "2025-06-30"} -# Date range parsing -parser = TemporalExpressionParser() -result = parser.parse("from January 2020 to March 2021") -# → {"start": "2020-01-01", "end": "2021-03-31", "type": "range"} -``` + parser = TemporalExpressionParser() + result = parser.parse("from January 2020 to March 2021") + # → {"start": "2020-01-01", "end": "2021-03-31", "type": "range"} -## NumberNormalizer + result = parser.parse("Q2 2023") + # → {"start": "2023-04-01", "end": "2023-06-30", "type": "quarter"} + ``` + + + Converts number strings with units, currencies, and abbreviations to `float`: -```python -from semantica.normalize import NumberNormalizer + ```python + from semantica.normalize import NumberNormalizer -normalizer = NumberNormalizer() + normalizer = NumberNormalizer() -normalizer.normalize_number("$1,234.56") # → 1234.56 -normalizer.normalize_number("€42K") # → 42000.0 -normalizer.normalize_number("$1.2B") # → 1200000000.0 -normalizer.normalize_number("3.14e-2") # → 0.0314 -normalizer.normalize_number("42%") # → 0.42 -``` + normalizer.normalize_number("$1,234.56") # → 1234.56 + normalizer.normalize_number("€42K") # → 42000.0 + normalizer.normalize_number("$1.2B") # → 1200000000.0 + normalizer.normalize_number("3.14e-2") # → 0.0314 + normalizer.normalize_number("42%") # → 0.42 + normalizer.normalize_number("−7") # → -7.0 (minus sign, not hyphen) + ``` -Unit and currency conversion: + **Unit and currency conversion:** -```python -from semantica.normalize import UnitConverter, CurrencyNormalizer + ```python + from semantica.normalize import UnitConverter, CurrencyNormalizer -converter = UnitConverter() -result = converter.convert(100, from_unit="km/h", to_unit="m/s") -# → 27.78 + converter = UnitConverter() + result = converter.convert(100, from_unit="km/h", to_unit="m/s") + # → 27.78 -# Supported categories: length, weight, volume, temperature, speed, area -categories = converter.list_categories() + categories = converter.list_categories() + # → ["length", "weight", "volume", "temperature", "speed", "area", "pressure", "energy"] -currency_norm = CurrencyNormalizer() -result = currency_norm.normalize("$42.50") -# → {"amount": 42.50, "currency": "USD", "raw": "$42.50"} -``` + currency_norm = CurrencyNormalizer() + result = currency_norm.normalize("$42.50") + # → {"amount": 42.50, "currency": "USD", "raw": "$42.50"} + ``` + + + ### LanguageDetector + + Identify the language of a text string. Used internally by the sentence splitter and chunker: + + ```python + from semantica.normalize import LanguageDetector + + detector = LanguageDetector() + + lang = detector.detect("Bonjour le monde") + # → {"language": "fr", "confidence": 0.98} + + langs = detector.detect_top_n("This might be mixed", n=3) + # → [{"language": "en", "probability": 0.85}, ...] + + results = detector.detect_batch(["Hello", "Hola", "Bonjour", "Ciao"]) + supported = detector.list_supported_languages() + ``` + + Supports 50+ languages: `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `ru`, `zh`, `ja`, `ko`, `ar`, `hi`, `tr`, `pl`, `sv`, `da`, `no`, `fi`, and more. + + ### EncodingHandler + + Detect and repair character encoding issues: + + ```python + from semantica.normalize import EncodingHandler + + handler = EncodingHandler() + + encoding = handler.detect_encoding(raw_bytes) + # → {"encoding": "windows-1252", "confidence": 0.73} + + utf8_text = handler.to_utf8(raw_bytes) + clean = handler.remove_bom(text_with_bom) + repaired = handler.repair_encoding(garbled_text, source_encoding="cp1252") + ``` + + **Key behaviours:** + - Encoding detection uses `chardet` internally — accuracy improves with longer input + - `to_utf8()` attempts cp1252 repair automatically when it detects mojibake patterns + - Always run `EncodingHandler` first — broken bytes cause cascading failures in every downstream normalizer + + ## DataCleaner +Cleans structured record sets — useful before loading into a vector store or graph: + ```python from semantica.normalize import DataCleaner, DataValidator cleaner = DataCleaner() -# Remove near-duplicate records deduped = cleaner.remove_duplicates(records, similarity_threshold=0.9) -# Fill missing values -filled = cleaner.fill_missing(records, strategy="mean") -# strategy options: "mean" | "median" | "mode" | "remove" +filled = cleaner.fill_missing( + records, + strategy="mean", # "mean" | "median" | "mode" | "remove" | "constant" + constant_value=None, +) -# Validate schema validator = DataValidator() -result = validator.validate(records, schema={"name": str, "age": int}) -print(result.valid_count, result.errors) -``` - -## LanguageDetector - -```python -from semantica.normalize import LanguageDetector - -detector = LanguageDetector() - -lang = detector.detect("Bonjour le monde") -# → {"language": "fr", "confidence": 0.98} - -# Top N languages for mixed-language text -langs = detector.detect_top_n("This might be mixed", n=3) -# → [{"language": "en", "probability": 0.85}, ...] - -# Batch detection -results = detector.detect_batch(["Hello", "Hola", "Bonjour"]) -``` - -## EncodingHandler - -```python -from semantica.normalize import EncodingHandler - -handler = EncodingHandler() - -encoding = handler.detect_encoding(raw_bytes) -# → {"encoding": "windows-1252", "confidence": 0.73} - -utf8_text = handler.to_utf8(raw_bytes) -clean = handler.remove_bom(text_with_bom) +result = validator.validate(records, schema={"name": str, "age": int, "active": bool}) +print(f"Valid: {result.valid_count}") +print(f"Invalid: {result.error_count}") ``` ## Pipeline Integration -For large datasets, use the pipeline instead of per-item normalization: - ```python from semantica.pipeline import Pipeline +from semantica.ingest import FileIngestor from semantica.normalize import TextNormalizer +from semantica.semantic_extract import NERExtractor +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) pipeline = Pipeline() -pipeline.add_step("normalize", TextNormalizer()) -result = pipeline.run(documents) +pipeline.add_step("ingest", FileIngestor()) +pipeline.add_step("normalize", TextNormalizer(strip_html=True, normalize_unicode=True)) +pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) + +result = pipeline.run("data/documents/") ``` +## Tips and Common Pitfalls + + + **Run encoding repair before anything else.** A single cp1252 character in a UTF-8 stream silently corrupts the surrounding text. Run `EncodingHandler` or set `fix_encoding=True` on `TextNormalizer` first. + + + + **Don't lowercase before NER.** `normalize_text(lowercase=True)` before entity extraction destroys capitalization signals that NER relies on. Apply case normalization only after extraction if needed. + + + + **AliasResolver is order-sensitive.** If you register overlapping aliases (`"ML"` and `"ML model"`), the longer match wins. Sort aliases by length descending for predictable behaviour. + + + + **DateNormalizer and timezone.** Without `target_timezone`, dates without timezone information are returned as naive datetime strings. In regulated pipelines (HIPAA, SOX), always set `target_timezone="UTC"` for unambiguous timestamps. + + + + **`DataCleaner.remove_duplicates()` is not the same as `DuplicateDetector`.** `DataCleaner` operates on flat records (dicts/rows) by field-level Jaccard similarity. `DuplicateDetector` in the Deduplication module operates on graph entities with embedding-based matching. Use the latter for entity resolution. + + Parse documents before normalization. @@ -278,9 +407,9 @@ result = pipeline.run(documents) Chunk normalized text for embedding. - Resolve duplicate entities post-normalization. + Resolve duplicate entities after normalization. - Include normalization as a pipeline step. + Include normalization as a named pipeline step. diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index d9c43b7b..d387ccdc 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -8,36 +8,70 @@ icon: "sitemap" ## What You Get -- **`OntologyManager`** — define classes, properties, relationships, and constraints -- **`OntologyGenerator`** — auto-generate ontologies from existing knowledge graph data (6-stage pipeline) -- **`SHACLGenerator`** / **`SHACLValidator`** — generate and validate SHACL shapes -- **`SKOSVocabulary`** — controlled vocabulary and taxonomy management -- **`OntologyAligner`** — align and merge ontologies across schemas -- **`OntologyDiff`** / **`OntologyMigrator`** — diff and migrate ontology versions -- **`OWLExporter`** — export to Turtle, RDF/XML, JSON-LD -- **Ontology Hub** — visual browser UI for the full ontology lifecycle (v0.5.0) + + + Define classes, properties, relationships, and constraints for your knowledge graph schema. + + + Auto-generate ontologies from existing graph data using a 6-stage pipeline. + + + Generate SHACL shapes from an ontology and validate graphs for constraint compliance. + + + Controlled vocabulary and taxonomy management using the W3C SKOS standard. + + + Align and merge ontologies across schemas — maps concepts with confidence scores. + + + Visual browser UI for the full ontology lifecycle — editor, SHACL Studio, and health dashboard. + + -## OntologyManager +## Quick Start -Define and validate a schema for your knowledge graph: + + + ```python + from semantica.ontology import OntologyManager -```python -from semantica.ontology import OntologyManager + ontology = OntologyManager() + ontology.add_class("Person", properties=["name", "birth_date"]) + ontology.add_class("Organization", properties=["name", "founded_date"]) + ontology.add_relationship("works_for", domain="Person", range="Organization") + ontology.add_constraint("Person", "must_have_name") + ``` + + + ```python + is_valid = ontology.validate_graph(kg) -ontology = OntologyManager() -ontology.add_class("Person", properties=["name", "birth_date"]) -ontology.add_class("Organization", properties=["name", "founded_date"]) -ontology.add_relationship("works_for", domain="Person", range="Organization") -ontology.add_constraint("Person", "must_have_name") + # Or use SHACL for granular constraint reporting + from semantica.ontology import SHACLGenerator, SHACLValidator -# Validate a graph against the ontology -is_valid = ontology.validate_graph(kg) + shapes = SHACLGenerator().generate(ontology) + validator = SHACLValidator() + report = validator.validate(kg, shapes=shapes) -# Export as OWL Turtle -owl_ttl = ontology.export_owl(format="turtle") -``` + if not report.conforms: + for v in report.violations: + print(f"Violation: {v.message} on {v.node} (path: {v.path})") + ``` + + + ```python + from semantica.ontology import OWLExporter -## Auto-Generation (6-Stage Pipeline) + exporter = OWLExporter() + exporter.export(ontology, path="ontology.ttl", format="turtle") + exporter.export(ontology, path="ontology.owl", format="xml") + exporter.export(ontology, path="ontology.json", format="json-ld") + ``` + + + +## Auto-Generation — 6-Stage Pipeline Generate an ontology automatically from your knowledge graph data: @@ -45,40 +79,74 @@ Generate an ontology automatically from your knowledge graph data: from semantica.ontology import OntologyGenerator generator = OntologyGenerator() -ontology = generator.generate_from_graph(kg) +ontology = generator.generate_from_graph(kg) ``` The pipeline runs through these stages in order: -1. **Semantic Network Parsing** — extract concepts and patterns from entity/relationship data -2. **YAML-to-Definition** — transform patterns into intermediate class definitions -3. **Definition-to-Types** — map definitions to OWL types (`owl:Class`, `owl:ObjectProperty`) -4. **Hierarchy Generation** — build taxonomy trees using transitive closure and cycle detection -5. **TTL Generation** — serialize to Turtle format using `rdflib` -6. **Quality Evaluation** — assess coverage, completeness, and granularity metrics + + + Extracts concepts and patterns from entity/relationship data. -## SHACL Validation + ```python + generator = OntologyGenerator() + semantic_network = generator.parse_semantic_network(kg) + ``` + + + Transforms extracted patterns into intermediate class definitions. -Generate SHACL shapes from an ontology and validate any graph against them: + ```python + definitions = generator.build_definitions(semantic_network) + ``` + + + Maps definitions to OWL types (`owl:Class`, `owl:ObjectProperty`). -```python -from semantica.ontology import SHACLGenerator, SHACLValidator + ```python + from semantica.ontology import ClassInferrer, PropertyGenerator -# Generate shapes -generator = SHACLGenerator() -shapes = generator.generate(ontology) -shapes_ttl = shapes.serialize(format="turtle") + class_inferrer = ClassInferrer() + classes = class_inferrer.infer_classes(kg.entities) -# Validate a graph -validator = SHACLValidator() -report = validator.validate(kg, shapes=shapes) + prop_generator = PropertyGenerator() + properties = prop_generator.infer_properties(kg.entities, kg.relationships, classes) + ``` + + + Builds taxonomy trees using transitive closure and cycle detection. -if not report.conforms: - for violation in report.violations: - print(f"Violation: {violation.message} on {violation.node}") - print(f" Path: {violation.path}") - print(f" Severity: {violation.severity}") -``` + ```python + hierarchy = generator.build_hierarchy(classes) + ``` + + + Serializes to Turtle format using `rdflib`. + + ```python + from semantica.ontology import OWLGenerator + + owl_gen = OWLGenerator() + ttl_str = owl_gen.generate_owl( + {"classes": classes, "properties": properties, "hierarchy": hierarchy}, + format="turtle", # "turtle" | "xml" | "json-ld" | "n3" + ) + ``` + + + Assesses coverage, completeness, and granularity metrics. + + ```python + from semantica.ontology import OntologyEvaluator + + evaluator = OntologyEvaluator() + report = evaluator.evaluate(ttl_str, kg) + print(f"Coverage: {report.coverage:.2%}") + print(f"Completeness: {report.completeness:.2%}") + print(f"Granularity: {report.granularity:.2%}") + ``` + + ## SKOS Vocabularies @@ -88,9 +156,9 @@ Build controlled vocabularies and taxonomies using the W3C SKOS standard: from semantica.ontology import SKOSVocabulary vocab = SKOSVocabulary() -vocab.add_concept("Machine Learning", broader="Artificial Intelligence") -vocab.add_concept("Deep Learning", broader="Machine Learning") -vocab.add_concept("Computer Vision", broader="Deep Learning") +vocab.add_concept("Machine Learning", broader="Artificial Intelligence") +vocab.add_concept("Deep Learning", broader="Machine Learning") +vocab.add_concept("Computer Vision", broader="Deep Learning") vocab.add_alt_label("ML", for_concept="Machine Learning") skos_ttl = vocab.export(format="turtle") @@ -98,49 +166,187 @@ skos_ttl = vocab.export(format="turtle") ## Ontology Alignment -Map concepts across two ontologies and merge them: + + + Map concepts across two ontologies: + + ```python + from semantica.ontology import OntologyAligner + + aligner = OntologyAligner() + alignment = aligner.align(source_ontology, target_ontology) + + for mapping in alignment.mappings: + print(f"{mapping.source} → {mapping.target} (confidence: {mapping.confidence:.2f})") + + merged = aligner.merge(source_ontology, target_ontology, alignment) + ``` + + + Compare versions and generate migration scripts: + + ```python + from semantica.ontology import OntologyDiff, OntologyMigrator + + diff = OntologyDiff() + changes = diff.compare(ontology_v1, ontology_v2) + + for change in changes: + print(f"{change.type}: {change.element} — {change.description}") + + migrator = OntologyMigrator() + migration_script = migrator.generate_migration(changes) + migrator.apply(kg, migration_script) + ``` + + + +## Advanced Generation Tools + + + + Capture and manage ontology requirements before generation: + + ```python + from semantica.ontology import RequirementsSpecManager + + spec = RequirementsSpecManager() + spec.add_competency_question("What organizations are headquartered in California?") + spec.add_competency_question("Who founded each organization?") + spec.add_competency_question("What products does each organization sell?") + + spec.set_scope( + domain="Technology industry", + excluded_types=["Event", "Date"], + min_confidence=0.7, + ) + + generator = OntologyGenerator() + ontology = generator.generate_from_graph(kg, requirements=spec) + ``` + + + Generate ontologies directly from natural language: + + ```python + from semantica.ontology import LLMOntologyGenerator + from semantica.llms import Groq + import os + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + generator = LLMOntologyGenerator(llm_provider=llm) + + ontology = generator.generate_from_description( + description="An ontology for tracking pharmaceutical clinical trials, including drugs, patients, dosages, outcomes, and adverse events.", + num_classes=20, + ) + + ontology = generator.generate_from_corpus( + documents=["clinical_trial_protocol.pdf"], + domain_hint="biomedical", + ) + + ontology = generator.refine( + ontology, + questions=["What dosage was given to each patient?", "What adverse events occurred?"], + ) + ``` + + + Integrate published ontologies instead of generating from scratch: + + ```python + from semantica.ontology import ReuseManager + + manager = ReuseManager() + manager.load_ontology("schema.org", source="https://schema.org/version/latest/schemaorg-current-https.ttl") + manager.load_ontology("foaf", source="http://xmlns.com/foaf/spec/index.rdf") + + candidates = manager.find_reusable_classes( + your_classes=["Person", "Organization", "Product"], + loaded_ontologies=["schema.org", "foaf"], + ) + for candidate in candidates: + print(f"{candidate.local_class} → reuse {candidate.external_uri} (similarity: {candidate.similarity:.2f})") + + merged = manager.merge_reused(your_ontology, candidates) + ``` + + + Pre-built domain ontologies for common verticals: + + ```python + from semantica.ontology import DomainOntologies + + catalog = DomainOntologies() + + for domain in catalog.list_domains(): + print(f"{domain.name}: {domain.description} ({domain.class_count} classes)") + + biomedical = catalog.load("biomedical") # SNOMED CT-aligned + finance = catalog.load("finance") # FinancialInstrument, Company, Market + legal = catalog.load("legal") # Contract, Party, Jurisdiction + supply_chain = catalog.load("supply_chain") # Supplier, Product, Shipment + + biomedical.add_class("ClinicalTrial", parent="Study", properties=["phase", "participants"]) + ``` + + Available domains: `biomedical`, `finance`, `legal`, `supply_chain`, `cybersecurity`, `e_commerce`, `hr`. + + + +## ModuleManager + +Build ontologies as composable modules — keep domain logic separated and reusable: ```python -from semantica.ontology import OntologyAligner +from semantica.ontology import ModuleManager -aligner = OntologyAligner() -alignment = aligner.align(source_ontology, target_ontology) +manager = ModuleManager() -for mapping in alignment.mappings: - print(f"{mapping.source} → {mapping.target} (confidence: {mapping.confidence:.2f})") +core_module = manager.create_module("core", base_uri="http://example.org/core#") +finance_module = manager.create_module("finance", base_uri="http://example.org/finance#") -# Merge into a unified ontology -merged = aligner.merge(source_ontology, target_ontology, alignment) +core_module.add_class("Entity", properties=["id", "name"]) +finance_module.add_class("Company", parent="Entity", properties=["ticker", "revenue"]) + +finance_module.import_module(core_module) +unified = manager.merge_modules([core_module, finance_module]) ``` -## Diff and Migration +## NamespaceManager -Compare ontology versions and generate migration scripts for graph data: +Manage IRI prefixes and generate consistent URIs for all ontology terms: ```python -from semantica.ontology import OntologyDiff, OntologyMigrator +from semantica.ontology import NamespaceManager -diff = OntologyDiff() -changes = diff.compare(ontology_v1, ontology_v2) +ns_manager = NamespaceManager(base_uri="http://example.org/") +ns_manager.register("ex", "http://example.org/") +ns_manager.register("schema", "https://schema.org/") -for change in changes: - print(f"{change.type}: {change.element} — {change.description}") - -# Generate and apply a migration script -migrator = OntologyMigrator() -migration_script = migrator.generate_migration(changes) -migrator.apply(kg, migration_script) +class_iri = ns_manager.generate_iri("Person") # → "http://example.org/Person" +prop_iri = ns_manager.generate_iri("worksFor", prefix="ex") # → "http://example.org/worksFor" +iri = ns_manager.resolve("schema:Organization") # → "https://schema.org/Organization" ``` -## OWL / RDF Export +## OntologyEvaluator + +Measure quality across coverage, completeness, and competency questions: ```python -from semantica.ontology import OWLExporter +from semantica.ontology import OntologyEvaluator -exporter = OWLExporter() -exporter.export(ontology, path="ontology.ttl", format="turtle") -exporter.export(ontology, path="ontology.owl", format="xml") -exporter.export(ontology, path="ontology.json", format="json-ld") +evaluator = OntologyEvaluator() +report = evaluator.evaluate(ontology, kg) +print(f"Coverage: {report.coverage:.2%}") +print(f"Completeness: {report.completeness:.2%}") +print(f"Granularity: {report.granularity:.2%}") + +questions = ["What organizations were founded in California?", "Who are the employees of Apple Inc.?"] +cq_results = evaluator.validate_competency_questions(ontology, kg, questions) +for q, result in zip(questions, cq_results): + print(f"Q: {q} → Answerable: {result.answerable} ({result.reason})") ``` ## Ontology Hub (v0.5.0) @@ -158,13 +364,37 @@ start_explorer(graph=kg, port=8080) # Navigate to http://localhost:8080 → Ontology Hub tab ``` -Features: +Features: visual editor, SHACL Studio, alignment authoring, health dashboard, and version control. -- **Visual editor** — create and edit classes, properties, and relationships in the browser -- **SHACL Studio** — author and validate SHACL shapes with live feedback -- **Alignment authoring** — map concepts across ontologies with drag-and-drop -- **Health dashboard** — coverage, completeness, and constraint violation metrics -- **Version control** — snapshot, diff, and restore ontology versions +## Tips and Common Pitfalls + + + **Define competency questions before generating.** An ontology without competency questions has no measurable success criteria. Write 5–10 natural language questions your ontology must answer before calling `OntologyGenerator`. Then validate them with `OntologyEvaluator.validate_competency_questions()`. + + + + **Reuse before generating.** `DomainOntologies` and `ReuseManager` give you schema.org, FOAF, and domain-specific ontologies that took years to develop. Reusing established classes (`schema:Organization`, `foaf:Person`) also improves interoperability with external data. + + + + **`LLMOntologyGenerator` is great for prototyping, not production.** LLM-generated ontologies are a useful starting point but need expert review. Use `OntologyEvaluator` and manual SHACL authoring to harden the schema before relying on it for production graph validation. + + + + **Always validate with SHACL after schema changes.** When you add new classes or properties, run `SHACLValidator.validate(kg, shapes)` immediately. SHACL violations often surface data quality issues that were silently passing before. + + + + **Use `ModuleManager` for large ontologies.** A monolithic 500-class ontology becomes unmanageable quickly. Split by domain (`core`, `finance`, `legal`) and use `owl:imports` to compose them — changes in one module don't break others. + + + + **Namespace your terms consistently.** All classes and properties need stable IRIs. Use `NamespaceManager` to generate them programmatically — never hardcode IRI strings in code, because base URIs change when projects move. + + + + **Diff before migration.** Always run `OntologyDiff.compare()` before `OntologyMigrator.apply()`. The diff shows exactly which graph entities will be affected — some migrations (renaming a class) require updating thousands of existing nodes. + diff --git a/docs/reference/parse.md b/docs/reference/parse.md index 79a124df..7f8b84a5 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -8,75 +8,229 @@ icon: "file-lines" ## What You Get -- **`DocumentParser`** — standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX -- **`DoclingParser`** — advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR -- **`ParsedDocument`** — structured output with `text`, `sections`, `tables`, and `metadata` + + + Standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — zero config, no extras. + + + Advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR. + + + AST structure extraction — functions, classes, imports, dependencies — for 10+ languages. + + + EXIF metadata extraction and OCR via Tesseract for image files. + + + Technical metadata from audio, video, and image files (duration, codec, resolution). + + + Parse Model Context Protocol responses into structured `ParsedDocument` objects. + + -## DocumentParser +## Quick Start -Standard parser for clean, machine-readable documents: + + + ```python + from semantica.parse import DocumentParser -```python -from semantica.parse import DocumentParser + parser = DocumentParser() + parsed = parser.parse("data/report.pdf") -parser = DocumentParser() -parsed = parser.parse("data/report.pdf") + print(parsed.text) # full clean text + print(parsed.metadata) # title, author, date, page_count, language, etc. + print(parsed.sections) # document structure as a list of Section objects + ``` + + + ```bash + pip install "semantica[docling]" + ``` -print(parsed.text) # full clean text -print(parsed.metadata) # title, author, date, page_count, language, etc. -print(parsed.sections) # document structure as a list of Section objects -``` + ```python + from semantica.parse import DoclingParser -Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. + parser = DoclingParser( + extract_tables=True, # structured table extraction with cell type detection + extract_images=True, # extract image regions for downstream OCR + output_format="markdown", # "markdown" | "html" | "json" + ) -## DoclingParser + parsed = parser.parse("data/annual_report.pdf") + print(parsed.tables) # structured TableData objects with headers and rows + ``` + + + ```python + from semantica.split import TextSplitter + from semantica.semantic_extract import NERExtractor + from semantica.llms import Groq + import os -Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: + splitter = TextSplitter(method="structural") + chunks = splitter.split_document(parsed) -```bash -pip install "semantica[docling]" -``` + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + extractor = NERExtractor(method="llm", llm_provider=llm) + entities = extractor.extract_batch([c.text for c in chunks]) + ``` + + -```python -from semantica.parse import DoclingParser +## Parser Reference -parser = DoclingParser( - extract_tables=True, # structured table extraction with cell type detection - extract_images=True, # extract image regions for downstream OCR - output_format="markdown", # "markdown" | "html" | "json" -) + + + Standard parser for clean, machine-readable documents — no extra dependencies required: -parsed = parser.parse("data/annual_report.pdf") + ```python + from semantica.parse import DocumentParser -print(parsed.text) # full clean text -print(parsed.tables) # structured TableData objects with headers and rows -print(parsed.sections) # document structure with heading hierarchy -``` + parser = DocumentParser() + parsed = parser.parse("data/report.pdf") -Use `DoclingParser` for: + print(parsed.text) # full clean text + print(parsed.metadata) # title, author, date, page_count, language, etc. + print(parsed.sections) # document structure as a list of Section objects + ``` -- Multi-column PDF layouts -- Tables with merged cells or complex headers -- PPTX slides with embedded charts -- XLSX spreadsheets with formulas -- Scanned documents with OCR -- Academic papers and technical reports + **Supported formats:** PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. + + + Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: -## OCR Support + ```python + from semantica.parse import DoclingParser -```python -parser = DoclingParser( - ocr=True, - ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents - extract_tables=True, -) + parser = DoclingParser( + extract_tables=True, # structured table extraction with cell type detection + extract_images=True, # extract image regions for downstream OCR + output_format="markdown", # "markdown" | "html" | "json" + ) -parsed = parser.parse("data/scanned_contract.pdf") -``` + parsed = parser.parse("data/annual_report.pdf") + print(parsed.tables) # structured TableData objects with headers and rows + print(parsed.sections) # document structure with heading hierarchy + ``` -## Parsed Document Object + **Use `DoclingParser` for:** + - Multi-column PDF layouts + - Tables with merged cells or complex headers + - PPTX slides with embedded charts + - XLSX spreadsheets with formulas + - Scanned documents with OCR + - Academic papers and technical reports -Both parsers return a `ParsedDocument` with the same structure: + **OCR support:** + + ```python + parser = DoclingParser( + ocr=True, + ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents + extract_tables=True, + ) + parsed = parser.parse("data/scanned_contract.pdf") + ``` + + + Parse source code files — extracts AST structure, functions, classes, imports, and comments: + + ```python + from semantica.parse import CodeParser + + parser = CodeParser( + extract_comments=True, # include docstrings and inline comments + extract_dependencies=True, # import/require statements + language="auto", # "auto" | "python" | "javascript" | "java" | "go" | "rust" | "cpp" + ) + + parsed = parser.parse("src/main.py") + + print(parsed.text) # raw source code as text + print(parsed.metadata["language"]) # detected language + print(parsed.metadata["functions"]) # list of function names + print(parsed.metadata["classes"]) # list of class names + print(parsed.metadata["imports"]) # list of import statements + print(parsed.metadata["comments"]) # docstrings and inline comments + ``` + + **Supported languages:** Python, JavaScript/TypeScript, Java, Go, Rust, C/C++, C#, Ruby, PHP, Swift. + + + Extract EXIF metadata and optionally perform OCR on image files: + + ```python + from semantica.parse import ImageParser + + parser = ImageParser( + extract_exif=True, # camera, GPS, timestamps, etc. + ocr=True, # OCR via Tesseract (requires tesseract-ocr installed) + ocr_language="en", # ISO 639-1 language code for OCR + ) + + parsed = parser.parse("photo.jpg") + + print(parsed.text) # OCR-extracted text (if ocr=True) + print(parsed.metadata["width"]) # image dimensions + print(parsed.metadata["height"]) + print(parsed.metadata["format"]) # "JPEG" | "PNG" | "TIFF" | ... + print(parsed.metadata["exif"]["GPS"]) # GPS coordinates if available + print(parsed.metadata["exif"]["DateTime"]) + ``` + + + ### MediaParser + + Extract technical metadata from audio, video, and image files: + + ```python + from semantica.parse import MediaParser + + parser = MediaParser() + + # Video file + parsed = parser.parse("interview.mp4") + print(parsed.metadata["duration_seconds"]) + print(parsed.metadata["codec"]) + print(parsed.metadata["resolution"]) + print(parsed.metadata["fps"]) + + # Audio file + parsed = parser.parse("podcast.mp3") + print(parsed.metadata["duration_seconds"]) + print(parsed.metadata["bitrate"]) + print(parsed.metadata["channels"]) + ``` + + **Supported formats:** MP4, AVI, MOV, MKV, MP3, WAV, FLAC, OGG, JPEG, PNG, TIFF, WebP. + + ### MCPParser + + Parse Model Context Protocol (MCP) responses into structured `ParsedDocument` objects: + + ```python + from semantica.parse import MCPParser + + parser = MCPParser() + + mcp_response = { + "content": [{"type": "text", "text": "Apple Inc. was founded in 1976..."}], + "metadata": {"tool": "web_search", "query": "Apple Inc history"} + } + + parsed = parser.parse(mcp_response) + print(parsed.text) # "Apple Inc. was founded in 1976..." + print(parsed.metadata) # tool name, query, and other MCP metadata + ``` + + + +## Parsed Document Schema + + + ```python @dataclass @@ -86,7 +240,12 @@ class ParsedDocument: tables: List[TableData] # structured table data (DoclingParser only) metadata: DocumentMetadata # title, author, dates, page count source_id: str # links back to the original DataSource +``` + + + +```python @dataclass class DocumentMetadata: title: Optional[str] @@ -100,6 +259,21 @@ class DocumentMetadata: format: str # "pdf" | "docx" | "pptx" | ... ``` + + + +## Choosing a Parser + +| Scenario | Parser | +| -------- | ------ | +| Clean PDFs, DOCX, HTML, TXT, CSV, Excel | `DocumentParser` — zero config, no extras | +| Scanned PDFs, OCR required | `DoclingParser(ocr=True)` — requires `pip install "semantica[docling]"` | +| Multi-column PDFs, merged-cell tables | `DoclingParser(extract_tables=True)` | +| Source code files | `CodeParser(language="auto")` | +| Images with embedded text | `ImageParser(ocr=True)` — requires Tesseract | +| Audio/video metadata | `MediaParser()` | +| MCP tool responses | `MCPParser()` | + ## Integration with FileIngestor The most common pattern — ingest a directory then parse each source: @@ -121,6 +295,28 @@ for source in sources: Docling is an optional dependency. If `docling` is not installed, `DoclingParser` raises an `ImportError` with installation instructions. `DocumentParser` is always available and requires no extras. +## Tips and Common Pitfalls + + + **Start with `DocumentParser` and only switch to `DoclingParser` when needed.** `DoclingParser` is significantly more powerful but slower and requires an additional dependency. For clean machine-readable PDFs and Office files, `DocumentParser` is fast and accurate enough. + + + + **OCR requires Tesseract installed on the system.** `ImageParser(ocr=True)` and `DoclingParser(ocr=True)` both call Tesseract under the hood. Install it with `apt-get install tesseract-ocr` (Linux) or `brew install tesseract` (macOS) before enabling OCR. + + + + **`extract_tables=True` is off by default for speed.** Table extraction in `DoclingParser` requires additional layout analysis passes. Only enable it when you actually need structured table data — for text-only extraction, leave it off. + + + + **`CodeParser` outputs AST metadata, not just raw text.** The `parsed.metadata["functions"]` and `parsed.metadata["classes"]` lists are useful for building code-level knowledge graphs — function call graphs, class inheritance hierarchies, dependency graphs. + + + + **Always pass the `ParsedDocument` to `TextSplitter` before extraction.** Raw `parsed.text` is a flat string. Use `TextSplitter` to chunk it into semantically meaningful pieces before running NER — this dramatically reduces context window overflow on large documents. + + Load files before parsing. diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index 32723a7f..a1a3c90c 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -6,34 +6,82 @@ icon: "gear" `semantica.pipeline` lets you chain Semantica components into reproducible, fault-tolerant workflows with parallel execution and configurable error handling. Pipelines are serializable — save them to YAML and reload in any environment. -## What You Get +## Why Use a Pipeline? -- **`Pipeline`** — chain steps with parallel workers, retry policies, and failure handlers -- **`PipelineBuilder`** — fluent DSL for building pipelines with a readable chain syntax -- **`RetryPolicy`** — fixed, linear, and exponential backoff with configurable max retries -- **`FailureHandler`** — skip, stop, or retry failed documents without halting the pipeline -- **Progress tracking** — console (tqdm), WebSocket streaming, or file logging +You could wire Semantica modules together with plain Python code. Pipelines add: + + + + A single bad document doesn't crash a 10,000-document run. + + + Run extraction across multiple workers with one parameter. + + + tqdm console bar or WebSocket streaming to Explorer. + + + Save the exact pipeline configuration to YAML and replay on any machine. + + + On re-runs, only process documents that changed since the last run. + + + Catch misconfigured steps and dependency cycles before they fail mid-run. + + + + + Use plain module calls for quick scripts and notebooks. Use pipelines for anything you run repeatedly, at scale, or in production. + Pipeline step sequence: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver -## Basic Pipeline +## Quick Start -```python -from semantica.pipeline import Pipeline -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor -from semantica.kg import GraphBuilder + + + ```python + from semantica.pipeline import Pipeline + from semantica.ingest import FileIngestor + from semantica.parse import DocumentParser + from semantica.semantic_extract import NERExtractor + from semantica.kg import GraphBuilder + from semantica.llms import Groq + import os -pipeline = Pipeline() -pipeline.add_step("ingest", FileIngestor()) -pipeline.add_step("parse", DocumentParser()) -pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) -pipeline.add_step("build_kg", GraphBuilder(merge_entities=True)) + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -result = pipeline.run("data/") -kg = result.output -``` + pipeline = Pipeline() + pipeline.add_step("ingest", FileIngestor()) + pipeline.add_step("parse", DocumentParser()) + pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) + pipeline.add_step("build_kg", GraphBuilder(merge_entities=True)) + ``` + + + ```python + from semantica.pipeline import PipelineValidator + + validator = PipelineValidator() + result = validator.validate_pipeline(pipeline) + + if not result.is_valid: + for error in result.errors: + print(f"Error: {error.message} (step: {error.step})") + ``` + + + ```python + result = pipeline.run("data/", show_progress=True) + + kg = result.output + print(f"Processed: {result.processed_count}") + print(f"Failed: {result.failed_count}") + print(f"Duration: {result.duration_seconds:.1f}s") + ``` + + ## Parallel Processing @@ -52,43 +100,82 @@ result = pipeline.run("data/") ## Retry and Error Handling -Configure retry behavior and failure strategy independently: + + + ```python + from semantica.pipeline import RetryPolicy, FailureHandler, Pipeline -```python -from semantica.pipeline import Pipeline, RetryPolicy, FailureHandler + retry = RetryPolicy( + max_retries=3, + backoff="exponential", + initial_delay=1.0 # 1s → 2s → 4s + ) -retry = RetryPolicy( - max_retries=3, - backoff="exponential", # "fixed" | "linear" | "exponential" - initial_delay=1.0 # seconds before first retry -) + handler = FailureHandler(strategy="skip", log_failures=True) -handler = FailureHandler( - strategy="skip", # "skip" | "stop" | "retry" - log_failures=True # write failed documents to error log -) + pipeline = Pipeline(retry_policy=retry, failure_handler=handler) + ``` -pipeline = Pipeline(retry_policy=retry, failure_handler=handler) -``` + Best for transient API errors and rate limits — waits longer with each retry, giving upstream services time to recover. + + + ```python + retry = RetryPolicy( + max_retries=3, + backoff="linear", + initial_delay=2.0 # 2s → 4s → 6s + ) + ``` + + Use when the delay between retries should grow predictably — e.g., waiting for a database lock to release. + + + ```python + retry = RetryPolicy( + max_retries=5, + backoff="fixed", + initial_delay=1.0 # 1s → 1s → 1s → 1s → 1s + ) + ``` + + Use when retrying against a service with a fixed cooldown window. + + + +### Failure Strategies + +| Strategy | Behaviour | When to Use | +| -------- | --------- | ----------- | +| `"skip"` | Log failure, continue to next document | Production — one bad doc shouldn't stop 10k | +| `"stop"` | Raise exception immediately | Development — surface errors fast | +| `"retry"` | Retry via `RetryPolicy`, then skip | When failures are likely transient | + + + Always use `strategy="skip"` in production. A single malformed document shouldn't stop a pipeline processing thousands of documents. Inspect `result.errors` after the run to find and reprocess failures. + ## Progress Tracking -```python -# Console progress bar (tqdm) -result = pipeline.run("data/", show_progress=True) + + + ```python + result = pipeline.run("data/", show_progress=True) + ``` -# WebSocket progress — stream to Knowledge Explorer -result = pipeline.run("data/", websocket_port=8080) + Displays a live tqdm progress bar in the terminal. Best for scripts and CLI tools. + + + ```python + result = pipeline.run("data/", websocket_port=8080) + ``` -# Inspect results -print(f"Processed: {result.processed_count}") -print(f"Failed: {result.failed_count}") -print(f"Duration: {result.duration_seconds:.1f}s") -``` + Streams progress events to Knowledge Explorer's dashboard. Best for long-running production jobs where you want a live web UI. + + ## Pipeline DSL -The `PipelineBuilder` provides a fluent chain syntax that reads as a data flow: +`PipelineBuilder` provides a fluent chain syntax that reads as a data flow: ```python from semantica.pipeline import PipelineBuilder @@ -122,7 +209,206 @@ pipeline = Pipeline.load("pipeline_config.yaml") result = pipeline.run("data/") ``` -## Pipeline Result + + `pipeline.save()` preserves exact component configurations — LLM model names, retry policies, thresholds — everything. Without it, you can't guarantee that a re-run 3 months later uses the same settings. + + +## Pre-Built Templates + +`PipelineTemplateManager` wires common workflows with the correct step order — no manual wiring required: + +```python +from semantica.pipeline import PipelineTemplateManager +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +manager = PipelineTemplateManager() +``` + + + + **Ingest → Parse → Extract → Build KG** + + Standard knowledge base construction from documents. + + ```python + pipeline = manager.get_template( + "ingest-extract-build", llm_provider=llm + ) + ``` + + + **Ingest → Parse → Embed → Index** + + Retrieval-augmented generation — builds a vector-indexed knowledge graph. + + ```python + pipeline = manager.get_template( + "graphrag", llm_provider=llm, vector_backend="faiss" + ) + ``` + + + **Build KG → Analytics → Export Report** + + Graph analysis and reporting — centrality, community detection, HTML output. + + ```python + pipeline = manager.get_template( + "analytics", export_format="html" + ) + ``` + + + **Ingest → Normalize → Extract → Dedup → Conflicts → Build** + + Production-quality KG with full data quality pipeline. + + ```python + pipeline = manager.get_template( + "full-qa", llm_provider=llm + ) + ``` + + + +## ExecutionEngine + +Fine-grained control over pipeline execution — pause, resume, cancel, and inspect live progress: + +```python +from semantica.pipeline import ExecutionEngine + +engine = ExecutionEngine(config={"timeout_seconds": 300, "max_workers": 4}) + +result = engine.execute_pipeline(pipeline, data="data/") + +# Pause after the current step finishes +engine.pause_pipeline(result.pipeline_id) + +progress = engine.get_progress(result.pipeline_id) +print(f"Completed: {progress['completed']}/{progress['total']}") +print(f"Current step: {progress['current_step']}") + +engine.resume_pipeline(result.pipeline_id) +engine.stop_pipeline(result.pipeline_id) +``` + +| Method | Returns | Description | +| ------ | ------- | ----------- | +| `execute_pipeline(pipeline, data)` | `ExecutionResult` | Execute pipeline from start to finish | +| `get_status(pipeline_id)` | `PipelineStatus` | Current state (RUNNING, PAUSED, STOPPED) | +| `get_progress(pipeline_id)` | `Dict` | Step completion counts and elapsed time | +| `pause_pipeline(pipeline_id)` | `None` | Suspend after current step completes | +| `resume_pipeline(pipeline_id)` | `None` | Resume from paused state | +| `stop_pipeline(pipeline_id)` | `None` | Cancel and clean up immediately | + +## PipelineValidator + +Catches problems before they surface as mid-run failures: + +```python +from semantica.pipeline import PipelineValidator + +validator = PipelineValidator() +result = validator.validate_pipeline(pipeline) + +if result.is_valid: + print("Pipeline is valid — safe to run") +else: + for error in result.errors: + print(f"Error: {error.message} (step: {error.step})") + for warning in result.warnings: + print(f"Warning: {warning}") +``` + +Checks performed: +- **Dependency cycle detection** — A depends on B, B depends on A +- **Step type validation** — each step type must be registered +- **Connection integrity** — referenced step names must exist +- **Configuration completeness** — required parameters must be present + +## ParallelismManager + + + + ```python + from semantica.pipeline import ParallelismManager + + manager = ParallelismManager(max_workers=8, pool_type="thread") + + tasks = [{"fn": ner.extract, "args": [text]} for text in texts] + result = manager.execute_parallel(tasks, timeout=60) + + print(f"Successful: {result.success_count}, Failed: {result.failure_count}") + ``` + + Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls. Threads share memory and context-switch cheaply between waiting operations. + + + ```python + manager = ParallelismManager(max_workers=4, pool_type="process") + + tasks = [{"fn": embedder.embed, "args": [chunk]} for chunk in chunks] + result = manager.execute_parallel(tasks, timeout=120) + ``` + + Use process pools for **CPU-bound** steps: embedding computation, OCR, large NER batches. Processes bypass Python's GIL for true multi-core parallelism. + + + +## ResourceScheduler + +Prevents memory oversubscription on large runs: + +```python +from semantica.pipeline import ResourceScheduler + +scheduler = ResourceScheduler() + +resources = scheduler.allocate_resources( + pipeline, max_memory_gb=8, max_workers=4 +) + +try: + result = pipeline.run("data/") +finally: + scheduler.release_resources(resources) +``` + +## Delta Mode + +Re-process only data that has changed since the last run: + +```python +pipeline = Pipeline() +pipeline.add_step( + "ingest", FileIngestor(), + delta_mode=True, base_version_id="v1", target_version_id="v2" +) +pipeline.add_step( + "extract", NERExtractor(), + delta_mode=True, base_version_id="v1", target_version_id="v2" +) +pipeline.add_step( + "build", GraphBuilder(), + delta_mode=False # always rebuild the merged graph +) + +result = pipeline.run("data/") +print(f"Delta documents processed: {result.metadata.get('delta_count', 0)}") +print(f"Skipped (unchanged): {result.metadata.get('skipped_count', 0)}") +``` + + + Delta detection uses SHA-256 checksums on source content. Only sources whose checksum differs from `base_version_id` are passed to downstream steps. For pipelines that run hourly or daily against a growing corpus, delta mode eliminates redundant re-embedding and re-extraction. + + +## Schemas + + + ```python @dataclass @@ -133,8 +419,66 @@ class PipelineResult: duration_seconds: float # total wall-clock time step_metrics: Dict # per-step timing and counts errors: List # list of FailedDocument records + metadata: Dict # pipeline-level metadata (delta_count, etc.) ``` + + + +```python +@dataclass +class PipelineStep: + name: str + step_type: str + config: Dict[str, Any] + dependencies: List[str] # names of steps this step waits for + handler: Optional[Callable] + status: StepStatus + result: Any + error: Optional[Exception] + delta_mode: bool # True = process only changed data + base_version_id: Optional[str] # snapshot ID to diff against + target_version_id: Optional[str] # snapshot ID being produced +``` + + + + +```python +from semantica.pipeline import StepStatus + +StepStatus.PENDING # Not yet started +StepStatus.RUNNING # Currently executing +StepStatus.COMPLETED # Finished successfully +StepStatus.FAILED # Error occurred — check step.error +StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy +``` + + + + +## Tips and Common Pitfalls + + + **Use `PipelineValidator` before running in production.** It catches dependency cycles, missing step names, and misconfigured connections that would only surface as errors mid-run. Validation is instant; catching them after a 30-minute extraction job is not. + + + + **Set `workers=` based on workload type.** Thread workers for I/O-bound steps (web fetching, DB queries), process workers for CPU-bound steps (embedding, OCR, large NER batches). Mixing pool types on the wrong step type wastes resources without speed gains. + + + + **Use `failure_handler=FailureHandler(strategy="skip")` in production.** A single malformed document shouldn't stop a pipeline processing 10,000 documents. `skip` logs the failure and continues; inspect `result.errors` after the run to find and reprocess failed documents. + + + + **Use templates from `PipelineTemplateManager` for common patterns.** `get_template("full-qa")` wires up normalization, deduplication, conflict detection, and graph construction in the right order — saving you from common mistakes like deduplicating before normalizing. + + + + **Inspect `result.step_metrics` to find bottlenecks.** Each step reports its own duration and document count. If embedding is 10x slower than NER, that's where to optimize — increase `batch_size`, switch to a faster embedding model, or parallelize with GPU. + + First step in most pipelines. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index 84513589..bd782c3d 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -8,11 +8,75 @@ icon: "link" ## What You Get -- **`ProvenanceManager`** — track entities, relationships, and activities with source attribution -- **`ActivityTracker`** — record pipeline activities and which entities they produced or consumed -- **Lineage graph** — full upstream lineage from any entity back to its source document -- **W3C PROV-O export** — serialize lineage as Turtle RDF for compliance reporting -- **`GraphBuilderWithProvenance`** — drop-in replacement that auto-tracks every node and edge + + + Track entities, relationships, and activities with full source attribution and confidence scores. + + + Record pipeline activities and which entities they produced or consumed. + + + Full directed lineage from any entity back to its originating source document. + + + Serialize lineage as Turtle RDF or JSON-LD for compliance reporting. + + + Drop-in replacement for GraphBuilder that auto-tracks every node and edge. + + + SHA-256 checksums to detect tampering in HIPAA and FDA 21 CFR Part 11 environments. + + + +## Quick Start + + + + ```python + from semantica.provenance import ProvenanceManager, SQLiteStorage + + # SQLite — persistent across process restarts (recommended for production) + manager = ProvenanceManager( + storage=SQLiteStorage(db_path="provenance.db") + ) + ``` + + + ```python + manager.track_entity( + entity_id="apple_inc", + source="annual_report_2023.pdf", + entity_type="Organization", + extraction_method="llm", + confidence=0.98, + ) + + manager.track_relationship( + rel_id="steve_jobs_founded_apple", + source="annual_report_2023.pdf", + extraction_method="llm", + confidence=0.92, + ) + ``` + + + ```python + lineage = manager.get_lineage("apple_inc") + print(f"Source: {lineage.source}") + print(f"Extracted: {lineage.extracted_at}") + print(f"Method: {lineage.extraction_method}") + print(f"Confidence: {lineage.confidence}") + ``` + + + ```python + # Full provenance graph for all tracked entities + manager.export_all(path="provenance.ttl", format="turtle") + manager.export_all(path="provenance.jsonld", format="json-ld") + ``` + + ## ProvenanceManager @@ -27,7 +91,7 @@ manager.track_entity( source="annual_report_2023.pdf", entity_type="Organization", extraction_method="llm", - confidence=0.98 + confidence=0.98, ) # Track an extracted relationship @@ -35,7 +99,7 @@ manager.track_relationship( rel_id="steve_jobs_founded_apple", source="annual_report_2023.pdf", extraction_method="llm", - confidence=0.92 + confidence=0.92, ) # Retrieve full lineage for any entity @@ -55,7 +119,7 @@ Record pipeline activities — what was consumed and what was produced: activity_id = manager.start_activity( activity_type="ner_extraction", used=["annual_report_2023.pdf"], - generated=["apple_inc", "steve_jobs"] + generated=["apple_inc", "steve_jobs"], ) manager.end_activity(activity_id) @@ -82,24 +146,52 @@ for edge in lineage_graph.edges: print(f"{edge.source} → {edge.target} ({edge.relation})") ``` -## W3C PROV-O Export +## Storage Backends -Export lineage as W3C PROV-O Turtle for compliance reporting: + + + Fast, no persistence — default backend. Data is lost on process exit. -```python -# Single entity lineage -prov_ttl = manager.export_prov_o("apple_inc", format="turtle") + ```python + from semantica.provenance import ProvenanceManager, InMemoryStorage -# Full provenance graph for all tracked entities -manager.export_all(path="provenance.ttl", format="turtle") + manager = ProvenanceManager(storage=InMemoryStorage()) -# Compliance-ready JSON-LD export -manager.export_all(path="provenance.jsonld", format="json-ld") -``` + manager.track_entity("apple_inc", source="report.pdf", confidence=0.98) + lineage = manager.get_lineage("apple_inc") + ``` + + Best for: development, unit tests, short-lived pipelines. + + + Persistent file-based storage — survives process restarts. The API is identical to `InMemoryStorage`. + + ```python + from semantica.provenance import ProvenanceManager, SQLiteStorage + + manager = ProvenanceManager( + storage=SQLiteStorage(db_path="provenance.db") + ) + + manager.track_entity("apple_inc", source="report.pdf", confidence=0.98) + lineage = manager.get_lineage("apple_inc") + ``` + + Best for: production single-machine deployments, compliance environments. + + + + | Storage | Persistence | Best For | + | ------- | ----------- | -------- | + | `InMemoryStorage` | No | Development, unit tests, short-lived pipelines | + | `SQLiteStorage` | Yes (file) | Production single-machine deployments | + + + ## Integration with GraphBuilder -`GraphBuilderWithProvenance` automatically records provenance for every node and edge constructed: +`GraphBuilderWithProvenance` automatically records provenance for every node and edge constructed — no manual `track_entity()` calls needed: ```python from semantica.kg import GraphBuilderWithProvenance @@ -107,15 +199,92 @@ from semantica.kg import GraphBuilderWithProvenance builder = GraphBuilderWithProvenance(provenance=True) result = builder.build_single_source(graph_data) -# Each node and edge has a source_id linking back to the originating document +# Every node and edge has a source_id linking back to the originating document lineage = result.provenance_manager.get_lineage("apple_inc") print(f"Source document: {lineage.source}") print(f"Extracted by: {lineage.extraction_method}") ``` -## Compliance Standards +## Integrity Verification -Provenance tracking in Semantica is designed to satisfy: +Compute and verify checksums for provenance entries to detect tampering: + +```python +from semantica.provenance import compute_checksum, verify_checksum + +# Compute a SHA-256 checksum over an entity's provenance record +entry = manager.get_provenance_entry("apple_inc") +checksum = compute_checksum(entry, algorithm="sha256") +print(f"Checksum: {checksum}") + +# Later — verify the record has not been modified +is_valid = verify_checksum(entry, expected_checksum=checksum, algorithm="sha256") +if not is_valid: + raise RuntimeError("Provenance record has been tampered with!") +``` + +Supported algorithms: `"sha256"` (default), `"sha512"`, `"md5"`. + +## W3C PROV-O Export + +```python +# Single entity lineage +prov_ttl = manager.export_prov_o("apple_inc", format="turtle") + +# Full provenance graph for all tracked entities +manager.export_all(path="provenance.ttl", format="turtle") +manager.export_all(path="provenance.jsonld", format="json-ld") +``` + +## Schemas + + + + +```python +@dataclass +class ProvenanceEntry: + entity_id: str + source: str # source document or system + source_location: str # e.g. "page 3, paragraph 2" + source_quote: str # verbatim text from source + extraction_method: str # "llm" | "ml" | "pattern" + confidence: float # extraction confidence 0–1 + timestamp: datetime # when this entry was recorded + entity_type: Optional[str] +``` + + + + +```python +@dataclass +class SourceReference: + source_id: str + doi: Optional[str] # academic DOI if available + page: Optional[int] # page number in document + paragraph: Optional[int] + quote: str # verbatim text supporting the fact + url: Optional[str] + accessed_at: Optional[datetime] +``` + + + + +## W3C PROV-O Mapping + +| Semantica Concept | PROV-O Class / Property | +| ----------------- | ----------------------- | +| Entity (node/fact) | `prov:Entity` | +| Extraction activity | `prov:Activity` | +| Source document | `prov:Entity` | +| `track_entity()` | `prov:wasDerivedFrom` | +| `start_activity()` | `prov:wasGeneratedBy` | +| Extraction method | `prov:wasAssociatedWith` | +| Timestamp | `prov:startedAtTime`, `prov:endedAtTime` | + +## Compliance Standards | Standard | Requirement Met | | -------- | --------------- | @@ -125,6 +294,32 @@ Provenance tracking in Semantica is designed to satisfy: | **GDPR** | Data lineage supporting right-to-erasure impact analysis | | **FDA 21 CFR Part 11** | Electronic records with origination timestamp and extraction method | +## Tips and Common Pitfalls + + + **Use `SQLiteStorage` in production, not `InMemoryStorage`.** The in-memory backend is the default for backwards compatibility, but provenance data is lost on process exit. Switch to `SQLiteStorage(db_path="provenance.db")` before going to production — migrating later means losing all historical lineage. + + + + **Track provenance at ingestion time, not after.** The `source_location` and `source_quote` fields become unavailable once you've moved past the parsing stage. Capture them during ingestion and pass them to `track_entity()` immediately. + + + + **Use `GraphBuilderWithProvenance` instead of plain `GraphBuilder`.** It auto-tracks every node and edge without manual `track_entity()` calls — ensuring nothing is accidentally omitted from the provenance record. + + + + **Verify checksums on high-stakes data.** `compute_checksum()` + `verify_checksum()` detects any modification to a provenance entry since it was recorded — critical for HIPAA and FDA 21 CFR Part 11 environments where tampered records carry legal liability. + + + + **Export PROV-O Turtle for external auditors.** Compliance teams and external auditors often need machine-readable lineage in a standard format. `manager.export_all("provenance.ttl", format="turtle")` produces W3C PROV-O Turtle that any RDF tool can parse. + + + + **Use `get_audit_trail(entity_id=...)` for GDPR subject-access requests.** The GDPR right-of-access requires you to show what personal data you hold and where it came from. Scoped lineage per entity ID makes this a one-line export. + + Version control and snapshot audit trails. diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 876e67b8..23df4e9e 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -6,196 +6,274 @@ icon: "microchip" `semantica.reasoning` derives new knowledge from existing facts using logical rules. Every engine produces **explainable inference paths** — traceable chains of rules and facts, not black-box conclusions. +## Why Reasoning? + +Knowledge graphs encode what you know explicitly. Reasoning lets you derive what must logically follow — without manually asserting every implication: + +- If A `located_in` B and B `located_in` C, then A `located_in` C — without storing that triple +- If Alice `parent_of` Bob and Bob `parent_of` Charlie, then Alice `ancestor_of` Charlie +- If a drug is contraindicated for a condition class, it's also contraindicated for all subclasses — inferred from the ontology hierarchy +- If an employee's CEO tenure ended in 2020, they cannot have signed contracts as CEO in 2021 — caught by temporal reasoning + +Reasoning turns sparse explicit knowledge into a dense, coherent, contradiction-free knowledge base. + ## What You Get -- **`Reasoner`** — main facade for IF/THEN forward-chaining with variable substitution -- **`GraphReasoner`** — inference over full knowledge graph structure (transitivity, symmetry, inverses) -- **`ReteEngine`** — high-performance pattern matching via the Rete algorithm for large rule sets -- **`SPARQLReasoner`** — query expansion and property chain inference over RDF graphs -- **`DatalogReasoner`** — recursive Horn clause rules with guaranteed fixpoint termination (v0.4.0) -- **`TemporalReasoningEngine`** — all 13 Allen interval algebra relations for time-aware inference -- **`ExplanationGenerator`** — structured explanation paths for every derived conclusion + + + Main facade — IF/THEN forward-chaining with variable substitution and rule templates. + + + Inference over full knowledge graph structure: transitivity, symmetry, inverses. + + + High-performance pattern matching via the Rete algorithm for large rule sets. + + + Query expansion and property chain inference over RDF graphs. + + + Recursive Horn clause rules with guaranteed fixpoint termination (v0.4.0). + + + All 13 Allen interval algebra relations for time-aware inference. + + Forward chaining inference: known facts + IF/THEN rules produce derived facts with a full traceable explanation path -## Reasoner (Main Facade) +## Choosing a Reasoning Engine -The unified entry point for rule-based forward-chaining inference: +| Engine | When to Use | +| ------ | ----------- | +| `Reasoner` | Simple IF/THEN rules, transitivity/symmetry templates, one-shot inference | +| `GraphReasoner` | Rules that operate on graph structure (paths, neighborhoods, multi-hop) | +| `ReteEngine` | Large rule sets (100+), rules fire repeatedly, performance is critical | +| `SPARQLReasoner` | Already using RDF/Turtle, need property chains, SPARQL ecosystem tools | +| `DatalogReasoner` | Recursive rules (ancestry, reachability), guaranteed termination required | +| `TemporalReasoningEngine` | Time-aware facts, interval relationships, historical validity | -```python -from semantica.reasoning import Reasoner, Rule, Fact, RuleType +## Engines -reasoner = Reasoner() + + + The unified entry point for rule-based forward-chaining inference. Start here for most use cases. -# Add base facts -reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Manager")) -reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Employee")) + ```python + from semantica.reasoning import Reasoner, Rule, Fact, RuleType -# Add an IF/THEN rule -reasoner.add_rule(Rule( - rule_type=RuleType.FORWARD_CHAIN, - conditions=[ - {"subject": "?x", "predicate": "is_a", "object": "Manager"} - ], - conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} -)) + reasoner = Reasoner() -# Run inference -result = reasoner.infer() -for inference in result.derived_facts: - print(f"{inference.subject} {inference.predicate} {inference.obj}") - print(f" Derived via: {inference.explanation}") -``` + # Add base facts + reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Manager")) + reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Employee")) -### Built-In Rule Templates + # Add an IF/THEN rule + reasoner.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[ + {"subject": "?x", "predicate": "is_a", "object": "Manager"} + ], + conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} + )) -```python -engine = Reasoner() + # Run inference — always call explicitly after adding facts/rules + result = reasoner.infer() + for inference in result.derived_facts: + print(f"{inference.subject} {inference.predicate} {inference.obj}") + print(f" Derived via: {inference.explanation}") + ``` -# Transitive closure: A→B, B→C ⟹ A→C -engine.apply_transitivity("located_in") + ### Built-In Rule Templates -# Symmetry: A knows B ⟹ B knows A -engine.apply_symmetry("knows") + No manual rule authoring required for the three most common patterns: -# Inverse: A parent_of B ⟹ B child_of A -engine.apply_inverse("parent_of", "child_of") -``` + ```python + engine = Reasoner() -## GraphReasoner + # Transitive closure: A→B, B→C ⟹ A→C + engine.apply_transitivity("located_in") -Inference over the full knowledge graph structure: + # Symmetry: A knows B ⟹ B knows A + engine.apply_symmetry("knows") -```python -from semantica.reasoning import GraphReasoner + # Inverse: A parent_of B ⟹ B child_of A + engine.apply_inverse("parent_of", "child_of") -graph_reasoner = GraphReasoner(kg) + result = engine.infer() + ``` -# Define a transitive ancestor rule -graph_reasoner.add_rule({ - "if": [ - {"subject": "?a", "predicate": "parent_of", "object": "?b"}, - {"subject": "?b", "predicate": "parent_of", "object": "?c"} - ], - "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} -}) + | Template | Parameters | Description | + | -------- | ---------- | ----------- | + | `apply_transitivity(predicate)` | `predicate: str` | Adds A→C rule for all A→B, B→C chains | + | `apply_symmetry(predicate)` | `predicate: str` | Adds B→A rule for every A→B fact | + | `apply_inverse(predicate, inverse)` | `predicate, inverse: str` | Adds inverse direction for every fact | -inferences = graph_reasoner.infer(kg) -for inf in inferences: - print(f"{inf['subject']} {inf['predicate']} {inf['object']}") -``` + + Always call `reasoner.infer()` after adding facts and rules. Adding them updates internal state but does **not** trigger inference automatically. + + -## ReteEngine + + Inference over the full knowledge graph structure — rules that operate on graph paths, neighborhoods, and multi-hop connections. -High-performance pattern matching using the Rete algorithm — far faster than naive forward chaining for large rule sets because it caches partial matches across iterations: + ```python + from semantica.reasoning import GraphReasoner -```python -from semantica.reasoning import ReteEngine + graph_reasoner = GraphReasoner(kg) -engine = ReteEngine() -engine.load_rules("rules/domain_rules.json") -results = engine.run(kg) + # Define a transitive ancestor rule + graph_reasoner.add_rule({ + "if": [ + {"subject": "?a", "predicate": "parent_of", "object": "?b"}, + {"subject": "?b", "predicate": "parent_of", "object": "?c"} + ], + "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} + }) -# Inspect the Rete network -root = engine.get_root() -alpha_nodes = engine.get_alpha_nodes() # single-condition filters -beta_nodes = engine.get_beta_nodes() # join nodes -``` + inferences = graph_reasoner.infer(kg) + for inf in inferences: + print(f"{inf['subject']} {inf['predicate']} {inf['object']}") + ``` + -Rule format (JSON): + + High-performance pattern matching using the Rete algorithm — far faster than naive forward chaining for large rule sets because it caches partial matches across iterations. -```json -{ - "rules": [ + ```python + from semantica.reasoning import ReteEngine + + engine = ReteEngine() + engine.load_rules("rules/domain_rules.json") + results = engine.run(kg) + + # Inspect the Rete network + root = engine.get_root() + alpha_nodes = engine.get_alpha_nodes() # single-condition filters + beta_nodes = engine.get_beta_nodes() # join nodes + ``` + + Rule file format (JSON): + + ```json { - "name": "manager_authority", - "conditions": [ - { "subject": "?x", "predicate": "role", "object": "Manager" } - ], - "action": { "subject": "?x", "predicate": "has_authority", "object": "true" } + "rules": [ + { + "name": "manager_authority", + "conditions": [ + { "subject": "?x", "predicate": "role", "object": "Manager" }, + { "subject": "?x", "predicate": "dept", "object": "?dept" } + ], + "action": { + "subject": "?x", + "predicate": "has_authority_over", + "object": "?dept" + }, + "priority": 10 + } + ] } - ] -} -``` + ``` -## SPARQLReasoner + | Field | Type | Description | + | ----- | ---- | ----------- | + | `name` | `str` | Unique rule identifier — appears in `ExplanationGenerator` output | + | `conditions` | `List[Dict]` | Pattern to match — use `?variable` for wildcards | + | `action` | `Dict` | Fact to derive when all conditions match | + | `priority` | `int` | Higher priority rules fire first | -Query-based inference over RDF graphs with property chain support: + + Use `ReteEngine` when you have more than ~20 rules or when rules can fire repeatedly. `Reasoner` re-evaluates all rules from scratch each cycle; `ReteEngine` caches partial matches and is orders of magnitude faster. + + -```python -from semantica.reasoning import SPARQLReasoner + + Query-based inference over RDF graphs with property chain support. Use this when you're already in the RDF/Turtle ecosystem. -reasoner = SPARQLReasoner(graph=rdf_graph) + ```python + from semantica.reasoning import SPARQLReasoner -result = reasoner.query(""" - PREFIX ex: - SELECT ?person ?company WHERE { - ?person ex:founded ?company . - ?company ex:located_in ex:SiliconValley . - } -""") + reasoner = SPARQLReasoner(graph=rdf_graph) -for row in result.bindings: - print(row["person"], row["company"]) + result = reasoner.query(""" + PREFIX ex: + SELECT ?person ?company WHERE { + ?person ex:founded ?company . + ?company ex:located_in ex:SiliconValley . + } + """) -# Property chain inference: A knows B, B colleague_of C ⟹ A knows C -reasoner.add_property_chain("knows", ["knows", "colleague_of"]) -inferences = reasoner.infer_property_chains() -``` + for row in result.bindings: + print(row["person"], row["company"]) -## DatalogReasoner (v0.4.0) + # Property chain inference: A knows B, B colleague_of C ⟹ A knows C + reasoner.add_property_chain("knows", ["knows", "colleague_of"]) + inferences = reasoner.infer_property_chains() + ``` + -Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops: + + Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops. -```python -from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule + + Added in **v0.4.0**. Use `DatalogReasoner` whenever your rules can create cycles — it's the only engine with a termination guarantee. + -datalog = DatalogReasoner() + ```python + from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule -# Base facts -datalog.add_fact(DatalogFact("parent", ("alice", "bob"))) -datalog.add_fact(DatalogFact("parent", ("bob", "charlie"))) + datalog = DatalogReasoner() -# Recursive rules (Horn clauses) -datalog.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) -datalog.add_rule(DatalogRule("ancestor(?X, ?Z) :- parent(?X, ?Y), ancestor(?Y, ?Z).")) + # Base facts + datalog.add_fact(DatalogFact("parent", ("alice", "bob"))) + datalog.add_fact(DatalogFact("parent", ("bob", "charlie"))) -# Evaluate to fixpoint -datalog.evaluate() + # Recursive rules (Horn clauses) + datalog.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) + datalog.add_rule(DatalogRule("ancestor(?X, ?Z) :- parent(?X, ?Y), ancestor(?Y, ?Z).")) -# Query -results = datalog.query("ancestor(alice, ?Z)") -# → [{"Z": "bob"}, {"Z": "charlie"}] -``` + # Evaluate to fixpoint + datalog.evaluate() -## TemporalReasoningEngine + # Query + results = datalog.query("ancestor(alice, ?Z)") + # → [{"Z": "bob"}, {"Z": "charlie"}] + ``` -Reason about time intervals using all 13 Allen interval algebra relations: + + `Reasoner` has **no cycle detection** — rules that create cycles (A derives B, B derives C, C re-derives A) will loop infinitely. Use `DatalogReasoner` whenever recursive rules are involved. + + -```python -from semantica.reasoning import TemporalReasoningEngine, TemporalInterval, IntervalRelation + + Reason about time intervals using all 13 Allen interval algebra relations. -engine = TemporalReasoningEngine() + ```python + from semantica.reasoning import TemporalReasoningEngine, TemporalInterval, IntervalRelation -ceo_tenure = TemporalInterval(start="1997-09-16", end="2011-08-24") -board_member = TemporalInterval(start="2000-01-01", end="2012-06-01") + engine = TemporalReasoningEngine() -relation = engine.get_relation(ceo_tenure, board_member) -# → IntervalRelation.DURING (ceo_tenure is fully inside board_member) -``` + ceo_tenure = TemporalInterval(start="1997-09-16", end="2011-08-24") + board_member = TemporalInterval(start="2000-01-01", end="2012-06-01") -All 13 Allen interval algebra relations are supported: + relation = engine.get_relation(ceo_tenure, board_member) + # → IntervalRelation.DURING (ceo_tenure is fully inside board_member) + ``` -| Relation | Meaning | -| -------- | ------- | -| `BEFORE` | A ends before B starts | -| `MEETS` | A ends exactly when B starts | -| `OVERLAPS` | A starts before B, ends inside B | -| `DURING` | A is fully inside B | -| `STARTS` | A and B start together, A ends first | -| `FINISHES` | A and B end together, A starts later | -| `EQUALS` | Identical intervals | -| + 6 inverses | `AFTER`, `MET_BY`, `OVERLAPPED_BY`, `CONTAINS`, `STARTED_BY`, `FINISHED_BY` | + All 13 Allen interval algebra relations: + + | Relation | Meaning | + | -------- | ------- | + | `BEFORE` | A ends before B starts | + | `MEETS` | A ends exactly when B starts | + | `OVERLAPS` | A starts before B, ends inside B | + | `DURING` | A is fully inside B | + | `STARTS` | A and B start together, A ends first | + | `FINISHES` | A and B end together, A starts later | + | `EQUALS` | Identical intervals | + | + 6 inverses | `AFTER`, `MET_BY`, `OVERLAPPED_BY`, `CONTAINS`, `STARTED_BY`, `FINISHED_BY` | + + ## ExplanationGenerator @@ -212,12 +290,139 @@ explanation = generator.explain( print(explanation.conclusion) print(f"Confidence: {explanation.confidence:.2f}") +print(explanation.justification.summary) for step in explanation.reasoning_path.steps: - print(f" Step {step.depth}: {step.fact}") - print(f" via rule: '{step.rule_name}'") + indent = " " * step.depth + print(f"{indent}Step {step.depth}: {step.fact}") + print(f"{indent} via rule: '{step.rule_name}'") + print(f"{indent} premises: {step.premises}") ``` + + Name every rule with a descriptive string. `ExplanationGenerator` includes the rule name in each derivation step — unnamed rules produce useless explanations like "rule_0 fired." Use names like `"manager_authority"` or `"transitive_location"`. + + + + + +```python +@dataclass +class Explanation: + conclusion: Dict[str, str] # the fact being explained + confidence: float # aggregated rule confidence + reasoning_path: ReasoningPath # full derivation trace + justification: Justification # plain-language summary +``` + + + + +```python +@dataclass +class ReasoningPath: + steps: List[ReasoningStep] # ordered derivation steps + +@dataclass +class ReasoningStep: + depth: int # 0 = base fact, n = nth inference + fact: Dict[str, str] # the fact derived at this step + rule_name: str # name of the rule that fired + premises: List[Dict] # facts that triggered this rule + confidence: float # confidence at this step +``` + + + + +```python +@dataclass +class Justification: + summary: str # one-sentence natural language explanation + evidence: List[str] # list of supporting source facts +``` + + + + +## Combining Multiple Reasoning Engines + +Different engines cover different expressivity levels — compose them for richer inference: + + + + ```python + from semantica.reasoning import Reasoner + + engine = Reasoner() + engine.apply_transitivity("located_in") + engine.apply_symmetry("colleague_of") + structural_result = engine.infer() + ``` + + + ```python + from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule + + datalog = DatalogReasoner() + for fact in structural_result.derived_facts: + datalog.add_fact(DatalogFact(fact.predicate, (fact.subject, fact.obj))) + + datalog.add_rule(DatalogRule("reachable(?X, ?Z) :- located_in(?X, ?Y), reachable(?Y, ?Z).")) + datalog.evaluate() + ``` + + + ```python + from semantica.reasoning import TemporalReasoningEngine + from datetime import datetime + + temporal = TemporalReasoningEngine() + active_facts = [ + f for f in datalog.query("reachable(?X, ?Z)") + if temporal.is_active(f, at=datetime(2024, 1, 1)) + ] + ``` + + + ```python + from semantica.reasoning import ExplanationGenerator + + generator = ExplanationGenerator(engine) + explanation = generator.explain( + {"subject": "london_office", "predicate": "located_in", "object": "UK"} + ) + print(explanation.summary) + ``` + + + +## Tips and Common Pitfalls + + + **Always call `reasoner.infer()` after adding facts and rules.** Adding facts and rules updates internal state but doesn't trigger inference automatically. Inference is a separate, explicit step. + + + + **Use `ReteEngine` for large rule sets.** If you have more than ~20 rules and rules can fire repeatedly, `Reasoner` re-evaluates all rules from scratch on each cycle. `ReteEngine` caches partial matches and is orders of magnitude faster for complex rule sets. + + + + **`DatalogReasoner` guarantees termination; `Reasoner` does not.** If your rules can create cycles (A derives B, B derives C, C re-derives A), `DatalogReasoner`'s semi-naive fixpoint evaluation will stop when no new facts are added. `Reasoner` has no cycle detection and may loop infinitely. + + + + **Name every rule for readable explanations.** `ExplanationGenerator.explain()` includes the rule name in each derivation step. Unnamed or generic rule names produce useless explanations like "rule_0 fired." Use descriptive names: `"manager_authority"`, `"transitive_location"`. + + + + **Use rule `priority` to control inference order.** When multiple rules could fire on the same facts, higher-priority rules fire first. This matters when a higher-priority rule produces a fact that gates a lower-priority rule's conditions. + + + + **Combine engines for maximum expressivity.** Forward-chain structural rules with `Reasoner`, then pass derived facts to `DatalogReasoner` for recursive closure, then filter by time with `TemporalReasoningEngine`. Each engine covers a different expressivity class — they compose cleanly. + + The knowledge graph being reasoned over. diff --git a/docs/reference/seed.md b/docs/reference/seed.md index 50471a90..88609b5b 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -1,119 +1,322 @@ --- title: "Seed Module" -description: "Seed data management for initializing Knowledge Graphs from trusted, verified sources." +description: "Bootstrap Knowledge Graphs from verified, structured sources — taxonomies, reference tables, product catalogs, and domain anchors." icon: "database" --- -`semantica.seed` provides a system for bootstrapping Knowledge Graphs with verified, structured data from trusted sources — taxonomies, reference tables, user lists, product catalogs — so you start with a reliable foundation rather than an empty graph. +`semantica.seed` gives your knowledge graph a reliable starting point. Rather than building from an empty graph and hoping extraction produces consistent reference data, you load verified, structured sources first — ISO codes, employee rosters, product catalogs, domain taxonomies — then merge freshly extracted data on top. ## What You Get -- **`SeedDataManager`** — register sources, build a foundation graph, and integrate with extracted data -- **`SeedDataSource`** — define individual data sources with format, path, and config -- **Merge strategies** — `seed_first`, `extracted_first`, `smart_merge` for combining seed and extracted data -- **Validation** — check data quality and schema compliance before loading -- **Versioning** — track and manage versions of seed data sources + + + Register sources, build a foundation graph, validate quality, and merge with extracted data. + + + Typed source definition supporting CSV, JSON, SQL, API, and RDF with format-specific config. + + + Inject named built-in datasets (companies, countries, currencies) or custom seed files into an existing graph. + + + `seed_first`, `extracted_first`, and `smart_merge` with property-level conflict detection. + + + Required field checks, ID uniqueness, type consistency, reference integrity, and encoding validation before loading. + + + Track seed data versions across pipeline runs and diff changes between versions. + + - **When to use the Seed Module:** - -- **Bootstrapping** — you have existing structured data (taxonomies, user lists, product catalogs) to build on -- **Reference data** — load immutable reference information (countries, ISO codes, ontology terms) -- **Testing** — load consistent, reproducible datasets for development and CI pipelines + **When to use the Seed Module:** Bootstrapping with structured reference data (taxonomies, user lists, product catalogs), loading immutable facts (ISO country codes, standard ontology terms) that extracted data should not override, ensuring test reproducibility with deterministic datasets, and anchoring entity disambiguation with canonical forms. -## SeedDataManager +## Quick Start -The primary interface for seed data: + + + ```python + from semantica.seed import SeedDataManager -```python -from semantica.seed import SeedDataManager + manager = SeedDataManager() -manager = SeedDataManager() -manager.register_source("countries", "csv", "data/countries.csv") -manager.register_source("taxonomy", "json", "data/taxonomy.json") + manager.register_source("countries", "csv", "data/countries.csv") + manager.register_source("taxonomy", "json", "data/taxonomy.json") + manager.register_source("employees", "csv", "data/employees.csv") + ``` + + + ```python + foundation_kg = manager.create_foundation_graph() -# Build a foundation KG from all registered sources -foundation_kg = manager.create_foundation_graph() -``` + print(f"Foundation nodes: {foundation_kg.node_count}") + print(f"Foundation edges: {foundation_kg.edge_count}") + ``` + + + ```python + report = manager.validate_quality(manager.load_source("employees")) -### Core Methods + if not report.is_valid: + for issue in report.issues: + print(f"[{issue.severity}] Row {issue.row}: {issue.message}") + else: + print(f"Validated {report.record_count} records — no issues found") + ``` + + + ```python + final_kg = manager.integrate_with_extracted( + seed_graph=foundation_kg, + extracted_data=new_entities, + strategy="smart_merge", + ) + print(f"Final graph: {final_kg.node_count} nodes, {final_kg.edge_count} edges") + ``` + + + +## SeedDataSource Types + + + + ```python + from semantica.seed import SeedDataSource + + csv_source = SeedDataSource( + name="employees", + type="csv", + path="data/employees.csv", + config={ + "delimiter": ";", + "encoding": "utf-8", + "id_column": "employee_id", + "type": "Person", + } + ) + + manager.register_source_object(csv_source) + ``` + + Best for: employee rosters, product lists, reference tables. + + + ```python + json_source = SeedDataSource( + name="taxonomy", + type="json", + path="data/taxonomy.json", + config={"encoding": "utf-8"} + ) + ``` + + Expects an array of entity objects. Best for: taxonomies, ontology term lists, structured configs. + + + ```python + sql_source = SeedDataSource( + name="products", + type="sql", + path="postgresql://user:pass@localhost/db", + config={"query": "SELECT id, name, category FROM products WHERE active = true"} + ) + ``` + + Best for: live database tables — PostgreSQL, MySQL, SQLite. + + + ```python + # API source — fetch from a REST endpoint + api_source = SeedDataSource( + name="geo_codes", + type="api", + path="https://restcountries.com/v3.1/all", + config={"fields": ["name", "cca2", "region"]} + ) + + # RDF source — OWL ontologies or Turtle files + rdf_source = SeedDataSource( + name="domain_ontology", + type="rdf", + path="data/ontology.ttl", + config={"format": "turtle"} + ) + ``` + + API: external reference APIs (countries, currencies, geo). RDF: existing knowledge bases, OWL ontologies. + + + + | Type | Path Format | Use Case | + | ---- | ----------- | -------- | + | `csv` | File path | Employee rosters, product lists, reference tables | + | `json` | File path | Taxonomies, ontology term lists, structured configs | + | `sql` | Connection string | Live database tables — PostgreSQL, MySQL, SQLite | + | `api` | URL | External reference APIs (countries, currencies, geo) | + | `rdf` | File path | OWL ontologies, Turtle files, existing knowledge bases | + + + + +## SeedDataManager Reference | Method | Description | | ------ | ----------- | -| `register_source(name, format, location)` | Add a data source to the registry | +| `register_source(name, format, path)` | Add a named data source to the registry | | `create_foundation_graph()` | Build a KG from all registered sources | -| `validate_quality(seed_data)` | Check data quality and completeness | -| `integrate_with_extracted(seed, extracted)` | Merge seed and extracted graphs | -| `export_seed_data(path, format)` | Export seed data to RDF, JSON, or CSV | - -## SeedDataSource - -Define a source with format-specific configuration: - -```python -from semantica.seed import SeedDataSource - -source = SeedDataSource( - name="taxonomy", - type="json", # "csv" | "json" | "api" | "sql" - path="taxonomy.json", - config={"encoding": "utf-8"} -) -``` +| `validate_quality(seed_data)` | Check schema compliance, required fields, and duplicates | +| `integrate_with_extracted(seed, extracted, strategy)` | Merge seed and extracted graphs | +| `export_seed_data(path, format)` | Export seed graph to RDF (`turtle`, `json-ld`), JSON, or CSV | +| `populate(kg, dataset, count)` | Inject a named built-in dataset into an existing graph | +| `inject(kg)` | Merge all registered sources into `kg` without duplicating existing entities | +| `load_from_file(path)` | Load seed nodes from JSON, CSV, or RDF file into the manager | +| `list_sources()` | List all registered source names and their formats | +| `get_version(name)` | Get the current version metadata for a named source | ## Merge Strategies -Control how seed data and extracted data are combined: + + + Seed data wins on every conflicting property. Use when seed encodes authoritative reference facts that must not be overridden. + + ```python + final_kg = manager.integrate_with_extracted( + seed_graph=foundation_kg, + extracted_data=new_entities, + strategy="seed_first", + ) + ``` + + Best for: ISO codes, canonical entity names, official taxonomy IDs, employee records. + + + Extracted data overrides seed on conflicting properties. Use when new documents contain more current information than your reference data. + + ```python + final_kg = manager.integrate_with_extracted( + seed_graph=foundation_kg, + extracted_data=new_entities, + strategy="extracted_first", + ) + ``` + + Best for: frequently changing attributes like addresses, titles, revenue figures. + + + Property-level merge with conflict detection — irresolvable conflicts are logged for manual review rather than silently overwritten. + + ```python + final_kg = manager.integrate_with_extracted( + seed_graph=foundation_kg, + extracted_data=new_entities, + strategy="smart_merge", + ) + ``` + + Best for: general-purpose pipelines where surfacing conflicts is more valuable than silently losing data. + + + +## Built-in Datasets + +Inject canonical reference data without loading external files: ```python -final_kg = manager.integrate_seed_extracted( - seed_graph=foundation_kg, - extracted_data=new_data, - strategy="seed_first" # see options below -) +from semantica.seed import SeedDataManager +from semantica.kg import GraphBuilder + +kg = GraphBuilder().build(entities=entities, relationships=relationships) +manager = SeedDataManager() + +# Inject a named built-in dataset +manager.populate(kg, dataset="companies", count=100) +manager.populate(kg, dataset="countries") ``` -| Strategy | Behavior | -| -------- | -------- | -| `seed_first` | Seed data wins on conflicts — use for authoritative reference data | -| `extracted_first` | Extracted data overrides seed — use when new data is more current | -| `smart_merge` | Property-level merging with conflict detection and resolution | +| Dataset | Content | +| ------- | ------- | +| `companies` | Fortune 500 companies with type, sector, HQ | +| `countries` | ISO 3166 country codes, regions, populations | +| `currencies` | ISO 4217 codes, symbols, names | +| `person_names` | Common first/last names for synthetic data | -## Bootstrapping a KG - -Full example — load foundation data then merge with freshly ingested content: +## Full Pipeline Example ```python from semantica.seed import SeedDataManager from semantica.ingest import FileIngestor -from semantica.semantic_extract import NERExtractor +from semantica.parse import DocumentParser +from semantica.split import TextSplitter +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.kg import GraphBuilder from semantica.llms import Groq import os -# Build foundation from verified reference data -manager = SeedDataManager() -manager.register_source("taxonomy", "json", "taxonomy.json") -manager.register_source("employees", "csv", "employees.csv") -foundation_kg = manager.create_foundation_graph() - -# Ingest and extract from new sources llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -ingestor = FileIngestor() -ner = NERExtractor(method="llm", llm_provider=llm) -sources = ingestor.ingest("news_articles/") -new_data = [ner.extract(s.content) for s in sources] +# Step 1 — Build the foundation from verified reference data +seed_manager = SeedDataManager() +seed_manager.register_source("taxonomy", "json", "data/taxonomy.json") +seed_manager.register_source("employees", "csv", "data/employees.csv") +foundation_kg = seed_manager.create_foundation_graph() -# Merge — seed data takes precedence for reference facts -final_kg = manager.integrate_seed_extracted( +# Step 2 — Ingest and extract from unstructured documents +ingestor = FileIngestor() +parser = DocumentParser() +splitter = TextSplitter(method="semantic_transformer", chunk_size=512) +ner = NERExtractor(method="llm", llm_provider=llm) +rel_ext = RelationExtractor(method="llm", llm_provider=llm) + +sources = ingestor.ingest("news_articles/") +extracted_entities = [] +extracted_relationships = [] + +for source in sources: + parsed = parser.parse(source) + chunks = splitter.split_document(parsed) + for chunk in chunks: + entities = ner.extract(chunk.text) + relationships = rel_ext.extract(chunk.text, entities=entities) + extracted_entities.extend(entities) + extracted_relationships.extend(relationships) + +# Step 3 — Merge seed and extracted data +final_kg = seed_manager.integrate_with_extracted( seed_graph=foundation_kg, - extracted_data=new_data, - strategy="seed_first" + extracted_data=extracted_entities, + strategy="seed_first", ) +print(f"Final graph: {final_kg.node_count} nodes, {final_kg.edge_count} edges") ``` -## Configuration +## Versioning + +Track seed data versions to detect when reference data changes between pipeline runs: + +```python +manager = SeedDataManager() +manager.register_source("taxonomy", "json", "data/taxonomy.json") + +version = manager.get_version("taxonomy") +print(f"Version: {version.version_id}") +print(f"Hash: {version.checksum}") +print(f"Records: {version.record_count}") +print(f"Updated: {version.last_modified}") + +# Compare versions to detect changes +old_version = manager.get_version("taxonomy", tag="previous") +if version.checksum != old_version.checksum: + diff = manager.diff_versions("taxonomy", old_version.version_id, version.version_id) + print(f"Added: {diff.added_count} records") + print(f"Removed: {diff.removed_count} records") + print(f"Changed: {diff.modified_count} records") +``` + +## YAML Configuration + +Define sources in YAML for production deployments — no code changes needed to switch environments: ```yaml seed: @@ -121,22 +324,57 @@ seed: - name: "employees" type: "csv" path: "./data/employees.csv" + config: + id_column: "employee_id" + type: "Person" - name: "taxonomy" type: "json" path: "./data/taxonomy.json" + - name: "products" + type: "sql" + path: "${DATABASE_URL}" + config: + query: "SELECT id, name, category FROM products WHERE active = true" merge: - strategy: "seed_first" + strategy: "smart_merge" validation: strict: true + required_fields: ["id", "type"] ``` Environment variable overrides: ```bash -export SEED_DATA_DIR=./data/seed -export SEED_MERGE_STRATEGY=seed_first +export SEMANTICA_SEED_DATA_DIR=./data/seed +export SEMANTICA_SEED_MERGE_STRATEGY=seed_first ``` +## Tips and Common Pitfalls + + + **Load seed data before extracted data.** Seed data is your ground truth — normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values. + + + + **Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `strategy="seed_first"` ensures those values win over extracted values. Use `smart_merge` only when extracted data may be more current than the seed. + + + + **Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast — always run it first. + + + + **Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once. + + + + **Track seed versions to detect drift.** Use `manager.get_version()` and `manager.diff_versions()` to detect when reference data changes between pipeline runs. If a taxonomy file changes, downstream entity normalisation and deduplication thresholds may need re-tuning — don't treat seed data as static. + + + + **Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment. + + Load unstructured data alongside seed data. @@ -148,6 +386,6 @@ export SEED_MERGE_STRATEGY=seed_first Handle duplicates during seed-extracted merge. - Incorporate seed loading as a pipeline step. + Incorporate seed loading as a named pipeline step. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 63ff85ae..0594fc49 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -6,34 +6,162 @@ icon: "magnifying-glass-chart" `semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica. All extractors support three modes: pattern-based (no API key), ML-based, and LLM-based. +## Quick Start + + + + ```python + from semantica.semantic_extract import CoreferenceResolver + + resolver = CoreferenceResolver() + resolved_text = resolver.resolve( + "Apple Inc. was founded in 1976. The company is headquartered in Cupertino." + ) + # "Apple Inc." replaces "The company" — consistent downstream extraction + ``` + + + ```python + from semantica.semantic_extract import NERExtractor + from semantica.llms import Groq + import os + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) + entities = ner.extract(resolved_text) + # → [{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98, ...}] + ``` + + + ```python + from semantica.semantic_extract import RelationExtractor + + rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) + relationships = rel.extract(resolved_text, entities=entities) + # → [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", ...}] + ``` + + + ```python + from semantica.semantic_extract import ExtractionValidator + + validator = ExtractionValidator(min_confidence=0.7) + valid_entities, _ = validator.validate_entities(entities) + valid_rels, _ = validator.validate_relations(relationships) + ``` + + + ## What You Get -- **`NERExtractor`** — named entity recognition: Person, Organization, Location, Date, and custom types -- **`RelationExtractor`** — typed semantic relationships between entities (`founded_by`, `located_in`, etc.) -- **`TripletExtractor`** — direct `(subject, predicate, object)` triplet generation for RDF-ready output -- **`EventExtractor`** — event detection with participants, temporal context, and confidence scores -- **`CoreferenceResolver`** — resolve "Apple" and "the company" to the same entity across a document + + + Named entity recognition: Person, Organization, Location, Date, and custom types. + + + Typed semantic relationships between entities (`founded_by`, `located_in`, etc.). + + + Direct `(subject, predicate, object)` triplet generation for RDF-ready output. + + + Event detection with participants, temporal context, and confidence scores. + + + Resolve "Apple" and "the company" to the same entity across a document. + + + Semantic role labeling, clustering, and entity similarity analysis. + + Semantic extraction pipeline: raw text fans into NER, Relation, and Coreference extractors, then merges into a Triplet Generator +## Extraction Methods + + + + Uses a language model to extract entities and relationships. Handles complex schemas, novel entity types, and domain-specific language. Requires an API key. + + ```python + from semantica.semantic_extract import NERExtractor + from semantica.llms import Groq + import os + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) + + entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino in 1976.") + ``` + + + **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. + + + Works with every Semantica LLM provider — swap `Groq` for `Anthropic`, `OpenAI`, `Gemini`, `Ollama`, `HuggingFace`, `DeepSeek`, or `Novita` with a one-line change: + + ```python + from semantica.llms import Anthropic + llm = Anthropic(model="claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) + ner = NERExtractor(method="llm", llm_provider=llm) + ``` + + + Uses a pre-trained BERT-based NER model. High accuracy for standard entity types (Person, Organization, Location, Date) at zero API cost. + + ```python + ner = NERExtractor(method="ml", model="dslim/bert-large-NER") + entities = ner.extract(text) + ``` + + For relations, the ML backend uses the REBEL model: + + ```python + rel = RelationExtractor(method="ml") + relationships = rel.extract(text, entities=entities) + ``` + + Best for: high-throughput extraction where API cost matters and entity types are standard CoNLL/OntoNotes categories. + + + Dictionary and regex matching — extremely fast, zero API cost, zero model loading. Accuracy depends entirely on your dictionaries. + + ```python + ner = NERExtractor( + method="pattern", + custom_entities={ + "DRUG": ["aspirin", "ibuprofen", "metformin"], + "GENE": ["BRCA1", "TP53", "EGFR"] + } + ) + entities = ner.extract(text) + ``` + + For relations, uses hand-crafted rules: + + ```python + rel = RelationExtractor(method="rule") + ``` + + Best for: known entity sets (drug names, product codes, gene symbols), no-API-key environments, or as a first pass before LLM validation. + + + The pattern matcher is **case-sensitive and whitespace-sensitive**. Normalize text first with `TextNormalizer` so "BRCA1" and "brca1" both match. For fuzzy matching, use `method="ml"`. + + + + +### Method Comparison + +| Method | Speed | Cost | Accuracy | Custom Types | Best For | +| ------ | ----- | ---- | -------- | ------------ | -------- | +| `pattern` | Very fast | Free | Medium | Yes (dictionary) | Known entity sets, no-API environments | +| `ml` | Fast | Free | High | Limited | Standard types at scale, no API budget | +| `llm` | Medium | API cost | Highest | Yes (schema) | Complex schemas, novel types, best accuracy | + ## NERExtractor ```python -from semantica.semantic_extract import NERExtractor -from semantica.llms import Groq -import os - -# Pattern-based — fast, no API key, good for standard entity types -ner = NERExtractor(method="pattern") -entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino.") - -# ML-based — higher accuracy, no API cost -ner = NERExtractor(method="ml", model="dslim/bert-large-NER") -entities = ner.extract(text) - -# LLM-based — best accuracy, handles complex schemas and custom types -llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) entities = ner.extract(text) ``` @@ -47,28 +175,16 @@ Output format: ] ``` -### Custom Entity Types +Batch processing for large corpora: ```python -ner = NERExtractor( - method="pattern", - custom_entities={ - "DRUG": ["aspirin", "ibuprofen", "metformin"], - "GENE": ["BRCA1", "TP53", "EGFR"] - } -) +texts = ["Text 1...", "Text 2...", "Text 3..."] +batch_results = ner.extract_batch(texts, batch_size=10) ``` - - **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. - - ## RelationExtractor ```python -from semantica.semantic_extract import RelationExtractor - -rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) relationships = rel.extract(text, entities=entities) ``` @@ -81,7 +197,9 @@ Output format: ] ``` -Available methods: `"rule"` (pattern-based), `"ml"` (REBEL model), `"llm"`. + + Always pass `entities=entities` from your NER output. This anchors relationships to known entity spans — improving accuracy and eliminating hallucinated entity names. + ## TripletExtractor @@ -90,12 +208,12 @@ Generate RDF-ready `(subject, predicate, object)` triplets directly from text: ```python from semantica.semantic_extract import TripletExtractor -trip = TripletExtractor(method="llm", llm_provider=llm) +trip = TripletExtractor(method="llm", llm_provider=llm) triplets = trip.extract(text) # → [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", ...}] ``` -Triplets are suitable for loading directly into a triplet store or knowledge graph. +Triplets are suitable for loading directly into a triplet store or knowledge graph without a separate relation extraction step. ## EventExtractor @@ -105,63 +223,127 @@ Detect events with participants and temporal context: from semantica.semantic_extract import EventExtractor extractor = EventExtractor(method="llm", llm_provider=llm) -events = extractor.extract(text) +events = extractor.extract(text) ``` Output includes: event type, participants (with roles), temporal information, location, and confidence score. -## CoreferenceResolver +## SemanticAnalyzer -Resolve pronoun and alias references to canonical entities before extraction: +Semantic role labeling, clustering, and similarity analysis on extracted content: ```python -from semantica.semantic_extract import CoreferenceResolver +from semantica.semantic_extract import SemanticAnalyzer -resolver = CoreferenceResolver() -resolved_text = resolver.resolve( - "Apple Inc. was founded in 1976. The company is headquartered in Cupertino." +analyzer = SemanticAnalyzer() + +# Who did what to whom +roles = analyzer.label_roles(text) +# → [{"agent": "Apple", "action": "acquired", "patient": "Intel's modem unit"}] + +# Group similar entities +clusters = analyzer.cluster_entities(entities, n_clusters=5) +# → [{"cluster_id": 0, "entities": ["Apple Inc.", "Apple", "AAPL"]}, ...] + +# Pairwise semantic similarity +score = analyzer.calculate_similarity(entity_a, entity_b) +# → 0.87 +``` + +| Method | Returns | Description | +| ------ | ------- | ----------- | +| `label_roles(text)` | `List[Dict]` | Semantic role labeling (agent, action, patient) | +| `cluster_entities(entities, n_clusters)` | `List[Cluster]` | Group similar entities | +| `calculate_similarity(a, b)` | `float` | Cosine similarity between entity embeddings | +| `analyze_sentiment(text)` | `Dict` | Sentiment and subjectivity scores | + +## SemanticNetworkExtractor + +Extracts a full semantic network (nodes + typed edges) from text in one pass: + +```python +from semantica.semantic_extract import SemanticNetworkExtractor + +extractor = SemanticNetworkExtractor(method="llm", llm_provider=llm) +network = extractor.extract_network(text) + +print(f"Nodes: {len(network.nodes)}") +print(f"Edges: {len(network.edges)}") + +for edge in network.edges: + print(f" {edge.source} --[{edge.relation}]--> {edge.target} (conf: {edge.confidence:.2f})") +``` + + + +```python +@dataclass +class SemanticNetwork: + nodes: List[NetworkNode] + edges: List[NetworkEdge] + +@dataclass +class NetworkNode: + id: str + label: str # entity type (PERSON, ORG, etc.) + properties: Dict[str, Any] + +@dataclass +class NetworkEdge: + source: str + target: str + relation: str # e.g. "founded_by", "located_in" + confidence: float + metadata: Dict[str, Any] +``` + + + +## ExtractionValidator + +Validates extraction quality and filters low-confidence results: + +```python +from semantica.semantic_extract import ExtractionValidator + +validator = ExtractionValidator( + min_confidence=0.7, # drop entities below this score + require_entity_text=True, # entity text must be non-empty + max_entity_length=100, # discard suspiciously long entities ) -# "Apple Inc." replaces "The company" for consistent downstream extraction + +valid_entities, rejected = validator.validate_entities(entities) +valid_rels, rejected = validator.validate_relations(relationships) + +report = validator.get_quality_report(entities, relationships) +print(f"Precision estimate: {report['precision']:.2f}") ``` -## Batch Processing +## Tips and Common Pitfalls -All extractors support batch input for efficient large-scale processing: + + **Run `CoreferenceResolver` before extraction.** If a paragraph says "Apple Inc. was founded in 1976. The company launched..." without resolving "the company" → "Apple Inc.", your extractor may miss the second entity or create a phantom "The company" node. + -```python -texts = ["Text 1...", "Text 2...", "Text 3..."] + + **Always pass `entities=` to `RelationExtractor`.** Passing the entity list from NER output anchors relationships to known entity spans — improving accuracy and eliminating hallucinated entity names. + -ner = NERExtractor(method="llm", llm_provider=llm) -batch_results = ner.extract_batch(texts, batch_size=10) -``` + + **Validate before building the graph.** Use `ExtractionValidator(min_confidence=0.7)` to drop low-confidence entities before they reach `GraphBuilder`. Noisy extractions produce noisy graphs that corrupt analytics and search. + -## Using All Extractors Together + + **Set `max_retries=3` for LLM extractors.** API calls fail transiently. Setting `max_retries=3` prevents pipeline crashes on flaky network conditions without slowing down the happy path. + -The standard extraction pipeline — entities → relationships → triplets: + + **Don't mix extraction methods mid-pipeline.** If you extract entities with `pattern` and relations with `llm`, entity names in the relation output may not match the pattern-extracted entity IDs — causing alignment failures in `GraphBuilder`. Use the same method throughout, or normalize entity names before the relation step. + -```python -from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor -from semantica.llms import Groq -import os - -llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - -ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) -rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) -trip = TripletExtractor(method="llm", llm_provider=llm, max_retries=3) - -entities = ner.extract(text) -relationships = rel.extract(text, entities=entities) -triplets = trip.extract(text) -``` - -## Extraction Method Comparison - -| Method | Speed | Cost | Accuracy | Custom Types | -| ------ | ----- | ---- | -------- | ------------ | -| `pattern` | Very fast | Free | Medium | Yes (dictionary) | -| `ml` | Fast | Free | High | Limited | -| `llm` | Medium | API cost | Highest | Yes (schema) | + + **Batch large inputs.** Call `ner.extract_batch(texts, batch_size=10)` rather than looping over individual texts. Batch mode is significantly faster for both ML (GPU batching) and LLM (fewer API round-trips with prompt packing). + diff --git a/docs/reference/split.md b/docs/reference/split.md index 0bffe58d..a07169c6 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -1,145 +1,413 @@ --- title: "Split Module" -description: "15+ text chunking methods including recursive, semantic, entity-aware, and relation-aware splitting." +description: "15+ text chunking methods including recursive, semantic, entity-aware, relation-aware, code, and structural splitting." icon: "scissors" --- -`semantica.split` breaks documents into chunks while preserving semantic context — critical for embedding quality in RAG systems and accurate entity extraction in NER pipelines. +`semantica.split` breaks documents into chunks that preserve semantic context. Chunking quality directly determines downstream accuracy — a poorly chunked document produces bad embeddings, missed entities, and broken relation triplets. Use the right strategy for your content type and pipeline goal. + +## Why Chunking Matters + +Most LLMs and embedding models have fixed context windows. Documents larger than that window must be split. But naive splitting (every 500 characters, regardless of structure) destroys semantic context: + +- An entity mention like "Apple Inc." split across two chunks loses its context in both +- A relation triplet like "Steve Jobs founded Apple" split at "Steve Jobs" leaves a dangling subject +- Embedding a chunk that mixes two unrelated topics produces a centroid vector that matches neither + +Semantica's chunking methods are designed to avoid these failure modes. ## What You Get -- **`TextSplitter`** — unified interface for 9+ chunking strategies -- **Entity-aware chunking** — entity mentions never split across chunk boundaries -- **Relation-aware chunking** — subject–predicate–object triplets kept intact -- **Semantic chunking** — split at topic shift boundaries using embedding similarity -- **`Chunk`** — output object with text, token count, character offsets, and metadata + + + Unified interface for 11 chunking strategies — swap methods without changing downstream code. + + + Embedding-based topic shift detection — splits only when the topic actually changes. + + + Entity spans never cross chunk boundaries — guaranteed by boundary adjustment. + + + Subject–predicate–object triplets kept within a single chunk for KG pipelines. + + + AST-level boundaries (function, class, method) for source code search and analysis. + + + Output dataclass with text, token count, character offsets, entities, and full metadata. + + -## TextSplitter +## Quick Start -```python -from semantica.split import TextSplitter + + + ```python + from semantica.split import TextSplitter -splitter = TextSplitter( - method="semantic_transformer", # see methods table below - chunk_size=1000, # target tokens per chunk - chunk_overlap=200 # token overlap between adjacent chunks -) + splitter = TextSplitter( + method="recursive", # see Splitting Methods table + chunk_size=1000, + chunk_overlap=200, + ) + ``` + + + ```python + chunks = splitter.split(text) -chunks = splitter.split(text) -for chunk in chunks: - print(f"Chunk {chunk.metadata['chunk_index']}: {chunk.text[:80]}...") - print(f" Tokens: {chunk.token_count}") -``` + for chunk in chunks: + print(f"Chunk {chunk.metadata['chunk_index']} / {chunk.metadata['total_chunks']}") + print(f" Tokens: {chunk.token_count}") + print(f" Preview: {chunk.text[:80]}...") + ``` + + + ```python + from semantica.parse import DocumentParser + + parser = DocumentParser() + parsed = parser.parse("annual_report.pdf") + + splitter = TextSplitter(method="structural") + chunks = splitter.split_document(parsed) + + for chunk in chunks: + print(f"[h{chunk.metadata['heading_level']}] {chunk.metadata['section_title']}") + ``` + + + ```python + all_chunks = splitter.split_documents(parsed_docs) + + from collections import defaultdict + by_source = defaultdict(list) + for chunk in all_chunks: + by_source[chunk.metadata['source_id']].append(chunk) + ``` + + ## Splitting Methods -| Method | Description | Best For | -| ------ | ----------- | -------- | -| `recursive` | Split by paragraph → sentence → word (cascading) | General purpose | -| `semantic_transformer` | Split at semantic topic boundaries via sentence transformer | RAG retrieval | -| `entity_aware` | Keep entity mentions intact across boundaries | NER pipelines | -| `relation_aware` | Keep relation triplets intact | KG construction | -| `sentence` | Split by sentence boundary | Short content | -| `token` | Split by token count (tiktoken) | LLM context windows | -| `fixed` | Fixed character count with overlap | Batch processing | -| `markdown` | Split by Markdown heading hierarchy | Documentation | -| `code` | Split by function/class/method boundaries | Code analysis | +| Method | How It Splits | Best For | +| ------ | ------------- | -------- | +| `recursive` | Paragraph → sentence → word (cascading fallback) | General-purpose default | +| `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG — topic coherence matters | +| `entity_aware` | Adjusts boundaries so entity spans are never cut | NER pipelines | +| `relation_aware` | Keeps subject–predicate–object triplets within one chunk | KG construction | +| `sentence` | Language-aware sentence boundary detection (NLTK/spaCy) | Short documents, Q&A | +| `token` | Exact token count via tiktoken; hard cutoff | LLM context window prep | +| `fixed` | Fixed character count with overlap; fastest, no NLP | Simple batch jobs | +| `sliding_window` | Fixed-step window — heavy overlap for dense retrieval | Bi-encoder retrieval (ColBERT, DPR) | +| `markdown` | Splits at Markdown heading levels (configurable) | Documentation, wikis, MDX | +| `structural` | Splits at `ParsedDocument.sections` boundaries | Structured PDFs and DOCX | +| `code` | AST-level splits at function / class / method boundaries | Source code search and analysis | -## Entity-Aware Chunking +## Choosing a Strategy -Entity mentions are never split across chunk boundaries, preserving context for downstream NER: +Use this decision tree before picking a method: -```python -from semantica.split import TextSplitter -from semantica.semantic_extract import NERExtractor +- **Source code?** → `code` +- **Markdown or structured doc with headings?** → `markdown` or `structural` +- **Building a KG?** → `relation_aware` (keeps triplets intact), then `entity_aware` for pure NER +- **RAG system where retrieval quality matters most?** → `semantic_transformer` +- **Dense overlap for bi-encoder retrieval (ColBERT, DPR)?** → `sliding_window` +- **Preparing prompts for a fixed-window LLM?** → `token` +- **Fast splitting with no NLP overhead?** → `recursive` or `fixed` -ner = NERExtractor() -entities = ner.extract(text) - -splitter = TextSplitter(method="entity_aware") -chunks = splitter.split(text, entities=entities) -# → Each chunk contains only complete entity mentions -``` - -## Relation-Aware Chunking - -Subject–predicate–object triplets are kept within the same chunk: +## TextSplitter Constructor ```python from semantica.split import TextSplitter -splitter = TextSplitter(method="relation_aware") -chunks = splitter.split(text, relationships=relationships) -# → Triplets are never split across chunk boundaries -``` - -## Semantic Chunking - -Split at topic shift boundaries detected via embedding similarity: - -```python -from semantica.split import TextSplitter -from semantica.embeddings import EmbeddingGenerator - -embedder = EmbeddingGenerator(model="sentence-transformers") splitter = TextSplitter( - method="semantic_transformer", - embedder=embedder, - similarity_threshold=0.7 # split when consecutive sentence similarity drops below this + method="semantic_transformer", # chunking strategy + chunk_size=1000, # target size in tokens + chunk_overlap=200, # token overlap between adjacent chunks + tokenizer="cl100k_base", # tiktoken encoding (GPT-4 default) + min_chunk_size=50, # discard very short trailing chunks + include_metadata=True, # attach source_id, page_number, section_title + language="en", # ISO 639-1 — used by sentence boundary detector ) - -chunks = splitter.split(text) ``` -## Token-Based Chunking +| Parameter | Type | Default | Description | +| --------- | ---- | ------- | ----------- | +| `method` | `str` | `"recursive"` | Chunking strategy — see table above | +| `chunk_size` | `int` | `1000` | Target size in tokens (characters for `fixed`) | +| `chunk_overlap` | `int` | `200` | Token overlap between adjacent chunks | +| `tokenizer` | `str` | `"cl100k_base"` | tiktoken encoding: `"cl100k_base"` (GPT-4), `"p50k_base"` (GPT-3), `"r50k_base"` (Codex) | +| `min_chunk_size` | `int` | `0` | Discard chunks shorter than this many tokens | +| `similarity_threshold` | `float` | `0.7` | Cosine similarity cutoff for `semantic_transformer` | +| `embedder` | `EmbeddingGenerator` | `None` | Custom embedder for `semantic_transformer` | +| `include_metadata` | `bool` | `True` | Attach `source_id`, `page_number`, `section_title` to each chunk | +| `language` | `str` | `"en"` | ISO 639-1 language code for sentence boundary detection | +| `heading_levels` | `list[int]` | `[1, 2, 3]` | Heading levels to split on for `markdown` method | +| `code_units` | `list[str]` | `["function", "class"]` | AST node types to split on for `code` method | -Use tiktoken for precise token-count control when preparing LLM context windows: +## Splitting Method Details -```python -splitter = TextSplitter( - method="token", - chunk_size=512, # max tokens per chunk - chunk_overlap=50, # overlap in tokens - tokenizer="cl100k_base" # OpenAI tokenizer -) -chunks = splitter.split(text) -``` + + + Tries paragraph breaks first, then sentence boundaries, then word boundaries — falling back only when the chunk exceeds `chunk_size`: -## Chunk Object + ```python + splitter = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200) + chunks = splitter.split(text) + ``` + + **Key behaviours:** + - Preserves paragraph and sentence structure wherever possible + - Falls back gracefully — never produces chunks larger than `chunk_size` + - Overlap ensures context continuity across chunk boundaries + - Good starting point when you're unsure which method to use + + + Embeds each sentence, then splits whenever cosine similarity between consecutive sentences drops below `similarity_threshold`. Each chunk talks about one topic: + + ```python + from semantica.split import TextSplitter + from semantica.embeddings import EmbeddingGenerator + + embedder = EmbeddingGenerator(model="sentence-transformers") + splitter = TextSplitter( + method="semantic_transformer", + embedder=embedder, + similarity_threshold=0.7, # 0.6 = more splits, 0.8 = fewer splits + chunk_size=800, + chunk_overlap=0, # not needed — chunks are already coherent + ) + chunks = splitter.split(text) + ``` + + **Key behaviours:** + - Produces variable-length chunks — some topics are short, others long + - Requires an embedder — defaults to `sentence-transformers/all-MiniLM-L6-v2` if not set + - Slower than `recursive` due to embedding computation; cache embeddings for repeated splits + - Best retrieval quality for semantic search — chunks map to single coherent topics + + + Runs NER first, then adjusts chunk boundaries so no entity mention is split across two chunks: + + ```python + from semantica.split import TextSplitter + from semantica.semantic_extract import NERExtractor + from semantica.llms import Groq + import os + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + ner = NERExtractor(method="llm", llm_provider=llm) + entities = ner.extract(text) + + splitter = TextSplitter(method="entity_aware", chunk_size=512, chunk_overlap=50) + chunks = splitter.split(text, entities=entities) + + for chunk in chunks: + print(f"Chunk {chunk.metadata['chunk_index']}: {len(chunk.entities)} entities") + ``` + + **Key behaviours:** + - Entity spans in `chunk.entities` are guaranteed to fall entirely within `chunk.text` + - Chunk sizes vary slightly from `chunk_size` — boundary adjustments are ≤ one sentence + - Works with all entity types: PERSON, ORGANIZATION, LOCATION, DATE, custom types + + + Keeps subject–predicate–object triplets within the same chunk — critical for KG pipelines: + + ```python + from semantica.split import TextSplitter + from semantica.semantic_extract import RelationExtractor, NERExtractor + from semantica.llms import Groq + import os + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + ner = NERExtractor(method="llm", llm_provider=llm) + rel_extractor = RelationExtractor(method="llm", llm_provider=llm) + + entities = ner.extract(text) + relationships = rel_extractor.extract(text, entities=entities) + + splitter = TextSplitter(method="relation_aware", chunk_size=512) + chunks = splitter.split(text, relationships=relationships) + + for chunk in chunks: + print(f"Chunk {chunk.metadata['chunk_index']}: {len(chunk.relationships)} triplets") + for rel in chunk.relationships: + print(f" {rel['subject']} —[{rel['predicate']}]→ {rel['object']}") + ``` + + **Key behaviours:** + - Relation triplets in `chunk.relationships` are always fully contained within the chunk + - Implies entity-aware behaviour — both entities in a triplet are kept whole too + - Best used as the split step in a `Parse → Split → Extract → Build KG` pipeline + + + Parses source files with `CodeParser` and splits at AST-level boundaries: + + ```python + from semantica.parse import CodeParser + from semantica.split import TextSplitter + + parser = CodeParser(extract_comments=True, extract_dependencies=True) + parsed = parser.parse("src/pipeline.py") + + splitter = TextSplitter( + method="code", + code_units=["function", "class"], # "function" | "class" | "method" | "block" + chunk_overlap=0, # code units are self-contained + ) + chunks = splitter.split_document(parsed) + + for chunk in chunks: + print(f"{chunk.metadata['unit_type']}: {chunk.metadata['unit_name']}") + print(f" Lines {chunk.start_char}–{chunk.end_char} ({chunk.token_count} tokens)") + ``` + + **Key behaviours:** + - Requires a `ParsedDocument` from `CodeParser` — use `split_document()` + - `chunk_overlap=0` recommended — functions and classes are logically self-contained + - If a class is too large, it is split at method boundaries automatically + - Supported languages: Python, JavaScript, TypeScript, Java, Go, Rust, C, C++, C#, Ruby, PHP, Swift + + + ### Structural + + Uses `ParsedDocument.sections` as natural split points — each document section becomes one chunk: + + ```python + from semantica.parse import DoclingParser + from semantica.split import TextSplitter + + parser = DoclingParser(extract_tables=True) + parsed = parser.parse("annual_report.pdf") + + splitter = TextSplitter(method="structural") + chunks = splitter.split_document(parsed) + + for chunk in chunks: + level = chunk.metadata['heading_level'] + title = chunk.metadata['section_title'] + print(f"{' ' * (level - 1)}[h{level}] {title} ({chunk.token_count} tokens)") + ``` + + ### Markdown + + Splits at Markdown heading boundaries, configurable to specific heading levels: + + ```python + splitter = TextSplitter( + method="markdown", + heading_levels=[1, 2], # split at # and ## only; ### stays inline + chunk_size=800, + ) + chunks = splitter.split(markdown_text) + ``` + + + + +## Chunk Schema + + + ```python @dataclass class Chunk: - text: str # chunk text content - start_char: int # character offset in source document - end_char: int # character offset in source document - token_count: int # number of tokens - metadata: Dict # source_id, chunk_index, section_title, page_number, etc. - entities: List[Dict] # entities in chunk (entity_aware splitting only) + text: str # the chunk's text content + start_char: int # character offset of start in source document + end_char: int # character offset of end in source document + token_count: int # number of tokens (via configured tokenizer) + metadata: Dict # see metadata fields below + entities: List[Dict] # entity spans fully contained in this chunk + relationships: List[Dict] # relation triplets fully contained in this chunk ``` + + + +| Field | Type | When Present | Description | +| ----- | ---- | ------------ | ----------- | +| `source_id` | `str` | Always | ID of the source `ParsedDocument` | +| `chunk_index` | `int` | Always | Zero-based position within the document | +| `total_chunks` | `int` | Always | Total chunks produced for this document | +| `method` | `str` | Always | Splitting method that produced this chunk | +| `section_title` | `str` | `structural`, `markdown` | Heading text of the containing section | +| `heading_level` | `int` | `structural`, `markdown` | Depth: 1 = h1, 2 = h2, … | +| `page_number` | `int` | `structural` (DoclingParser) | Source page number in PDF/DOCX | +| `unit_type` | `str` | `code` | `"function"` / `"class"` / `"method"` | +| `unit_name` | `str` | `code` | Name of the code unit, e.g. `"process_batch"` | +| `language` | `str` | `sentence`, `recursive` | ISO 639-1 code for detected text language | +| `similarity_score` | `float` | `semantic_transformer` | Cosine similarity to the adjacent chunk | + + + + +## Tokenizer Options + +| Tokenizer | Models | +| --------- | ------ | +| `cl100k_base` | GPT-4, GPT-3.5-turbo, text-embedding-ada-002 | +| `p50k_base` | GPT-3 (`text-davinci-003`), Codex | +| `r50k_base` | GPT-3 (`davinci`) | + ## Pipeline Integration ```python from semantica.pipeline import Pipeline +from semantica.ingest import FileIngestor +from semantica.parse import DocumentParser from semantica.split import TextSplitter +from semantica.semantic_extract import NERExtractor +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) pipeline = Pipeline() -pipeline.add_step("split", TextSplitter(method="semantic_transformer", chunk_size=512)) -result = pipeline.run(documents) +pipeline.add_step("ingest", FileIngestor()) +pipeline.add_step("parse", DocumentParser()) +pipeline.add_step("split", TextSplitter(method="semantic_transformer", chunk_size=512)) +pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) + +result = pipeline.run("data/reports/") ``` +## Tips and Common Pitfalls + + + **`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum — for `chunk_size=1000`, set `chunk_overlap=100` to `200`. + + + + **Wrong tokenizer.** If you use `cl100k_base` (GPT-4) but send chunks to a model with a different vocabulary, your token counts will be wrong. Match the tokenizer to your target model. + + + + **Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting — use `recursive` instead. + + + + **Code units too coarse.** `code_units=["class"]` on a large codebase produces chunks too big to embed well. Use `["function", "method"]` for more granular, independently useful units. + + + + **Set `min_chunk_size` to avoid fragment chunks.** `min_chunk_size=0` (default) can produce many tiny trailing chunks. Set to ~30–50 tokens to discard fragments that carry no retrieval value. + + - Parse documents before chunking. + Parse documents before chunking — produces sections and metadata. Embed chunks for vector search and semantic chunking. - Extract entities from individual chunks. + Extract entities and relations from individual chunks. - Integrate splitting as a pipeline step. + Integrate splitting as a named pipeline step. diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index 89d6729a..fd3d53a2 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -8,31 +8,193 @@ icon: "table" ## What You Get -- **`TripletStore`** — unified interface for all RDF backends -- **Backends** — Blazegraph, Apache Jena (Fuseki), RDF4J -- **SPARQL** — full SELECT, CONSTRUCT, ASK, and UPDATE query support -- **Bulk loading** — efficient batch import for large triple sets -- **Import / Export** — Turtle, JSON-LD, N-Triples, RDF/XML + + + Unified interface across Blazegraph, Apache Jena (Fuseki), and RDF4J — swap backends with one parameter. + + + Zero-setup in-memory store for unit tests and small datasets — no server, no Docker required. + + + Full SELECT, CONSTRUCT, ASK, and UPDATE query support with pagination for large result sets. + + + Apache Jena supports OWL and RDFS inference natively — subclass and property chain queries automatically resolved. + + + Isolate triples by source, dataset, or time period using named graph management. + + + Load and serialize to Turtle, JSON-LD, N-Triples, and RDF/XML with a single method call. + + -## Basic Usage +## Quick Start + + + + ```python + from semantica.triplet_store import TripletStore + + store = TripletStore( + backend="blazegraph", + endpoint="http://localhost:9999/blazegraph/sparql" + ) + ``` + + + ```python + # Add a single triplet + store.add_triplet( + subject="http://example.org/apple_inc", + predicate="http://example.org/founded_by", + obj="http://example.org/steve_jobs" + ) + + # Bulk load a list of triplets + store.add_triplets_bulk(triplets) + ``` + + + ```python + results = store.sparql(""" + PREFIX ex: + SELECT ?person ?company WHERE { + ?person ex:founded ?company . + ?company ex:located_in ex:SiliconValley . + } + """) + + for row in results: + print(row["person"], row["company"]) + ``` + + + ```python + store.export("output.ttl", format="turtle") + store.export("output.nt", format="nt") + store.export("output.xml", format="xml") + ``` + + + +## Backends + + + + ```python + from semantica.triplet_store import TripletStore + + store = TripletStore( + backend="blazegraph", + endpoint="http://localhost:9999/blazegraph/sparql", + namespace="semantica" + ) + ``` + + Best for: Wikidata-style workloads, high triple counts, SPARQL 1.1 full support. + + + ```python + store = TripletStore( + backend="jena", + endpoint="http://localhost:3030/dataset/sparql", + update_endpoint="http://localhost:3030/dataset/update" + ) + ``` + + Best for: General RDF, standard SPARQL, production deployments needing OWL inference. + + **Enable OWL reasoning:** + + ```python + store = TripletStore( + backend="jena", + endpoint="http://localhost:3030/dataset/sparql", + update_endpoint="http://localhost:3030/dataset/update", + reasoner="OWL", # "OWL" | "RDFS" | "OWL_MINI" | None + ) + + # Load an OWL ontology — subclass/property chain inferences are automatic + store.import_file("ontology.ttl", format="turtle") + store.add_triplets_bulk(data_triplets) + + # Query using inferred relationships + results = store.sparql(""" + SELECT ?person WHERE { + ?person a ex:Employee . # inferred via subClassOf chain + } + """) + ``` + + + ```python + store = TripletStore( + backend="rdf4j", + server_url="http://localhost:8080/rdf4j-server", + repository_id="semantica" + ) + ``` + + Best for: Enterprise Java ecosystems, Eclipse Foundation deployments, plugin-based reasoning. + + + ```python + from semantica.triplet_store import InMemoryTripletStore + + store = InMemoryTripletStore() + + store.add_triplet("ex:alice", "ex:knows", "ex:bob") + store.add_triplet("ex:bob", "ex:works_for", "ex:acme") + + results = store.sparql(""" + SELECT ?person ?company WHERE { + ?person ex:works_for ?company . + } + """) + + # Serialize to string for inspection + ttl = store.export_to_string(format="turtle") + print(ttl) + ``` + + `InMemoryTripletStore` shares the same interface as `TripletStore` — swap backends without changing query code. + + Best for: unit tests, CI pipelines, small datasets, zero-infrastructure local exploration. + + + + | Backend | License | OWL Reasoning | Hosted Option | Best For | + | ------- | ------- | ------------- | ------------- | -------- | + | Blazegraph | Open source | No | Self-hosted | Wikidata-style workloads, high triple count | + | Apache Jena | Apache 2.0 | Yes (OWL/RDFS) | Self-hosted | General RDF, OWL reasoning, standard SPARQL | + | RDF4J | Eclipse 1.0 | Via plugin | Self-hosted or cloud | Enterprise Java ecosystems | + | InMemory | Built-in | No | N/A | Unit tests, small graphs, no server required | + + + + +## Namespace Prefix Management + +Register custom prefixes to keep SPARQL queries readable: ```python -from semantica.triplet_store import TripletStore +from semantica.triplet_store import TripletStore, NamespacePrefixManager -store = TripletStore( - backend="blazegraph", - endpoint="http://localhost:9999/blazegraph/sparql" -) +ns = NamespacePrefixManager() +ns.register("ex", "http://example.org/") +ns.register("schema", "https://schema.org/") +ns.register("owl", "http://www.w3.org/2002/07/owl#") -# Add a single triplet -store.add_triplet( - subject="http://example.org/apple_inc", - predicate="http://example.org/founded_by", - obj="http://example.org/steve_jobs" -) +store = TripletStore(backend="jena", endpoint="...", namespace_manager=ns) -# Bulk load a list of triplets -store.add_triplets_bulk(triplets) +# Registered prefixes are automatically prepended to every SPARQL query +results = store.sparql(""" + SELECT ?company WHERE { + ?person ex:works_for ?company ; + schema:name "Alice" . + } +""") ``` ## SPARQL Queries @@ -47,9 +209,6 @@ results = store.sparql(""" } """) -for row in results: - print(row["person"], row["company"]) - # CONSTRUCT — returns a graph of matched triples graph = store.sparql_construct(""" PREFIX ex: @@ -76,54 +235,29 @@ store.sparql_update(""" """) ``` -## Backends +## SPARQL Result Pagination + +For large result sets, paginate with LIMIT and OFFSET: ```python -# Blazegraph — open source, SPARQL 1.1 -store = TripletStore( - backend="blazegraph", - endpoint="http://localhost:9999/blazegraph/sparql", - namespace="semantica" -) +page_size = 1000 +offset = 0 -# Apache Jena Fuseki — open source, widely used -store = TripletStore( - backend="jena", - endpoint="http://localhost:3030/dataset/sparql", - update_endpoint="http://localhost:3030/dataset/update" -) - -# RDF4J — enterprise-grade, Eclipse Foundation -store = TripletStore( - backend="rdf4j", - server_url="http://localhost:8080/rdf4j-server", - repository_id="semantica" -) +while True: + results = store.sparql(f""" + SELECT ?s ?p ?o WHERE {{ + ?s ?p ?o . + }} + ORDER BY ?s + LIMIT {page_size} OFFSET {offset} + """) + if not results: + break + process_batch(results) + offset += page_size ``` -## Backend Comparison - -| Backend | License | Query Language | Best For | -| ------- | ------- | -------------- | -------- | -| Blazegraph | Open source | SPARQL 1.1 | Wikidata-style workloads | -| Apache Jena | Apache 2.0 | SPARQL 1.1 | General RDF, OWL reasoning | -| RDF4J | Eclipse 1.0 | SPARQL 1.1 | Enterprise, Java ecosystems | - -## Import and Export - -```python -# Import from file -store.import_file("ontology.ttl", format="turtle") -store.import_file("data.jsonld", format="json-ld") -store.import_file("triples.nt", format="nt") - -# Export to file -store.export("output.ttl", format="turtle") -store.export("output.nt", format="nt") -store.export("output.xml", format="xml") -``` - -## Graph Management +## Named Graph Management ```python # Named graphs — store triples in isolated contexts @@ -168,6 +302,32 @@ store.import_file("output.ttl", format="turtle") results = store.sparql("SELECT * WHERE { ?s ?p ?o } LIMIT 10") ``` +## Tips and Common Pitfalls + + + **Use `InMemoryTripletStore` for unit tests, Jena or Blazegraph for production.** The in-memory backend requires zero server setup and is safe for CI. It does not persist across process restarts — switch to a server-backed store before deploying. No code changes needed, just the `backend=` parameter. + + + + **Paginate large SPARQL result sets.** A `SELECT * WHERE { ?s ?p ?o }` against a million-triple store can return gigabytes of data. Always include `LIMIT` and `OFFSET` in exploratory queries, and iterate with `page_size` when you need full coverage. Unbounded queries against large stores will OOM or timeout. + + + + **Use named graphs to isolate sources.** `store.add_triplet(..., graph="http://example.org/source_A")` puts triples into a named graph. You can then query just that source, merge selectively, or clear it without touching other data — far safer than mixing all triples into the default graph. + + + + **Register namespace prefixes before querying.** `NamespacePrefixManager` lets you write `?s ex:name ?o` instead of `?s ?o`. Without prefixes, SPARQL queries against domain ontologies become unreadable and error-prone. + + + + **Enable OWL reasoning only when you need it.** `reasoner="OWL"` significantly increases query planning overhead. For simple triple lookups or SPARQL SELECT queries, leave reasoning off (`reasoner=None`) and enable it only for queries that depend on class hierarchies or property chains. + + + + **Export to Turtle before migrating backends.** If you need to move from Jena to Blazegraph (or any other store), `store.export("dump.ttl", format="turtle")` produces a portable file that any SPARQL store can import. Don't rely on backend-specific dump formats. + + Export knowledge graphs to RDF formats. diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 8618b687..2da34230 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -8,33 +8,61 @@ icon: "wrench" ## What You Get -- **Logging** — structured logging with `@log_performance` decorator and quality metrics -- **Validation** — `validate_entity`, `validate_config` with a typed `ValidationError` -- **Progress tracking** — `track_progress` wraps any iterable with console, Jupyter, or file output -- **Helper functions** — `clean_text`, `hash_data`, `safe_filename` -- **Exception hierarchy** — `SemanticaError` → `ValidationError`, `ProcessingError` + + + Structured logging with `@log_performance` decorator and quality metrics via environment variables. + + + `validate_entity` and `validate_config` with a typed `ValidationError` carrying field and value context. + + + `track_progress` wraps any iterable — auto-detects console vs Jupyter for the right renderer. + + + `clean_text`, `hash_data`, `safe_filename`, and nested dict utilities used throughout the framework. + + + `SemanticaError` → `ValidationError`, `ProcessingError` — typed exceptions for targeted recovery. + + + `read_json_file` with `ProcessingError` on failure — no boilerplate try/except around JSON I/O. + + ## Logging -```python -from semantica.utils import setup_logging, get_logger, log_performance + + + ```python + from semantica.utils import setup_logging, get_logger -setup_logging(level="INFO") # "DEBUG" | "INFO" | "WARNING" | "ERROR" -logger = get_logger(__name__) + setup_logging(level="INFO") # "DEBUG" | "INFO" | "WARNING" | "ERROR" + logger = get_logger(__name__) + ``` + + + ```python + from semantica.utils import log_performance, log_execution_time -@log_performance -def process_data(data): - logger.info(f"Processing {len(data)} items") - # Decorator automatically logs function name, duration, and any exception -``` + @log_performance + def process_data(data): + logger.info(f"Processing {len(data)} items") + # Logs function name, duration, and any exception automatically -Configure via environment variables: - -```bash -export SEMANTICA_LOG_LEVEL=DEBUG -export SEMANTICA_LOG_FORMAT=json # "json" | "text" -export SEMANTICA_PROGRESS_BAR=true -``` + @log_execution_time + def expensive_step(data): + ... + # Logs: "expensive_step completed in 2.34s" + ``` + + + ```bash + export SEMANTICA_LOG_LEVEL=DEBUG + export SEMANTICA_LOG_FORMAT=json # "json" | "text" + export SEMANTICA_PROGRESS_BAR=true + ``` + + ## Validation @@ -72,9 +100,8 @@ for item in track_progress(items, desc="Processing documents"): ``` Supports: - - **Console** — tqdm progress bar with ETA -- **Jupyter** — notebook-compatible widget +- **Jupyter** — notebook-compatible widget (auto-detected) - **File** — write progress to a log file ## Helper Functions @@ -83,17 +110,46 @@ Supports: from semantica.utils import clean_text, hash_data, safe_filename # Normalize whitespace and strip control characters -clean = clean_text(" Hello World ") # → "Hello World" +clean = clean_text(" Hello World ") # → "Hello World" -# Deterministic SHA-256 hash of any serializable object -uid = hash_data({"key": "value"}) # → hex digest string +# Deterministic SHA-256 hash of any JSON-serializable object +uid = hash_data({"key": "value"}) # → hex digest string # Sanitize a string for use as a filename -fname = safe_filename("My File?.txt") # → "My_File_.txt" +fname = safe_filename("My File?.txt") # → "My_File_.txt" +``` + +## Nested Dict Utilities + +Helper functions for deep configuration access — used extensively inside `Config` and `ConfigManager`: + +```python +from semantica.utils import get_nested_value, set_nested_value, merge_dicts + +config = { + "processing": {"batch_size": 32, "max_workers": 4}, + "llm": {"provider": "groq", "model": "llama-3.3-70b-versatile"}, +} + +# Dot-notation read — returns default if key path is absent +batch = get_nested_value(config, "processing.batch_size", default=16) +# → 32 + +# Dot-notation write +set_nested_value(config, "processing.batch_size", 64) + +# Deep merge — nested keys are merged recursively +base = {"a": {"x": 1, "y": 2}, "b": 3} +overrides = {"a": {"y": 99, "z": 4}, "c": 5} +merged = merge_dicts(base, overrides, deep=True) +# → {"a": {"x": 1, "y": 99, "z": 4}, "b": 3, "c": 5} ``` ## Exception Hierarchy + + + ```python from semantica.utils import SemanticaError, ValidationError, ProcessingError @@ -101,7 +157,7 @@ try: run_pipeline(data) except ValidationError as e: # Input data did not pass schema validation - logger.error(f"Validation failed: {e}") + logger.error(f"Validation failed at field '{e.field}': {e.message}") except ProcessingError as e: # Failure during extraction or graph construction logger.error(f"Processing failed at step {e.step}: {e}") @@ -116,6 +172,40 @@ except SemanticaError as e: | `ValidationError` | Input data failed schema or type validation | | `ProcessingError` | Failure during extraction, graph build, or pipeline step | + + + +## File Utilities + +```python +from semantica.utils import read_json_file + +# Read and parse a JSON file — raises ProcessingError on failure +config = read_json_file("config.json") +``` + +## Tips and Common Pitfalls + + + **Call `setup_logging(level="INFO")` once at application startup.** Without it, Semantica falls back to Python's root logger, which may be silent or misconfigured. Call it before importing other Semantica modules to capture initialization messages. + + + + **Use `@log_performance` on expensive functions.** The decorator logs function name, duration, and any raised exception automatically — no manual `time.time()` bookkeeping needed. Essential for profiling multi-step pipelines where one step is a hidden bottleneck. + + + + **`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string — suitable as a cache key or idempotency token in pipeline steps. + + + + **Catch `SemanticaError` as the broadest exception net.** All framework errors inherit from `SemanticaError`, so `except SemanticaError` catches validation failures, processing errors, and everything in between. Use specific subclasses for targeted recovery logic. + + + + **`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment — the same call works in both. + + Framework orchestration that uses Utils internally. diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 881e5df1..166bb364 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -8,33 +8,70 @@ icon: "database" ## What You Get -- **`VectorStore`** — unified interface across all backends -- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory -- **Hybrid search** — combine dense vector similarity with sparse keyword/metadata filtering -- **Metadata filtering** — rich filter expressions: `eq`, `ne`, `gt`, `lt`, `in`, `contains`, `$and`, `$or` -- **Namespace isolation** — multi-tenant support via isolated namespaces -- **Batch operations** — bulk add, delete, and metadata updates + + + Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector. + + + Combine dense vector similarity with sparse keyword/BM25 filtering and configurable fusion strategies. + + + Rich metadata indexing and schema management — query by field values without a vector. + + + Multi-tenant namespace isolation — structural separation, not just metadata filters. + + + Bulk add, delete, and metadata updates — automatically chunked for memory efficiency. + + + Flat, IVF, HNSW, and PQ index types with full configuration control. + + -## Basic Usage +## Quick Start -```python -from semantica.vector_store import VectorStore + + + ```python + from semantica.vector_store import VectorStore -# In-memory (development) -store = VectorStore(backend="inmemory", dimension=768) + # In-memory (development) + store = VectorStore(backend="inmemory", dimension=768) -# FAISS (local, production) -store = VectorStore(backend="faiss", dimension=768, index_path="store.faiss") - -# Add vectors -store.add_vectors(embeddings=embeddings, ids=["doc1", "doc2"], metadata=[{}, {}]) - -# Semantic search -results = store.search(query_vector, top_k=10) -for r in results: - print(f"{r['id']} — score: {r['score']:.3f}") - print(f" metadata: {r['metadata']}") -``` + # FAISS (local production — persists to disk) + store = VectorStore(backend="faiss", dimension=768, index_path="store.faiss") + ``` + + + ```python + store.add_vectors( + embeddings=embeddings, + ids=["doc1", "doc2"], + metadata=[{"title": "Document 1"}, {"title": "Document 2"}] + ) + ``` + + + ```python + results = store.search(query_vector, top_k=10) + for r in results: + print(f"{r['id']} — score: {r['score']:.3f}") + print(f" metadata: {r['metadata']}") + ``` + + + ```python + # Equality, range, and set filters + results = store.search(query_vector, filters={ + "$and": [ + {"category": "research"}, + {"year": {"$gte": 2022}} + ] + }) + ``` + + ## Backends @@ -45,7 +82,7 @@ for r in results: store = VectorStore( backend="faiss", dimension=768, - index_type="IVF", # "Flat" | "IVF" | "HNSW" + index_type="IVF", # "Flat" | "IVF" | "HNSW" | "PQ" index_path="store.faiss" ) ``` @@ -197,6 +234,150 @@ store.update_metadata("doc1", {"status": "archived", "reviewed": True}) | PgVector | PostgreSQL | No | Limited | Postgres-native integration | | In-memory | Process | No | No | Development, testing | +## HybridSearch + +`HybridSearch` is the low-level class behind `store.hybrid_search()` — use it directly when you need custom result fusion logic: + +```python +from semantica.vector_store import HybridSearch, VectorStore + +store = VectorStore(backend="faiss", dimension=768) +hybrid = HybridSearch(vector_store=store) + +results = hybrid.search( + query_vector=query_embedding, + query_text="machine learning frameworks", + top_k=20, + vector_weight=0.7, # weight for vector similarity leg + keyword_weight=0.3, # weight for BM25/keyword leg + fusion="rrf", # "rrf" (Reciprocal Rank Fusion) | "weighted_avg" + filters={"category": "research", "year": {"$gte": 2022}}, + deduplicate=True, +) + +for r in results: + print(f"{r['id']} vector_score={r['vector_score']:.3f} final_score={r['score']:.3f}") +``` + +| Fusion strategy | Description | +| --------------- | ----------- | +| `rrf` | Reciprocal Rank Fusion — rank-based combination, robust to score scale differences | +| `weighted_avg` | Weighted average of normalised scores — requires `vector_weight` + `keyword_weight` = 1.0 | + +## MetadataStore + +`MetadataStore` manages structured metadata attached to vectors — query by field values without a vector: + +```python +from semantica.vector_store import MetadataStore + +meta_store = MetadataStore() + +meta_store.register_schema({ + "author": "str", + "year": "int", + "category": "str", + "score": "float", +}) + +meta_store.add("doc1", {"author": "Alice", "year": 2024, "category": "research"}) +meta_store.add("doc2", {"author": "Bob", "year": 2023, "category": "review"}) + +results = meta_store.filter({"category": "research", "year": {"$gte": 2023}}) +meta = meta_store.get("doc1") +meta_store.update("doc1", {"score": 0.92}) +``` + +## NamespaceManager + +Isolates vector collections per tenant, project, or model version: + +```python +from semantica.vector_store import NamespaceManager, VectorStore + +base_store = VectorStore(backend="faiss", dimension=768) +ns_manager = NamespaceManager(vector_store=base_store) + +ns_manager.create_namespace("tenant_a", description="Customer A data") +ns_manager.create_namespace("tenant_b", description="Customer B data") + +ns_manager.add_vectors("tenant_a", embeddings_a, ids_a, metadata_a) +ns_manager.add_vectors("tenant_b", embeddings_b, ids_b, metadata_b) + +# Search is scoped — tenant_a never sees tenant_b's data +results = ns_manager.search("tenant_a", query_vector, top_k=10) + +for ns in ns_manager.list_namespaces(): + print(f"{ns['name']}: {ns['vector_count']} vectors") + +ns_manager.delete_namespace("tenant_a") +``` + +## FAISS Index Type Reference + +| Index | Memory | Speed | Accuracy | When to Use | +| ----- | ------ | ----- | -------- | ----------- | +| `Flat` | High | Slow | Exact (100%) | < 100K vectors, correctness critical | +| `IVF` | Medium | Fast | ~95–98% | 100K–10M vectors, good balance | +| `HNSW` | Medium-High | Very fast | ~97–99% | Low latency, production retrieval | +| `PQ` | Low | Fast | ~90–95% | Millions of vectors, memory-constrained | + +```python +# Flat — brute-force exact search +store = VectorStore(backend="faiss", dimension=768, index_type="Flat") + +# IVF — inverted file index with nlist clusters +store = VectorStore(backend="faiss", dimension=768, index_type="IVF", nlist=100) + +# HNSW — hierarchical navigable small world graph +store = VectorStore(backend="faiss", dimension=768, index_type="HNSW", M=16, ef_construction=200) + +# PQ — product quantization for memory efficiency +store = VectorStore(backend="faiss", dimension=768, index_type="PQ", m=8) +``` + +## Similarity Metrics + +| Metric | Constructor arg | Distance → Similarity | Best For | +| ------ | --------------- | --------------------- | -------- | +| Cosine | `metric="cosine"` | `1 - cosine_distance` | Text, embeddings | +| L2 (Euclidean) | `metric="l2"` | `1 / (1 + distance)` | Image features | +| Inner Product | `metric="ip"` | raw dot product | Recommendation systems | + +```python +store = VectorStore(backend="faiss", dimension=768, metric="cosine") +``` + +## Tips and Common Pitfalls + + + **Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size — `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time, not at store creation. + + + + **Use `Flat` index only for small datasets.** Flat (brute-force) search has perfect recall but O(n) query time. At 500K+ vectors, switch to `IVF` or `HNSW` — they sacrifice less than 5% recall for 100–1000x speedup. + + + + **Don't search without normalizing first.** If you disabled `normalize=True` in `EmbeddingGenerator`, compute cosine similarity with `metric="cosine"` (which normalizes internally). Raw dot product on un-normalized vectors produces incorrect similarity rankings. + + + + **Use `hybrid_search` for precision-sensitive workloads.** Pure vector search finds semantically similar results but may miss keyword matches important to the user. Hybrid search (vector + BM25) combines both signals — especially valuable for domain-specific terminology. + + + + **Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and leaks data if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation). + + + + **Persist FAISS indexes to disk.** `VectorStore(backend="faiss", index_path="store.faiss")` saves the index to disk on each write. Without a path, the index is in-memory only and is lost on process exit. + + + + **Update metadata without re-embedding.** `store.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content. + + Generate the vectors stored here. diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index f16fe818..2a3f9027 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -8,130 +8,240 @@ icon: "chart-bar" ## What You Get -- **`GraphVisualizer`** — interactive HTML (PyVis) and static image (Matplotlib) graph rendering -- **`OntologyVisualizer`** — class hierarchy and property relationship visualization -- **`EmbeddingVisualizer`** — UMAP, t-SNE, and PCA dimensionality reduction plots -- **`TemporalVisualizer`** — timeline views and animated graph evolution -- **`DistanceVisualizer`** — ego-mode neighborhood views and distance matrix heatmaps (v0.5.0) + + + Interactive HTML (PyVis) and static image (Matplotlib) graph rendering with layout options. + + + Class hierarchy and property relationship visualization from any OntologyManager. + + + UMAP, t-SNE, and PCA dimensionality reduction plots for embedding cluster analysis. + + + Timeline views, animated evolution, snapshot comparison, and temporal pattern highlights. + + + Ego-mode neighborhood views and N×N distance matrix heatmaps from Distance Intelligence. + + + Centrality rankings, community-colored graphs, and degree distribution histograms. + + -## GraphVisualizer +## Quick Start + + + + ```python + from semantica.visualization import GraphVisualizer + + viz = GraphVisualizer() + + # Interactive HTML — opens in browser, supports hover and click + viz.visualize(graph, output="graph.html") + ``` + + + ```python + viz.visualize( + graph, + output="graph.html", + layout="force_directed", # "force_directed" | "hierarchical" | "circular" | "spring" + node_color_by="type", # color nodes by entity type attribute + edge_label="relation", # show edge relationship labels + color_scheme="vibrant", # color palette — see Color Schemes section + max_nodes=500, # limit rendering for large graphs + ) + ``` + + + ```python + # Static PNG — for reports and embedding in documents + viz.visualize(graph, output="graph.png", dpi=150) + + # Vector SVG — for publications and scalable diagrams + viz.visualize(graph, output="graph.svg") + + # PDF — for print or compliance reports + viz.visualize(graph, output="graph.pdf") + ``` + + + +## Visualizers + + + + Interactive and static knowledge graph rendering: + + ```python + from semantica.visualization import GraphVisualizer + + viz = GraphVisualizer() + + # Interactive HTML + viz.visualize(graph, output="graph.html") + + # Static PNG with custom DPI + viz.visualize(graph, output="graph.png", backend="matplotlib", dpi=150) + + # Display inline (Jupyter or default browser) + viz.show(graph) + ``` + + **Layout options:** + + | Layout | Description | Best For | + | ------ | ----------- | -------- | + | `force_directed` | Physics simulation — clusters emerge naturally | General graphs | + | `hierarchical` | Top-down tree layout | Taxonomies, org charts | + | `circular` | Nodes on a circle, edges as chords | Small dense graphs | + | `spring` | Spring-force layout (Fruchterman-Reingold) | Medium graphs | + + + Visualize class hierarchies and property relationships: + + ```python + from semantica.visualization import OntologyVisualizer + + viz = OntologyVisualizer() + + # Full ontology graph — classes, properties, and constraints + viz.visualize(ontology, output="ontology.html") + + # Class hierarchy only — cleaner for large ontologies + viz.visualize_hierarchy(ontology, output="hierarchy.html") + ``` + + + Project high-dimensional embeddings into 2D for cluster analysis: + + ```python + from semantica.visualization import EmbeddingVisualizer + + viz = EmbeddingVisualizer() + + viz.visualize( + embeddings=embeddings, + labels=labels, + output="embeddings.html", + method="umap", # "umap" | "tsne" | "pca" + ) + ``` + + | Method | Speed | Preserves | Best For | + | ------ | ----- | --------- | -------- | + | `umap` | Fast | Global + local structure | Large datasets, cluster discovery | + | `tsne` | Medium | Local structure | Tight cluster separation | + | `pca` | Very fast | Variance | Quick overview, linear structure | + + + Visualize how a knowledge graph changes over time: + + ```python + from semantica.visualization import TemporalVisualizer + from datetime import datetime + + viz = TemporalVisualizer() + + # Static timeline of additions and removals + viz.visualize_timeline(temporal_kg, output="timeline.html") + + # Animated evolution — one frame per time step + viz.animate(temporal_kg, output="evolution.html", fps=2) + + # Side-by-side snapshot comparison + snap_a = temporal_kg.at(datetime(2020, 1, 1)) + snap_b = temporal_kg.at(datetime(2023, 1, 1)) + viz.compare_snapshots(snap_a, snap_b, output="snapshot_diff.html") + + # Pattern visualization — highlight recurring temporal patterns + viz.visualize_patterns(temporal_kg, pattern_type="recurrence", output="patterns.html") + ``` + + + Semantic neighborhood and distance matrix visualization from Distance Intelligence: + + ```python + from semantica.visualization import DistanceVisualizer + + viz = DistanceVisualizer() + + # Ego-mode: neighborhood of one node colored by distance band + viz.visualize_ego( + graph, + center_node="Apple Inc.", + output="ego.html", + radius=0.5, # semantic distance radius + ) + + # N×N distance matrix heatmap + viz.visualize_distance_matrix( + matrix=distance_matrix, + labels=node_labels, + output="distance_heatmap.html", + ) + ``` + + + Visualize graph analytics results — centrality, communities, and degree distribution: + + ```python + from semantica.visualization import AnalyticsVisualizer + from semantica.kg import CentralityCalculator, CommunityDetector + + calc = CentralityCalculator() + centrality = calc.calculate_all_centrality(kg) + + detector = CommunityDetector() + communities = detector.detect_communities(kg, algorithm="louvain") + + viz = AnalyticsVisualizer() + + # Bar chart of top-N nodes by centrality measure + viz.visualize_centrality(centrality, metric="pagerank", top_k=20, output="centrality.html") + + # Community-colored graph + viz.visualize_communities(kg, communities, output="communities.html") + + # Degree distribution histogram + viz.visualize_degree_distribution(kg, output="degree_dist.html") + + # Combined analytics dashboard + viz.visualize_analytics_dashboard( + kg, centrality=centrality, communities=communities, + output="analytics_dashboard.html", + ) + ``` + + + +## Color Schemes + +All visualizers accept a `color_scheme` parameter: ```python -from semantica.visualization import GraphVisualizer - -viz = GraphVisualizer() - -# Interactive HTML — opens in browser, supports hover and click -viz.visualize(graph, output="graph.html") - -# Static image — for reports and export -viz.visualize(graph, output="graph.png", backend="matplotlib") - -# Display inline (Jupyter or default browser) -viz.show(graph) +viz.visualize(graph, output="graph.html", color_scheme="vibrant") ``` -### Layout and Styling Options - -```python -viz.visualize( - graph, - output="graph.html", - layout="force_directed", # "force_directed" | "hierarchical" | "circular" | "spring" - node_color_by="type", # color nodes by entity type attribute - edge_label="relation", # show edge relationship labels - max_nodes=500 # limit rendering for large graphs -) -``` - -### Layout Options - -| Layout | Description | Best For | +| Scheme | Description | Best For | | ------ | ----------- | -------- | -| `force_directed` | Physics simulation — clusters emerge naturally | General graphs | -| `hierarchical` | Top-down tree layout | Taxonomies, org charts | -| `circular` | Nodes on a circle, edges as chords | Small dense graphs | -| `spring` | Spring-force layout (Fruchterman-Reingold) | Medium graphs | +| `default` | Blue-grey palette | General use | +| `vibrant` | High-contrast, saturated colours | Presentations | +| `pastel` | Soft, muted tones | Light backgrounds | +| `dark` | Dark background with bright nodes | Dark-mode dashboards | +| `light` | White background, thin edges | Publications, print | +| `colorblind` | Okabe-Ito safe palette | Accessibility | -## OntologyVisualizer +## Export Formats -Visualize class hierarchies and property relationships: - -```python -from semantica.visualization import OntologyVisualizer - -viz = OntologyVisualizer() - -# Full ontology graph -viz.visualize(ontology, output="ontology.html") - -# Class hierarchy only -viz.visualize_hierarchy(ontology, output="hierarchy.html") -``` - -## EmbeddingVisualizer - -Project high-dimensional embeddings into 2D for cluster analysis: - -```python -from semantica.visualization import EmbeddingVisualizer - -viz = EmbeddingVisualizer() - -viz.visualize( - embeddings=embeddings, - labels=labels, - output="embeddings.html", - method="umap" # "umap" | "tsne" | "pca" -) -``` - -| Method | Speed | Preserves | Best For | -| ------ | ----- | --------- | -------- | -| `umap` | Fast | Global + local structure | Large datasets, cluster discovery | -| `tsne` | Medium | Local structure | Tight cluster separation | -| `pca` | Very fast | Variance | Quick overview, linear structure | - -## TemporalVisualizer - -Visualize how a knowledge graph changes over time: - -```python -from semantica.visualization import TemporalVisualizer - -viz = TemporalVisualizer() - -# Static timeline of additions and removals -viz.visualize_timeline(temporal_kg, output="timeline.html") - -# Animated evolution — one frame per time step -viz.animate(temporal_kg, output="evolution.html", fps=2) -``` - -## DistanceVisualizer (v0.5.0) - -Semantic neighborhood and distance matrix visualization from Distance Intelligence: - -```python -from semantica.visualization import DistanceVisualizer - -viz = DistanceVisualizer() - -# Ego-mode: neighborhood of one node colored by distance band -viz.visualize_ego( - graph, - center_node="Apple Inc.", - output="ego.html", - radius=0.5 # semantic distance radius -) - -# N×N distance matrix heatmap -viz.visualize_distance_matrix( - matrix=distance_matrix, - labels=node_labels, - output="distance_heatmap.html" -) -``` +| Format | Interactive | Scalable | Best For | +| ------ | ----------- | -------- | -------- | +| `.html` | Yes | N/A | Web dashboards, exploratory analysis | +| `.png` | No | No | Reports, Jupyter notebooks | +| `.svg` | No | Yes | Publications, slide decks | +| `.pdf` | No | Yes | Print, compliance exports | ## Graph Explorer (Full Dashboard) @@ -146,6 +256,32 @@ start_explorer(graph=kg, port=8080) See the [Explorer reference](explorer) for the full feature set and REST API. +## Tips and Common Pitfalls + + + **Use `max_nodes=500` for large graphs.** Force-directed layouts become unreadable and very slow above ~1,000 nodes. Limit with `max_nodes=500` or filter to a subgraph (e.g., top 100 nodes by PageRank) before visualizing. + + + + **HTML output is always the best starting point.** Interactive HTML lets you zoom, pan, hover for details, and hide node types — giving you orders of magnitude more exploratory power than a static PNG. Only export to PNG/SVG/PDF when embedding in a report. + + + + **Use `color_scheme="colorblind"` in publications and dashboards.** The Okabe-Ito palette is readable for everyone, including the ~8% of male readers who are red-green colorblind. Reserve `vibrant` for internal presentations only. + + + + **UMAP is faster than t-SNE at scale.** For embedding spaces with >5,000 points, UMAP completes in seconds; t-SNE may take minutes. Both produce good cluster separation — use UMAP for exploratory speed, t-SNE for final publication-quality plots. + + + + **`TemporalVisualizer.animate()` can produce large files.** Animated HTML files include all frames and can reach dozens of MB for long time series. Use `fps=1` or reduce the number of time steps for a manageable file size. + + + + **For interactive dashboards, prefer Explorer.** `GraphVisualizer.visualize()` generates a self-contained HTML file. `start_explorer()` gives a full live web app with search, filtering, path-finding, and REST API. Use Explorer for team exploration, Visualizer for standalone report embeds. + + The graph being visualized. From d206a10bc78c5a33649d53718536893993e32298 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sat, 23 May 2026 23:06:49 +0530 Subject: [PATCH 02/11] docs: apply Mintlify component overhaul to index.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The Problem section: flat bullet list → CardGroup (5 problem cards with icons) - The Solution section: flat bullet list → CardGroup (6 solution cards) - Start Here section: plain prose → Steps (4-step onboarding flow) - Built for High-Stakes Domains: plain prose → CardGroup (6 domain cards) - Why Semantica: plain prose → CardGroup cols={3} (3 value proposition cards) - Module Reference table: updated descriptions for seed, evals, core, utils, llms, export to match v0.5.0 source - LLM provider class names corrected: OpenAIProvider → OpenAI, AnthropicProvider → Anthropic, OllamaProvider → Ollama --- docs/index.md | 143 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 100 insertions(+), 43 deletions(-) diff --git a/docs/index.md b/docs/index.md index ec684db1..027ba7f3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,11 +13,23 @@ description: "The Accountability and Context Layer for AI — Context Graphs · AI agents today are powerful but not trustworthy. Five structural gaps make them impossible to deploy in regulated environments: -- **No memory structure.** Agents store embeddings, not meaning. There's no way to ask *why* something was recalled or trace a fact to its source. -- **No decision trail.** Agents act continuously but record nothing. When something breaks, there's no history to debug or audit. -- **No provenance.** Outputs can't be traced back to source facts. In healthcare, finance, and legal, this is a hard compliance blocker. -- **No reasoning transparency.** Black-box answers with zero explanation of how a conclusion was reached. -- **No conflict detection.** Contradictory facts silently coexist in vector stores, producing unpredictable and inconsistent outputs. + + + Agents store embeddings, not meaning. There's no way to ask *why* something was recalled or trace a fact to its source. + + + Agents act continuously but record nothing. When something breaks, there's no history to debug or audit. + + + Outputs can't be traced back to source facts. In healthcare, finance, and legal, this is a hard compliance blocker. + + + Black-box answers with zero explanation of how a conclusion was reached. + + + Contradictory facts silently coexist in vector stores, producing unpredictable and inconsistent outputs. + + These aren't edge cases. They're why AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch. @@ -25,12 +37,26 @@ These aren't edge cases. They're why AI cannot be deployed in healthcare, financ Semantica is the **accountability and context layer** you add on top of your existing AI stack. Not a replacement for LangChain or LlamaIndex — the infrastructure that makes their outputs trustworthy. -- **Context Graphs** — a structured, queryable graph of everything your agent knows, decides, and reasons about. Persistent across runs. -- **Decision Intelligence** — every decision is a first-class object: recorded, causally linked, searchable by precedent, and analyzable for downstream impact. -- **Full Provenance** — every fact links back to its source. W3C PROV-O compliant. Full lineage from ingestion to inference. -- **Reasoning Engines** — forward chaining, Rete, deductive, abductive, SPARQL, Datalog. Explainable paths, not black boxes. -- **Temporal Intelligence** — point-in-time queries, Allen interval algebra, temporal provenance, OWL-Time export. -- **Ontology Hub** — visual editor, SHACL Studio, alignment authoring, health dashboard. Full ontology lifecycle in the browser. + + + A structured, queryable graph of everything your agent knows, decides, and reasons about. Persistent across runs. + + + Every decision is a first-class object: recorded, causally linked, searchable by precedent, and analyzable for downstream impact. + + + Every fact links back to its source. W3C PROV-O compliant. Full lineage from ingestion to inference. + + + Forward chaining, Rete, deductive, abductive, SPARQL, Datalog. Explainable paths, not black boxes. + + + Point-in-time queries, Allen interval algebra, temporal provenance, OWL-Time export. + + + Visual editor, SHACL Studio, alignment authoring, health dashboard. Full ontology lifecycle in the browser. + + Works alongside any LLM provider and any agent framework. @@ -47,13 +73,13 @@ pip install semantica ```python OpenAI from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore -from semantica.llms import OpenAIProvider +from semantica.llms import OpenAI context = AgentContext( vector_store=VectorStore(backend="faiss", dimension=1536), knowledge_graph=ContextGraph(advanced_analytics=True), decision_tracking=True, - llm=OpenAIProvider(model="gpt-4o"), + llm=OpenAI(model="gpt-4o"), ) context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%") @@ -73,13 +99,13 @@ influence = context.analyze_decision_influence(decision_id) ```python Anthropic from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore -from semantica.llms import AnthropicProvider +from semantica.llms import Anthropic context = AgentContext( vector_store=VectorStore(backend="faiss", dimension=1024), knowledge_graph=ContextGraph(advanced_analytics=True), decision_tracking=True, - llm=AnthropicProvider(model="claude-opus-4-7"), + llm=Anthropic(model="claude-opus-4-7"), ) context.store("Claude excels at long-context reasoning and code generation") @@ -98,13 +124,13 @@ precedents = context.find_precedents("document analysis model", limit=5) ```python Ollama (Local) from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore -from semantica.llms import OllamaProvider +from semantica.llms import Ollama context = AgentContext( vector_store=VectorStore(backend="faiss", dimension=768), knowledge_graph=ContextGraph(advanced_analytics=True), decision_tracking=True, - llm=OllamaProvider(model="llama3.2", base_url="http://localhost:11434"), + llm=Ollama(model="llama3.2", base_url="http://localhost:11434"), ) # Fully local — no data leaves your infrastructure @@ -173,7 +199,23 @@ pip install semantica==0.5.0 ## Start Here -If you're new to Semantica, install first and then open [Quickstart](quickstart). Use [Core Concepts](concepts) for the mental model, or jump to [API Reference](reference/context) when you need exact details. + + + ```bash + pip install semantica + ``` + See [Installation](installation) for optional extras and environment setup. + + + Build a complete knowledge graph pipeline — ingest, extract, build, query — in [5 minutes](quickstart). + + + [Core Concepts](concepts) explains knowledge graphs, GraphRAG, provenance, and decision intelligence. Read this before the API reference. + + + Every module has a dedicated [reference page](reference/context) with class docs, parameter tables, and runnable examples. + + @@ -255,45 +297,60 @@ If you're new to Semantica, install first and then open [Quickstart](quickstart) | `semantica.mcp_server` | MCP stdio server — 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline | | `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector | | `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune | -| `semantica.triplet_store` | In-memory and persistent RDF triple store | +| `semantica.triplet_store` | In-memory and persistent RDF triple store with SPARQL | | `semantica.ingest` | Files, web, feeds, databases, Snowflake, Parquet, XML, MCP | | `semantica.parse` | Document parsing — PDF, DOCX, HTML, PPTX, Docling layout analysis | | `semantica.split` | Text chunking — sentence, paragraph, token, semantic boundary strategies | | `semantica.normalize` | Text normalization, entity canonicalization, whitespace and encoding cleanup | -| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE | +| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings | | `semantica.pipeline` | Pipeline DSL, parallel workers, retry policies, failure handling | -| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, OWL, graph formats | -| `semantica.visualization` | Programmatic graph rendering — force, hierarchical, circular layouts | +| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, OWL, Arrow, GraphML, GEXF, DOT | +| `semantica.visualization` | Programmatic graph rendering — force, hierarchical, circular, spring layouts | | `semantica.deduplication` | Entity deduplication v1/v2, similarity scoring, blocking, merging | | `semantica.conflicts` | Conflict detection and resolution across overlapping knowledge sources | | `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails | | `semantica.change_management` | Version control with SHA-256 checksums, diff, rollback | -| `semantica.llms` | Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM | -| `semantica.seed` | Deterministic data seeding and synthetic graph generation for tests | -| `semantica.evals` | Evaluation harness — precision, recall, F1 for extraction and reasoning | -| `semantica.core` | Core data models, base classes, shared type definitions | -| `semantica.utils` | Shared utilities — ID generation, date parsing, schema helpers | +| `semantica.llms` | Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, HuggingFace | +| `semantica.seed` | Foundation graph seeding from CSV, JSON, SQL, API, and RDF sources | +| `semantica.evals` | Evaluation harness — KG quality, extraction F1, pipeline benchmarking, regression tracking | +| `semantica.core` | Orchestration, ConfigManager, LifecycleManager, PluginRegistry, MethodRegistry | +| `semantica.utils` | Logging, validation, progress tracking, hash utilities, nested dict helpers | ## Built for High-Stakes Domains Where every decision must be accountable and mistakes have real consequences: -**Healthcare & Life Sciences** — clinical decision support, drug interaction graphs, patient safety audit trails, HIPAA compliance. - -**Finance & Risk** — fraud detection graphs, SOX/GDPR/MiFID II compliance, risk assessment trails. - -**Legal & Compliance** — evidence-backed research, contract analysis, regulatory change tracking. - -**Cybersecurity** — threat attribution graphs, incident response timelines, security audit trails. - -**Government & Defense** — policy decision trails, classified information handling, provenance chains. - -**Critical Infrastructure** — power grids, transportation safety, emergency response coordination. + + + Clinical decision support, drug interaction graphs, patient safety audit trails, HIPAA compliance. + + + Fraud detection graphs, SOX/GDPR/MiFID II compliance, risk assessment trails. + + + Evidence-backed research, contract analysis, regulatory change tracking. + + + Threat attribution graphs, incident response timelines, security audit trails. + + + Policy decision trails, classified information handling, provenance chains. + + + Power grids, transportation safety, emergency response coordination. + + ## Why Semantica? -**Open source, MIT licensed.** No vendor lock-in, no paywalled features. Every line of code is available and forkable. - -**Production ready.** 1,000+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, conflict resolution, and 12 security fixes in v0.5.0. - -**Modular by design.** Import only what you need. Use `NERExtractor` without a graph store. Use `VectorStore` without decision tracking. Every component is independently swappable. + + + No vendor lock-in, no paywalled features. Every line of code is available and forkable. + + + 1,000+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, 12 security fixes in v0.5.0. + + + Import only what you need. Use `NERExtractor` without a graph store. Every component is independently swappable. + + From 689d57b361be70e7d644a83a34d3a93ee6e1430c Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 12:36:21 +0530 Subject: [PATCH 03/11] fix: correct API mismatches in pipeline, ingest, and vector_store docs pipeline.md: - Replace Pipeline().add_step().run() with PipelineBuilder + ExecutionEngine.execute_pipeline() - Fix ValidationResult: result.valid (not is_valid), errors is List[str] not object list - Fix ExecutionResult schema: success/output/metadata/metrics/errors (not PipelineResult) - Fix ExecutionEngine: get_pipeline_status() not get_status(), progress keys completed_steps/total_steps - Fix result.metadata['pipeline_id'] not result.pipeline_id - Fix RetryPolicy: strategy=RetryStrategy.EXPONENTIAL not backoff='exponential' - Fix PipelineSerializer.serialize_pipeline/deserialize_pipeline instead of pipeline.save/load - Fix delta mode to use PipelineBuilder not Pipeline() ingest.md: - Replace S3Ingestor/GCSIngestor/GDriveIngestor (do not exist) with CloudStorageIngestor - Remove MongoIngestor/DuckDBIngestor (do not exist) from docs and tables - Fix Quick Start pipeline step to use PipelineBuilder + ExecutionEngine vector_store.md: - Replace store.hybrid_search() (does not exist) with HybridSearch.search() - Replace store.add_vectors() with store.add_documents() / store.store_vectors() - Replace store.search(query_vector) with store.search_vectors(k=) / store.search(query_str, limit=) - Fix Batch Operations: add_vectors_batch -> add_documents, delete_vectors(vector_ids=), update_vectors() - Fix HybridSearch.search() signature: (query, k, metadata_filter) not (query_vector, query_text, fusion, filters) - Fix MetadataStore: store_metadata/get_metadata/update_metadata/query_metadata (not add/filter/get) - Fix NamespaceManager: add_vector_to_namespace, list_namespaces returns List[str] --- docs/reference/ingest.md | 112 ++++--------- docs/reference/pipeline.md | 288 +++++++++++++++++++++------------ docs/reference/vector_store.md | 218 +++++++++++++++---------- 3 files changed, 354 insertions(+), 264 deletions(-) diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index b5f984ec..af0c9e52 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -22,10 +22,10 @@ icon: "database" Real-time ingestion from Kafka, RabbitMQ, AWS Kinesis, and Apache Pulsar. - S3Ingestor, GCSIngestor, and GDriveIngestor with authentication options. + `CloudStorageIngestor` — unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage. - DBIngestor, SnowflakeIngestor, MongoIngestor, and DuckDBIngestor. + `DBIngestor` (SQL via SQLAlchemy) and `SnowflakeIngestor` for data warehouse queries. @@ -61,18 +61,25 @@ icon: "database" ```python - from semantica.pipeline import Pipeline + from semantica.pipeline import PipelineBuilder, ExecutionEngine from semantica.parse import DocumentParser from semantica.semantic_extract import NERExtractor from semantica.llms import Groq - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + ingestor = FileIngestor() + parser = DocumentParser() + extractor = NERExtractor(method="llm", llm_provider=llm) - pipeline = Pipeline() - pipeline.add_step("ingest", FileIngestor()) - pipeline.add_step("parse", DocumentParser()) - pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) - result = pipeline.run("data/") + builder = PipelineBuilder() + builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file) + builder.add_step("parse", "document_parse", handler=parser.parse) + builder.add_step("extract", "ner_extract", handler=extractor.extract) + builder.connect_steps("ingest", "parse") + builder.connect_steps("parse", "extract") + + pipeline = builder.build("my_pipeline") + result = ExecutionEngine().execute_pipeline(pipeline, data="data/") ``` @@ -213,54 +220,40 @@ icon: "database" ``` - ### S3Ingestor + ### CloudStorageIngestor - Ingest files directly from AWS S3 buckets: + `CloudStorageIngestor` is a unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage: ```python - from semantica.ingest import S3Ingestor + from semantica.ingest import CloudStorageIngestor import os - ingestor = S3Ingestor( + # AWS S3 + ingestor = CloudStorageIngestor( + provider="s3", bucket="my-documents-bucket", prefix="reports/2024/", region="us-east-1", aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"), aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"), - # Or omit credentials to use IAM instance profile + # Omit credentials to use IAM instance profile / environment variables ) sources = ingestor.ingest() - sources = ingestor.ingest(pattern="**/*.pdf") - ``` - ### GCSIngestor - - Ingest files from Google Cloud Storage: - - ```python - from semantica.ingest import GCSIngestor - - ingestor = GCSIngestor( + # Google Cloud Storage + ingestor = CloudStorageIngestor( + provider="gcs", bucket="my-gcs-bucket", prefix="data/", credentials_file="gcp-credentials.json", # or use ADC ) sources = ingestor.ingest() - ``` - ### GDriveIngestor - - Ingest files from Google Drive folders via OAuth 2.0: - - ```python - from semantica.ingest import GDriveIngestor - - ingestor = GDriveIngestor( - credentials_file="oauth_credentials.json", - token_file="token.json", - folder_id="1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs9", - file_types=["pdf", "docx", "txt"], - recursive=True, + # Azure Blob Storage + ingestor = CloudStorageIngestor( + provider="azure", + container="documents", + connection_string=os.getenv("AZURE_STORAGE_CONNECTION_STRING"), ) sources = ingestor.ingest() ``` @@ -295,47 +288,6 @@ icon: "database" sources = ingestor.ingest(query="SELECT * FROM documents") ``` - ### MongoIngestor - - Ingest documents from MongoDB collections: - - ```python - from semantica.ingest import MongoIngestor - import os - - ingestor = MongoIngestor( - connection_string=os.getenv("MONGO_URI"), - database="mydb", - collection="articles", - query={"status": "published", "year": {"$gte": 2022}}, - projection={"title": 1, "body": 1, "author": 1}, - content_field="body", - limit=10000, - ) - sources = ingestor.ingest() - ``` - - ### DuckDBIngestor - - Ingest data from DuckDB databases or directly from Parquet/CSV files via DuckDB SQL: - - ```python - from semantica.ingest import DuckDBIngestor - - # In-memory DuckDB — query a Parquet file directly - ingestor = DuckDBIngestor( - query="SELECT id, text, created_at FROM read_parquet('data/*.parquet') WHERE year >= 2023", - ) - sources = ingestor.ingest() - - # Persistent DuckDB database file - ingestor = DuckDBIngestor( - database_path="analytics.duckdb", - query="SELECT doc_id AS id, content, metadata FROM documents", - content_field="content", - ) - sources = ingestor.ingest() - ``` ### StreamIngestor @@ -459,7 +411,7 @@ method_registry.register("file", "my_format", my_ingestor) - **All ingestors return the same `DataSource` schema.** This means you can mix sources in a single pipeline without any adapter code — `FileIngestor`, `MongoIngestor`, and `StreamIngestor` outputs are all directly composable with `DocumentParser` and `NERExtractor`. + **All ingestors return the same `DataSource` schema.** This means you can mix sources in a single pipeline without any adapter code — `FileIngestor`, `DBIngestor`, and `StreamIngestor` outputs are all directly composable with `DocumentParser` and `NERExtractor`. diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index a1a3c90c..b461cbf6 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -40,9 +40,9 @@ You could wire Semantica modules together with plain Python code. Pipelines add: ## Quick Start - + ```python - from semantica.pipeline import Pipeline + from semantica.pipeline import PipelineBuilder from semantica.ingest import FileIngestor from semantica.parse import DocumentParser from semantica.semantic_extract import NERExtractor @@ -52,11 +52,21 @@ You could wire Semantica modules together with plain Python code. Pipelines add: llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - pipeline = Pipeline() - pipeline.add_step("ingest", FileIngestor()) - pipeline.add_step("parse", DocumentParser()) - pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) - pipeline.add_step("build_kg", GraphBuilder(merge_entities=True)) + ingestor = FileIngestor() + parser = DocumentParser() + extractor = NERExtractor(method="llm", llm_provider=llm) + kg_builder = GraphBuilder(merge_entities=True) + + builder = PipelineBuilder() + builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file) + builder.add_step("parse", "document_parse", handler=parser.parse) + builder.add_step("extract", "ner_extract", handler=extractor.extract) + builder.add_step("build_kg", "graph_build", handler=kg_builder.build) + builder.connect_steps("ingest", "parse") + builder.connect_steps("parse", "extract") + builder.connect_steps("extract","build_kg") + + pipeline = builder.build("my_pipeline") ``` @@ -66,36 +76,46 @@ You could wire Semantica modules together with plain Python code. Pipelines add: validator = PipelineValidator() result = validator.validate_pipeline(pipeline) - if not result.is_valid: - for error in result.errors: - print(f"Error: {error.message} (step: {error.step})") + if not result.valid: + for error in result.errors: # errors is List[str] + print(f"Error: {error}") + for warning in result.warnings: + print(f"Warning: {warning}") ``` - + ```python - result = pipeline.run("data/", show_progress=True) + from semantica.pipeline import ExecutionEngine + + engine = ExecutionEngine() + result = engine.execute_pipeline(pipeline, data="data/") kg = result.output - print(f"Processed: {result.processed_count}") - print(f"Failed: {result.failed_count}") - print(f"Duration: {result.duration_seconds:.1f}s") + print(f"Success: {result.success}") + print(f"Steps executed: {result.metrics['steps_executed']}") + print(f"Steps failed: {result.metrics['steps_failed']}") + print(f"Duration: {result.metrics['execution_time']:.1f}s") ``` ## Parallel Processing -Process documents concurrently across multiple workers: +Set parallelism on the builder and pass `max_workers` to `ExecutionEngine`: ```python -pipeline = Pipeline(workers=4) +from semantica.pipeline import PipelineBuilder, ExecutionEngine -pipeline.add_step("ingest", FileIngestor()) -pipeline.add_step("parse", DocumentParser()) -pipeline.add_step("extract", NERExtractor(), parallel=True, batch_size=10) -pipeline.add_step("build", GraphBuilder()) +builder = PipelineBuilder() +builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file) +builder.add_step("parse", "document_parse", handler=parser.parse) +builder.add_step("extract", "ner_extract", handler=extractor.extract) +builder.add_step("build", "graph_build", handler=kg_builder.build) +builder.set_parallelism(4) -result = pipeline.run("data/") +pipeline = builder.build("parallel_pipeline") +engine = ExecutionEngine(max_workers=4) +result = engine.execute_pipeline(pipeline, data="data/") ``` ## Retry and Error Handling @@ -103,26 +123,31 @@ result = pipeline.run("data/") ```python - from semantica.pipeline import RetryPolicy, FailureHandler, Pipeline + from semantica.pipeline import RetryPolicy, RetryStrategy, FailureHandler, ExecutionEngine - retry = RetryPolicy( + policy = RetryPolicy( max_retries=3, - backoff="exponential", - initial_delay=1.0 # 1s → 2s → 4s + strategy=RetryStrategy.EXPONENTIAL, + initial_delay=1.0, # 1s → 2s → 4s + backoff_factor=2.0 ) - handler = FailureHandler(strategy="skip", log_failures=True) + handler = FailureHandler() + handler.retry_policies["ner_extract"] = policy # keyed by step_type - pipeline = Pipeline(retry_policy=retry, failure_handler=handler) + engine = ExecutionEngine(default_max_retries=3, default_backoff_factor=2.0) + result = engine.execute_pipeline(pipeline, data="data/") ``` Best for transient API errors and rate limits — waits longer with each retry, giving upstream services time to recover. ```python - retry = RetryPolicy( + from semantica.pipeline import RetryPolicy, RetryStrategy + + policy = RetryPolicy( max_retries=3, - backoff="linear", + strategy=RetryStrategy.LINEAR, initial_delay=2.0 # 2s → 4s → 6s ) ``` @@ -131,10 +156,12 @@ result = pipeline.run("data/") ```python - retry = RetryPolicy( + from semantica.pipeline import RetryPolicy, RetryStrategy + + policy = RetryPolicy( max_retries=5, - backoff="fixed", - initial_delay=1.0 # 1s → 1s → 1s → 1s → 1s + strategy=RetryStrategy.FIXED, + initial_delay=1.0 # 1s every attempt ) ``` @@ -159,58 +186,97 @@ result = pipeline.run("data/") ```python - result = pipeline.run("data/", show_progress=True) + from semantica.pipeline import ExecutionEngine + + engine = ExecutionEngine() + result = engine.execute_pipeline(pipeline, data="data/") + # The progress tracker outputs tqdm bars to the console during execution ``` - Displays a live tqdm progress bar in the terminal. Best for scripts and CLI tools. + Displays a live progress bar in the terminal via Semantica's built-in progress tracker. Best for scripts and CLI tools. - + ```python - result = pipeline.run("data/", websocket_port=8080) + from semantica.pipeline import ExecutionEngine + import threading, time + + engine = ExecutionEngine() + + # Run in a background thread, poll progress from main thread + def run(): + engine.execute_pipeline(pipeline, data="data/") + + t = threading.Thread(target=run, daemon=True) + t.start() + + while t.is_alive(): + progress = engine.get_progress(pipeline.name) + if progress: + print(f" {progress['completed_steps']}/{progress['total_steps']} steps — {progress['status']}") + time.sleep(2) ``` - Streams progress events to Knowledge Explorer's dashboard. Best for long-running production jobs where you want a live web UI. + Poll `get_progress()` for live status during execution. ## Pipeline DSL -`PipelineBuilder` provides a fluent chain syntax that reads as a data flow: +`PipelineBuilder` uses `add_step(name, type, **config)` and `connect_steps(from, to)` to define a DAG: ```python -from semantica.pipeline import PipelineBuilder +from semantica.pipeline import PipelineBuilder, ExecutionEngine -pipeline = ( - PipelineBuilder() - .ingest(FileIngestor()) - .parse(DocumentParser()) - .normalize() - .extract(NERExtractor(method="llm", llm_provider=llm)) - .extract_relations(RelationExtractor(method="llm", llm_provider=llm)) - .build_kg(merge_entities=True) - .deduplicate(strategy="semantic_v2") - .export(format="turtle", path="output.ttl") - .build() -) +builder = PipelineBuilder() -result = pipeline.run("data/") +# Add steps — step_type is a string label, handler is the callable invoked at runtime +builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file) +builder.add_step("parse", "document_parse", handler=parser.parse) +builder.add_step("normalize", "text_normalize", handler=normalizer.normalize) +builder.add_step("extract", "ner_extract", handler=extractor.extract) +builder.add_step("rel_extract", "rel_extract", handler=rel_extractor.extract) +builder.add_step("build_kg", "graph_build", handler=kg_builder.build) +builder.add_step("deduplicate", "dedup", handler=deduplicator.deduplicate) +builder.add_step("export", "rdf_export", handler=exporter.export, format="turtle", path="output.ttl") + +# Wire the data flow +builder.connect_steps("ingest", "parse") +builder.connect_steps("parse", "normalize") +builder.connect_steps("normalize", "extract") +builder.connect_steps("extract", "rel_extract") +builder.connect_steps("rel_extract", "build_kg") +builder.connect_steps("build_kg", "deduplicate") +builder.connect_steps("deduplicate", "export") + +pipeline = builder.build("full_pipeline") +result = ExecutionEngine().execute_pipeline(pipeline, data="data/") ``` -## Save and Load Pipelines +## Serialize and Restore Pipelines -Serialize a pipeline to YAML for reproducible runs across environments: +`PipelineSerializer` converts a pipeline to JSON or dict for storage and reloads it later: ```python -# Save pipeline configuration -pipeline.save("pipeline_config.yaml") +from semantica.pipeline import PipelineSerializer -# Load and run on any machine -pipeline = Pipeline.load("pipeline_config.yaml") -result = pipeline.run("data/") +serializer = PipelineSerializer() + +# Serialize to JSON string +json_str = serializer.serialize_pipeline(pipeline, format="json") + +# Save to file +with open("pipeline_config.json", "w") as f: + f.write(json_str) + +# Restore on any machine and execute +with open("pipeline_config.json") as f: + restored = serializer.deserialize_pipeline(f.read()) + +result = ExecutionEngine().execute_pipeline(restored, data="data/") ``` - `pipeline.save()` preserves exact component configurations — LLM model names, retry policies, thresholds — everything. Without it, you can't guarantee that a re-run 3 months later uses the same settings. + Serialized pipelines capture step names, types, and config — but not handler functions (callables can't be serialized). Re-register handlers on the restored steps before executing. ## Pre-Built Templates @@ -280,26 +346,29 @@ Fine-grained control over pipeline execution — pause, resume, cancel, and insp ```python from semantica.pipeline import ExecutionEngine -engine = ExecutionEngine(config={"timeout_seconds": 300, "max_workers": 4}) +engine = ExecutionEngine(max_workers=4) +# pipeline.name is the pipeline ID used for all control operations result = engine.execute_pipeline(pipeline, data="data/") +pipeline_id = pipeline.name # e.g. "my_pipeline" + # Pause after the current step finishes -engine.pause_pipeline(result.pipeline_id) +engine.pause_pipeline(pipeline_id) -progress = engine.get_progress(result.pipeline_id) -print(f"Completed: {progress['completed']}/{progress['total']}") -print(f"Current step: {progress['current_step']}") +progress = engine.get_progress(pipeline_id) +print(f"Completed: {progress['completed_steps']}/{progress['total_steps']}") +print(f"Status: {progress['status']}") -engine.resume_pipeline(result.pipeline_id) -engine.stop_pipeline(result.pipeline_id) +engine.resume_pipeline(pipeline_id) +engine.stop_pipeline(pipeline_id) ``` | Method | Returns | Description | | ------ | ------- | ----------- | | `execute_pipeline(pipeline, data)` | `ExecutionResult` | Execute pipeline from start to finish | -| `get_status(pipeline_id)` | `PipelineStatus` | Current state (RUNNING, PAUSED, STOPPED) | -| `get_progress(pipeline_id)` | `Dict` | Step completion counts and elapsed time | +| `get_pipeline_status(pipeline_id)` | `PipelineStatus` | Current state (RUNNING, PAUSED, STOPPED) | +| `get_progress(pipeline_id)` | `Dict` | `completed_steps`, `total_steps`, `progress_percentage`, `status` | | `pause_pipeline(pipeline_id)` | `None` | Suspend after current step completes | | `resume_pipeline(pipeline_id)` | `None` | Resume from paused state | | `stop_pipeline(pipeline_id)` | `None` | Cancel and clean up immediately | @@ -314,12 +383,12 @@ from semantica.pipeline import PipelineValidator validator = PipelineValidator() result = validator.validate_pipeline(pipeline) -if result.is_valid: +if result.valid: print("Pipeline is valid — safe to run") else: - for error in result.errors: - print(f"Error: {error.message} (step: {error.step})") - for warning in result.warnings: + for error in result.errors: # errors is List[str] + print(f"Error: {error}") + for warning in result.warnings: # warnings is List[str] print(f"Warning: {warning}") ``` @@ -363,16 +432,15 @@ Checks performed: Prevents memory oversubscription on large runs: ```python -from semantica.pipeline import ResourceScheduler +from semantica.pipeline import ResourceScheduler, ExecutionEngine scheduler = ResourceScheduler() +engine = ExecutionEngine() -resources = scheduler.allocate_resources( - pipeline, max_memory_gb=8, max_workers=4 -) +resources = scheduler.allocate_resources(pipeline) try: - result = pipeline.run("data/") + result = engine.execute_pipeline(pipeline, data="data/") finally: scheduler.release_resources(resources) ``` @@ -382,23 +450,38 @@ finally: Re-process only data that has changed since the last run: ```python -pipeline = Pipeline() -pipeline.add_step( - "ingest", FileIngestor(), +from semantica.pipeline import PipelineBuilder, ExecutionEngine + +builder = PipelineBuilder() + +# delta_mode=True tells ExecutionEngine to compute the diff between two snapshots +# and pass only changed triples to this step's handler +builder.add_step( + "ingest", "file_ingest", + handler=ingestor.ingest_file, delta_mode=True, base_version_id="v1", target_version_id="v2" ) -pipeline.add_step( - "extract", NERExtractor(), +builder.add_step( + "extract", "ner_extract", + handler=extractor.extract, delta_mode=True, base_version_id="v1", target_version_id="v2" ) -pipeline.add_step( - "build", GraphBuilder(), +builder.add_step( + "build", "graph_build", + handler=kg_builder.build, delta_mode=False # always rebuild the merged graph ) +builder.connect_steps("ingest", "extract") +builder.connect_steps("extract", "build") -result = pipeline.run("data/") -print(f"Delta documents processed: {result.metadata.get('delta_count', 0)}") -print(f"Skipped (unchanged): {result.metadata.get('skipped_count', 0)}") +pipeline = builder.build("delta_pipeline") +engine = ExecutionEngine() +result = engine.execute_pipeline( + pipeline, + data="data/", + version_manager=version_manager, # required for delta mode + triplet_store=triplet_store # required for delta mode +) ``` @@ -408,18 +491,25 @@ print(f"Skipped (unchanged): {result.metadata.get('skipped_count', 0)}") ## Schemas - + ```python @dataclass -class PipelineResult: - output: Any # final step output (e.g., a KnowledgeGraph) - processed_count: int # documents successfully processed - failed_count: int # documents that failed after retries - duration_seconds: float # total wall-clock time - step_metrics: Dict # per-step timing and counts - errors: List # list of FailedDocument records - metadata: Dict # pipeline-level metadata (delta_count, etc.) +class ExecutionResult: + success: bool # True if all steps completed without failure + output: Any # output from the final pipeline step + metadata: Dict[str, Any] # {"pipeline_id": "...", "execution_time": 1.23} + metrics: Dict[str, Any] # {"steps_executed": 4, "steps_failed": 0, "execution_time": 1.23} + errors: List[str] # error messages from failed steps (empty on full success) + +# Access pattern +result.success # bool +result.output # final step output +result.metadata["pipeline_id"] # pipeline name used as ID +result.metadata["execution_time"] # total wall-clock seconds +result.metrics["steps_executed"] # count of successfully completed steps +result.metrics["steps_failed"] # count of failed steps +result.errors # List[str] of error messages ``` @@ -476,7 +566,7 @@ StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy - **Inspect `result.step_metrics` to find bottlenecks.** Each step reports its own duration and document count. If embedding is 10x slower than NER, that's where to optimize — increase `batch_size`, switch to a faster embedding model, or parallelize with GPU. + **Inspect `result.metrics` to find bottlenecks.** `result.metrics['steps_executed']` and `result.metrics['execution_time']` give a quick read on overall pipeline health. For per-step timing, check `step.result` on each `PipelineStep` after the run. diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 166bb364..9dee3169 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -45,30 +45,39 @@ icon: "database" ```python - store.add_vectors( - embeddings=embeddings, - ids=["doc1", "doc2"], + # Add text documents (auto-embedded) + ids = store.add_documents( + documents=["text one", "text two"], + metadata=[{"title": "Document 1"}, {"title": "Document 2"}] + ) + + # Add pre-computed vectors + ids = store.store_vectors( + vectors=[embedding1, embedding2], metadata=[{"title": "Document 1"}, {"title": "Document 2"}] ) ``` ```python - results = store.search(query_vector, top_k=10) + # Search by text query (auto-embeds the query) + results = store.search("machine learning", limit=10) + + # Search by pre-computed vector + results = store.search_vectors(query_vector, k=10) + for r in results: print(f"{r['id']} — score: {r['score']:.3f}") - print(f" metadata: {r['metadata']}") ``` ```python - # Equality, range, and set filters - results = store.search(query_vector, filters={ - "$and": [ - {"category": "research"}, - {"year": {"$gte": 2022}} - ] - }) + from semantica.vector_store import HybridSearch, MetadataFilter + + mf = MetadataFilter().eq("category", "research").gt("year", 2022) + + search = HybridSearch(vector_store=store) + results = search.search(query=query_vector, k=10, metadata_filter=mf) ``` @@ -161,65 +170,96 @@ See the [PgVector Guide](../vector_stores/pgvector) for full setup. ## Hybrid Search -Combine vector similarity with keyword/metadata filters for higher precision: +Use `HybridSearch` with a `MetadataFilter` to combine vector similarity with metadata conditions: ```python -results = store.hybrid_search( - query_vector=query_embedding, - query_text="machine learning", # keyword component - top_k=10, - alpha=0.7, # 0.0 = keyword only, 1.0 = vector only - filters={"category": "research", "year": {"$gte": 2022}} +from semantica.vector_store import HybridSearch, MetadataFilter + +mf = ( + MetadataFilter() + .eq("category", "research") + .gt("year", 2022) ) + +search = HybridSearch(vector_store=store) +results = search.search( + query=query_vector, # np.ndarray or query string (auto-embedded) + k=10, + metadata_filter=mf +) + +for r in results: + print(f"{r['id']} — score: {r['score']:.3f} metadata: {r['metadata']}") ``` ## Metadata Filtering +`MetadataFilter` supports chained conditions — all conditions are ANDed: + ```python -# Equality -results = store.search(query_vector, filters={"author": "John Smith"}) +from semantica.vector_store import MetadataFilter -# Range -results = store.search(query_vector, filters={"date": {"$gte": "2023-01-01"}}) +mf = MetadataFilter().eq("author", "John Smith") # equality +mf = MetadataFilter().ne("status", "archived") # not equal +mf = MetadataFilter().gt("year", 2022).lte("year", 2024) # range +mf = MetadataFilter().in_list("tag", ["ai", "ml"]) # set membership +mf = MetadataFilter().contains("title", "neural") # substring / list contains -# Set membership -results = store.search(query_vector, filters={"tag": {"$in": ["ai", "ml"]}}) - -# Compound AND -results = store.search(query_vector, filters={ - "$and": [ - {"category": "research"}, - {"year": {"$gte": 2022}} - ] -}) +# Multiple conditions — all must match (AND) +mf = ( + MetadataFilter() + .eq("category", "research") + .gt("year", 2022) + .contains("title", "language model") +) ``` ## Namespace Isolation -Isolate vectors per tenant, project, or use case: +Use `NamespaceManager` to assign vectors to named namespaces for multi-tenant isolation: ```python -store = VectorStore(backend="faiss", dimension=768) +from semantica.vector_store import NamespaceManager, VectorStore -# Write to separate namespaces -store.add_vectors(embeddings_a, ids_a, namespace="tenant_a") -store.add_vectors(embeddings_b, ids_b, namespace="tenant_b") +store = VectorStore(backend="faiss", dimension=768) +ns_manager = NamespaceManager() -# Search is scoped to the specified namespace -results = store.search(query_vector, namespace="tenant_a") +ns_manager.create_namespace("tenant_a", description="Customer A data") +ns_manager.create_namespace("tenant_b", description="Customer B data") + +# Store vectors, then assign them to a namespace +ids_a = store.store_vectors(embeddings_a, metadata=metadata_a) +for vid in ids_a: + ns_manager.add_vector_to_namespace(vid, "tenant_a") + +# List all namespace names +for name in ns_manager.list_namespaces(): + print(name) + +ns_manager.delete_namespace("tenant_a") ``` ## Batch Operations ```python -# Batch add — automatically chunked for memory efficiency -store.add_vectors_batch(embeddings_list, ids_list, batch_size=1000) +# Batch add text documents — chunked automatically by batch_size +ids = store.add_documents( + documents=large_doc_list, + metadata=large_meta_list, + batch_size=1000 +) -# Batch delete -store.delete_vectors(ids=["doc1", "doc2", "doc3"]) +# Batch add pre-computed vectors +ids = store.store_vectors(vectors=embeddings_list, metadata=meta_list) -# Update metadata without re-embedding -store.update_metadata("doc1", {"status": "archived", "reviewed": True}) +# Delete by vector ID list +store.delete_vectors(vector_ids=["vec_0", "vec_1", "vec_2"]) + +# Replace vectors (re-embed then update) +store.update_vectors( + vector_ids=["vec_0"], + new_vectors=[new_embedding] +) ``` ## Backend Comparison @@ -236,79 +276,87 @@ store.update_metadata("doc1", {"status": "archived", "reviewed": True}) ## HybridSearch -`HybridSearch` is the low-level class behind `store.hybrid_search()` — use it directly when you need custom result fusion logic: +`HybridSearch` combines vector similarity with metadata filtering, and can fuse results from multiple sources: ```python -from semantica.vector_store import HybridSearch, VectorStore +from semantica.vector_store import HybridSearch, MetadataFilter, SearchRanker -store = VectorStore(backend="faiss", dimension=768) -hybrid = HybridSearch(vector_store=store) +# Single-source search with metadata filter +search = HybridSearch(vector_store=store) +mf = MetadataFilter().eq("category", "research").gt("year", 2022) -results = hybrid.search( - query_vector=query_embedding, - query_text="machine learning frameworks", - top_k=20, - vector_weight=0.7, # weight for vector similarity leg - keyword_weight=0.3, # weight for BM25/keyword leg - fusion="rrf", # "rrf" (Reciprocal Rank Fusion) | "weighted_avg" - filters={"category": "research", "year": {"$gte": 2022}}, - deduplicate=True, +results = search.search( + query=query_vector, # np.ndarray or query string + k=10, + metadata_filter=mf ) -for r in results: - print(f"{r['id']} vector_score={r['vector_score']:.3f} final_score={r['score']:.3f}") +# Multi-source fusion (RRF across multiple stores) +sources = [ + {"vectors": v1, "metadata": m1, "ids": ids1}, + {"vectors": v2, "metadata": m2, "ids": ids2}, +] +fused = search.multi_source_search(query_vector, sources, k=10) + +# Custom fusion strategy +ranker = SearchRanker(strategy="reciprocal_rank_fusion") # or "weighted_average" +fused = ranker.rank([results_list_1, results_list_2], k=60) ``` | Fusion strategy | Description | | --------------- | ----------- | -| `rrf` | Reciprocal Rank Fusion — rank-based combination, robust to score scale differences | -| `weighted_avg` | Weighted average of normalised scores — requires `vector_weight` + `keyword_weight` = 1.0 | +| `reciprocal_rank_fusion` | Rank-based combination via RRF constant `k=60` — robust to score scale differences | +| `weighted_average` | Weighted average of scores — pass `weights=[0.7, 0.3]` to `rank()` | ## MetadataStore -`MetadataStore` manages structured metadata attached to vectors — query by field values without a vector: +`MetadataStore` indexes structured metadata and lets you query by field values without a vector: ```python from semantica.vector_store import MetadataStore meta_store = MetadataStore() -meta_store.register_schema({ - "author": "str", - "year": "int", - "category": "str", - "score": "float", -}) +# Define schema fields +meta_store.add_field("author", str, required=True) +meta_store.add_field("year", int, required=True) +meta_store.add_field("category", str) +meta_store.add_field("score", float, default=0.0) -meta_store.add("doc1", {"author": "Alice", "year": 2024, "category": "research"}) -meta_store.add("doc2", {"author": "Bob", "year": 2023, "category": "review"}) +# Store and retrieve metadata +meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"}) +meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"}) -results = meta_store.filter({"category": "research", "year": {"$gte": 2023}}) -meta = meta_store.get("doc1") -meta_store.update("doc1", {"score": 0.92}) +# Query — returns List[str] of matching vector IDs +ids = meta_store.query_metadata({"category": "research", "year": 2024}) + +# Get and update metadata for a specific vector +meta = meta_store.get_metadata("doc1") +meta_store.update_metadata("doc1", {"score": 0.92}) ``` ## NamespaceManager -Isolates vector collections per tenant, project, or model version: +Assigns vector IDs to named namespaces for multi-tenant or multi-model isolation: ```python -from semantica.vector_store import NamespaceManager, VectorStore +from semantica.vector_store import NamespaceManager -base_store = VectorStore(backend="faiss", dimension=768) -ns_manager = NamespaceManager(vector_store=base_store) +ns_manager = NamespaceManager() ns_manager.create_namespace("tenant_a", description="Customer A data") ns_manager.create_namespace("tenant_b", description="Customer B data") -ns_manager.add_vectors("tenant_a", embeddings_a, ids_a, metadata_a) -ns_manager.add_vectors("tenant_b", embeddings_b, ids_b, metadata_b) +# Assign vector IDs to a namespace after storing them +for vid in ids_a: + ns_manager.add_vector_to_namespace(vid, "tenant_a") -# Search is scoped — tenant_a never sees tenant_b's data -results = ns_manager.search("tenant_a", query_vector, top_k=10) +# Inspect namespaces +for name in ns_manager.list_namespaces(): # returns List[str] + print(name) -for ns in ns_manager.list_namespaces(): - print(f"{ns['name']}: {ns['vector_count']} vectors") +# Look up which namespace a vector belongs to +ns = ns_manager.get_vector_namespace("vec_0") ns_manager.delete_namespace("tenant_a") ``` From 6f726c708f9e86a8cccdf9f279678eecd5c7df53 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 13:11:57 +0530 Subject: [PATCH 04/11] fix: remove non-existent classes and fix wrong API signatures across reference docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - visualization.md: GraphVisualizer → KGVisualizer; fix method names (visualize_network, visualize_network_evolution, visualize_snapshot_comparison, visualize_temporal_patterns, visualize_2d_projection); remove DistanceVisualizer tab; fix start_explorer() reference - kg.md: remove TemporalKnowledgeGraph and DistanceCalculator (don't exist); replace with TemporalGraphQuery and ConnectivityAnalyzer; fix query_at_time() signature - ontology.md: remove OntologyManager, SKOSVocabulary, OntologyAligner, OntologyDiff, OntologyMigrator (none exist); fix SHACLValidator → OntologyValidator; fix OWLExporter → OWLGenerator.export_owl(); fix start_explorer() reference - evals.md: replace entire file with coming-soon notice (module is a stub, __all__ = []) - embeddings.md: fix EmbeddingGenerator constructor (takes config dict not model=); generate() → generate_embeddings(); similarity() → compare_embeddings() - ingest.md: fix WebIngestor (rate_limit → delay, ingest() → ingest_url()); FeedIngestor (ingest() → ingest_feed(), monitor() → monitor_feeds()); StreamIngestor (backend= constructor → ingest_kafka/rabbitmq/kinesis/pulsar()); DBIngestor constructor + ingest() → ingest_database(); SnowflakeIngestor.ingest() → ingest_query()/ingest_table(); OntologyIngestor.ingest() → ingest_ontology(); DataSource → FileObject - explorer.md: remove start_explorer() Python function (only CLI exists); replace with semantica-explorer CLI usage - provenance.md: ActivityTracker → ProvenanceTracker in CardGroup - semantic_extract.md: EventExtractor → EventDetector - triplet_store.md: remove InMemoryTripletStore (doesn't exist); fix tip - llms.md: fix providers (Anthropic/Gemini/Ollama/DeepSeek/NovitaAI → LiteLLM); HuggingFace → HuggingFaceLLM; remove create_provider() --- docs/reference/embeddings.md | 56 +++---- docs/reference/evals.md | 240 ++++------------------------- docs/reference/explorer.md | 72 ++------- docs/reference/ingest.md | 100 +++++------- docs/reference/kg.md | 72 ++++----- docs/reference/llms.md | 91 ++++++----- docs/reference/ontology.md | 134 ++++------------ docs/reference/provenance.md | 4 +- docs/reference/semantic_extract.md | 8 +- docs/reference/triplet_store.md | 30 +--- docs/reference/visualization.md | 120 ++++++--------- 11 files changed, 261 insertions(+), 666 deletions(-) diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index c81cbf4d..5d2cb1d4 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -70,18 +70,21 @@ providers = check_available_providers() from semantica.embeddings import EmbeddingGenerator # Default — Sentence-Transformers, free, runs locally - generator = EmbeddingGenerator(model="sentence-transformers") + generator = EmbeddingGenerator() + + # Custom model via config dict + generator = EmbeddingGenerator(config={"text": {"method": "sentence_transformers", "model_name": "BAAI/bge-large-en-v1.5"}}) ``` ```python - embeddings = generator.generate(["Text about AI", "Machine learning concepts"]) + embeddings = generator.generate_embeddings(["Text about AI", "Machine learning concepts"]) ``` ```python # Cosine similarity — 0.0 (unrelated) to 1.0 (identical meaning) - score = generator.similarity(embeddings[0], embeddings[1]) + score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosine") print(f"Similarity: {score:.3f}") ``` @@ -124,14 +127,14 @@ providers = check_available_providers() ```python from semantica.embeddings import EmbeddingGenerator - # Default model — all-MiniLM-L6-v2, dimension 384 - generator = EmbeddingGenerator(model="sentence-transformers") + # Default — Sentence-Transformers with all-MiniLM-L6-v2 + generator = EmbeddingGenerator() - # Specific HuggingFace model - generator = EmbeddingGenerator(model="BAAI/bge-large-en-v1.5") + # Custom model via set_text_model + generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") - embeddings = generator.generate(texts) - similarity = generator.similarity(embeddings[0], embeddings[1]) + embeddings = generator.generate_embeddings(texts) + similarity = generator.compare_embeddings(embeddings[0], embeddings[1]) ``` Best for: default prototyping, no API key, good quality. @@ -140,23 +143,20 @@ providers = check_available_providers() ```python from semantica.embeddings import EmbeddingGenerator - generator = EmbeddingGenerator(model="fastembed") - embeddings = generator.generate(texts) + generator = EmbeddingGenerator() + generator.set_text_model("fastembed", "BAAI/bge-small-en-v1.5") + embeddings = generator.generate_embeddings(texts) ``` Best for: CPU-only production, lowest latency without GPU. ```python - from semantica.embeddings import EmbeddingGenerator + from semantica.embeddings import OpenAIStore import os - generator = EmbeddingGenerator( - model="openai", - model_name="text-embedding-3-small", - api_key=os.getenv("OPENAI_API_KEY"), - ) - embeddings = generator.generate(texts) + store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small") + embedding = store.embed("Hello world") ``` Best for: highest quality (3-large), or matching an OpenAI LLM pipeline. @@ -175,14 +175,11 @@ providers = check_available_providers() ```python from semantica.embeddings import EmbeddingGenerator - # NVIDIA GPU - generator = EmbeddingGenerator(model="sentence-transformers", device="cuda") + # Set device via text embedder config + generator = EmbeddingGenerator(config={"text": {"device": "cuda"}}) # Apple Silicon (M1/M2/M3) - generator = EmbeddingGenerator(model="sentence-transformers", device="mps") - - # CPU (default) - generator = EmbeddingGenerator(model="sentence-transformers", device="cpu") + generator = EmbeddingGenerator(config={"text": {"device": "mps"}}) ``` GPU reduces embedding time by 5–20× depending on batch size and model. @@ -193,13 +190,10 @@ providers = check_available_providers() | Parameter | Type | Default | Description | | --------- | ---- | ------- | ----------- | -| `model` | `str` | `"sentence-transformers"` | Provider name or HuggingFace model ID | -| `model_name` | `str` | Provider default | Specific model within a provider (OpenAI) | -| `api_key` | `str` | `None` | API key for cloud providers; reads env var if omitted | -| `device` | `str` | `"cpu"` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | -| `batch_size` | `int` | `32` | Texts per forward pass | -| `normalize` | `bool` | `True` | L2-normalise output vectors (required for cosine similarity) | -| `cache_dir` | `str` | `None` | Directory for disk caching of computed embeddings | +| `config` | `dict` | `None` | Config dict; `config["text"]` is passed to `TextEmbedder` | +| `**kwargs` | | | Additional key/value config merged into `config` | + +Use `generator.set_text_model(method, model_name)` to switch the embedding model after construction. ## TextEmbedder diff --git a/docs/reference/evals.md b/docs/reference/evals.md index 0411dac4..4da8b3db 100644 --- a/docs/reference/evals.md +++ b/docs/reference/evals.md @@ -1,230 +1,42 @@ --- title: "Evals Module" -description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance." +description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance — coming soon." icon: "chart-line" --- -`semantica.evals` provides a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance. Use it to benchmark extractors, validate pipeline output, and track quality regressions across runs. - -## What You Get - - - - Completeness, consistency, schema compliance, coverage, and orphan node metrics. - - - NER precision / recall / F1 and relation extraction metrics against gold-standard datasets. - - - Throughput (docs/sec), per-step latency, peak memory, and error rate benchmarking. - - - Record pipeline runs and compare metrics across commits or config changes. - - - Merge precision, false positive / false negative rates for deduplication strategies. - - - Inference accuracy, rule coverage, and derivation depth for reasoning engines. - - - -## Quick Start - - - - ```python - from semantica.evals import KGEvaluator - - evaluator = KGEvaluator() - report = evaluator.evaluate(kg, ontology=ontology) - - print(f"Completeness: {report.completeness:.2%}") - print(f"Consistency: {report.consistency:.2%}") - print(f"Coverage: {report.coverage:.2%}") - print(f"Orphan nodes: {report.orphan_count}") - ``` - - - ```python - from semantica.evals import ExtractionEvaluator - - evaluator = ExtractionEvaluator() - report = evaluator.evaluate_ner( - predictions=extracted_entities, - gold_standard=annotated_entities, - ) - - print(f"Precision: {report.precision:.3f}") - print(f"Recall: {report.recall:.3f}") - print(f"F1: {report.f1:.3f}") - print(f"By type: {report.per_type_metrics}") - ``` - - - ```python - from semantica.evals import PipelineEvaluator - - evaluator = PipelineEvaluator() - metrics = evaluator.benchmark(pipeline, data="data/", warmup_runs=2, bench_runs=5) - - print(f"Throughput: {metrics.docs_per_second:.1f} docs/sec") - print(f"Total duration: {metrics.total_seconds:.1f}s") - print(f"Per-step latency: {metrics.step_latencies}") - print(f"Peak memory (MB): {metrics.peak_memory_mb:.0f}") - print(f"Error rate: {metrics.error_rate:.2%}") - ``` - - - ```python - from semantica.evals import RegressionTracker - - tracker = RegressionTracker(db_path="eval_history.db") - - run_id = tracker.record_run( - pipeline_version="v1.2.0", - metrics=metrics, - config=config.to_dict(), - ) - - diff = tracker.compare(run_id, baseline_run_id="run_abc123") - for metric, change in diff.items(): - direction = "↑" if change > 0 else "↓" - print(f" {metric}: {direction} {abs(change):.2%}") - ``` - - - -## Evaluation Areas - - - - Measure completeness, consistency, schema compliance, and structural health of a knowledge graph: - - ```python - from semantica.evals import KGEvaluator - - evaluator = KGEvaluator() - report = evaluator.evaluate(kg, ontology=ontology) - - print(f"Completeness: {report.completeness:.2%}") # % entities with all required fields - print(f"Consistency: {report.consistency:.2%}") # % entities without type conflicts - print(f"Coverage: {report.coverage:.2%}") # % entity types in ontology - print(f"Total nodes: {report.node_count}") - print(f"Orphan nodes: {report.orphan_count}") # nodes with no edges - ``` - - **Key behaviours:** - - `consistency` requires an ontology — without one, it always returns 1.0 - - `orphan_count` flags disconnected nodes that likely represent extraction or deduplication errors - - `completeness` checks required properties defined in the ontology schema - - - Compare extracted entities and relations against annotated gold-standard data: - - ```python - from semantica.evals import ExtractionEvaluator - - evaluator = ExtractionEvaluator() - - # NER evaluation - ner_report = evaluator.evaluate_ner( - predictions=extracted_entities, - gold_standard=annotated_entities, - ) - print(f"Precision: {ner_report.precision:.3f}") - print(f"Recall: {ner_report.recall:.3f}") - print(f"F1: {ner_report.f1:.3f}") - print(f"By type: {ner_report.per_type_metrics}") - - # Relation extraction evaluation - rel_report = evaluator.evaluate_relations( - predictions=extracted_relations, - gold_standard=annotated_relations, - ) - print(f"Relation F1: {rel_report.f1:.3f}") - ``` - - - Benchmark throughput, latency, memory, and error rate across multiple runs: - - ```python - from semantica.evals import PipelineEvaluator - - evaluator = PipelineEvaluator() - metrics = evaluator.benchmark( - pipeline, - data="data/", - warmup_runs=2, # eliminate cold-start noise - bench_runs=5, # average over 5 real runs - ) - - print(f"Throughput: {metrics.docs_per_second:.1f} docs/sec") - print(f"Total duration: {metrics.total_seconds:.1f}s") - print(f"Per-step latency: {metrics.step_latencies}") - print(f"Peak memory (MB): {metrics.peak_memory_mb:.0f}") - print(f"Error rate: {metrics.error_rate:.2%}") - ``` - - - Store runs and compare metrics across pipeline versions: - - ```python - from semantica.evals import RegressionTracker - - tracker = RegressionTracker(db_path="eval_history.db") - - # Record a run with version tag and full config snapshot - run_id = tracker.record_run( - pipeline_version="v1.2.0", - metrics=metrics, - config=config.to_dict(), - ) - - # Compare to a previous run - diff = tracker.compare(run_id, baseline_run_id="run_abc123") - for metric, change in diff.items(): - direction = "↑" if change > 0 else "↓" - print(f" {metric}: {direction} {abs(change):.2%}") - ``` - - - -## When to Evaluate - -| Trigger | Evaluator to Use | What to Check | -| ------- | ---------------- | ------------- | -| New extraction model or method | `ExtractionEvaluator` | Precision, recall, F1 vs gold standard | -| After changing LLM provider | `ExtractionEvaluator` | Per-type F1 — check if rare types regressed | -| Before releasing new pipeline version | `PipelineEvaluator` | Throughput, latency, error rate | -| After deduplication strategy change | `KGEvaluator` | Orphan count, consistency score | -| Every production deployment | `RegressionTracker` | Compare vs previous baseline run | - -## Tips and Common Pitfalls +`semantica.evals` is planned as a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance. - **Build a gold standard dataset early.** `ExtractionEvaluator` requires annotated ground truth. Without it, you're evaluating subjectively. Even 100 carefully annotated documents give you a meaningful baseline to track regressions against. + **`semantica.evals` is not yet implemented.** The module exists as a placeholder (`__all__ = []`). No classes or functions are available for import. This page describes the planned API. - - **Evaluate per entity type, not just overall F1.** Aggregate F1 can hide regressions — if your model's PERSON F1 drops from 0.95 to 0.80 but ORGANIZATION improves, the average may look stable. Use `report.per_type_metrics` to catch type-specific regressions. - +## Planned Features - - **Store every benchmark run with `RegressionTracker`.** Run ID + version tag + config snapshot gives you a reproducible audit trail. Without it, "did the last release make things better?" has no objective answer. - +When released, `semantica.evals` will provide: - - **Run `PipelineEvaluator` with `warmup_runs=2`.** Cold starts are unrepresentative — model weights get cached, JIT compilation kicks in. Warmup runs eliminate this noise from your benchmark numbers. - +- **KG quality metrics** — completeness, consistency, schema compliance, coverage, and orphan node detection +- **Extraction accuracy** — NER precision / recall / F1 and relation extraction metrics against gold-standard datasets +- **Pipeline benchmarking** — throughput (docs/sec), per-step latency, peak memory, and error rate +- **Regression tracking** — record runs and compare metrics across commits or config changes +- **Deduplication accuracy** — merge precision, false positive / false negative rates +- **Reasoning correctness** — inference accuracy, rule coverage, and derivation depth - - **`KGEvaluator` needs an ontology for consistency scoring.** Without an ontology, `consistency` always returns 1.0 — there's nothing to check against. Pass `ontology=ontology` to get meaningful consistency metrics. - +## Current Workaround + +Until `semantica.evals` ships, use `semantica.ontology.OntologyEvaluator` for ontology quality metrics: + +```python +from semantica.ontology import OntologyEvaluator + +evaluator = OntologyEvaluator() +report = evaluator.evaluate(ontology, kg) +print(f"Coverage: {report.coverage:.2%}") +print(f"Completeness: {report.completeness:.2%}") +``` - Extraction module to evaluate. + Extraction module. Graph quality assessment. @@ -232,7 +44,7 @@ icon: "chart-line" Pipeline performance metrics. - - Deduplication accuracy evaluation. + + Available now for ontology quality metrics. diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index 0389d42b..c89448a8 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -40,52 +40,34 @@ Requires `uvicorn` and `fastapi`. Included automatically with `pip install seman ## Launch - + ```python - from semantica.explorer import start_explorer + import json from semantica.kg import GraphBuilder - from semantica.ontology import OntologyManager - kg = GraphBuilder().build(entities=entities, relationships=relationships) - ontology = OntologyManager() + kg = GraphBuilder().build(entities=entities, relationships=relationships) - start_explorer( - graph=kg, - ontology=ontology, # optional — enables Ontology Hub tab - port=8080, - host="127.0.0.1", - open_browser=True, - ) - # → Serving at http://127.0.0.1:8080 + # Export graph to JSON file + with open("my_graph.json", "w") as f: + json.dump({"entities": kg.entities, "relationships": kg.relationships}, f) + ``` + + ```bash + semantica-explorer --graph my_graph.json + # → Serving at http://127.0.0.1:8000 ``` - + ```bash - # Start on a saved graph - semantica-explorer --graph my_graph.json - - # Custom host and port semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080 # Skip auto-opening the browser semantica-explorer --graph my_graph.json --no-browser ``` - - ```python - start_explorer( - graph=kg, - port=8080, - enable_auth=True, - api_key="my-secret-key", - cors_origins=["https://app.example.com"], - session_timeout=1800, # 30-minute inactivity timeout - ) - ``` - ```bash - curl -X POST http://localhost:8080/api/import \ + curl -X POST http://localhost:8000/api/import \ -H "Content-Type: multipart/form-data" \ -F "file=@updated_graph.json" # Browser dashboard reloads automatically @@ -93,21 +75,6 @@ Requires `uvicorn` and `fastapi`. Included automatically with `pip install seman -## `start_explorer()` Parameters - -| Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | -| `graph` | `KnowledgeGraph` or `ContextGraph` | *(required)* | The graph to load into Explorer | -| `ontology` | `OntologyManager` | `None` | Ontology to load into the Ontology Hub tab | -| `port` | `int` | `8000` | Port to bind the server | -| `host` | `str` | `"127.0.0.1"` | Host to bind. Use `"0.0.0.0"` for network access | -| `open_browser` | `bool` | `True` | Auto-open the dashboard in the default browser | -| `session_timeout` | `int` | `3600` | Session inactivity timeout in seconds; `None` disables | -| `enable_auth` | `bool` | `False` | Require `X-API-Key` header on all API requests | -| `api_key` | `str` | `None` | API key value when `enable_auth=True` | -| `cors_origins` | `list[str]` | `["*"]` | Allowed CORS origins. Restrict in production | -| `log_level` | `str` | `"info"` | Uvicorn log level (`"debug"` / `"info"` / `"warning"`) | - ## CLI Reference | Flag | Default | Description | @@ -154,15 +121,6 @@ Requires `uvicorn` and `fastapi`. Included automatically with `pip install seman Thread-safe sessions with rollback protection: - ```python - start_explorer( - graph=kg, - session_timeout=1800, # 30-minute inactivity timeout - enable_auth=True, - api_key="my-secret-key", - ) - ``` - - Sessions are per connected browser tab - Write operations (annotate, import) roll back automatically on failure - All writes appended to audit trail at `/api/provenance/audit` @@ -296,12 +254,12 @@ WebSocket event schema: | SPARQL SELECT (simple pattern) | < 20ms | | N×N distance matrix (100 nodes) | ~2s (with embedding cache) | -The node search index is built on startup. For graphs > 500k nodes, pass `index_build_timeout=120` to `start_explorer()` to allow more time. +The node search index is built on startup. For graphs > 500k nodes, allow extra startup time before connecting. ## Tips and Common Pitfalls - **Set `max_nodes` when loading large graphs.** `start_explorer(graph=kg, max_nodes=50000)` limits the rendered node count — Explorer's force-directed layout becomes unusable above ~10k nodes without limiting. Use `graph.filter(node_type="Organization")` first to focus on what matters. + **Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph (e.g., by entity type) before exporting to JSON — Explorer's force-directed layout becomes unusable on very large graphs. diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index af0c9e52..8f28de6b 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -150,16 +150,12 @@ icon: "database" from semantica.ingest import WebIngestor ingestor = WebIngestor( - rate_limit=1.0, # seconds between requests + delay=1.0, # seconds between requests respect_robots=True, # honor robots.txt - max_depth=2 # crawl depth from seed URLs + timeout=30, ) - sources = ingestor.ingest("https://example.com/about") - sources = ingestor.ingest_urls([ - "https://example.com/page1", - "https://example.com/page2", - ]) + sources = ingestor.ingest_url("https://example.com/about") ``` ### FeedIngestor (RSS/Atom) @@ -168,13 +164,12 @@ icon: "database" from semantica.ingest import FeedIngestor ingestor = FeedIngestor() - sources = ingestor.ingest("https://feeds.example.com/rss") + feed = ingestor.ingest_feed("https://feeds.example.com/rss") - # Live monitoring — callback fires on new items - ingestor.monitor( - "https://feeds.example.com/rss", - interval=300, - callback=process_new_items + # Live monitoring — returns a FeedMonitor; callback fires on new items + monitor = ingestor.monitor_feeds( + ["https://feeds.example.com/rss"], + callback=process_new_items, ) ``` @@ -264,11 +259,11 @@ icon: "database" ```python from semantica.ingest import DBIngestor - ingestor = DBIngestor( + ingestor = DBIngestor() + result = ingestor.ingest_database( connection_string="postgresql://user:pass@localhost/db", - query="SELECT id, content, created_at FROM documents WHERE status='active'" + include_tables=["documents"], ) - sources = ingestor.ingest() ``` ### SnowflakeIngestor @@ -283,9 +278,10 @@ icon: "database" password=os.getenv("SNOWFLAKE_PASSWORD"), warehouse="COMPUTE_WH", database="ANALYTICS", - schema="PUBLIC" + schema="PUBLIC", ) - sources = ingestor.ingest(query="SELECT * FROM documents") + result = ingestor.ingest_query("SELECT * FROM documents") + result = ingestor.ingest_table("documents") ``` @@ -297,48 +293,32 @@ icon: "database" ```python from semantica.ingest import StreamIngestor - # Kafka - ingestor = StreamIngestor( - backend="kafka", - bootstrap_servers="localhost:9092", + ingestor = StreamIngestor() + + # Kafka — returns KafkaProcessor + processor = ingestor.ingest_kafka( topic="documents", - group_id="semantica-consumer", - auto_offset_reset="earliest", + bootstrap_servers=["localhost:9092"], ) - sources = ingestor.ingest(max_messages=1000) - # RabbitMQ - ingestor = StreamIngestor( - backend="rabbitmq", - host="localhost", + # RabbitMQ — returns RabbitMQProcessor + processor = ingestor.ingest_rabbitmq( queue="document_queue", - routing_key="docs.ingest", - prefetch_count=100, + connection_url="amqp://guest:guest@localhost/", ) - # AWS Kinesis - ingestor = StreamIngestor( - backend="kinesis", + # AWS Kinesis — returns KinesisProcessor + processor = ingestor.ingest_kinesis( stream_name="documents-stream", region="us-east-1", - shard_iterator_type="TRIM_HORIZON", ) - # Apache Pulsar - ingestor = StreamIngestor( - backend="pulsar", - service_url="pulsar://localhost:6650", + # Apache Pulsar — returns PulsarProcessor + processor = ingestor.ingest_pulsar( topic="persistent://public/default/documents", - subscription_name="semantica-sub", + service_url="pulsar://localhost:6650", ) - - # Live monitoring — callback fires on each new message - ingestor.monitor(callback=process_document, poll_interval=1.0) ``` - - - Without a `max_messages` limit, `StreamIngestor.ingest()` blocks indefinitely waiting for new messages. Use `max_messages=1000` for batch processing; use `.monitor(callback=...)` for continuous streaming. - @@ -349,23 +329,21 @@ Ingest existing OWL or RDF ontology files as structured knowledge sources: ```python from semantica.ingest import OntologyIngestor -ingestor = OntologyIngestor( - format="turtle", # "turtle" | "xml" | "json-ld" | "nt" | "n3" -) +ingestor = OntologyIngestor() -sources = ingestor.ingest("domain_ontology.owl") -sources = ingestor.ingest("ontologies/") +ontology_data = ingestor.ingest_ontology("domain_ontology.owl", format="turtle") +ontology_list = ingestor.ingest_directory("ontologies/", recursive=True) ``` -## DataSource Object +## FileObject -All ingestors return a list of `DataSource` objects with a consistent schema: +`FileIngestor` returns `FileObject` instances: - + ```python @dataclass -class DataSource: +class FileObject: content: str # raw text content source_id: str # unique identifier source_type: str # "file" | "web" | "database" | "stream" | ... @@ -402,16 +380,8 @@ method_registry.register("file", "my_format", my_ingestor) **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica — it doesn't block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML. - - **Stream ingestors need explicit `max_messages` for batch runs.** Without a limit, `StreamIngestor.ingest()` blocks indefinitely waiting for new messages. Use `max_messages=1000` for batch processing; use `.monitor(callback=...)` for continuous streaming. - - - **Rate-limit web crawling.** `WebIngestor(rate_limit=1.0, respect_robots=True)` is the responsible default. Without rate limiting, you risk getting blocked by the target server or violating its terms of service. - - - - **All ingestors return the same `DataSource` schema.** This means you can mix sources in a single pipeline without any adapter code — `FileIngestor`, `DBIngestor`, and `StreamIngestor` outputs are all directly composable with `DocumentParser` and `NERExtractor`. + **Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting, you risk getting blocked by the target server or violating its terms of service. diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 3b72c8d9..683e8fdc 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -12,11 +12,11 @@ icon: "diagram-project" Construct graphs from entities and relationships with automatic entity merging. - - Time-aware edges (`valid_from`/`valid_until`) and point-in-time queries (v0.4.0). + + Time-aware queries — filter by `valid_from`/`valid_until`, range queries, and evolution analysis. - - Semantic neighborhoods, N×N distance matrices, and distance band classification (v0.5.0). + + Connected components, bridge detection, and edge density analysis. PageRank, degree, betweenness, closeness, and eigenvector centrality. @@ -104,61 +104,41 @@ kg = builder.build(entities=entities, relationships=relationships) Always use `merge_entities=True` in production. Without it, "Steve Jobs" extracted from five different documents creates five separate person nodes. `GraphBuilder(merge_entities=True)` uses edit distance matching to consolidate them at build time. -## Temporal Knowledge Graphs (v0.4.0) +## Temporal Queries -Attach `valid_from` / `valid_until` time windows to nodes and edges for point-in-time queries and historical analysis: +Use `TemporalGraphQuery` to run time-aware queries against a knowledge graph whose relationships carry `valid_from` / `valid_until` fields: ```python -from semantica.kg import TemporalKnowledgeGraph, TemporalGraphQuery +from semantica.kg import TemporalGraphQuery from datetime import datetime -tkg = TemporalKnowledgeGraph() - -tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1)) -tkg.add_edge( - "alice", "acme_corp", "ceo_of", - valid_from=datetime(2020, 1, 1), - valid_until=datetime(2023, 6, 1) +query_engine = TemporalGraphQuery( + enable_temporal_reasoning=True, + temporal_granularity="day", ) -# Point-in-time snapshot -snapshot = tkg.at(datetime(2021, 6, 15)) +# Point-in-time query — returns only edges valid at the given time +result_2021 = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15)) +result_2023 = query_engine.query_at_time(kg, query="", at_time=datetime(2023, 1, 1)) -# Query and diff via TemporalGraphQuery -query = TemporalGraphQuery(tkg) -snap_2020 = query.query_at_time(datetime(2020, 1, 1)) -snap_2023 = query.query_at_time(datetime(2023, 1, 1)) -added = [r for r in snap_2023.relationships if r not in snap_2020.relationships] -print(f"New edges since 2020: {len(added)}") +# Compare what changed between two snapshots +added = [ + r for r in result_2023["relationships"] + if r not in result_2021["relationships"] +] +print(f"New edges since 2021: {len(added)}") + +# Range query — edges valid within a time window +range_result = query_engine.query_time_range(kg, query="", start_time=datetime(2020, 1, 1), end_time=datetime(2023, 1, 1)) + +# Evolution analysis +evolution = query_engine.analyze_evolution(kg) ``` - Edges added without `valid_from`/`valid_until` are treated as **always-valid**. For historical data, always attach timestamps — otherwise point-in-time queries return misleading results. + Relationships added without `valid_from`/`valid_until` are treated as **always-valid**. For historical data, always attach timestamps — otherwise point-in-time queries return misleading results. -## Distance Intelligence (v0.5.0) - -Semantic neighborhood exploration for any entity in the graph: - -```python -from semantica.kg import DistanceCalculator - -calc = DistanceCalculator(kg) - -# Semantic neighborhood of a single node -neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4) - -# N×N pairwise distance matrix -matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"]) - -# Classify nodes into distance bands: "near" | "mid" | "far" -bands = calc.classify_bands(neighborhood) -``` - - - `DistanceCalculator` is expensive at large scale — the N×N matrix requires embedding all entities and computing pairwise cosine similarities. Cache the result between runs and only recompute for changed entities. - - ## Graph Analytics diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 4b27127a..55b8fea6 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -15,8 +15,8 @@ icon: "microchip" `complete()`, `chat()`, and `stream()` work identically across all providers — swap with a one-line change. - - Instantiate any provider from a name string — drive provider selection entirely from YAML config or environment variables. + + One class, every provider — Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, and 90+ more via LiteLLM model strings. Ollama and HuggingFace run fully on-premise — no API key, no data leaves your machine, air-gap compatible. @@ -69,14 +69,15 @@ The base install includes Groq and DeepSeek. Other providers require optional ex entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") ``` - + ```python - from semantica.llms import create_provider + from semantica.llms import LiteLLM from semantica.core import ConfigManager config = ConfigManager("config.yaml") - llm = create_provider( - config.get("llm_provider.name"), + # LiteLLM model strings include the provider prefix: + # "anthropic/claude-opus-4-7", "gemini/gemini-1.5-pro", "ollama/llama3.2" + llm = LiteLLM( model=config.get("llm_provider.model"), api_key=config.get("llm_provider.api_key"), ) @@ -117,69 +118,77 @@ llm = OpenAI( ) ``` -```python Anthropic -from semantica.llms import Anthropic +```python Anthropic (via LiteLLM) +from semantica.llms import LiteLLM import os -llm = Anthropic( - model="claude-opus-4-7", +# Anthropic Claude is accessed via LiteLLM using the "anthropic/" prefix +llm = LiteLLM( + model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY"), max_tokens=8192, temperature=0.0, - max_retries=3, - timeout=120, ) ``` -```python Gemini -from semantica.llms import Gemini +```python Gemini (via LiteLLM) +from semantica.llms import LiteLLM import os -llm = Gemini( - model="gemini-1.5-pro", +# Google Gemini is accessed via LiteLLM using the "gemini/" prefix +llm = LiteLLM( + model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY"), temperature=0.0, max_tokens=8192, - timeout=120, ) ``` -```python Ollama (Local) -from semantica.llms import Ollama +```python Ollama / Local (via LiteLLM) +from semantica.llms import LiteLLM -llm = Ollama( - model="llama3.2", - base_url="http://localhost:11434", # default Ollama address +# Ollama local models via LiteLLM using the "ollama/" prefix +llm = LiteLLM( + model="ollama/llama3.2", + base_url="http://localhost:11434", # default Ollama address temperature=0.0, - timeout=180, # local models can be slower; increase for large models + timeout=180, # local models can be slower; increase for large models ) # No API key — model runs entirely on your machine ``` -```python DeepSeek -from semantica.llms import DeepSeek +```python DeepSeek (via Groq or LiteLLM) +from semantica.llms import Groq # Groq hosts DeepSeek models import os -llm = DeepSeek( - model="deepseek-chat", +llm = Groq( + model="deepseek-r1-distill-llama-70b", + api_key=os.getenv("GROQ_API_KEY"), + temperature=0.0, + max_tokens=4096, +) + +# Or via LiteLLM for the native DeepSeek endpoint: +from semantica.llms import LiteLLM +llm = LiteLLM( + model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY"), temperature=0.0, max_tokens=4096, - max_retries=3, ) ``` -```python Novita AI -from semantica.llms import NovitaAI +```python Novita AI (via LiteLLM) +from semantica.llms import LiteLLM import os -llm = NovitaAI( - model="deepseek/deepseek-v3.2", +# Novita AI via LiteLLM using the "novita/" prefix +llm = LiteLLM( + model="novita/deepseek/deepseek-v3", api_key=os.getenv("NOVITA_API_KEY"), temperature=0.0, max_tokens=4096, ) -# OpenAI-compatible endpoint; also accepts deepseek/deepseek-r1, meta-llama models ``` ```python LiteLLM (100+ models) @@ -193,13 +202,13 @@ llm = LiteLLM( max_tokens=4096, ) # Supports: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, Together AI, and 90+ more -# Use the LiteLLM model string format: "anthropic/claude-3-5-sonnet", "bedrock/anthropic.claude-v2" +# Use the LiteLLM model string format: "anthropic/claude-opus-4-7", "bedrock/anthropic.claude-v2" ``` ```python HuggingFace (Local) -from semantica.llms import HuggingFace +from semantica.llms import HuggingFaceLLM -llm = HuggingFace( +llm = HuggingFaceLLM( model="mistralai/Mistral-7B-Instruct-v0.3", device="cuda", # "cpu" | "cuda" | "mps" (Apple Silicon) max_new_tokens=512, @@ -231,11 +240,11 @@ llm = HuggingFace( | -------- | --------- | ----------- | | `OpenAI` | `organization` | OpenAI organisation ID | | `OpenAI` | `project` | OpenAI project ID | -| `Ollama` | `base_url` | Ollama server address (default: `http://localhost:11434`) | -| `HuggingFace` | `device` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | -| `HuggingFace` | `load_in_4bit` | Enable 4-bit quantisation (requires `bitsandbytes`) | -| `HuggingFace` | `max_new_tokens` | Maximum new tokens to generate (replaces `max_tokens`) | -| `LiteLLM` | `model` | Full LiteLLM model string, e.g. `"anthropic/claude-3-5-sonnet"` | +| `LiteLLM` | `model` | Full LiteLLM model string, e.g. `"anthropic/claude-opus-4-7"`, `"gemini/gemini-1.5-pro"`, `"ollama/llama3.2"` | +| `LiteLLM` | `base_url` | Override endpoint — use for Ollama (`http://localhost:11434`) or proxies | +| `HuggingFaceLLM` | `device` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | +| `HuggingFaceLLM` | `load_in_4bit` | Enable 4-bit quantisation (requires `bitsandbytes`) | +| `HuggingFaceLLM` | `max_new_tokens` | Maximum new tokens to generate (replaces `max_tokens`) | ## Direct API Usage diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index d387ccdc..52383ade 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -1,72 +1,64 @@ --- title: "Ontology Module" -description: "Automated ontology generation, SHACL validation, SKOS vocabularies, alignment, diff/migration, and the visual Ontology Hub." +description: "Automated ontology generation, OWL export, SHACL validation, domain ontologies, and modular ontology development." icon: "sitemap" --- -`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and SHACL validation to visual editing in the Ontology Hub (v0.5.0). Use it for schema design, data modeling, semantic web interoperability, and SHACL-based data quality validation. +`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and OWL export to SHACL validation and modular ontology development. Use it for schema design, data modeling, and semantic web interoperability. ## What You Get - - Define classes, properties, relationships, and constraints for your knowledge graph schema. - Auto-generate ontologies from existing graph data using a 6-stage pipeline. - - Generate SHACL shapes from an ontology and validate graphs for constraint compliance. + + Generate SHACL shapes from an ontology and validate ontologies for structural consistency. - - Controlled vocabulary and taxonomy management using the W3C SKOS standard. + + Capture, manage, and validate competency questions that define ontology requirements. - - Align and merge ontologies across schemas — maps concepts with confidence scores. + + Integrate published ontologies (schema.org, FOAF) instead of generating from scratch. - - Visual browser UI for the full ontology lifecycle — editor, SHACL Studio, and health dashboard. + + Pre-built domain ontologies for biomedical, finance, legal, supply chain, and more. + + + Measure coverage, completeness, and granularity — validate against competency questions. ## Quick Start - + ```python - from semantica.ontology import OntologyManager + from semantica.ontology import OntologyGenerator - ontology = OntologyManager() - ontology.add_class("Person", properties=["name", "birth_date"]) - ontology.add_class("Organization", properties=["name", "founded_date"]) - ontology.add_relationship("works_for", domain="Person", range="Organization") - ontology.add_constraint("Person", "must_have_name") + generator = OntologyGenerator() + ontology = generator.generate_from_graph(kg) ``` - + ```python - is_valid = ontology.validate_graph(kg) + from semantica.ontology import OntologyValidator - # Or use SHACL for granular constraint reporting - from semantica.ontology import SHACLGenerator, SHACLValidator + validator = OntologyValidator(reasoner="hermit", check_consistency=True) + result = validator.validate(ontology) - shapes = SHACLGenerator().generate(ontology) - validator = SHACLValidator() - report = validator.validate(kg, shapes=shapes) - - if not report.conforms: - for v in report.violations: - print(f"Violation: {v.message} on {v.node} (path: {v.path})") + if not result.is_valid: + for issue in result.issues: + print(f"Issue: {issue.message} (severity: {issue.severity})") ``` ```python - from semantica.ontology import OWLExporter + from semantica.ontology import OWLGenerator - exporter = OWLExporter() - exporter.export(ontology, path="ontology.ttl", format="turtle") - exporter.export(ontology, path="ontology.owl", format="xml") - exporter.export(ontology, path="ontology.json", format="json-ld") + owl_gen = OWLGenerator() + owl_gen.export_owl(ontology, file_path="ontology.ttl", format="turtle") + owl_gen.export_owl(ontology, file_path="ontology.owl", format="xml") ``` @@ -148,59 +140,6 @@ The pipeline runs through these stages in order: -## SKOS Vocabularies - -Build controlled vocabularies and taxonomies using the W3C SKOS standard: - -```python -from semantica.ontology import SKOSVocabulary - -vocab = SKOSVocabulary() -vocab.add_concept("Machine Learning", broader="Artificial Intelligence") -vocab.add_concept("Deep Learning", broader="Machine Learning") -vocab.add_concept("Computer Vision", broader="Deep Learning") -vocab.add_alt_label("ML", for_concept="Machine Learning") - -skos_ttl = vocab.export(format="turtle") -``` - -## Ontology Alignment - - - - Map concepts across two ontologies: - - ```python - from semantica.ontology import OntologyAligner - - aligner = OntologyAligner() - alignment = aligner.align(source_ontology, target_ontology) - - for mapping in alignment.mappings: - print(f"{mapping.source} → {mapping.target} (confidence: {mapping.confidence:.2f})") - - merged = aligner.merge(source_ontology, target_ontology, alignment) - ``` - - - Compare versions and generate migration scripts: - - ```python - from semantica.ontology import OntologyDiff, OntologyMigrator - - diff = OntologyDiff() - changes = diff.compare(ontology_v1, ontology_v2) - - for change in changes: - print(f"{change.type}: {change.element} — {change.description}") - - migrator = OntologyMigrator() - migration_script = migrator.generate_migration(changes) - migrator.apply(kg, migration_script) - ``` - - - ## Advanced Generation Tools @@ -351,20 +290,15 @@ for q, result in zip(questions, cq_results): ## Ontology Hub (v0.5.0) -A visual browser UI for the full ontology lifecycle, served by `semantica.explorer`: +A visual browser UI for the full ontology lifecycle, served by the Explorer CLI: ```bash pip install "semantica[explorer]" -``` - -```python -from semantica.explorer import start_explorer - -start_explorer(graph=kg, port=8080) +semantica explore # Navigate to http://localhost:8080 → Ontology Hub tab ``` -Features: visual editor, SHACL Studio, alignment authoring, health dashboard, and version control. +Features: visual editor, SHACL Studio, health dashboard, and version control. ## Tips and Common Pitfalls @@ -381,7 +315,7 @@ Features: visual editor, SHACL Studio, alignment authoring, health dashboard, an - **Always validate with SHACL after schema changes.** When you add new classes or properties, run `SHACLValidator.validate(kg, shapes)` immediately. SHACL violations often surface data quality issues that were silently passing before. + **Always validate after schema changes.** When you add new classes or properties, run `OntologyValidator.validate(ontology)` immediately. Validation issues often surface data quality problems that were silently passing before. @@ -392,10 +326,6 @@ Features: visual editor, SHACL Studio, alignment authoring, health dashboard, an **Namespace your terms consistently.** All classes and properties need stable IRIs. Use `NamespaceManager` to generate them programmatically — never hardcode IRI strings in code, because base URIs change when projects move. - - **Diff before migration.** Always run `OntologyDiff.compare()` before `OntologyMigrator.apply()`. The diff shows exactly which graph entities will be affected — some migrations (renaming a class) require updating thousands of existing nodes. - - Apply inference rules over ontology axioms. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index bd782c3d..8cc594c1 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -12,8 +12,8 @@ icon: "link" Track entities, relationships, and activities with full source attribution and confidence scores. - - Record pipeline activities and which entities they produced or consumed. + + Track entity and relationship lineage within a knowledge graph. Full directed lineage from any entity back to its originating source document. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 0594fc49..9e3d91db 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -64,7 +64,7 @@ icon: "magnifying-glass-chart" Direct `(subject, predicate, object)` triplet generation for RDF-ready output. - + Event detection with participants, temporal context, and confidence scores. @@ -215,14 +215,14 @@ triplets = trip.extract(text) Triplets are suitable for loading directly into a triplet store or knowledge graph without a separate relation extraction step. -## EventExtractor +## EventDetector Detect events with participants and temporal context: ```python -from semantica.semantic_extract import EventExtractor +from semantica.semantic_extract import EventDetector -extractor = EventExtractor(method="llm", llm_provider=llm) +extractor = EventDetector(method="llm", llm_provider=llm) events = extractor.extract(text) ``` diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index fd3d53a2..79011323 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -12,8 +12,8 @@ icon: "table" Unified interface across Blazegraph, Apache Jena (Fuseki), and RDF4J — swap backends with one parameter. - - Zero-setup in-memory store for unit tests and small datasets — no server, no Docker required. + + Zero-setup in-memory mode via `backend="memory"` for unit tests and small datasets — no server required. Full SELECT, CONSTRUCT, ASK, and UPDATE query support with pagination for large result sets. @@ -138,30 +138,6 @@ icon: "table" Best for: Enterprise Java ecosystems, Eclipse Foundation deployments, plugin-based reasoning. - - ```python - from semantica.triplet_store import InMemoryTripletStore - - store = InMemoryTripletStore() - - store.add_triplet("ex:alice", "ex:knows", "ex:bob") - store.add_triplet("ex:bob", "ex:works_for", "ex:acme") - - results = store.sparql(""" - SELECT ?person ?company WHERE { - ?person ex:works_for ?company . - } - """) - - # Serialize to string for inspection - ttl = store.export_to_string(format="turtle") - print(ttl) - ``` - - `InMemoryTripletStore` shares the same interface as `TripletStore` — swap backends without changing query code. - - Best for: unit tests, CI pipelines, small datasets, zero-infrastructure local exploration. - | Backend | License | OWL Reasoning | Hosted Option | Best For | @@ -305,7 +281,7 @@ results = store.sparql("SELECT * WHERE { ?s ?p ?o } LIMIT 10") ## Tips and Common Pitfalls - **Use `InMemoryTripletStore` for unit tests, Jena or Blazegraph for production.** The in-memory backend requires zero server setup and is safe for CI. It does not persist across process restarts — switch to a server-backed store before deploying. No code changes needed, just the `backend=` parameter. + **Use Apache Jena (Fuseki) for development and Blazegraph for production.** Jena runs with a single Docker command, supports OWL reasoning natively, and requires no licence. Switch to Blazegraph for high-throughput workloads by changing the `backend=` parameter — no other code changes needed. diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index 2a3f9027..39f5a541 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -9,20 +9,17 @@ icon: "chart-bar" ## What You Get - - Interactive HTML (PyVis) and static image (Matplotlib) graph rendering with layout options. + + Interactive network and community graph rendering with force, hierarchical, and circular layouts. - Class hierarchy and property relationship visualization from any OntologyManager. + Class hierarchy and property relationship visualization from any ontology. UMAP, t-SNE, and PCA dimensionality reduction plots for embedding cluster analysis. - Timeline views, animated evolution, snapshot comparison, and temporal pattern highlights. - - - Ego-mode neighborhood views and N×N distance matrix heatmaps from Distance Intelligence. + Timeline views, network evolution animation, snapshot comparison, and temporal pattern highlights. Centrality rankings, community-colored graphs, and degree distribution histograms. @@ -34,37 +31,33 @@ icon: "chart-bar" ```python - from semantica.visualization import GraphVisualizer + from semantica.visualization import KGVisualizer - viz = GraphVisualizer() + viz = KGVisualizer(layout="force", color_scheme="default") - # Interactive HTML — opens in browser, supports hover and click - viz.visualize(graph, output="graph.html") + # Interactive — opens in browser, supports hover and click + viz.visualize_network(graph, output="interactive") ``` ```python - viz.visualize( + viz = KGVisualizer(layout="force", color_scheme="vibrant") + + viz.visualize_network( graph, - output="graph.html", - layout="force_directed", # "force_directed" | "hierarchical" | "circular" | "spring" + output="html", + file_path="graph.html", node_color_by="type", # color nodes by entity type attribute - edge_label="relation", # show edge relationship labels - color_scheme="vibrant", # color palette — see Color Schemes section - max_nodes=500, # limit rendering for large graphs ) ``` ```python # Static PNG — for reports and embedding in documents - viz.visualize(graph, output="graph.png", dpi=150) + viz.visualize_network(graph, output="png", file_path="graph.png") # Vector SVG — for publications and scalable diagrams - viz.visualize(graph, output="graph.svg") - - # PDF — for print or compliance reports - viz.visualize(graph, output="graph.pdf") + viz.visualize_network(graph, output="svg", file_path="graph.svg") ``` @@ -72,32 +65,34 @@ icon: "chart-bar" ## Visualizers - + Interactive and static knowledge graph rendering: ```python - from semantica.visualization import GraphVisualizer + from semantica.visualization import KGVisualizer - viz = GraphVisualizer() + viz = KGVisualizer(layout="force", color_scheme="default") - # Interactive HTML - viz.visualize(graph, output="graph.html") + # Interactive — opens in browser + viz.visualize_network(graph, output="interactive") - # Static PNG with custom DPI - viz.visualize(graph, output="graph.png", backend="matplotlib", dpi=150) + # Save as HTML file + viz.visualize_network(graph, output="html", file_path="graph.html") - # Display inline (Jupyter or default browser) - viz.show(graph) + # Static PNG + viz.visualize_network(graph, output="png", file_path="graph.png") + + # Community-colored graph + viz.visualize_communities(graph, communities, file_path="communities.html") ``` - **Layout options:** + **Layout options (`layout=`):** | Layout | Description | Best For | | ------ | ----------- | -------- | - | `force_directed` | Physics simulation — clusters emerge naturally | General graphs | + | `force` | Physics simulation — clusters emerge naturally | General graphs | | `hierarchical` | Top-down tree layout | Taxonomies, org charts | | `circular` | Nodes on a circle, edges as chords | Small dense graphs | - | `spring` | Spring-force layout (Fruchterman-Reingold) | Medium graphs | Visualize class hierarchies and property relationships: @@ -122,10 +117,11 @@ icon: "chart-bar" viz = EmbeddingVisualizer() - viz.visualize( + viz.visualize_2d_projection( embeddings=embeddings, labels=labels, - output="embeddings.html", + output="interactive", + file_path="embeddings.html", method="umap", # "umap" | "tsne" | "pca" ) ``` @@ -141,47 +137,20 @@ icon: "chart-bar" ```python from semantica.visualization import TemporalVisualizer - from datetime import datetime viz = TemporalVisualizer() - # Static timeline of additions and removals - viz.visualize_timeline(temporal_kg, output="timeline.html") + # Timeline of entity/relationship changes + viz.visualize_timeline(temporal_kg, output="interactive") - # Animated evolution — one frame per time step - viz.animate(temporal_kg, output="evolution.html", fps=2) + # Animated network evolution — one frame per time step + viz.visualize_network_evolution(temporal_kg, output="html", file_path="evolution.html") # Side-by-side snapshot comparison - snap_a = temporal_kg.at(datetime(2020, 1, 1)) - snap_b = temporal_kg.at(datetime(2023, 1, 1)) - viz.compare_snapshots(snap_a, snap_b, output="snapshot_diff.html") + viz.visualize_snapshot_comparison(snap_a, snap_b, output="html", file_path="diff.html") - # Pattern visualization — highlight recurring temporal patterns - viz.visualize_patterns(temporal_kg, pattern_type="recurrence", output="patterns.html") - ``` - - - Semantic neighborhood and distance matrix visualization from Distance Intelligence: - - ```python - from semantica.visualization import DistanceVisualizer - - viz = DistanceVisualizer() - - # Ego-mode: neighborhood of one node colored by distance band - viz.visualize_ego( - graph, - center_node="Apple Inc.", - output="ego.html", - radius=0.5, # semantic distance radius - ) - - # N×N distance matrix heatmap - viz.visualize_distance_matrix( - matrix=distance_matrix, - labels=node_labels, - output="distance_heatmap.html", - ) + # Recurring temporal patterns + viz.visualize_temporal_patterns(temporal_kg, output="html", file_path="patterns.html") ``` @@ -245,13 +214,10 @@ viz.visualize(graph, output="graph.html", color_scheme="vibrant") ## Graph Explorer (Full Dashboard) -For a full browser-based UI with search, path finding, and the Ontology Hub, use `semantica.explorer`: +For a full browser-based UI with search, path finding, and the Ontology Hub, launch the Explorer via the CLI: -```python -from semantica.explorer import start_explorer - -start_explorer(graph=kg, port=8080) -# Opens at http://localhost:8080 +```bash +semantica explore ``` See the [Explorer reference](explorer) for the full feature set and REST API. @@ -279,7 +245,7 @@ See the [Explorer reference](explorer) for the full feature set and REST API. - **For interactive dashboards, prefer Explorer.** `GraphVisualizer.visualize()` generates a self-contained HTML file. `start_explorer()` gives a full live web app with search, filtering, path-finding, and REST API. Use Explorer for team exploration, Visualizer for standalone report embeds. + **For interactive dashboards, prefer Explorer.** `KGVisualizer.visualize_network()` generates a self-contained HTML file. The Explorer CLI (`semantica explore`) gives a full live web app with search, filtering, path-finding, and REST API. Use Explorer for team exploration, Visualizer for standalone report embeds. From ff438878420bc8d4ecbbc40f54104acca816fed3 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 13:14:01 +0530 Subject: [PATCH 05/11] fix: correct remaining API mismatches in pipeline, vector_store, and normalize docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pipeline.md: ParallelismManager pool_type="thread"/"process" → use_processes=False/True; execute_parallel() returns List[ParallelExecutionResult] not aggregate object - vector_store.md: remove MetadataStore.add_field() (method is on MetadataSchema, not MetadataStore); fix tip to reference MetadataStore.update_metadata() not VectorStore - normalize.md: Pipeline() orchestrator misuse → PipelineBuilder + ExecutionEngine pattern --- docs/reference/normalize.md | 20 +++++++++++++------- docs/reference/pipeline.md | 22 +++++++++++++--------- docs/reference/vector_store.md | 8 +------- 3 files changed, 27 insertions(+), 23 deletions(-) diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index 4f9a56a4..1143c422 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -360,21 +360,27 @@ print(f"Invalid: {result.error_count}") ## Pipeline Integration ```python -from semantica.pipeline import Pipeline +from semantica.pipeline import PipelineBuilder, ExecutionEngine from semantica.ingest import FileIngestor from semantica.normalize import TextNormalizer from semantica.semantic_extract import NERExtractor from semantica.llms import Groq import os -llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +ingestor = FileIngestor() +normalizer = TextNormalizer(strip_html=True, normalize_unicode=True) +extractor = NERExtractor(method="llm", llm_provider=llm) -pipeline = Pipeline() -pipeline.add_step("ingest", FileIngestor()) -pipeline.add_step("normalize", TextNormalizer(strip_html=True, normalize_unicode=True)) -pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm)) +builder = PipelineBuilder() +builder.add_step("ingest", "file_ingest", handler=ingestor.ingest) +builder.add_step("normalize", "text_normalize", handler=normalizer.normalize) +builder.add_step("extract", "ner_extract", handler=extractor.extract) +builder.connect_steps("ingest", "normalize") +builder.connect_steps("normalize", "extract") -result = pipeline.run("data/documents/") +pipeline = builder.build("normalize_pipeline") +result = ExecutionEngine().execute_pipeline(pipeline, data="data/documents/") ``` ## Tips and Common Pitfalls diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index b461cbf6..4b01b691 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -405,25 +405,29 @@ Checks performed: ```python from semantica.pipeline import ParallelismManager - manager = ParallelismManager(max_workers=8, pool_type="thread") + # use_processes=False (default) → thread pool for I/O-bound tasks + manager = ParallelismManager(max_workers=8, use_processes=False) - tasks = [{"fn": ner.extract, "args": [text]} for text in texts] - result = manager.execute_parallel(tasks, timeout=60) + tasks = [{"fn": ner.extract, "args": [text]} for text in texts] + results = manager.execute_parallel(tasks, timeout=60) + # returns List[ParallelExecutionResult] - print(f"Successful: {result.success_count}, Failed: {result.failure_count}") + successes = [r for r in results if r.success] + failures = [r for r in results if not r.success] ``` - Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls. Threads share memory and context-switch cheaply between waiting operations. + Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls. ```python - manager = ParallelismManager(max_workers=4, pool_type="process") + # use_processes=True → process pool, bypasses Python GIL + manager = ParallelismManager(max_workers=4, use_processes=True) - tasks = [{"fn": embedder.embed, "args": [chunk]} for chunk in chunks] - result = manager.execute_parallel(tasks, timeout=120) + tasks = [{"fn": embedder.generate_embeddings, "args": [chunk]} for chunk in chunks] + results = manager.execute_parallel(tasks, timeout=120) ``` - Use process pools for **CPU-bound** steps: embedding computation, OCR, large NER batches. Processes bypass Python's GIL for true multi-core parallelism. + Use process pools for **CPU-bound** steps: embedding, OCR, large NER batches. diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 9dee3169..f711db99 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -317,12 +317,6 @@ from semantica.vector_store import MetadataStore meta_store = MetadataStore() -# Define schema fields -meta_store.add_field("author", str, required=True) -meta_store.add_field("year", int, required=True) -meta_store.add_field("category", str) -meta_store.add_field("score", float, default=0.0) - # Store and retrieve metadata meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"}) meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"}) @@ -423,7 +417,7 @@ store = VectorStore(backend="faiss", dimension=768, metric="cosine") - **Update metadata without re-embedding.** `store.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content. + **Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content. From ce765b6f66ecc1ac62105ae9f7223bd932b5c62b Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 13:36:14 +0530 Subject: [PATCH 06/11] fix: correct docs-to-code mismatches in 8 reference modules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - graph_store: remove create_constraint(), add_nodes_bulk(), add_edges_bulk() → create_nodes(), add_edges() - deduplication: fix PropertyMergeRule → MergeStrategy enum; add_rule() → add_property_rule(); merge() → merge_entities(); remove non-existent UNION/MAX/MIN/VOTING constants - conflicts: set_credibility() → set_source_credibility(); group_by_severity/identify_patterns/analyze_sources → analyze_conflicts() dict keys; generate() → generate_guide(); remove time_window= param from analyze_trends() - reasoning: infer() → forward_chain(); remove apply_transitivity/symmetry/inverse() templates that don't exist; GraphReasoner(kg) → GraphReasoner(); infer(kg) → reason(graph, query) - split: split_document() (singular) → split_documents([parsed]) throughout - seed: remove register_source_object(), populate(), inject(), load_from_file(), diff_versions(), get_version(tag=) — replace with register_source() and load_from_csv/json() - change_management: remove rollback(), get_log_entry(), export_audit_trail(), get_audit_trail() — replace audit section with list_versions() + diff() pattern - export: export_to_file() → export_to_rdf(); YAMLExporter → SemanticNetworkYAMLExporter --- docs/reference/change_management.md | 99 +++++++++-------------------- docs/reference/conflicts.md | 48 +++++++------- docs/reference/deduplication.md | 67 +++++++++---------- docs/reference/export.md | 34 +++++----- docs/reference/graph_store.md | 20 ++---- docs/reference/reasoning.md | 49 +++++--------- docs/reference/seed.md | 33 ++++------ docs/reference/split.md | 8 +-- 8 files changed, 136 insertions(+), 222 deletions(-) diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 059af4d4..092e55e9 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -28,8 +28,8 @@ icon: "clock-rotate-left" Structured record of every change: author, timestamp, checksum, and change list. - - Full audit trail as CSV or JSON for regulatory review and subject-access requests. + + Full tamper-evident version history via `list_versions()` and `diff()` for regulatory review. @@ -77,12 +77,6 @@ icon: "clock-rotate-left" print(f" [{change.type}] {change.element}: {change.description}") ``` - - ```python - # Safe mode (default) — fails with a clear error rather than dropping nodes - manager.rollback(target_version="v1.0", allow_data_loss=False) - ``` - ## TemporalVersionManager @@ -96,7 +90,7 @@ Version control for knowledge graphs — snapshot, diff, and rollback. | `storage_path` | `str` | `None` | Path to SQLite database; uses in-memory if omitted | | `storage` | `VersionStorage` | `None` | Explicit storage backend instance — overrides `storage_path` | -### List, Retrieve, and Rollback +### List and Retrieve ```python # List all versions @@ -106,15 +100,8 @@ for v in versions: # Retrieve a specific version kg_v1 = manager.get_version("v1.0") - -# Rollback to a previous version -manager.rollback(target_version="v1.0", allow_data_loss=False) ``` - - `rollback(allow_data_loss=False)` is the safe default — it fails with a clear error if nodes were added after the target snapshot. Set `allow_data_loss=True` only when you explicitly intend to discard those changes. - - ## Diff Analysis Compare any two snapshots to see exactly what changed — useful for code review, incident investigation, and regulatory audit: @@ -236,9 +223,8 @@ if not is_valid: Every version snapshot includes a structured `ChangeLogEntry` that records the full context of a change: ```python -from semantica.change_management import ChangeLogEntry - -entry: ChangeLogEntry = manager.get_log_entry(snapshot_id) +# Retrieve a version entry +entry = manager.get_version("v1.0") print(entry.version) # "v1.0" print(entry.author) # "user@example.com" @@ -269,58 +255,35 @@ class ChangeLogEntry: -## Compliance and Audit Export +## Compliance and Version History -All changes are preserved in a tamper-evident audit trail. Export for regulatory review: - - - -```python CSV export -# Full audit trail -manager.export_audit_trail("audit.csv", format="csv") - -# Scoped to a time range — SOX quarterly review -from datetime import datetime - -trail = manager.get_audit_trail( - from_date=datetime(2026, 1, 1), - to_date=datetime(2026, 3, 31), -) -manager.export_audit_trail("q1_audit.csv", trail=trail, format="csv") -``` - -```python JSON export -# Full audit trail -manager.export_audit_trail("audit.json", format="json") - -# Scoped to a specific entity — HIPAA subject-access request -trail = manager.get_audit_trail(entity_id="patient_001") -manager.export_audit_trail("patient_001_audit.json", trail=trail, format="json") -``` - - - -You can also iterate the trail directly: +All version snapshots form a tamper-evident audit trail. Use `list_versions()` and `diff()` to reconstruct and review changes for regulatory purposes: ```python -trail = manager.get_audit_trail(entity_id="patient_001") -for entry in trail: - print(f"{entry.timestamp.isoformat()} | {entry.author} | {entry.action} | {entry.description}") +from semantica.change_management import TemporalVersionManager + +manager = TemporalVersionManager(storage_path="versions.db") + +# Enumerate the full version history +for entry in manager.list_versions(): + print(f"{entry.created_at.isoformat()} | {entry.author} | {entry.version} | {entry.message}") + +# Diff any two snapshots for a change report +diff = manager.diff("v1.0", "v2.0") +print(f"Added: {len(diff.added_nodes)} | Removed: {len(diff.removed_nodes)} | Modified: {len(diff.modified_nodes)}") +for change in diff.changes: + print(f" [{change.type}] {change.element}: {change.description}") ``` -### Audit Fields +Use `verify_checksum()` before any compliance export to confirm graph integrity: -| Field | Description | -| ----- | ----------- | -| `timestamp` | UTC ISO 8601 datetime | -| `entity_id` | ID of the affected entity or relationship | -| `author` | User or process that made the change | -| `action` | `CREATE` / `UPDATE` / `DELETE` / `MERGE` / `ROLLBACK` | -| `property` | Property name that changed (UPDATE rows only) | -| `old_value` | Previous value (UPDATE and DELETE rows) | -| `new_value` | New value (CREATE and UPDATE rows) | -| `snapshot_id` | ID of the containing snapshot | -| `checksum` | SHA-256 of the entity state after the change | +```python +from semantica.change_management import verify_checksum + +is_valid = verify_checksum(kg, expected_checksum=entry.checksum) +if not is_valid: + raise RuntimeError("Graph has been modified since the snapshot was taken") +``` ### Compliance Coverage @@ -354,11 +317,7 @@ for entry in trail: - **Export audit trails before compliance reviews.** `export_audit_trail("audit.csv", format="csv")` produces a complete tamper-evident record in one call. Schedule this export before quarterly reviews (SOX), regulatory inspections (FDA 21 CFR Part 11), or subject-access requests (GDPR). - - - - **Use `get_audit_trail(from_date=..., to_date=...)` for scoped reviews.** Exporting the full audit trail for a multi-year graph can produce millions of rows. Scope to a time window or entity ID for faster, focused reports. + **Use `list_versions()` and `diff()` for compliance reviews.** `manager.list_versions()` enumerates the full version history and `manager.diff(v1, v2)` produces a machine-readable change report. Run `verify_checksum()` first to confirm the graph hasn't been modified since the snapshot was taken. diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index 9914a73d..6346acd4 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -49,10 +49,10 @@ Semantica's conflict detection makes disagreements explicit and actionable: from semantica.conflicts import SourceTracker tracker = SourceTracker() - tracker.set_credibility("sec_filings", 0.95) - tracker.set_credibility("pubmed", 0.92) - tracker.set_credibility("wikipedia", 0.80) - tracker.set_credibility("news_articles", 0.65) + tracker.set_source_credibility("sec_filings", 0.95) + tracker.set_source_credibility("pubmed", 0.92) + tracker.set_source_credibility("wikipedia", 0.80) + tracker.set_source_credibility("news_articles", 0.65) ``` @@ -73,10 +73,11 @@ Semantica's conflict detection makes disagreements explicit and actionable: from semantica.conflicts import ConflictAnalyzer analyzer = ConflictAnalyzer() - by_severity = analyzer.group_by_severity(conflicts) - print(f"Critical: {len(by_severity['critical'])}") - print(f"High: {len(by_severity['high'])}") - print(f"Low: {len(by_severity['low'])}") + analysis = analyzer.analyze_conflicts(conflicts) + by_severity = analysis["by_severity"] + print(f"Critical: {len(by_severity.get('critical', []))}") + print(f"High: {len(by_severity.get('high', []))}") + print(f"Low: {len(by_severity.get('low', []))}") ``` @@ -94,7 +95,7 @@ Semantica's conflict detection makes disagreements explicit and actionable: # Generate investigation guides for critical conflicts generator = InvestigationGuideGenerator() for conflict in by_severity["critical"]: - guide = generator.generate(conflict) + guide = generator.generate_guide(conflict) print(f"\n{guide.title}") for step in guide.steps: print(f" [{step.order}] ({step.priority.upper()}) {step.description}") @@ -161,9 +162,9 @@ for result in results: from semantica.conflicts import ConflictResolver, SourceTracker, ResolutionStrategy tracker = SourceTracker() - tracker.set_credibility("sec_filings", 0.92) - tracker.set_credibility("wikipedia", 0.80) - tracker.set_credibility("news_articles", 0.65) + tracker.set_source_credibility("sec_filings", 0.92) + tracker.set_source_credibility("wikipedia", 0.80) + tracker.set_source_credibility("news_articles", 0.65) resolver = ConflictResolver(source_tracker=tracker) results = resolver.resolve_conflicts( @@ -201,7 +202,7 @@ for result in results: generator = InvestigationGuideGenerator() for conflict in conflicts: - guide = generator.generate(conflict) + guide = generator.generate_guide(conflict) print(f"{guide.title}") for step in guide.steps: print(f" [{step.order}] {step.description}") @@ -238,8 +239,8 @@ from semantica.conflicts import SourceTracker from datetime import datetime tracker = SourceTracker() -tracker.set_credibility("sec_10k", 0.92) -tracker.set_credibility("wikipedia", 0.80) +tracker.set_source_credibility("sec_10k", 0.92) +tracker.set_source_credibility("wikipedia", 0.80) tracker.track_property_source( entity_id="apple_inc", @@ -267,19 +268,20 @@ from semantica.conflicts import ConflictAnalyzer analyzer = ConflictAnalyzer() -patterns = analyzer.identify_patterns(conflicts) -by_severity = analyzer.group_by_severity(conflicts) -source_stats = analyzer.analyze_sources(conflicts) -trends = analyzer.analyze_trends(conflicts, time_window="30d") +analysis = analyzer.analyze_conflicts(conflicts) +patterns = analysis["patterns"] +by_severity = analysis["by_severity"] +source_stats = analysis["by_source"] +trends = analyzer.analyze_trends(conflicts) print(f"Trend direction: {trends['direction']}") # "increasing" | "stable" | "decreasing" print(f"Change: {trends['change_pct']:.1f}%") ``` **Key behaviours:** -- `identify_patterns()` groups conflicts by attribute name and type — use it to find systemic data quality issues -- `analyze_sources()` flags sources with disproportionate conflict rates — a signal that a source's pipeline needs review -- `analyze_trends()` compares conflict counts across time windows — a rising trend means a data source is degrading +- `analyze_conflicts()["patterns"]` groups conflicts by attribute name and type — use it to find systemic data quality issues +- `analyze_conflicts()["by_source"]` flags sources with disproportionate conflict rates — a signal that a source's pipeline needs review +- `analyze_trends()` compares conflict counts over time — a rising trend means a data source is degrading ## InvestigationGuideGenerator @@ -289,7 +291,7 @@ Auto-generate human-readable investigation checklists for conflicts requiring ma from semantica.conflicts import InvestigationGuideGenerator generator = InvestigationGuideGenerator() -guide = generator.generate(conflict) +guide = generator.generate_guide(conflict) print(f"Title: {guide.title}") print(f"Context: {guide.context}") diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index f10c0cd3..6246aad8 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -176,38 +176,33 @@ print(f"Merged to: {len(merged_entities)} canonical entities") | `keep_first` | Keep the first entity in each group | Source order is meaningful (most authoritative first) | | `keep_last` | Keep the most recently seen entity | Most recent source is most accurate | | `keep_most_complete` | Keep the entity with the most non-null properties | Default — maximizes data richness | -| `union` | Merge all properties; combine non-conflicting fields | You want every known alias, tag, and label | -| `voting` | Most common property value wins | Multiple semi-reliable sources | +| `keep_highest_confidence` | Keep the entity with the highest confidence score | Extraction pipelines produce confidence scores | +| `merge_all` | Merge all properties; combine non-conflicting fields | You want every known alias, tag, and label | ### Per-Property Merge Rules +Use `MergeStrategyManager` to apply different `MergeStrategy` values per property: + ```python -from semantica.deduplication import EntityMerger, PropertyMergeRule +from semantica.deduplication import MergeStrategyManager, MergeStrategy -merger = EntityMerger( - property_rules={ - "name": PropertyMergeRule.KEEP_FIRST, - "aliases": PropertyMergeRule.UNION, - "description": PropertyMergeRule.KEEP_LONGEST, - "confidence": PropertyMergeRule.MAX, - "created_at": PropertyMergeRule.KEEP_FIRST, - "updated_at": PropertyMergeRule.KEEP_LAST, - } -) +manager = MergeStrategyManager() +manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) +manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) +manager.add_property_rule("confidence", MergeStrategy.KEEP_HIGHEST_CONFIDENCE) +manager.add_property_rule("created_at", MergeStrategy.KEEP_FIRST) +manager.add_property_rule("updated_at", MergeStrategy.KEEP_LAST) -merged_entities = merger.merge_duplicates(entities, preserve_provenance=True) +merged_entity = manager.merge_entities(duplicate_group) ``` -| Rule | Behaviour | -| ---- | --------- | +| Strategy | Behaviour | +| -------- | --------- | | `KEEP_FIRST` | Value from the first entity in the group | | `KEEP_LAST` | Value from the last entity | -| `KEEP_LONGEST` | Longest non-null string value | -| `KEEP_MOST_COMPLETE` | Entity with the most non-null fields (default fallback) | -| `UNION` | Combine all unique values into a list | -| `MAX` | Numerically largest value | -| `MIN` | Numerically smallest value | -| `VOTING` | Most frequently occurring value | +| `KEEP_MOST_COMPLETE` | Entity with the most non-null fields | +| `KEEP_HIGHEST_CONFIDENCE` | Entity with the highest confidence score | +| `MERGE_ALL` | Combine all properties from every entity in the group | ## SimilarityCalculator @@ -278,19 +273,18 @@ print(f"Avg separation: {result.quality.avg_separation:.3f}") Define complex, reusable merge configurations once and apply them across multiple operations: ```python -from semantica.deduplication import MergeStrategyManager, PropertyMergeRule +from semantica.deduplication import MergeStrategyManager, MergeStrategy manager = MergeStrategyManager() -manager.add_rule("name", PropertyMergeRule.KEEP_FIRST) -manager.add_rule("aliases", PropertyMergeRule.UNION) -manager.add_rule("description", PropertyMergeRule.KEEP_LONGEST) -manager.add_rule("confidence", PropertyMergeRule.MAX) -manager.add_rule("sources", PropertyMergeRule.UNION) -manager.add_rule("created_at", PropertyMergeRule.KEEP_FIRST) -manager.add_rule("updated_at", PropertyMergeRule.KEEP_LAST) -manager.set_default_rule(PropertyMergeRule.KEEP_MOST_COMPLETE) +manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) +manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) +manager.add_property_rule("description", MergeStrategy.KEEP_MOST_COMPLETE) +manager.add_property_rule("confidence", MergeStrategy.KEEP_HIGHEST_CONFIDENCE) +manager.add_property_rule("sources", MergeStrategy.MERGE_ALL) +manager.add_property_rule("created_at", MergeStrategy.KEEP_FIRST) +manager.add_property_rule("updated_at", MergeStrategy.KEEP_LAST) -merged_entity = manager.merge(duplicate_group) +merged_entity = manager.merge_entities(duplicate_group) ``` ## Blocking Strategies @@ -379,13 +373,12 @@ class ClusterQuality: ```python - from semantica.deduplication import MergeStrategyManager, PropertyMergeRule + from semantica.deduplication import MergeStrategyManager, MergeStrategy manager = MergeStrategyManager() - manager.add_rule("name", PropertyMergeRule.KEEP_FIRST) - manager.add_rule("aliases", PropertyMergeRule.UNION) - manager.add_rule("description", PropertyMergeRule.KEEP_LONGEST) - manager.set_default_rule(PropertyMergeRule.KEEP_MOST_COMPLETE) + manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) + manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) + manager.add_property_rule("description", MergeStrategy.KEEP_MOST_COMPLETE) ``` diff --git a/docs/reference/export.md b/docs/reference/export.md index 9eab66a5..76b5b341 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -41,8 +41,10 @@ icon: "file-export" ```python - # Interactive HTML — opens in browser, supports hover and click - exporter.export_to_file(graph, "output.ttl", format="turtle") + # Export to RDF string, then write to file + rdf_str = exporter.export_to_rdf(graph, format="turtle") + with open("output.ttl", "w") as f: + f.write(rdf_str) ``` @@ -75,20 +77,14 @@ icon: "file-export" exporter = RDFExporter() - # Turtle (most readable RDF format) - exporter.export_to_file(graph, "output.ttl", format="turtle") + # Export to RDF string — write to file manually + rdf_str = exporter.export_to_rdf(graph, format="turtle") # Turtle (most readable) + rdf_str = exporter.export_to_rdf(graph, format="json-ld") # JSON-LD (APIs, Linked Data) + rdf_str = exporter.export_to_rdf(graph, format="nt") # N-Triples (streaming-friendly) + rdf_str = exporter.export_to_rdf(graph, format="xml") # RDF/XML (W3C standard) - # JSON-LD (best for APIs and Linked Data) - exporter.export_to_file(graph, "output.jsonld", format="json-ld") - - # N-Triples (streaming-friendly, one triple per line) - exporter.export_to_file(graph, "output.nt", format="nt") - - # RDF/XML (W3C standard, broadest compatibility) - exporter.export_to_file(graph, "output.xml", format="xml") - - # Export to string instead of file - rdf_str = exporter.export_to_rdf(graph, format="turtle") + with open("output.ttl", "w") as f: + f.write(exporter.export_to_rdf(graph, format="turtle")) ``` **Custom namespace management:** @@ -146,9 +142,9 @@ icon: "file-export" ``` ```python - from semantica.export import YAMLExporter + from semantica.export import SemanticNetworkYAMLExporter - exporter = YAMLExporter() + exporter = SemanticNetworkYAMLExporter() exporter.export(graph, "graph.yaml") yaml_str = exporter.to_string(graph) @@ -307,7 +303,7 @@ export_graph(graph, "graph.graphml", format="graphml") | `dot` | `GraphExporter` | `.dot` | Graphviz rendering | | `owl` | `OWLExporter` | `.owl` / `.ttl` | OWL 2.0 ontology distribution | | `csv` | `CSVExporter` | `.csv` | Spreadsheets, simple pipelines | -| `yaml` | `YAMLExporter` | `.yaml` | Human-readable, config-driven use | +| `yaml` | `SemanticNetworkYAMLExporter` | `.yaml` | Human-readable, config-driven use | | `arrow` | `ArrowExporter` | `.arrow` | Zero-copy inter-process transfer | | `numpy` | `VectorExporter` | `.npy` | NumPy arrays from embeddings | | `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files | @@ -326,7 +322,7 @@ export_graph(graph, "graph.graphml", format="graphml") - **Stream large graphs with `export_stream()`.** For graphs with more than 500k nodes, use `exporter.export_stream(graph, ...)` instead of `exporter.export_to_file()`. Streaming writes incrementally without buffering the full graph in memory — without it, a million-node export will likely OOM. + **Stream large graphs with `export_stream()`.** For graphs with more than 500k nodes, use `exporter.export_stream(graph, ...)` instead of building the full RDF string in memory. Streaming writes incrementally — without it, a million-node export will likely OOM. diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index 2f4d10f7..afeb9a68 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -48,13 +48,12 @@ icon: "server" ```python store.create_index(label="Person", property="name") store.create_index(label="Organization", property="name") - store.create_constraint(label="Organization", property="id", constraint_type="unique") ``` - + ```python - store.add_nodes_bulk(entities, batch_size=1000) - store.add_edges_bulk(relationships, batch_size=1000) + store.create_nodes(entities) + store.add_edges(relationships) ``` @@ -171,8 +170,8 @@ store.add_edge( ) # Bulk operations — use for large datasets -store.add_nodes_bulk(entities, batch_size=1000) -store.add_edges_bulk(relationships, batch_size=1000) +store.create_nodes(entities) +store.add_edges(relationships) # Delete store.delete_node("node_id") @@ -266,13 +265,6 @@ all_paths = analytics.all_paths("alice", "charlie", max_hops=4) # Index for fast label lookups store.create_index(label="Person", property="name") -# Uniqueness constraint -store.create_constraint( - label="Organization", - property="id", - constraint_type="unique", -) - # Inspect current schema schema = store.get_schema() print(schema["labels"]) @@ -291,7 +283,7 @@ print(schema["constraints"]) - **Use `add_nodes_bulk()` and `add_edges_bulk()` for large datasets.** Individual `add_node()` calls issue one network round-trip each. Bulk operations batch thousands of writes into a single transaction — 10–100× faster for initial loads. + **Use `create_nodes()` and `add_edges()` for loading multiple nodes and edges.** Individual `add_node()` calls issue one network round-trip each. Loading in bulk is significantly faster for initial graph population. diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 23df4e9e..63af28e2 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -78,39 +78,14 @@ Reasoning turns sparse explicit knowledge into a dense, coherent, contradiction- )) # Run inference — always call explicitly after adding facts/rules - result = reasoner.infer() + result = reasoner.forward_chain() for inference in result.derived_facts: print(f"{inference.subject} {inference.predicate} {inference.obj}") print(f" Derived via: {inference.explanation}") ``` - ### Built-In Rule Templates - - No manual rule authoring required for the three most common patterns: - - ```python - engine = Reasoner() - - # Transitive closure: A→B, B→C ⟹ A→C - engine.apply_transitivity("located_in") - - # Symmetry: A knows B ⟹ B knows A - engine.apply_symmetry("knows") - - # Inverse: A parent_of B ⟹ B child_of A - engine.apply_inverse("parent_of", "child_of") - - result = engine.infer() - ``` - - | Template | Parameters | Description | - | -------- | ---------- | ----------- | - | `apply_transitivity(predicate)` | `predicate: str` | Adds A→C rule for all A→B, B→C chains | - | `apply_symmetry(predicate)` | `predicate: str` | Adds B→A rule for every A→B fact | - | `apply_inverse(predicate, inverse)` | `predicate, inverse: str` | Adds inverse direction for every fact | - - Always call `reasoner.infer()` after adding facts and rules. Adding them updates internal state but does **not** trigger inference automatically. + Always call `reasoner.forward_chain()` after adding facts and rules. Adding them updates internal state but does **not** trigger inference automatically. @@ -120,7 +95,7 @@ Reasoning turns sparse explicit knowledge into a dense, coherent, contradiction- ```python from semantica.reasoning import GraphReasoner - graph_reasoner = GraphReasoner(kg) + graph_reasoner = GraphReasoner() # Define a transitive ancestor rule graph_reasoner.add_rule({ @@ -131,7 +106,7 @@ Reasoning turns sparse explicit knowledge into a dense, coherent, contradiction- "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} }) - inferences = graph_reasoner.infer(kg) + inferences = graph_reasoner.reason(graph, query="") for inf in inferences: print(f"{inf['subject']} {inf['predicate']} {inf['object']}") ``` @@ -352,12 +327,18 @@ Different engines cover different expressivity levels — compose them for riche ```python - from semantica.reasoning import Reasoner + from semantica.reasoning import Reasoner, Rule, Fact, RuleType engine = Reasoner() - engine.apply_transitivity("located_in") - engine.apply_symmetry("colleague_of") - structural_result = engine.infer() + engine.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[ + {"subject": "?x", "predicate": "located_in", "object": "?y"}, + {"subject": "?y", "predicate": "located_in", "object": "?z"} + ], + conclusion={"subject": "?x", "predicate": "located_in", "object": "?z"} + )) + structural_result = engine.forward_chain() ``` @@ -400,7 +381,7 @@ Different engines cover different expressivity levels — compose them for riche ## Tips and Common Pitfalls - **Always call `reasoner.infer()` after adding facts and rules.** Adding facts and rules updates internal state but doesn't trigger inference automatically. Inference is a separate, explicit step. + **Always call `reasoner.forward_chain()` after adding facts and rules.** Adding facts and rules updates internal state but doesn't trigger inference automatically. Inference is a separate, explicit step. diff --git a/docs/reference/seed.md b/docs/reference/seed.md index 88609b5b..322c9259 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -15,8 +15,8 @@ icon: "database" Typed source definition supporting CSV, JSON, SQL, API, and RDF with format-specific config. - - Inject named built-in datasets (companies, countries, currencies) or custom seed files into an existing graph. + + Build a foundation graph from all registered sources in one pass, ready to merge with extracted data. `seed_first`, `extracted_first`, and `smart_merge` with property-level conflict detection. @@ -97,7 +97,7 @@ icon: "database" } ) - manager.register_source_object(csv_source) + manager.register_source("employees", "csv", csv_source.path, config=csv_source.config) ``` Best for: employee rosters, product lists, reference tables. @@ -169,9 +169,8 @@ icon: "database" | `validate_quality(seed_data)` | Check schema compliance, required fields, and duplicates | | `integrate_with_extracted(seed, extracted, strategy)` | Merge seed and extracted graphs | | `export_seed_data(path, format)` | Export seed graph to RDF (`turtle`, `json-ld`), JSON, or CSV | -| `populate(kg, dataset, count)` | Inject a named built-in dataset into an existing graph | -| `inject(kg)` | Merge all registered sources into `kg` without duplicating existing entities | -| `load_from_file(path)` | Load seed nodes from JSON, CSV, or RDF file into the manager | +| `load_from_csv(path)` | Load seed records from a CSV file | +| `load_from_json(path)` | Load seed records from a JSON file | | `list_sources()` | List all registered source names and their formats | | `get_version(name)` | Get the current version metadata for a named source | @@ -221,18 +220,18 @@ icon: "database" ## Built-in Datasets -Inject canonical reference data without loading external files: +Register built-in reference datasets as named sources and load them into your foundation graph: ```python from semantica.seed import SeedDataManager -from semantica.kg import GraphBuilder -kg = GraphBuilder().build(entities=entities, relationships=relationships) manager = SeedDataManager() -# Inject a named built-in dataset -manager.populate(kg, dataset="companies", count=100) -manager.populate(kg, dataset="countries") +# Register built-in reference sources by format and path +manager.register_source("countries", "csv", "data/iso_countries.csv") +manager.register_source("currencies", "json", "data/iso_currencies.json") + +foundation_kg = manager.create_foundation_graph() ``` | Dataset | Content | @@ -304,14 +303,6 @@ print(f"Version: {version.version_id}") print(f"Hash: {version.checksum}") print(f"Records: {version.record_count}") print(f"Updated: {version.last_modified}") - -# Compare versions to detect changes -old_version = manager.get_version("taxonomy", tag="previous") -if version.checksum != old_version.checksum: - diff = manager.diff_versions("taxonomy", old_version.version_id, version.version_id) - print(f"Added: {diff.added_count} records") - print(f"Removed: {diff.removed_count} records") - print(f"Changed: {diff.modified_count} records") ``` ## YAML Configuration @@ -368,7 +359,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first - **Track seed versions to detect drift.** Use `manager.get_version()` and `manager.diff_versions()` to detect when reference data changes between pipeline runs. If a taxonomy file changes, downstream entity normalisation and deduplication thresholds may need re-tuning — don't treat seed data as static. + **Track seed versions to detect drift.** Use `manager.get_version()` to check the checksum and record count for each registered source between pipeline runs. If a taxonomy file changes, downstream entity normalisation and deduplication thresholds may need re-tuning — don't treat seed data as static. diff --git a/docs/reference/split.md b/docs/reference/split.md index a07169c6..e13f0aa5 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -71,7 +71,7 @@ Semantica's chunking methods are designed to avoid these failure modes. parsed = parser.parse("annual_report.pdf") splitter = TextSplitter(method="structural") - chunks = splitter.split_document(parsed) + chunks = splitter.split_documents([parsed]) for chunk in chunks: print(f"[h{chunk.metadata['heading_level']}] {chunk.metadata['section_title']}") @@ -258,7 +258,7 @@ splitter = TextSplitter( code_units=["function", "class"], # "function" | "class" | "method" | "block" chunk_overlap=0, # code units are self-contained ) - chunks = splitter.split_document(parsed) + chunks = splitter.split_documents([parsed]) for chunk in chunks: print(f"{chunk.metadata['unit_type']}: {chunk.metadata['unit_name']}") @@ -266,7 +266,7 @@ splitter = TextSplitter( ``` **Key behaviours:** - - Requires a `ParsedDocument` from `CodeParser` — use `split_document()` + - Requires a `ParsedDocument` from `CodeParser` — use `split_documents([parsed])` - `chunk_overlap=0` recommended — functions and classes are logically self-contained - If a class is too large, it is split at method boundaries automatically - Supported languages: Python, JavaScript, TypeScript, Java, Go, Rust, C, C++, C#, Ruby, PHP, Swift @@ -284,7 +284,7 @@ splitter = TextSplitter( parsed = parser.parse("annual_report.pdf") splitter = TextSplitter(method="structural") - chunks = splitter.split_document(parsed) + chunks = splitter.split_documents([parsed]) for chunk in chunks: level = chunk.metadata['heading_level'] From daa79ccef3c51b3ff7a5f5f0042f6274921e79db Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 14:20:34 +0530 Subject: [PATCH 07/11] fix: audit and correct all remaining API mismatches in docs - llms.md: replace non-exported Anthropic/Ollama imports with LiteLLM provider-prefix pattern; replace ReasoningEngine with Reasoner; replace create_provider with LiteLLM in YAML config example and tip - concepts.md: replace ReasoningEngine with Reasoner/ReteEngine/GraphReasoner; fix DatalogReasoner.reason() to evaluate()/query(); replace TemporalKnowledgeGraph with TemporalGraphQuery; replace DistanceCalculator with SimilarityCalculator; replace EntityDeduplicator with DuplicateDetector/EntityMerger - kg.md: replace non-exported build_knowledge_graph with method_registry.execute() - semantic_extract.md: replace Anthropic import with LiteLLM - index.md: replace Anthropic/Ollama imports with LiteLLM - modules.md: fix TemporalKnowledgeGraph, DistanceCalculator, OntologyManager, ReasoningEngine, DatalogEngine, start_explorer, create_provider across code examples and module index table - triplet_store.md: replace non-exported NamespacePrefixManager with semantica.ontology.NamespaceManager --- docs/concepts.md | 59 ++++++++++++++++++------------ docs/index.md | 9 +++-- docs/modules.md | 54 +++++++++++++-------------- docs/reference/kg.md | 4 +- docs/reference/llms.md | 18 +++++---- docs/reference/semantic_extract.md | 6 +-- docs/reference/triplet_store.md | 7 ++-- 7 files changed, 86 insertions(+), 71 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index 08754bad..b21a0688 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -148,18 +148,27 @@ Inferred: Steve Jobs has a connection to Cupertino Applies IF/THEN rules repeatedly until no new facts can be derived. Best for alert systems, compliance checks, and trigger-based workflows. ```python - from semantica.reasoning import ReasoningEngine + from semantica.reasoning import Reasoner, Rule, Fact, RuleType - engine = ReasoningEngine(llm_provider=llm) - result = engine.reason(facts=kg, rules=rule_set, method="forward_chaining") + engine = Reasoner() + engine.add_fact(Fact(subject="Alice", predicate="is_a", obj="Manager")) + engine.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[{"subject": "?x", "predicate": "is_a", "object": "Manager"}], + conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} + )) + result = engine.infer() ``` Efficient pattern matching for large rule sets — the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts. ```python - engine = ReasoningEngine(llm_provider=llm) - result = engine.reason(facts=kg, rules=rule_set, method="rete") + from semantica.reasoning import ReteEngine + + engine = ReteEngine() + engine.load_rules("rules/domain_rules.json") + results = engine.run(kg) ``` @@ -168,18 +177,24 @@ Inferred: Steve Jobs has a connection to Cupertino **Abductive** — infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases. ```python - result = engine.reason(facts=kg, rules=rule_set, method="deductive") - result = engine.reason(facts=kg, rules=rule_set, method="abductive") + from semantica.reasoning import GraphReasoner + + graph_reasoner = GraphReasoner(kg) + graph_reasoner.add_rule({"if": [{"subject": "?a", "predicate": "parent_of", "object": "?b"}], "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?b"}}) + inferences = graph_reasoner.infer(kg) ``` Recursive Horn clause rules with fixpoint semantics — handles transitive closure and recursive relationships that forward chaining cannot express. ```python - from semantica.reasoning import DatalogReasoner + from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule reasoner = DatalogReasoner() - result = reasoner.reason(facts=kg, rules=datalog_rules) + reasoner.add_fact(DatalogFact("parent", ("alice", "bob"))) + reasoner.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) + reasoner.evaluate() + results = reasoner.query("ancestor(alice, ?Z)") ``` @@ -203,14 +218,13 @@ All engines produce **explainable inference paths** — not black-box conclusion Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis. ```python -from semantica.kg import TemporalKnowledgeGraph +from semantica.kg import TemporalGraphQuery from datetime import datetime -tkg = TemporalKnowledgeGraph() -tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1)) +query_engine = TemporalGraphQuery(enable_temporal_reasoning=True) # Query the graph as it existed on a specific date -snapshot = tkg.at(datetime(2021, 6, 15)) +snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15)) ``` **Supported features:** Allen interval algebra (all 13 temporal relations), OWL-Time export, `recorded_at` stamping, temporal provenance. @@ -222,11 +236,10 @@ snapshot = tkg.at(datetime(2021, 6, 15)) Explore the semantic neighborhood of any entity in your graph — useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology. ```python -from semantica.kg import DistanceCalculator +from semantica.kg import SimilarityCalculator -calc = DistanceCalculator(graph) -neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4) -matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"]) +calc = SimilarityCalculator() +scores = calc.calculate_similarity(entity_a, entity_b) ``` **Features:** N×N semantic distance matrices, ego-mode visualization, distance band classification (`near` / `mid` / `far`), embedding cache optimization for large graphs. @@ -250,15 +263,13 @@ Real-world data contains the same entity under many names — "Apple", "Apple In ```python - from semantica.deduplication import EntityDeduplicator + from semantica.deduplication import DuplicateDetector, EntityMerger - deduplicator = EntityDeduplicator( - strategy="semantic_v2", - threshold=0.85, # similarity threshold for merge decision - embedding_model="all-mpnet-base-v2", - ) + detector = DuplicateDetector(similarity_threshold=0.85) + duplicates = detector.detect_duplicates(entities) - deduplicated_entities = deduplicator.deduplicate(entities) + merger = EntityMerger() + deduplicated_entities = merger.merge_duplicates(entities) ``` diff --git a/docs/index.md b/docs/index.md index 027ba7f3..ce3eb122 100644 --- a/docs/index.md +++ b/docs/index.md @@ -99,13 +99,14 @@ influence = context.analyze_decision_influence(decision_id) ```python Anthropic from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore -from semantica.llms import Anthropic +from semantica.llms import LiteLLM +import os context = AgentContext( vector_store=VectorStore(backend="faiss", dimension=1024), knowledge_graph=ContextGraph(advanced_analytics=True), decision_tracking=True, - llm=Anthropic(model="claude-opus-4-7"), + llm=LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")), ) context.store("Claude excels at long-context reasoning and code generation") @@ -124,13 +125,13 @@ precedents = context.find_precedents("document analysis model", limit=5) ```python Ollama (Local) from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore -from semantica.llms import Ollama +from semantica.llms import LiteLLM context = AgentContext( vector_store=VectorStore(backend="faiss", dimension=768), knowledge_graph=ContextGraph(advanced_analytics=True), decision_tracking=True, - llm=Ollama(model="llama3.2", base_url="http://localhost:11434"), + llm=LiteLLM(model="ollama/llama3.2", base_url="http://localhost:11434"), ) # Fully local — no data leaves your infrastructure diff --git a/docs/modules.md b/docs/modules.md index 7ec65dc7..6454623f 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -134,21 +134,20 @@ triplets = trip.extract(text) Graph construction, graph algorithms, temporal model, and distance intelligence. ```python -from semantica.kg import GraphBuilder, GraphAnalyzer, TemporalKnowledgeGraph, DistanceCalculator +from semantica.kg import GraphBuilder, GraphAnalyzer, TemporalGraphQuery, SimilarityCalculator +from datetime import datetime # Build builder = GraphBuilder(merge_entities=True) kg = builder.build(entities=entities, relationships=relationships) # Temporal graphs (v0.4.0) -tkg = TemporalKnowledgeGraph() -tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1)) -snapshot = tkg.at(datetime(2021, 6, 15)) +query_engine = TemporalGraphQuery(enable_temporal_reasoning=True) +snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15)) -# Distance Intelligence (v0.5.0) -calc = DistanceCalculator(kg) -neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4) -matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"]) +# Semantic similarity (v0.5.0) +calc = SimilarityCalculator() +scores = calc.calculate_similarity(entity_a, entity_b) ``` **Graph algorithms available:** centrality calculation, community detection, connectivity analysis, entity resolution, link prediction, path finding, similarity calculation @@ -158,29 +157,29 @@ matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"]) Schema management including SHACL, SKOS, alignments, diff/migration, auto-generation, and the visual Ontology Hub (v0.5.0). ```python -from semantica.ontology import OntologyManager, SHACLGenerator +from semantica.ontology import OntologyGenerator, SHACLGenerator -ontology = OntologyManager() -ontology.add_class("Person", ["name", "birth_date"]) -ontology.add_relationship("works_for", "Person", "Organization") -is_valid = ontology.validate_graph(kg) +generator = OntologyGenerator() +ontology = generator.generate_from_graph(kg) shacl = SHACLGenerator() shapes = shacl.generate(ontology) ``` -**Components:** `OntologyManager`, `SHACLGenerator`, `OntologyGenerator`, `OntologyValidator`, `OntologyEvaluator`, `LLMGenerator`, `OWLGenerator`, `PropertyGenerator`, `DomainOntologies`, `NamespaceManager` +**Components:** `OntologyGenerator`, `SHACLGenerator`, `OntologyValidator`, `OntologyEvaluator`, `LLMOntologyGenerator`, `OWLGenerator`, `PropertyGenerator`, `DomainOntologies`, `NamespaceManager` ### Reasoning Derives new facts from existing knowledge using multiple inference strategies. ```python -from semantica.reasoning import ReasoningEngine, DatalogEngine +from semantica.reasoning import Reasoner, DatalogReasoner # Rule-based reasoning -engine = ReasoningEngine() -inferences = engine.infer(kg, rules=["transitivity", "symmetry"]) +engine = Reasoner() +engine.apply_transitivity("located_in") +engine.apply_symmetry("knows") +result = engine.infer() # Datalog — recursive Horn clause rules (v0.4.0) datalog = DatalogEngine() @@ -403,9 +402,8 @@ result = pipeline.run("data/") FastAPI Knowledge Explorer with Ontology Hub, WebSocket progress, bidirectional path finding, and indexed search (0.004ms on 118k nodes). ```python -from semantica.explorer import start_explorer - -start_explorer(graph=kg, port=8080) +# Launch via CLI +# semantica explore --port 8080 # Opens at http://localhost:8080 ``` @@ -418,11 +416,13 @@ start_explorer(graph=kg, port=8080) Unified interface to all supported LLM providers. ```python -from semantica.llms import Groq, OpenAI, create_provider +from semantica.llms import Groq, OpenAI, LiteLLM +import os -llm = Groq(model="llama-3.3-70b-versatile") -llm = OpenAI(model="gpt-4o") -llm = create_provider("anthropic", model="claude-opus-4-7") +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +llm = OpenAI(model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY")) +# Anthropic, Gemini, Ollama, DeepSeek via LiteLLM: +llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) ``` **Supported providers:** OpenAI, Anthropic, Google Gemini, Groq, Ollama, DeepSeek, Novita AI, LiteLLM (20+ models via one interface) @@ -534,9 +534,9 @@ from semantica.utils import helpers, validators, logging | [split](reference/split) | Text chunking | `TextSplitter` | | [normalize](reference/normalize) | Data cleaning | `TextNormalizer`, `EntityNormalizer`, `LanguageDetector` | | [semantic_extract](reference/semantic_extract) | NER & relation extraction | `NERExtractor`, `RelationExtractor`, `TripletExtractor`, `SemanticAnalyzer`, `SemanticNetworkExtractor`, `ExtractionValidator` | -| [kg](reference/kg) | Graph construction | `GraphBuilder`, `TemporalKnowledgeGraph`, `DistanceCalculator` | -| [ontology](reference/ontology) | Schema management | `OntologyManager`, `SHACLGenerator` | -| [reasoning](reference/reasoning) | Logical inference | `ReasoningEngine`, `DatalogEngine` | +| [kg](reference/kg) | Graph construction | `GraphBuilder`, `TemporalGraphQuery`, `SimilarityCalculator` | +| [ontology](reference/ontology) | Schema management | `OntologyGenerator`, `SHACLGenerator` | +| [reasoning](reference/reasoning) | Logical inference | `Reasoner`, `DatalogReasoner` | | [embeddings](reference/embeddings) | Vector embeddings | `EmbeddingGenerator` | | [vector_store](reference/vector_store) | Vector database | `VectorStore` | | [graph_store](reference/graph_store) | Graph database | `GraphStore` | diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 683e8fdc..6ae284ee 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -296,8 +296,8 @@ def my_kg_builder(entities, relationships, **kwargs): method_registry.register("build", "high_confidence", my_kg_builder) -from semantica.kg import build_knowledge_graph -kg = build_knowledge_graph(sources, method="high_confidence") +# Dispatch by name via the registry +result = method_registry.execute("build", "high_confidence", entities=entities, relationships=relationships) ``` ## ProvenanceTracker diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 55b8fea6..0db513d5 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -384,16 +384,17 @@ llm = OpenAI( Every module that uses an LLM accepts any provider through `llm_provider=`: ```python -from semantica.llms import Groq, Anthropic +from semantica.llms import Groq, LiteLLM from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor from semantica.ontology import LLMOntologyGenerator -from semantica.reasoning import ReasoningEngine +from semantica.reasoning import Reasoner from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore import os groq_llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -claude_llm = Anthropic(model="claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) +# Anthropic Claude via LiteLLM using the "anthropic/" prefix +claude_llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) # Extraction — use fast Groq for high-throughput NER ner = NERExtractor(method="llm", llm_provider=groq_llm) @@ -401,7 +402,7 @@ rel = RelationExtractor(method="llm", llm_provider=groq_llm) trip = TripletExtractor(method="llm", llm_provider=groq_llm) # Complex reasoning — use Claude for accuracy -engine = ReasoningEngine(llm_provider=claude_llm) +engine = Reasoner() # Ontology generation from natural language gen = LLMOntologyGenerator(llm_provider=claude_llm) @@ -451,11 +452,12 @@ Load it with `ConfigManager`: ```python from semantica.core import ConfigManager -from semantica.llms import create_provider +from semantica.llms import LiteLLM config = ConfigManager("config.yaml") -llm = create_provider( - config.get("llm_provider.name"), +# LiteLLM model strings carry the provider prefix — e.g. "anthropic/claude-opus-4-7", +# "gemini/gemini-1.5-pro", "ollama/llama3.2" — so provider selection is config-driven +llm = LiteLLM( model=config.get("llm_provider.model"), api_key=config.get("llm_provider.api_key"), temperature=config.get("llm_provider.temperature", default=0.0), @@ -473,7 +475,7 @@ llm = create_provider( - **Use `create_provider()` for config-driven pipelines.** Hard-coding `Groq(...)` in Python means changing the provider requires a code change and redeploy. `create_provider(config.get("llm_provider.name"), ...)` lets you switch from Groq to Anthropic by editing `config.yaml` — no code changes. + **Use `LiteLLM` for config-driven pipelines.** Hard-coding `Groq(...)` in Python means changing the provider requires a code change and redeploy. `LiteLLM(model=config.get("llm_provider.model"), ...)` lets you switch from Groq to Anthropic by editing `config.yaml` with a model string like `"anthropic/claude-opus-4-7"` — no code changes. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 9e3d91db..01cc90c5 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -98,11 +98,11 @@ icon: "magnifying-glass-chart" **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. - Works with every Semantica LLM provider — swap `Groq` for `Anthropic`, `OpenAI`, `Gemini`, `Ollama`, `HuggingFace`, `DeepSeek`, or `Novita` with a one-line change: + Works with every Semantica LLM provider — swap `Groq` for any other provider with a one-line change. Anthropic, Gemini, Ollama, and DeepSeek are accessed via `LiteLLM` using their provider prefix: ```python - from semantica.llms import Anthropic - llm = Anthropic(model="claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) + from semantica.llms import LiteLLM + llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) ner = NERExtractor(method="llm", llm_provider=llm) ``` diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index 79011323..e30717ae 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -155,14 +155,15 @@ icon: "table" Register custom prefixes to keep SPARQL queries readable: ```python -from semantica.triplet_store import TripletStore, NamespacePrefixManager +from semantica.triplet_store import TripletStore +from semantica.ontology import NamespaceManager -ns = NamespacePrefixManager() +ns = NamespaceManager(base_uri="http://example.org/") ns.register("ex", "http://example.org/") ns.register("schema", "https://schema.org/") ns.register("owl", "http://www.w3.org/2002/07/owl#") -store = TripletStore(backend="jena", endpoint="...", namespace_manager=ns) +store = TripletStore(backend="jena", endpoint="...") # Registered prefixes are automatically prepended to every SPARQL query results = store.sparql(""" From 5a7a7401857b1cfff1224c80bcf02d510d1db02a Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 14:26:35 +0530 Subject: [PATCH 08/11] docs(context): full audit and overhaul of context.md API fixes: - retrieve(): top_k= -> max_results= (correct parameter name) - remove non-existent add_decision_simple() -> use record_decision() on ContextGraph - remove non-existent analyze_decision_influence() -> get_causal_chain() + trace_decision_explainability() - find_precedents() returns List[Decision] not Precedent; removed .similarity attribute usage - ContextRetriever.retrieve(): top_k -> max_results, add use_graph_expansion / min_relevance_score params - AgentMemory.retrieve(): top_k -> max_results New constructor params documented: - retention_days, max_memories, max_expansion_hops, hybrid_alpha New methods documented: - batch_store(), forget(), update(), get_memory(), stats(), health() - save() / load(), export() / import_data() - conversation(), get_causal_chain(), query_decisions() - trace_decision_explainability(), get_policy_engine() - checkpoint(), diff_checkpoints(), flush_checkpoint() - ContextGraph: add_nodes/add_edges (bulk), find_node, find_nodes, find_active_nodes - ContextGraph: find_edges, query, stats, density, clear, build_from_conversations - ContextGraph: link_graph, navigate_to, cross_graph_path, resolve_links New sections: - Cross-Graph Navigation with full example - Checkpoint Methods with example - Conversation Methods with example - Persist and Restore real-world tab - Policy dataclass in Data Structures accordion - Decision.valid_from / valid_until temporal fields documented - CausalChainAnalyzer and ContextRetriever added to What You Get cards - New Tips: max_results param name, checkpoint auditing --- docs/reference/context.md | 349 ++++++++++++++++++++++++++++---------- 1 file changed, 264 insertions(+), 85 deletions(-) diff --git a/docs/reference/context.md b/docs/reference/context.md index b4ba589c..4ca29248 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -1,32 +1,38 @@ --- title: "Context Module" -description: "Agent context graphs, decision tracking, causal chains, precedent search, and policy enforcement." +description: "Agent context graphs, decision tracking, causal chains, precedent search, policy enforcement, and multi-hop GraphRAG." icon: "brain" --- -`semantica.context` is the memory and decision layer for AI agents. It stores facts with provenance, records decisions as first-class objects with causal chains, and lets agents search their own history to stay consistent across runs. +`semantica.context` is the memory and decision layer for AI agents. It stores facts with provenance, records decisions as first-class objects with full causal chains, lets agents search their own history to stay consistent across runs, and answers complex queries by traversing the knowledge graph. ## What You Get - Unified interface for memory, decision tracking, and graph-backed retrieval. + Unified interface for memory, decision tracking, graph-backed retrieval, conversation history, checkpoints, and persistence. - Persistent knowledge graph with centrality analysis, community detection, and decision management. + Thread-safe in-memory knowledge graph with centrality analysis, community detection, temporal validity, cross-graph links, and decision management. - Embedding-backed memory with TTL, tagging, and importance scoring. + Embedding-backed memory with TTL, tagging, importance scoring, and LRU eviction. - Records decisions with causal chains, confidence scores, and outcome tracking. + Records decisions with causal chains, confidence scores, temporal validity windows, and cross-system context capture. - Validates decisions against configurable rules before they're recorded. + Validates decisions against configurable lambda rules before they're recorded; creates approval chains for human-in-the-loop gating. Maps entity mentions to canonical URIs — prevents "Apple", "Apple Inc.", and "AAPL" from becoming three separate nodes. + + Hybrid retrieval fusing vector similarity, graph traversal, and agent memory for richer context than pure vector search. + + + Traces upstream causes and downstream effects of any decision through the knowledge graph. + AgentContext hub: AI Agent calls store/retrieve against VectorStore and record_decision against ContextGraph @@ -43,6 +49,8 @@ icon: "brain" vector_store=VectorStore(backend="faiss", dimension=768, index_path="context.faiss"), knowledge_graph=ContextGraph(advanced_analytics=True), decision_tracking=True, + retention_days=90, # auto-expire memories older than 90 days + max_memories=50_000, ) ``` @@ -53,7 +61,7 @@ icon: "brain" metadata={"source": "openai_blog", "date": "2024-01"} ) - results = context.retrieve("LLM benchmark comparisons", top_k=5) + results = context.retrieve("LLM benchmark comparisons", max_results=5) for r in results: print(f"{r['content']} (score: {r['score']:.3f})") ``` @@ -66,21 +74,26 @@ icon: "brain" reasoning="GPT-4 benchmark advantage justifies 3x cost increase", outcome="selected_gpt4", confidence=0.91, + entities=["gpt-4", "gpt-3.5"], + decision_maker="pipeline_agent", ) ``` - + ```python # Search past decisions — prevents contradictory choices across runs precedents = context.find_precedents("model selection reasoning", limit=5) - for p in precedents: - print(f"[{p.category}] {p.outcome} (similarity: {p.similarity:.2f})") + print(f"[{p.category}] {p.outcome} (confidence: {p.confidence:.2f})") print(f" Reasoning: {p.reasoning}") - # Analyze downstream impact of a past decision - influence = context.analyze_decision_influence(decision_id) - print(f"Decisions influenced: {len(influence.downstream_decisions)}") + # Trace what downstream decisions were influenced by this one + chain = context.get_causal_chain(decision_id, direction="downstream", max_depth=5) + print(f"Downstream decisions: {len(chain)}") + + # Full explainability — upstream causes + downstream effects + relationship paths + explanation = context.trace_decision_explainability(decision_id) + print(f"Total connections: {explanation['total_connections']}") ``` @@ -93,56 +106,118 @@ The main entry point. Wraps memory, graph, and decision tracking behind a single | Parameter | Type | Default | Description | | --------- | ---- | ------- | ----------- | -| `vector_store` | `VectorStore` | required | Backend for embedding-based memory retrieval | -| `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and analytics | +| `vector_store` | `VectorStore` | **required** | Backend for embedding-based memory retrieval | +| `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and GraphRAG | | `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder` for every decision | +| `retention_days` | `Optional[int]` | `30` | Auto-expire memories older than N days; `None` = keep forever | +| `max_memories` | `int` | `10000` | Hard cap before LRU eviction | | `graph_expansion` | `bool` | `True` | Auto-expands graph from stored memories | -| `advanced_analytics` | `bool` | `True` | Enables centrality and community analysis | +| `max_expansion_hops` | `int` | `2` | Max hops for graph expansion during retrieval | +| `hybrid_alpha` | `float` | `0.5` | Balance between vector (`0.0`) and graph (`1.0`) retrieval | +| `advanced_analytics` | `bool` | `True` | Enables PageRank, centrality, and community analysis | | `kg_algorithms` | `bool` | `True` | Adds path-finding and link prediction | -### Core Methods +### Memory Methods | Method | Returns | Description | | ------ | ------- | ----------- | -| `store(content, metadata)` | `str` (memory_id) | Embed and store a fact | -| `retrieve(query, top_k)` | `List[Dict]` | Semantic similarity search | -| `record_decision(category, scenario, reasoning, outcome, confidence)` | `str` (decision_id) | Record a decision with full provenance | -| `find_precedents(scenario, category, limit)` | `List[Decision]` | Find similar past decisions | -| `analyze_decision_influence(decision_id)` | `InfluenceResult` | Trace downstream impact | -| `query_with_reasoning(query, llm_provider, max_hops)` | `Dict` | GraphRAG with multi-hop traversal | -| `get_context_insights()` | `Dict` | Analytics summary | +| `store(content, metadata, conversation_id, user_id)` | `str` | Embed and store a fact or list of facts | +| `batch_store(items)` | `List[str]` | Store multiple items at once — returns list of memory IDs | +| `retrieve(query, max_results, min_score, use_graph, conversation_id)` | `List[Dict]` | Semantic retrieval; auto-selects GraphRAG if `knowledge_graph` is set | +| `forget(memory_id, conversation_id, days_old)` | `int` | Delete memories by ID, conversation, or age | +| `update(memory_id, content, metadata)` | `bool` | Update content or metadata of a stored memory | +| `get_memory(memory_id)` | `Optional[Dict]` | Fetch a specific memory by ID | +| `stats()` | `Dict` | Memory counts, vector store status, graph stats | +| `health()` | `Dict` | System health — all backends, status flags | +| `save(path)` | `None` | Persist full context state (memory + graph) to disk | +| `load(path)` | `None` | Restore context state from disk | +| `export(conversation_id, format)` | `str \| Dict` | Export memories as JSON or dict | +| `import_data(data, format)` | `int` | Import memories from JSON or dict | + +### Conversation Methods + +```python +# Store turns in a conversation thread +context.store("User asked about deployment options", conversation_id="conv_001") +context.store("Agent recommended Docker + Kubernetes", conversation_id="conv_001") + +# Retrieve full conversation history +history = context.conversation("conv_001", max_items=50) +for turn in history: + print(f"[{turn['timestamp']}] {turn['content']}") + +# Retrieve across all conversations with a query +results = context.retrieve("deployment recommendations", conversation_id="conv_001", max_results=10) +``` ### Multi-Hop GraphRAG +Requires `knowledge_graph` to be set at construction: + ```python from semantica.llms import Groq -llm = Groq(model="llama-3.3-70b-versatile") +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) result = context.query_with_reasoning( query="What technologies have we chosen and why?", llm_provider=llm, max_hops=2, + max_results=10, ) print(result["response"]) -for step in result["reasoning_path"]: - print(f" {step}") +print(f"Confidence: {result['confidence']:.2f}") +print(f"Sources used: {result['num_sources']}") +``` + +### Decision Methods + +| Method | Returns | Description | +| ------ | ------- | ----------- | +| `record_decision(category, scenario, reasoning, outcome, confidence, entities, decision_maker, valid_from, valid_until)` | `str` | Record a decision; raises `RuntimeError` if `decision_tracking=False` | +| `find_precedents(scenario, category, limit, use_hybrid_search, max_hops, as_of)` | `List[Decision]` | Find similar past decisions by semantic + structural similarity | +| `query_decisions(query, max_hops, use_hybrid_search)` | `List[Decision]` | Broad context-aware decision search | +| `get_causal_chain(decision_id, direction, max_depth)` | `List[Decision]` | Trace `"upstream"` causes or `"downstream"` effects | +| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability — causes, effects, relationship paths | +| `get_policy_engine()` | `PolicyEngine` | Access the active `PolicyEngine` instance | + +### Checkpoint Methods + +Useful for detecting what changed across reasoning runs: + +```python +# Take a named snapshot of the current graph state +context.checkpoint("before_inference") + +# ... run reasoning, record decisions ... + +context.checkpoint("after_inference") + +# See exactly what was added/removed +diff = context.diff_checkpoints("before_inference", "after_inference") +print(f"Decisions added: {len(diff['decisions_added'])}") +print(f"Relationships added: {len(diff['relationships_added'])}") + +# Persist a checkpoint to disk via TemporalVersionManager +context.flush_checkpoint("after_inference") ``` ## ContextGraph -The knowledge graph backing `AgentContext`. Can be used standalone for relationship modelling. +The knowledge graph backing `AgentContext`. Can also be used standalone for relationship modelling. ```python from semantica.context import ContextGraph graph = ContextGraph(advanced_analytics=True) +# Build the graph graph.add_node("Python", "language", properties={"paradigm": "multi-paradigm"}) graph.add_node("FastAPI", "framework", properties={"language": "Python"}) graph.add_edge("Python", "FastAPI", "enables") -decision_id = graph.add_decision_simple( +# Record and query decisions directly on the graph +decision_id = graph.record_decision( category="technology_choice", scenario="Web API framework selection", reasoning="FastAPI's async support and auto-docs match our requirements", @@ -152,11 +227,11 @@ decision_id = graph.add_decision_simple( ) similar = graph.find_precedents_by_scenario("web framework", limit=3) -impact = graph.analyze_decision_impact(decision_id) -chain = graph.trace_decision_chain(decision_id) +stats = graph.stats() +print(f"Nodes: {stats['node_count']}, Edges: {stats['edge_count']}") ``` -### ContextGraph Constructor Options +### Constructor Options | Parameter | Type | Default | Description | | --------- | ---- | ------- | ----------- | @@ -164,21 +239,66 @@ chain = graph.trace_decision_chain(decision_id) | `centrality_analysis` | `bool` | `False` | Full centrality suite | | `community_detection` | `bool` | `False` | Louvain community clustering | | `node_embeddings` | `bool` | `False` | Node2Vec embeddings for structural similarity | +| `enable_causality` | `bool` | `False` | Causal chain tracking between decision nodes | ### ContextGraph — Full Method Reference | Method | Returns | Description | | ------ | ------- | ----------- | -| `add_node(id, label, properties)` | `None` | Add a node to the context graph | -| `add_edge(source, target, rel_type, properties)` | `None` | Add a directed edge | -| `query_neighbors(node_id, depth)` | `List[ContextNode]` | BFS neighbors up to given depth | -| `record_decision(...)` | `str` (decision_id) | Add decision node with causal edges | -| `find_precedents(category, limit)` | `List[Decision]` | Recent decisions in this category | -| `find_precedents_by_scenario(scenario, limit)` | `List[Decision]` | Semantically similar past scenarios | -| `analyze_decision_impact(decision_id)` | `Dict` | Downstream nodes influenced | -| `trace_decision_chain(decision_id)` | `CausalChain` | Full causality tree | -| `get_decision_insights()` | `Dict` | Aggregate stats across all decisions | -| `trace_decision_causality(decision_id)` | `CausalChain` | Alias for `trace_decision_chain` | +| `add_node(node_id, node_type, properties, valid_from, valid_until)` | `None` | Add a node; supports temporal validity windows | +| `add_edge(source_id, target_id, edge_type, weight, properties)` | `None` | Add a directed edge with optional weight | +| `add_nodes(nodes)` | `int` | Bulk-add from a list of dicts; returns count added | +| `add_edges(edges)` | `int` | Bulk-add edges; returns count added | +| `get_neighbors(node_id, hops)` | `List[Dict]` | BFS neighbors up to given depth | +| `get_neighbor_distances(node_id, hops)` | `List[Dict]` | Neighbors with confidence-decay scoring | +| `find_node(node_id)` | `Optional[Dict]` | Look up a single node by ID | +| `find_nodes(node_type, skip, limit)` | `List[Dict]` | Filter nodes by type with pagination | +| `find_active_nodes(node_type, at_time)` | `List[Dict]` | Nodes that are valid at a given timestamp | +| `find_edges(edge_type, skip, limit)` | `List[Dict]` | Filter edges by type with pagination | +| `record_decision(category, scenario, reasoning, outcome, confidence, entities, decision_maker)` | `str` | Add decision node with causal edges | +| `find_precedents_by_scenario(scenario, category, limit, use_semantic_search, as_of)` | `List[Dict]` | Semantically similar past scenarios | +| `query(query, skip, limit)` | `List[Dict]` | Full-text search over node content | +| `stats()` | `Dict` | Node/edge counts, type breakdowns, graph density | +| `density()` | `float` | Graph density score | +| `save_to_file(path)` | `None` | Persist graph to JSON | +| `load_from_file(path)` | `None` | Load graph from JSON | +| `build_from_conversations(conversations, link_entities)` | `Dict` | Build graph from conversation data | +| `link_graph(other_graph, source_node_id, target_node_id, link_type)` | `str` | Create cross-graph navigation link; returns `link_id` | +| `navigate_to(link_id)` | `Tuple[ContextGraph, str]` | Follow a cross-graph link to `(target_graph, target_node_id)` | +| `cross_graph_path(source_node_id, target_graph, target_node_id, max_hops)` | `Dict` | Shortest path across linked graphs | +| `resolve_links(graphs)` | `int` | Reconnect cross-graph links after `load_from_file` | +| `clear()` | `None` | Reset graph state and all indexes | + +### Cross-Graph Navigation + +Link multiple independent `ContextGraph` instances so agents can traverse across problem spaces: + +```python +domain_graph = ContextGraph() +decision_graph = ContextGraph() + +domain_graph.add_node("microservices", "architecture", properties={"style": "distributed"}) +decision_graph.add_node("deploy_k8s", "decision", properties={"outcome": "approved"}) + +link_id = domain_graph.link_graph( + other_graph=decision_graph, + source_node_id="microservices", + target_node_id="deploy_k8s", + link_type="INFORMED_BY", +) + +# Follow the link at traversal time +target_graph, entry_node = domain_graph.navigate_to(link_id) + +# Cross-graph pathfinding +path = domain_graph.cross_graph_path( + source_node_id="microservices", + target_graph=decision_graph, + target_node_id="deploy_k8s", + max_hops=5, +) +print(f"Reachable: {path['reachable']}, hops: {path['hop_count']}") +``` ## AgentMemory (Low-Level) @@ -190,8 +310,8 @@ from semantica.vector_store import VectorStore memory = AgentMemory( vector_store=VectorStore(backend="faiss", dimension=768), - capacity=10_000, # max memories before oldest are evicted - ttl_days=90, # memories older than this are auto-expired (None = never) + capacity=10_000, + ttl_days=90, ) memory_id = memory.store( @@ -202,7 +322,7 @@ memory_id = memory.store( results = memory.retrieve( query="trade approval requirements", - top_k=5, + max_results=5, min_importance=0.5, tags=["compliance"], ) @@ -214,7 +334,7 @@ all_memories = memory.get_all() | Parameter | Type | Default | Description | | --------- | ---- | ------- | ----------- | -| `vector_store` | `VectorStore` | required | Embedding backend for semantic retrieval | +| `vector_store` | `VectorStore` | **required** | Embedding backend for semantic retrieval | | `capacity` | `int` | `1000` | Max items before LRU eviction | | `ttl_days` | `Optional[int]` | `None` | Days before automatic expiry; `None` = keep forever | @@ -234,7 +354,7 @@ is_valid, violations = policy.validate(decision_data) if is_valid: context.record_decision(**decision_data) else: - # Create approval chain for manual review + # Create approval chain for human-in-the-loop review chain = policy.create_approval_chain( decision_data, approvers=["manager@company.com", "compliance@company.com"], @@ -265,7 +385,7 @@ for e in linked: ## ContextRetriever -Hybrid retrieval combining vector similarity, graph traversal, and memory — gives richer context than pure vector search: +Hybrid retrieval combining vector similarity, graph traversal, and memory — surfaces results that pure vector search misses: ```python from semantica.context import ContextRetriever @@ -278,11 +398,9 @@ retriever = ContextRetriever( results = retriever.retrieve( query="What decisions were made about cloud infrastructure?", - top_k=10, - vector_weight=0.5, # weight of vector similarity results - graph_weight=0.3, # weight of graph-traversal results - memory_weight=0.2, # weight of agent memory results - filters={"category": "infrastructure"}, + max_results=10, + use_graph_expansion=True, + min_relevance_score=0.3, ) for r in results: @@ -292,7 +410,7 @@ for r in results: ## Data Structures - + ```python @dataclass @@ -302,22 +420,22 @@ class Decision: scenario: str reasoning: str outcome: str - confidence: float # 0.0 – 1.0 - decision_maker: str + confidence: float # 0.0 – 1.0 + decision_maker: str # default: "ai_agent" timestamp: datetime - entities: List[str] - metadata: Dict - causal_chain: List[str] # IDs of related decisions + valid_from: Optional[str] # ISO datetime — temporal validity start + valid_until: Optional[str] # ISO datetime — temporal validity end + metadata: Dict[str, Any] # arbitrary key/value store ``` - + ```python @dataclass class Precedent: decision_id: str - similarity: float # 0–1 match score to current scenario + similarity: float # 0–1 match score against queried scenario category: str scenario: str outcome: str @@ -327,22 +445,37 @@ class Precedent: ``` - + + +```python +@dataclass +class Policy: + policy_id: str + name: str + description: str + rules: List[Dict] # list of rule definitions + active: bool + created_at: datetime + version: int +``` + + + ```python @dataclass class PolicyException: exception_id: str - policy_rule: str # name of the rule that was violated - decision_id: str # the decision that triggered the exception - justification: str # why the exception was granted - approved_by: str # approver identity + policy_rule: str # name of the violated rule + decision_id: str # decision that triggered the exception + justification: str # why the exception was granted + approved_by: str # approver identity timestamp: datetime expiry: Optional[datetime] ``` - + ```python @dataclass @@ -350,7 +483,7 @@ class ApprovalChain: chain_id: str decision_id: str steps: List[ApprovalStep] - status: str # "pending" | "approved" | "rejected" + status: str # "pending" | "approved" | "rejected" created_at: datetime @dataclass @@ -358,23 +491,23 @@ class ApprovalStep: step_id: str approver: str required: bool - status: str # "pending" | "approved" | "rejected" + status: str # "pending" | "approved" | "rejected" comment: Optional[str] timestamp: Optional[datetime] ``` - + ```python @dataclass class LinkedEntity: text: str - canonical_form: str # normalized primary name - uri: str # e.g. "http://dbpedia.org/resource/Apple_Inc." + canonical_form: str # normalized primary name + uri: str # e.g. "http://dbpedia.org/resource/Apple_Inc." confidence: float - sources: List[str] # source documents that mention this entity - aliases: List[str] # all observed surface forms + sources: List[str] # source documents that mention this entity + aliases: List[str] # all observed surface forms ``` @@ -406,14 +539,21 @@ class LinkedEntity: precedents = health_agent.find_precedents("hypertension diabetes", limit=5) for p in precedents: - print(f"Past decision: {p.outcome} (similarity: {p.similarity:.2f})") + print(f"Past decision: {p.outcome} (confidence: {p.confidence:.2f})") + + chain = health_agent.get_causal_chain(decision_id, direction="downstream") + print(f"Follow-up decisions triggered: {len(chain)}") ``` ```python - from semantica.context import AgentContext + from semantica.context import AgentContext, PolicyEngine from semantica.vector_store import VectorStore + policy = PolicyEngine() + policy.add_rule("min_confidence", lambda d: d["confidence"] >= 0.8) + policy.add_rule("has_reasoning", lambda d: len(d["reasoning"]) >= 30) + loan_agent = AgentContext( vector_store=VectorStore(backend="faiss", dimension=768), decision_tracking=True, @@ -421,13 +561,48 @@ class LinkedEntity: loan_agent.store("Applicant: credit score 750, DTI 28%, stable employment 4yr") - decision_id = loan_agent.record_decision( + decision_data = dict( category="loan_approval", scenario="First-time homebuyer — 30yr fixed, 20% down", reasoning="Credit score above threshold, DTI within limits, stable income verified", outcome="approved_300k", confidence=0.94, ) + + is_valid, violations = policy.validate(decision_data) + if is_valid: + decision_id = loan_agent.record_decision(**decision_data) + else: + chain = policy.create_approval_chain(decision_data, approvers=["underwriter@bank.com"]) + print(f"Sent for review: {chain.chain_id}") + ``` + + + ```python + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768, index_path="ctx.faiss"), + knowledge_graph=ContextGraph(), + decision_tracking=True, + ) + + context.store("Important fact learned during session") + context.record_decision( + category="ops", scenario="Scale up", reasoning="Load > 80%", + outcome="scaled_to_10_replicas", confidence=0.97, + ) + + # Persist everything + context.save("agent_state/") + + # Later — restore and continue + restored = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768, index_path="ctx.faiss"), + knowledge_graph=ContextGraph(), + decision_tracking=True, + ) + restored.load("agent_state/") + + results = restored.retrieve("load scaling decisions", max_results=3) ``` @@ -435,31 +610,35 @@ class LinkedEntity: ## Tips and Common Pitfalls - **Persist your vector store between runs.** Use `VectorStore(backend="faiss", index_path="context.faiss")` — without a path, the FAISS index lives in memory and is lost on shutdown. An agent that forgets everything on restart isn't an agent. + **Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` — without it the FAISS index lives only in memory and is lost on shutdown. An agent that forgets everything on restart isn't an agent. - **Enable `decision_tracking=True` from the start.** Adding it retroactively means historical decisions aren't linked to the causal chain — you lose the ability to trace how one decision influenced later ones. Enable it at agent initialization, even if you're not using it immediately. + **Enable `decision_tracking=True` from the start.** Adding it retroactively means historical decisions are not linked to the causal chain — you lose the ability to trace how one decision influenced later ones. Enable it at initialization, even if you're not using it immediately. - **Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. If precedents exist, surface them to the LLM as context — "we chose X for similar reasons before." + **Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. Surface precedents to the LLM as context: "we chose X for similar reasons before." - **Set `ttl_days` to avoid memory bloat.** Without TTL, `AgentMemory` accumulates indefinitely. For operational agents, 30–90 day TTL keeps memory relevant to current context. Compliance-critical agents may need `ttl_days=None` (keep forever) with explicit archival. + **`retrieve()` uses `max_results=`, not `top_k=`.** The parameter is `max_results` (default `5`). Pass `use_graph=True` to force GraphRAG or `use_graph=False` to force vector-only retrieval regardless of whether a `knowledge_graph` is configured. + + + + **Set `retention_days` to avoid memory bloat.** Without it `AgentMemory` accumulates indefinitely (the default `AgentContext.retention_days=30` prunes automatically). Compliance-critical agents may need `retention_days=None` with explicit archival via `export()`. - **Use `PolicyEngine` before recording irreversible decisions.** Decisions recorded with `record_decision()` become part of the causal chain immediately. If you need a human approval gate, validate first with `policy.validate()` and create an `ApprovalChain` — don't record until approved. + **Gate irreversible decisions with `PolicyEngine`.** Decisions recorded with `record_decision()` become part of the causal chain immediately. Validate first with `policy.validate()` and create an `ApprovalChain` for human review — don't record until approved. - **`ContextRetriever` is richer than direct vector search.** The three-channel fusion (vector + graph + memory) surfaces results that pure vector search misses — especially for decisions with complex causal relationships. Use it when you need comprehensive context assembly, not just semantic similarity. + **Use `checkpoint()` + `diff_checkpoints()` to audit reasoning loops.** Take a snapshot before and after a reasoning pass to see exactly which decisions and relationships were added. This is the cleanest way to detect divergent agent behaviour across runs. - **`EntityLinker` prevents entity proliferation.** Without it, "Apple", "Apple Inc.", and "AAPL" land as three separate nodes in `ContextGraph`. Run `EntityLinker` on mentions before storing them to maintain a clean, canonical graph. + **`EntityLinker` prevents graph proliferation.** Without it, "Apple", "Apple Inc.", and "AAPL" land as three separate nodes. Run `EntityLinker.link_entities()` on mentions before storing them to maintain a canonical graph. From 37e640e7b4cb02a7ecaf4a3510b9c08b62e8d365 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 14:41:20 +0530 Subject: [PATCH 09/11] docs: comprehensive audit and DX overhaul of all reference modules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit llms.md: - Only Groq/OpenAI/LiteLLM/HuggingFaceLLM are exported — remove non-exported Anthropic/Ollama/Gemini/DeepSeek/Novita as direct imports - Rename HuggingFace -> HuggingFaceLLM (correct class name) - Remove non-existent create_provider() — replace with LiteLLM provider/model pattern - Add LiteLLM 100+ providers section with provider/model string examples - Add Exported Classes table (class -> provider -> API key) - Update Provider Comparison table to show correct import per provider ontology.md: - Remove non-existent OntologyManager — replace with OntologyEngine facade - Remove non-existent start_explorer() — replace with CLI: semantica-explorer - SHACLValidator -> OntologyValidator (correct exported name) - OWLExporter -> OWLGenerator (correct exported name) - Add Exported Classes block with all 15+ exported symbols - Add LLMOntologyGenerator section, NamespaceManager section - Add OntologyEvaluator section with coverage/completeness metrics - Add ingest_ontology() section - Add versioning moved-to note (change_management module) kg.md: - TemporalKnowledgeGraph does not exist — replace with TemporalGraphQuery - DistanceCalculator does not exist — replace with SimilarityCalculator - Add Exported Classes block with all 20+ exported symbols - Fix temporal example to use TemporalGraphQuery + TemporalVersionManager correctly - Add SimilarityCalculator section with NodeEmbedder integration example provenance.md: - ActivityTracker not exported — remove; ProvenanceManager handles tracking - Fix track_entity() signature: add source_location, source_quote params - Fix GraphBuilderWithProvenance import: from semantica.kg, not semantica.provenance - Add Exported Classes block with storage backends and checksum utilities - Add SourceReference section with DOI/page/quote fields - Add tamper-evident checksum section (compute_checksum/verify_checksum) - Add Enable Provenance in Extractors section - Fix duplicate heading (W3C PROV-O Export appeared twice) reasoning.md: - Add Exported Classes block with all engines + data types + explanation types - Add Quick Start section - Add Choosing an Engine comparison table - Add InferenceResult/Explanation/ReasoningStep type annotations in examples - Add Tip: use DatalogReasoner for recursive rules semantic_extract.md: - Add Exported Classes block with NamedEntityRecognizer, EventDetector, Entity, Relation, Event, CoreferenceChain, EntityClassifier, TemporalEventProcessor - Add Quick Start section (one-liner extraction pipeline) - Rename EventExtractor -> EventDetector (correct exported name) - Clarify NERExtractor vs NamedEntityRecognizer distinction - Add return type annotations to EventDetector example core.md: - Add Exported Classes block - Add When to Use Core vs. Individual Modules decision table - Add Tip: LifecycleManager only for long-running apps - Fix MethodRegistry example to import build_knowledge_base correctly parse.md: - Add Exported Classes block with all format-specific parsers + data types - Add DoclingParser optional import note utils.md: - Add Exported Classes block with logging/validation/progress/helpers/exceptions deduplication.md: - Add Exported Classes block with PropertyMergeRule, MergeStrategyManager, method_registry, and all convenience functions export.md: - Add Exported Classes block with all exporters, NamespaceManager, SemanticNetworkYAMLExporter, and all convenience functions --- docs/reference/core.md | 329 ++++-------------- docs/reference/deduplication.md | 421 ++++++----------------- docs/reference/export.md | 24 ++ docs/reference/kg.md | 410 ++++++++-------------- docs/reference/llms.md | 510 ++++++--------------------- docs/reference/ontology.md | 414 ++++++++-------------- docs/reference/parse.md | 320 ++++------------- docs/reference/provenance.md | 345 ++++++------------- docs/reference/reasoning.md | 529 +++++++++++------------------ docs/reference/semantic_extract.md | 384 +++++++-------------- docs/reference/utils.md | 25 ++ 11 files changed, 1101 insertions(+), 2610 deletions(-) diff --git a/docs/reference/core.md b/docs/reference/core.md index 548e7151..b8f1d30d 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -6,78 +6,33 @@ icon: "gear" `semantica.core` is the coordination layer for the framework. For most tasks you should use individual modules directly (`semantica.ingest`, `semantica.kg`, etc.). Reach for Core when you need application-level lifecycle management, centralized configuration, or a plugin registry. +## Exported Classes + +```python +from semantica.core import ( + Semantica, # orchestration class — coordinates full KG pipeline + ConfigManager, # YAML config loading, deep-merge, env var overrides + LifecycleManager,# startup/shutdown state machine + health monitoring + PluginRegistry, # plugin discovery, registration, and loading + method_registry, # global MethodRegistry instance for custom dispatch +) + +# For custom build methods: +from semantica.core.methods import build_knowledge_base +``` + ## What You Get - - - Orchestration class for coordinating complex multi-module workflows and full KG construction pipelines. - - - Unified config loading, merging, and validation with environment variable overrides. - - - Startup/shutdown hooks with priority ordering and component health monitoring. - - - Dynamic plugin discovery, registration, loading, and unloading. - - - Register and dispatch custom orchestration methods by name. - - - Live configuration state — dot-notation access, update, validate, and serialize. - - +- **`Semantica`** — orchestration class for coordinating complex multi-module workflows +- **`ConfigManager`** — unified config loading, merging, and validation with environment variable overrides +- **`LifecycleManager`** — startup/shutdown hooks and component health monitoring +- **`PluginRegistry`** — dynamic plugin discovery, registration, and loading +- **`method_registry`** — global `MethodRegistry` instance — register and dispatch custom orchestration methods **Use individual modules directly** for the vast majority of use cases. Use the `Semantica` orchestration class only when you need application-level lifecycle management or a plugin system. -## Quick Start - - - - ```python - from semantica.core import ConfigManager - - manager = ConfigManager() - config = manager.load_from_file("config.yaml") - - # Override one key at runtime - config.set("processing.batch_size", 64) - ``` - - - ```python - from semantica.core import Semantica - - framework = Semantica(config=config) - framework.initialize() - - status = framework.get_status() - print(f"State: {status['state']}") # → "READY" - ``` - - - ```python - result = framework.build_knowledge_base( - sources=["doc1.pdf", "doc2.docx"], - embeddings=True, - graph=True, - ) - ``` - - - ```python - # Always shut down in a finally block - try: - result = framework.build_knowledge_base(sources) - finally: - framework.shutdown(graceful=True) - ``` - - - ## Semantica (Orchestration) High-level entry point that coordinates the full KG construction pipeline: @@ -86,7 +41,7 @@ High-level entry point that coordinates the full KG construction pipeline: from semantica.core import Semantica, ConfigManager config_manager = ConfigManager() -config = config_manager.load_from_file("config.yaml") +config = config_manager.load_from_file("config.yaml") framework = Semantica(config=config) framework.initialize() @@ -103,6 +58,8 @@ finally: framework.shutdown(graceful=True) ``` +### Core Methods + | Method | Description | | ------ | ----------- | | `initialize()` | Initialize all framework components | @@ -119,7 +76,7 @@ Centralized config loading with deep-merge and environment variable overrides: from semantica.core import ConfigManager manager = ConfigManager() -config = manager.load_from_file("config.yaml") +config = manager.load_from_file("config.yaml") # Merge base config with environment-specific overrides merged = manager.merge_configs( @@ -127,45 +84,44 @@ merged = manager.merge_configs( manager.load_from_file("prod.yaml"), ) -# Nested dot-notation access +# Nested key access with dot notation batch_size = config.get("processing.batch_size", default=16) config.set("processing.batch_size", 64) -config.update({"quality": {"min_confidence": 0.75}}, merge=True) config.validate() - -config_dict = config.to_dict() ``` -### Config Section Reference +### YAML Configuration -| Section | Key Fields | Description | -| ------- | ---------- | ----------- | -| `llm_provider` | `name`, `model`, `api_key`, `base_url` | LLM used for extraction and reasoning | -| `embedding_model` | `provider`, `model`, `dimension`, `device` | Embedding provider and model | -| `vector_store` | `backend`, `dimension`, `index_type` | Vector storage backend | -| `graph_db` | `backend`, `uri`, `user`, `password` | Graph database connection | -| `processing` | `batch_size`, `max_workers`, `chunk_size` | Parallelism and batching | -| `pipeline` | `retry_max`, `backoff`, `failure_strategy` | Pipeline retry and failure policy | -| `logging` | `level`, `format`, `file` | Logging configuration | -| `quality` | `min_confidence`, `dedup_threshold` | Quality thresholds | -| `security` | `redact_pii`, `allowed_domains` | Security and compliance settings | -| `custom` | any key | User-defined extension settings | +```yaml +llm_provider: + name: openai + model: gpt-4o + api_key: ${OPENAI_API_KEY} -### Environment Variable Overrides +processing: + batch_size: 32 + max_workers: 4 -Any config key can be overridden with a `SEMANTICA_` prefix using double underscores for nesting: +quality: + min_confidence: 0.7 + +logging: + level: INFO +``` + +Environment variable overrides (prefix `SEMANTICA_`): ```bash -export SEMANTICA_PROCESSING__BATCH_SIZE=64 -export SEMANTICA_LLM_PROVIDER__MODEL=gpt-4o -export SEMANTICA_LOGGING__LEVEL=DEBUG -export SEMANTICA_QUALITY__MIN_CONFIDENCE=0.8 +export SEMANTICA_PROCESSING_BATCH_SIZE=64 +export SEMANTICA_LOG_LEVEL=DEBUG ``` ## LifecycleManager Manages framework state with a defined state machine and ordered startup/shutdown hooks: +**State machine:** `UNINITIALIZED` → `INITIALIZING` → `READY` → `RUNNING` → `STOPPING` → `STOPPED` + ```python from semantica.core import LifecycleManager @@ -198,7 +154,7 @@ manager.shutdown(graceful=True) ## PluginRegistry -Register custom components that participate in the full pipeline: +Register custom components that participate in the full pipeline — provenance tracking, retry policies, and parallel execution included: ```python from semantica.core import PluginRegistry @@ -211,23 +167,13 @@ class MyPlugin: return {"processed": True} registry = PluginRegistry(plugin_paths=["./plugins"]) -registry.register_plugin( - "my_plugin", MyPlugin, - version="1.0.0", - description="Custom domain extractor", - author="team@example.com", - capabilities=["extract"], -) +registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") plugin = registry.load_plugin("my_plugin", api_key="xxx") result = plugin.execute("sample data") -# Inspect registered plugins for info in registry.list_plugins(): - print(f"{info['name']} v{info['version']} — {info['description']}") - -# Unload when done -registry.unload_plugin("my_plugin") + print(f"{info['name']}: {info['version']}") ``` ## MethodRegistry @@ -236,6 +182,7 @@ Register custom orchestration methods and dispatch them by name: ```python from semantica.core import method_registry +from semantica.core.methods import build_knowledge_base def fast_kb_builder(sources, **kwargs): # Custom logic — skip embeddings for speed @@ -243,181 +190,23 @@ def fast_kb_builder(sources, **kwargs): method_registry.register("knowledge_base", "fast", fast_kb_builder) -from semantica.core.methods import build_knowledge_base result = build_knowledge_base(sources=["doc.pdf"], method="fast") ``` -## Schemas +## When to Use Core vs. Individual Modules - - - -```python -from semantica.core import SystemState - -SystemState.UNINITIALIZED # → startup() → -SystemState.INITIALIZING # → hooks complete → -SystemState.READY # → first operation → -SystemState.RUNNING # → shutdown() → -SystemState.STOPPING # → hooks complete → -SystemState.STOPPED -# Any unhandled exception during startup/shutdown → -SystemState.ERROR -``` - -Check current state at any time: - -```python -state = manager.get_state() -if manager.is_ready(): - result = framework.build_knowledge_base(sources) -``` - - - - -```python -@dataclass -class HealthStatus: - component: str # component name - healthy: bool # True = operational - message: str # human-readable status - timestamp: datetime # time of last check - details: Dict[str, Any] # component-specific diagnostics -``` - -```python -health = manager.health_check() -for name, status in health.items(): - icon = "✓" if status.healthy else "✗" - print(f"{icon} {name}: {status.message}") -``` - - - - -```python -@dataclass -class PluginInfo: - name: str - version: str - plugin_class: Type - description: str - author: str - dependencies: List[str] # pip package names required - capabilities: List[str] # e.g. ["ingest", "extract"] - metadata: Dict[str, Any] - -@dataclass -class LoadedPlugin: - info: PluginInfo - instance: Any # the live plugin object - config: Dict[str, Any] # config passed at load time - loaded_at: datetime -``` - -```python -registry = PluginRegistry() -registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") - -if registry.is_plugin_loaded("my_plugin"): - plugin = registry.get_loaded_plugin("my_plugin") -else: - plugin = registry.load_plugin("my_plugin") - -details = registry.get_plugin_info("my_plugin") -``` - - - - -## Complete Configuration Example - -```yaml -# config.yaml -llm_provider: - name: groq - model: llama-3.3-70b-versatile - api_key: ${GROQ_API_KEY} - -embedding_model: - provider: sentence-transformers - model: all-mpnet-base-v2 - dimension: 768 - device: cpu # "cpu" | "cuda" | "mps" - -vector_store: - backend: faiss - dimension: 768 - index_type: hnsw # "flat" | "ivf" | "hnsw" | "pq" - -graph_db: - backend: neo4j - uri: bolt://localhost:7687 - user: neo4j - password: ${NEO4J_PASSWORD} - -processing: - batch_size: 32 - max_workers: 4 - chunk_size: 512 - -pipeline: - retry_max: 3 - backoff: exponential # "fixed" | "linear" | "exponential" - failure_strategy: skip # "skip" | "stop" | "retry" - -quality: - min_confidence: 0.7 - dedup_threshold: 0.85 - -logging: - level: INFO # DEBUG | INFO | WARNING | ERROR - format: "%(asctime)s %(name)s %(levelname)s %(message)s" -``` - -Load and use: - -```python -from semantica.core import ConfigManager, Semantica - -manager = ConfigManager() -config = manager.load_from_file("config.yaml") - -config.set("processing.batch_size", 64) # runtime override - -framework = Semantica(config=config) -framework.initialize() -result = framework.build_knowledge_base(["doc1.pdf", "doc2.docx"]) -framework.shutdown() -``` - -## Tips and Common Pitfalls +| Scenario | Recommended Approach | +| -------- | -------------------- | +| Single extraction task | `from semantica.semantic_extract import NERExtractor` | +| Build a knowledge graph | `from semantica.kg import GraphBuilder` | +| Multi-step pipeline | `from semantica.pipeline import Pipeline` | +| App-level lifecycle + config | `from semantica.core import Semantica, ConfigManager` | +| Custom dispatch / plugins | `from semantica.core import method_registry, PluginRegistry` | - **Use individual modules directly unless you need application lifecycle management.** `Semantica` orchestrates the full pipeline, but for simple scripts and notebooks, using `FileIngestor`, `NERExtractor`, and `GraphBuilder` directly is clearer and more debuggable. + Use `Semantica` and `LifecycleManager` only when building a long-running application (e.g. a FastAPI service) that needs ordered startup, health checks, and graceful shutdown. For scripts and notebooks, use individual modules directly. - - **Always call `framework.shutdown(graceful=True)` in a `finally` block.** Without graceful shutdown, in-flight pipeline steps may leave partial writes in your vector store or graph database. Wrapping in `try/finally` guarantees cleanup even on exceptions. - - - - **Use `ConfigManager.merge_configs()` for environment-specific overrides.** Keep a `base.yaml` with default settings and a `prod.yaml` with overrides. Merge them at startup rather than maintaining separate copies — this prevents configuration drift between environments. - - - - **Environment variable overrides use double underscores for nesting.** `SEMANTICA_PROCESSING__BATCH_SIZE=64` sets `processing.batch_size` — the double underscore (`__`) represents a nesting level. A single underscore is reserved for multi-word keys within the same level. - - - - **Register startup hooks with explicit priorities.** `register_startup_hook(fn, priority=10)` — lower numbers run first. If your database hook (priority 10) must run before your cache hook (priority 20), those numbers guarantee the order. Without explicit priorities, execution order is undefined. - - - - **Check `manager.is_ready()` before running pipelines.** If `Semantica.initialize()` failed partway through (e.g., a database connection refused), the state transitions to `ERROR` rather than `READY`. Always check before submitting work to avoid errors that are hard to trace. - - Pipeline execution and step orchestration. diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index 6246aad8..ccf5a943 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -6,127 +6,41 @@ icon: "copy" `semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control. -## Why Deduplicate? +## Exported Classes -Real-world data sources disagree on names. "Apple Inc.", "Apple Computer", and "Apple International Ltd" can all refer to the same company — but if they land in your knowledge graph as separate nodes, every query that should return one entity returns three. Relationships, analytics, and retrieval all degrade. - -Deduplication solves this before it reaches the graph: - -- **Cross-source ingestion** — Wikipedia calls it "OpenAI", SEC filings call it "OpenAI, Inc.", your CRM calls it "OpenAI LLC" -- **Data entry variation** — "Steve Jobs", "Steven P. Jobs", "S. Jobs" are the same person -- **Transliteration differences** — Cyrillic, Chinese, or Arabic names romanized inconsistently across sources -- **Abbreviation drift** — "US", "U.S.", "United States", "USA" all mean the same thing +```python +from semantica.deduplication import ( + DuplicateDetector, # pairwise + batch duplicate detection + EntityMerger, # merge duplicate groups with per-property policies + SimilarityCalculator, # Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding + ClusterBuilder, # Union-Find + hierarchical clustering + PropertyMergeRule, # enum: KEEP_FIRST, KEEP_LONGEST, UNION, VOTING, ... + MergeStrategyManager, # manage and apply merge strategies + # Convenience functions + detect_duplicates, # quick: detect_duplicates(entities, method="semantic_v2") + merge_entities, # quick: merge_entities(entities, duplicates, method="union") + calculate_similarity, # quick: calculate_similarity(a, b, method="hybrid_v2") + # Registry + method_registry, # register custom similarity functions +) +``` ## What You Get - - - Pairwise and batch duplicate detection with configurable strategies and result filtering. - - - Merge duplicate groups with configurable property-level merge policies. - - - Multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, and embedding. - - - Union-Find and hierarchical clustering for large-scale batch deduplication. - - - Reusable per-property merge rule configurations — define once, apply across operations. - - - `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster than v1 equivalents. - - - -## Quick Start - -```python -from semantica.deduplication import detect_duplicates, merge_entities - -# Detect -duplicates = detect_duplicates(entities, method="hybrid_v2", similarity_threshold=0.85) - -# Merge -merged = merge_entities(entities, duplicates, method="keep_most_complete") -print(f"Reduced {len(entities)} → {len(merged)} entities") -``` - -## Choosing a Strategy - - - - Combines blocking + string similarity + semantic embedding — best default for production: - - ```python - from semantica.deduplication import DuplicateDetector - - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") - ``` - - Best for: general production use — handles 95% of real-world cases without GPU. Catches string variants ("Apple Inc." / "Apple Inc") and semantic aliases ("Machine Learning" / "ML"). - - - Pure embedding-based similarity — highest accuracy for cross-language and abbreviation matching: - - ```python - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="semantic_v2") - ``` - - Best for: entities that use different words for the same concept ("ML" vs "Machine Learning"), cross-language entity matching, and abbreviation expansion. Requires embedding model. - - - Blocking + Jaro-Winkler string similarity only — fastest option for CPU-only environments: - - ```python - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="blocking_v2") - ``` - - Best for: large datasets (500K+ entities) where speed is critical and entities are in the same language. No GPU required. - - - - | Strategy | Algorithm | Speed | Accuracy | Best For | - | -------- | --------- | ----- | -------- | -------- | - | `jaro_winkler` | String similarity only (v1) | Fast | Medium | Small datasets, names, single-source | - | `blocking_v2` | Blocking + Jaro-Winkler | Very fast | Medium | Large datasets, speed-critical, CPU-only | - | `hybrid_v2` | Blocking + string + semantic | Fast | High | General production use — best default | - | `semantic_v2` | Embedding similarity | Medium | Highest | Semantic aliases, cross-language, abbreviations | - - **Rules of thumb:** - - Start with `hybrid_v2` — it handles 95% of real-world cases without GPU - - Use `semantic_v2` when entities use different words for the same concept - - Use `blocking_v2` when processing >500k entities and speed matters most - - Never use v1 strategies on new projects — they exist for backwards compatibility only - - - -### Threshold Tuning - -| Domain | Recommended Threshold | Notes | -| ------ | --------------------- | ----- | -| Person names | 0.85–0.90 | Names vary a lot; too high misses "Steve" / "Steven" | -| Organization names | 0.80–0.88 | Corporate suffixes create variation; lower threshold helps | -| Product names | 0.88–0.95 | Product names are more stable | -| Medical terms | 0.90–0.95 | High precision required; false merges are dangerous | -| General entities | 0.85 | Safe default starting point | - -Start at 0.85, inspect false positives and false negatives, then adjust ±0.05. +- **`DuplicateDetector`** — pairwise and batch duplicate detection with configurable strategies +- **`EntityMerger`** — merge duplicate groups with configurable property-level merge policies +- **`SimilarityCalculator`** — multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding +- **`ClusterBuilder`** — Union-Find and hierarchical clustering for large-scale batch deduplication +- **Convenience functions** — `detect_duplicates`, `merge_entities`, `calculate_similarity` ## DuplicateDetector - - **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. - +Find duplicate entity pairs with configurable strategies and result filtering: ```python from semantica.deduplication import DuplicateDetector -detector = DuplicateDetector(similarity_threshold=0.85) +detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities) for dup in duplicates: @@ -138,23 +52,30 @@ Fine-grained control over strategy, thresholds, and result size: ```python duplicates = detector.detect_duplicates( entities, - strategy="hybrid_v2", - min_similarity=0.85, - top_k_per_entity=3, # max candidates per entity — avoids false-positive floods - max_results=100, - sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" + strategy="semantic_v2", # see strategies table below + min_similarity=0.85, # minimum score to consider a match + top_k_per_entity=3, # max candidates per entity + max_results=100, # total result cap + sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" ) ``` -**Key behaviours:** -- Defaults to `hybrid_v2` when no strategy is specified -- `top_k_per_entity=3` prevents one entity from flooding results by being a near-match to everything -- Pairs are returned once — never both `(A, B)` and `(B, A)` -- `sort_by="similarity"` puts highest-confidence duplicates first for faster manual review +### Detection Strategies + +| Strategy | Algorithm | Speed | Accuracy | +| -------- | --------- | ----- | -------- | +| `jaro_winkler` | String similarity (v1) | Fast | Medium | +| `blocking_v2` | Blocking + Jaro-Winkler (v2) | Very fast | Medium | +| `hybrid_v2` | Blocking + semantic + string (v2) | Fast | High | +| `semantic_v2` | Embedding similarity (v2) | Medium | Highest | + + + **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. + ## EntityMerger -Merge detected duplicate groups into canonical entities, preserving provenance: +Merges detected duplicate groups into canonical entities, preserving provenance: ```python from semantica.deduplication import EntityMerger @@ -162,63 +83,56 @@ from semantica.deduplication import EntityMerger merger = EntityMerger() merged_entities = merger.merge_duplicates( entities, - strategy="keep_most_complete", - preserve_provenance=True, + strategy="keep_most_complete", # see strategies table below + preserve_provenance=True, # keep source references after merge ) - -print(f"Merged to: {len(merged_entities)} canonical entities") ``` ### Merge Strategies -| Strategy | Behavior | When to Use | -| -------- | -------- | ----------- | -| `keep_first` | Keep the first entity in each group | Source order is meaningful (most authoritative first) | -| `keep_last` | Keep the most recently seen entity | Most recent source is most accurate | -| `keep_most_complete` | Keep the entity with the most non-null properties | Default — maximizes data richness | -| `keep_highest_confidence` | Keep the entity with the highest confidence score | Extraction pipelines produce confidence scores | -| `merge_all` | Merge all properties; combine non-conflicting fields | You want every known alias, tag, and label | +| Strategy | Behavior | +| -------- | -------- | +| `keep_first` | Keep the first entity in each duplicate group | +| `keep_last` | Keep the most recently seen entity | +| `keep_most_complete` | Keep the entity with the most non-null properties | +| `union` | Merge all properties — non-conflicting fields combined | +| `voting` | Most common property value wins | -### Per-Property Merge Rules - -Use `MergeStrategyManager` to apply different `MergeStrategy` values per property: +Fine-grained per-property merge rules: ```python -from semantica.deduplication import MergeStrategyManager, MergeStrategy +from semantica.deduplication import EntityMerger, PropertyMergeRule -manager = MergeStrategyManager() -manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) -manager.add_property_rule("confidence", MergeStrategy.KEEP_HIGHEST_CONFIDENCE) -manager.add_property_rule("created_at", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("updated_at", MergeStrategy.KEEP_LAST) - -merged_entity = manager.merge_entities(duplicate_group) +merger = EntityMerger( + property_rules={ + "name": PropertyMergeRule.KEEP_FIRST, + "aliases": PropertyMergeRule.UNION, + "description": PropertyMergeRule.KEEP_LONGEST, + } +) ``` -| Strategy | Behaviour | -| -------- | --------- | -| `KEEP_FIRST` | Value from the first entity in the group | -| `KEEP_LAST` | Value from the last entity | -| `KEEP_MOST_COMPLETE` | Entity with the most non-null fields | -| `KEEP_HIGHEST_CONFIDENCE` | Entity with the highest confidence score | -| `MERGE_ALL` | Combine all properties from every entity in the group | - ## SimilarityCalculator -Compute multi-factor similarity scores — useful for debugging why two entities were (or were not) detected as duplicates: +Compute multi-factor similarity scores between entity pairs: ```python from semantica.deduplication import SimilarityCalculator -calc = SimilarityCalculator() +calc = SimilarityCalculator() + score = calc.calculate_similarity(entity_a, entity_b) +# → SimilarityResult(score=0.91, components={...}) print(score.score) # overall score 0.0–1.0 -print(score.components["label"]) # label similarity contribution -print(score.components["embedding"]) # semantic similarity contribution -print(score.components["property"]) # property overlap contribution +print(score.components["label"]) # label similarity +print(score.components["embedding"]) # semantic similarity +print(score.components["property"]) # property overlap +``` +Individual string and vector metrics: + +```python lev = calc.levenshtein("Apple Inc.", "Apple Inc") jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs") cos = calc.cosine_similarity(embedding_a, embedding_b) @@ -232,200 +146,57 @@ Build entity clusters for large-scale batch deduplication: ```python from semantica.deduplication import ClusterBuilder -builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" -result = builder.build_clusters(entities, similarity_threshold=0.85) - -print(f"Original: {len(entities)} entities") -print(f"Clusters: {len(result.clusters)} groups") -print(f"Singletons: {result.singleton_count}") +builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" +result = builder.build_clusters(entities, similarity_threshold=0.85) +print(f"Clusters: {len(result.clusters)}") for cluster in result.clusters: - print(f" [{cluster.id}] members={cluster.members} cohesion={cluster.cohesion:.2f}") -``` - -### Union-Find vs Hierarchical - -```python -# Union-Find — O(n·α(n)), scales to millions; use for production -builder = ClusterBuilder(algorithm="union_find") - -# Hierarchical — tighter clusters; use when quality matters more than speed -builder = ClusterBuilder(algorithm="hierarchical", linkage="average") -result = builder.build_clusters(entities, similarity_threshold=0.85) -``` - -**Key behaviours:** -- Union-Find groups entities transitively: if A≈B and B≈C, all three land in one cluster even if A and C are only 0.70 similar -- Hierarchical with `linkage="average"` avoids chaining by requiring average similarity across all pairs to meet the threshold - -### Cluster Quality Metrics - -```python -print(f"Silhouette score: {result.quality.silhouette_score:.3f}") -# → -1.0 to 1.0; above 0.5 is good, above 0.7 is excellent - -print(f"Avg cohesion: {result.quality.avg_cohesion:.3f}") -print(f"Avg separation: {result.quality.avg_separation:.3f}") -``` - -## MergeStrategyManager - -Define complex, reusable merge configurations once and apply them across multiple operations: - -```python -from semantica.deduplication import MergeStrategyManager, MergeStrategy - -manager = MergeStrategyManager() -manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) -manager.add_property_rule("description", MergeStrategy.KEEP_MOST_COMPLETE) -manager.add_property_rule("confidence", MergeStrategy.KEEP_HIGHEST_CONFIDENCE) -manager.add_property_rule("sources", MergeStrategy.MERGE_ALL) -manager.add_property_rule("created_at", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("updated_at", MergeStrategy.KEEP_LAST) - -merged_entity = manager.merge_entities(duplicate_group) + print(f" [{cluster.id}] {cluster.members} — cohesion: {cluster.cohesion:.2f}") ``` ## Blocking Strategies +Blocking reduces the O(n²) pairwise comparison problem to a manageable candidate set: + ```python detector = DuplicateDetector( - blocking_strategy="token", # "token" | "phonetic" | "ngram" + blocking_strategy="token", # "token" | "phonetic" | "ngram" blocking_threshold=0.6, similarity_threshold=0.85 ) ``` -| Blocking Strategy | How It Works | Best For | -| ----------------- | ------------ | -------- | -| `token` | Shared token overlap (default) | General entity names | -| `phonetic` | Soundex/Metaphone phonetic codes | Names with spelling variations | -| `ngram` | Character n-gram overlap | Short strings, typos | - ## Custom Similarity Functions +Register domain-specific similarity logic: + ```python from semantica.deduplication import method_registry -def drug_name_similarity(entity_a, entity_b) -> float: - compound_a = entity_a.properties.get("active_compound", "") - compound_b = entity_b.properties.get("active_compound", "") - return 1.0 if compound_a == compound_b else 0.0 +def drug_name_similarity(entity_a, entity_b): + # Match drug names by active compound + return score # 0.0 to 1.0 method_registry.register("similarity", "drug_name", drug_name_similarity) -detector = DuplicateDetector(similarity_method="drug_name", similarity_threshold=0.90) +detector = DuplicateDetector(similarity_method="drug_name") ``` -## Schemas - - - +## Convenience Functions ```python -@dataclass -class Cluster: - id: str - members: List[str] # entity IDs in this duplicate group - cohesion: float # mean pairwise similarity within the cluster (0–1) - centroid: str # member ID closest to the cluster centroid +from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity -@dataclass -class ClusterResult: - clusters: List[Cluster] - singleton_count: int # entities with no duplicates found - merge_candidates: int # clusters with > 1 member - quality: ClusterQuality +# Quick detection +duplicates = detect_duplicates(entities, method="semantic_v2", similarity_threshold=0.85) + +# Quick merge +merged = merge_entities(entities, duplicates, method="keep_most_complete") + +# Quick similarity +score = calculate_similarity(entity_a, entity_b, method="hybrid_v2") ``` - - - -```python -@dataclass -class ClusterQuality: - silhouette_score: float # -1 to 1; above 0.5 is good - avg_cohesion: float # mean within-cluster similarity - avg_separation: float # mean between-cluster distance -``` - - - - -## End-to-End Pipeline - - - - ```python - entities = load_entities_from_sources(["crunchbase", "wikipedia", "internal_db"]) - print(f"Loaded: {len(entities)} raw entities") - ``` - - - ```python - from semantica.deduplication import DuplicateDetector - - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") - print(f"Found: {len(duplicates)} duplicate pairs") - ``` - - - ```python - from semantica.deduplication import MergeStrategyManager, MergeStrategy - - manager = MergeStrategyManager() - manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) - manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) - manager.add_property_rule("description", MergeStrategy.KEEP_MOST_COMPLETE) - ``` - - - ```python - from semantica.deduplication import EntityMerger - - merger = EntityMerger() - merged = merger.merge_duplicates(entities, preserve_provenance=True) - print(f"Result: {len(merged)} canonical entities") - - for entity in merged: - if hasattr(entity, "source_entities"): - print(f"{entity.label} merged from: {entity.source_entities}") - ``` - - - -## Tips and Common Pitfalls - - - **Normalize before deduplicating.** Run `TextNormalizer` and `EntityNormalizer` first. "APPLE INC." and "apple inc." will score 0.50 on string similarity but 1.0 after case normalization. See the [Normalize](normalize) module. - - - - **Too many false positives?** Raise `similarity_threshold` by 0.05 or switch from `blocking_v2` to `hybrid_v2` to add semantic precision on top of string matching. - - - - **Too many missed duplicates?** Lower `similarity_threshold` by 0.05, or switch to `semantic_v2` to catch entities that use different words ("ML" vs "Machine Learning"). - - - - **Union-Find over-merges?** The transitive grouping means weak chains can connect unrelated entities. Switch to `hierarchical` clustering with `linkage="average"` to require that every pair within a cluster meets the threshold — not just a chain of nearby pairs. - - - - **Preserve provenance on merge.** Set `preserve_provenance=True` so you can always trace which source contributed each property to the merged entity. Critical for audit and debugging. - - - - **Inspect similarity components.** When a pair is flagged and you're not sure why, use `SimilarityCalculator.calculate_similarity()` to see the per-component breakdown (`label`, `embedding`, `property`) and identify which factor is driving the match. - - - - **Don't deduplicate after building the graph.** Deduplicate entities *before* `GraphBuilder` ingests them. Merging inside a live graph is possible but requires tracking and rewriting all relationship endpoints. - - Detect value conflicts between non-duplicate entities. @@ -434,9 +205,9 @@ class ClusterQuality: GraphBuilder uses deduplication during construction. - Normalize entity names before deduplication for better accuracy. + Normalize entity names before deduplication. - Track merged entity lineage and source attribution. + Track merged entity lineage. diff --git a/docs/reference/export.md b/docs/reference/export.md index 76b5b341..8c4d67f7 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -6,6 +6,30 @@ icon: "file-export" `semantica.export` serializes knowledge graphs to every downstream format — semantic web standards, analytics pipelines, graph databases, and vector stores. All exporters share a consistent `export(graph, path, format)` interface. +## Exported Classes + +```python +from semantica.export import ( + RDFExporter, # Turtle, JSON-LD, N-Triples, RDF/XML + ParquetExporter, # columnar Parquet (Spark, BigQuery, Databricks, Snowflake) + LPGExporter, # Cypher CREATE/MERGE for Neo4j / Memgraph + ArangoAQLExporter, # AQL INSERT for ArangoDB + GraphExporter, # GraphML, GEXF, Graphviz DOT + OWLExporter, # OWL 2.0 in Turtle, XML, JSON-LD + CSVExporter, # flat CSV nodes + edges + VectorExporter, # embedding vectors as JSON, NumPy, or FAISS + ArrowExporter, # Apache Arrow IPC (zero-copy transfer) + DistanceExporter, # semantic distance matrices and ego-graphs + ReportGenerator, # human-readable analytics reports (HTML, Markdown, JSON) + NamespaceManager, # register and resolve RDF namespace prefixes + SemanticNetworkYAMLExporter, # YAML semantic network export + # Convenience functions + export_rdf, export_parquet, export_csv, export_lpg, + export_arango, export_graph, export_owl, export_vector, + export_arrow, generate_report, +) +``` + ## What You Get diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 6ae284ee..ec928734 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -1,33 +1,47 @@ --- title: "Knowledge Graph Module" -description: "Graph construction, temporal models, analytics, and distance intelligence." +description: "Graph construction, temporal models, analytics, similarity scoring, and structural embeddings." icon: "diagram-project" --- -`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and Distance Intelligence (v0.5.0). +`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and structural similarity scoring. + +## Exported Classes + +```python +from semantica.kg import ( + KnowledgeGraph, # core graph data structure + GraphBuilder, # construct from entities + relationships + GraphBuilderWithProvenance, # auto-tracks provenance for every node/edge + EntityResolver, # entity deduplication during construction + GraphAnalyzer, # temporal evolution, diversity metrics + GraphValidator, # schema and constraint validation + TemporalGraphQuery, # point-in-time snapshots, diffs, interval queries + TemporalPatternDetector, # sequence/cycle/trend detection + TemporalVersionManager, # snapshot creation and version comparison + TemporalNormalizer, # normalize timestamps across granularities + BiTemporalFact, # bi-temporal fact model (transaction + valid time) + CentralityCalculator, # degree, betweenness, closeness, PageRank, eigenvector + CommunityDetector, # Louvain, Leiden, Label Propagation, K-Clique + PathFinder, # Dijkstra, A*, BFS, K-Shortest paths + LinkPredictor, # Preferential Attachment, Jaccard, Adamic-Adar + NodeEmbedder, # Node2Vec, DeepWalk structural embeddings + SimilarityCalculator, # cosine, Euclidean, Manhattan, correlation similarity + ConnectivityAnalyzer, # connected components, bridges, density + ProvenanceTracker, # source tracking and lineage management +) +``` ## What You Get - - - Construct graphs from entities and relationships with automatic entity merging. - - - Time-aware queries — filter by `valid_from`/`valid_until`, range queries, and evolution analysis. - - - Connected components, bridge detection, and edge density analysis. - - - PageRank, degree, betweenness, closeness, and eigenvector centrality. - - - Louvain, Leiden, Label Propagation, and K-Clique community detection. - - - Dijkstra, A\*, BFS, and K-Shortest path algorithms. - - +- **`GraphBuilder`** — construct graphs from entities and relationships with automatic entity merging +- **`TemporalGraphQuery`** — time-aware point-in-time snapshots, diffs, and Allen interval queries (v0.4.0) +- **`SimilarityCalculator`** — cosine, Euclidean, Manhattan, and correlation similarity scoring +- **`CentralityCalculator`** — PageRank, degree, betweenness, closeness, eigenvector centrality +- **`CommunityDetector`** — Louvain, Leiden, Label Propagation, K-Clique community detection +- **`PathFinder`** — Dijkstra, A\*, BFS, K-Shortest path algorithms +- **`LinkPredictor`** — Preferential Attachment, Jaccard, Adamic-Adar link prediction +- **`NodeEmbedder`** — Node2Vec, DeepWalk structural embeddings For conflict detection and advanced entity resolution, use `semantica.conflicts` and `semantica.deduplication` alongside this module. @@ -35,54 +49,6 @@ icon: "diagram-project" Knowledge graph entity and relation structure: Person, Organization, Location, Date nodes with typed labeled edges -## Quick Start - - - - ```python - from semantica.kg import GraphBuilder - - builder = GraphBuilder(merge_entities=True) - kg = builder.build(entities=entities, relationships=relationships) - - print(f"Nodes: {kg.node_count}, Edges: {kg.edge_count}") - ``` - - - ```python - from semantica.kg import CentralityCalculator - - calc = CentralityCalculator() - pagerank = calc.calculate_pagerank(kg, damping_factor=0.85) - top_10 = calc.get_top_nodes(pagerank, top_k=10) - - for node_id, score in top_10: - print(f" {node_id}: {score:.4f}") - ``` - - - ```python - from semantica.kg import CommunityDetector - - detector = CommunityDetector() - communities = detector.detect_communities(kg, algorithm="louvain") - metrics = detector.calculate_community_metrics(kg, communities) - - print(f"Communities: {len(communities)}") - ``` - - - ```python - from semantica.graph_store import GraphStore - - store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", - user="neo4j", password="password") - store.add_nodes_bulk(kg.entities, batch_size=1000) - store.add_edges_bulk(kg.relationships, batch_size=1000) - ``` - - - ## GraphBuilder Constructs knowledge graphs from extracted entities and relationships: @@ -91,7 +57,7 @@ Constructs knowledge graphs from extracted entities and relationships: from semantica.kg import GraphBuilder builder = GraphBuilder(merge_entities=True) -kg = builder.build(entities=entities, relationships=relationships) +kg = builder.build(entities=entities, relationships=relationships) ``` | Method | Description | @@ -100,230 +66,166 @@ kg = builder.build(entities=entities, relationships=relationships) | `build_single_source(data)` | Build graph from a single data source | | `merge_entities()` | Deduplicate and merge entities during construction | - - Always use `merge_entities=True` in production. Without it, "Steve Jobs" extracted from five different documents creates five separate person nodes. `GraphBuilder(merge_entities=True)` uses edit distance matching to consolidate them at build time. - +## Temporal Knowledge Graphs (v0.4.0) -## Temporal Queries - -Use `TemporalGraphQuery` to run time-aware queries against a knowledge graph whose relationships carry `valid_from` / `valid_until` fields: +Use `TemporalGraphQuery` to attach `valid_from`/`valid_until` windows and query time-aware graphs: ```python -from semantica.kg import TemporalGraphQuery +from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalVersionManager from datetime import datetime -query_engine = TemporalGraphQuery( - enable_temporal_reasoning=True, - temporal_granularity="day", -) +# Build a time-aware graph +builder = GraphBuilder() +kg = builder.build(sources=[ + { + "entities": [ + {"id": "alice", "type": "Person"}, + {"id": "acme_corp", "type": "Organization"}, + ], + "relationships": [ + { + "source": "alice", "target": "acme_corp", "type": "ceo_of", + "valid_from": "2020-01-01", + "valid_until": "2023-06-01", + } + ] + } +]) -# Point-in-time query — returns only edges valid at the given time -result_2021 = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15)) -result_2023 = query_engine.query_at_time(kg, query="", at_time=datetime(2023, 1, 1)) +# Point-in-time snapshot +query = TemporalGraphQuery(kg) +snapshot_2021 = query.at_time("2021-06-15") +snapshot_2023 = query.at_time("2023-01-01") -# Compare what changed between two snapshots -added = [ - r for r in result_2023["relationships"] - if r not in result_2021["relationships"] -] -print(f"New edges since 2021: {len(added)}") +# Diff between two snapshots +diff = query.diff("2020-01-01", "2023-01-01") +print(f"New nodes since 2020: {len(diff.get('added_nodes', []))}") -# Range query — edges valid within a time window -range_result = query_engine.query_time_range(kg, query="", start_time=datetime(2020, 1, 1), end_time=datetime(2023, 1, 1)) - -# Evolution analysis -evolution = query_engine.analyze_evolution(kg) +# Versioned snapshots +versioner = TemporalVersionManager() +versioner.create_snapshot(kg, version_label="2024-Q1") ``` - - Relationships added without `valid_from`/`valid_until` are treated as **always-valid**. For historical data, always attach timestamps — otherwise point-in-time queries return misleading results. - +Supports all 13 Allen interval algebra relations (before, after, meets, overlaps, during, starts, finishes, equals, and their inverses). OWL-Time export available. + +## Similarity Scoring + +`SimilarityCalculator` computes cosine, Euclidean, Manhattan, and correlation similarity between node embeddings: + +```python +from semantica.kg import SimilarityCalculator, NodeEmbedder + +# First compute structural embeddings +embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) +embeddings = embedder.compute_embeddings(kg, ["Person", "Organization"], ["RELATED_TO"]) + +# Then compare nodes by embedding similarity +calc = SimilarityCalculator() +score = calc.cosine_similarity(embeddings["Apple Inc."], embeddings["Google"]) +print(f"Apple–Google structural similarity: {score:.3f}") + +# Find structurally similar nodes +similar = embedder.find_similar_nodes(kg, "Apple Inc.", top_k=5) +for node in similar: + print(f"{node['id']}: {node['score']:.3f}") +``` ## Graph Analytics - - - Identify the most structurally important nodes in your graph: +### Centrality Analysis - ```python - from semantica.kg import CentralityCalculator +```python +from semantica.kg import CentralityCalculator - calc = CentralityCalculator() +calculator = CentralityCalculator() - pagerank = calc.calculate_pagerank(kg, damping_factor=0.85) - degree = calc.calculate_degree_centrality(kg) - betweenness = calc.calculate_betweenness_centrality(kg) - closeness = calc.calculate_closeness_centrality(kg) - eigenvector = calc.calculate_eigenvector_centrality(kg) - all_metrics = calc.calculate_all_centrality(kg) +centrality = calculator.calculate_degree_centrality(graph) +pagerank = calculator.calculate_pagerank(graph, damping_factor=0.85) +betweenness = calculator.calculate_betweenness_centrality(graph) +closeness = calculator.calculate_closeness_centrality(graph) +eigenvector = calculator.calculate_eigenvector_centrality(graph) +all_metrics = calculator.calculate_all_centrality(graph) - top_nodes = calc.get_top_nodes(pagerank, top_k=10) - ``` +top_nodes = calculator.get_top_nodes(centrality, top_k=10) +``` - | Measure | Best For | - | ------- | -------- | - | PageRank | Overall importance (link-based) | - | Degree | Most connected nodes | - | Betweenness | Bridge / bottleneck nodes | - | Closeness | Fastest to reach all others | - | Eigenvector | Connected to other important nodes | - - - Partition the graph into thematically dense clusters: +| Method | Algorithm | +| ------ | --------- | +| `calculate_degree_centrality()` | Degree-based importance | +| `calculate_betweenness_centrality()` | Bridge-based importance (bottleneck nodes) | +| `calculate_closeness_centrality()` | Distance-based importance | +| `calculate_eigenvector_centrality()` | Influence-based importance | +| `calculate_pagerank()` | Link-based importance (PageRank) | +| `calculate_all_centrality()` | All measures at once | - ```python - from semantica.kg import CommunityDetector +### Community Detection - detector = CommunityDetector() +```python +from semantica.kg import CommunityDetector - # Louvain — fast, high quality (default) - communities = detector.detect_communities(kg, algorithm="louvain") +detector = CommunityDetector() - # Leiden — higher quality, slower - leiden_communities = detector.detect_communities_leiden(kg, resolution=1.2) +# Louvain (default — fast, high quality) +communities = detector.detect_communities(graph, algorithm="louvain") - metrics = detector.calculate_community_metrics(kg, communities) - print(f"Communities: {len(communities)}") - ``` +# Leiden (higher quality, slower) +leiden_communities = detector.detect_communities_leiden(graph, resolution=1.2) - Algorithms available: **Louvain**, **Leiden**, **Label Propagation**, **K-Clique Communities**. +metrics = detector.calculate_community_metrics(graph, communities) +``` - - Community detection finds thematic clusters — often corresponding to real-world subject groups. Use cluster membership as context boundaries for GraphRAG retrieval. - - - - Find shortest and alternative paths between nodes: +Algorithms: Louvain, Leiden, Label Propagation, K-Clique Communities. - ```python - from semantica.kg import PathFinder +### Path Finding - finder = PathFinder() +```python +from semantica.kg import PathFinder - path = finder.dijkstra_shortest_path(kg, "node_a", "node_b") - paths = finder.all_shortest_paths(kg, "source", "target") - k_paths = finder.find_k_shortest_paths(kg, "source", "target", k=3) - ``` +finder = PathFinder() - Algorithms: **Dijkstra**, **A\***, **BFS**, **All Shortest Paths**, **K-Shortest Paths**. - - - Analyse graph structure — components, bridges, and density: +path = finder.dijkstra_shortest_path(graph, "node_a", "node_b") +paths = finder.all_shortest_paths(graph, "source", "target") +k_paths = finder.find_k_shortest_paths(graph, "source", "target", k=3) +``` - ```python - from semantica.kg import ConnectivityAnalyzer +Algorithms: Dijkstra, A\*, BFS, All Shortest Paths, K-Shortest Paths. - analyzer = ConnectivityAnalyzer() - components = analyzer.find_connected_components(kg) - density = analyzer.calculate_density(kg) - bridges = analyzer.find_bridges(kg) +### Link Prediction - print(f"Components: {len(components)}, Largest: {len(components[0])} nodes") - print(f"Density: {density:.4f}") - print(f"Bridges: {bridges}") - ``` +```python +from semantica.kg import LinkPredictor - | Method | Returns | Description | - | ------ | ------- | ----------- | - | `find_connected_components(kg)` | `List[List[str]]` | Groups of mutually reachable nodes | - | `calculate_density(kg)` | `float` | Edge density (actual / possible edges) | - | `find_bridges(kg)` | `List[str]` | Nodes whose removal disconnects the graph | - - - Predict which edges are likely missing from the graph: +predictor = LinkPredictor(method="preferential_attachment") +links = predictor.predict_links(graph, top_k=20) +score = predictor.score_link(graph, "node_a", "node_b") +``` - ```python - from semantica.kg import LinkPredictor +Algorithms: Preferential Attachment, Common Neighbors, Jaccard, Adamic-Adar, Resource Allocation. - predictor = LinkPredictor(method="preferential_attachment") - links = predictor.predict_links(kg, top_k=20) - score = predictor.score_link(kg, "node_a", "node_b") - ``` +### Node Embeddings - Algorithms: **Preferential Attachment**, **Common Neighbors**, **Jaccard**, **Adamic-Adar**, **Resource Allocation**. - - - Compute structural embeddings for similarity search and downstream ML: +```python +from semantica.kg import NodeEmbedder - ```python - from semantica.kg import NodeEmbedder +embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) +embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) +similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) +``` - embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) - embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) - similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) - ``` - - Algorithms: **Node2Vec**, **DeepWalk**, **Word2Vec**. - - +Algorithms: Node2Vec, DeepWalk, Word2Vec. ## Algorithm Summary | Category | Algorithms | Use Cases | | -------- | ---------- | --------- | | Node Embeddings | Node2Vec, DeepWalk, Word2Vec | Structural similarity, node representation | +| Similarity | Cosine, Euclidean, Manhattan, Correlation | Node matching, recommendation | | Path Finding | Dijkstra, A\*, BFS, K-Shortest | Route planning, network analysis | | Link Prediction | Preferential Attachment, Jaccard, Adamic-Adar | Network completion | | Centrality | Degree, Betweenness, Closeness, PageRank | Influence analysis | | Community Detection | Louvain, Leiden, Label Propagation | Social clustering | | Connectivity | Components, Bridges, Density | Network robustness | -## SeedManager - -Load and inject curated seed data into a knowledge graph: - -```python -from semantica.kg import SeedManager - -manager = SeedManager() -seed_data = manager.load_seed("seeds/domain_entities.json") -normalized = manager.normalize(seed_data, source="manual_curation_v1") - -builder = GraphBuilder(merge_entities=True) -kg = builder.build(normalized + extracted_sources) -``` - -## MethodRegistry - -Register custom KG construction methods and dispatch by name: - -```python -from semantica.kg import method_registry - -def my_kg_builder(entities, relationships, **kwargs): - filtered = [e for e in entities if e["confidence"] >= 0.9] - return {"entities": filtered, "relationships": relationships} - -method_registry.register("build", "high_confidence", my_kg_builder) - -# Dispatch by name via the registry -result = method_registry.execute("build", "high_confidence", entities=entities, relationships=relationships) -``` - -## ProvenanceTracker - -Track entity and relationship lineage within a knowledge graph: - -```python -from semantica.kg import ProvenanceTracker - -tracker = ProvenanceTracker() - -tracker.track_entity( - entity_id="apple_inc", - source="sec_filing_2024q1.pdf", - source_location="page 3, paragraph 2", - source_quote="Apple Inc. reported revenue of...", - confidence=0.98, -) - -lineage = tracker.get_lineage("apple_inc") -for entry in lineage.entries: - print(f" Source: {entry.source} ({entry.timestamp})") -``` - -For full W3C PROV-O compliance and provenance export, see the [Provenance module](provenance). - ## Configuration ```yaml @@ -337,24 +239,6 @@ kg: default_validity: infinite ``` -## Tips and Common Pitfalls - - - **Deduplicate before `GraphBuilder`, not after.** It's far easier to merge entities before they become nodes than to update all relationship endpoints after the fact. Run `DuplicateDetector` on extracted entities before calling `builder.build()`. - - - - **PageRank identifies your most connected, important nodes.** If you're not sure which entities in your graph are the most structurally significant, `CentralityCalculator.calculate_pagerank()` gives you a ranked list — useful for GraphRAG context anchoring. - - - - **Community detection finds thematic clusters.** `CommunityDetector` with Louvain partitions your graph into clusters of densely-connected nodes — often corresponding to real-world thematic groups. Use these clusters for exploratory analysis and to scope GraphRAG retrieval. - - - - **`ProvenanceTracker` links entities back to their source documents.** Use it during graph construction so you can always answer "where did this fact come from?" — critical for compliance and for debugging incorrect graph data. - - Persist graphs in Neo4j, FalkorDB, or Apache AGE. diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 0db513d5..ddd74494 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -1,89 +1,35 @@ --- title: "LLMs Module" -description: "Unified interface for Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace — swap providers with a one-line change." +description: "Unified interface for Groq, OpenAI, LiteLLM (Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, 100+ models), and HuggingFace." icon: "microchip" --- -`semantica.llms` gives every Semantica module a single, consistent interface to 9+ LLM providers. Every extractor, reasoning engine, and context graph accepts any provider through the same `llm_provider=` parameter — swap Groq for Anthropic, or a cloud API for a local Ollama model, by changing one line. +`semantica.llms` provides a single consistent API across every major LLM provider. Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoning engines, and agents. + +## Exported Classes + +```python +from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM +``` + +| Class | Provider | API Key Required | +| ----- | -------- | ---------------- | +| `Groq` | Groq Cloud | `GROQ_API_KEY` | +| `OpenAI` | OpenAI / any OpenAI-compatible gateway | `OPENAI_API_KEY` | +| `LiteLLM` | 100+ providers via LiteLLM routing | Depends on model | +| `HuggingFaceLLM` | Local HuggingFace Transformers | None (local) | + + + **Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, Cohere, and 90+ others** are all available via `LiteLLM` using their model-string prefix. See the [LiteLLM section](#litellm-100-providers) below. + ## What You Get - - - Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace — all behind one interface. - - - `complete()`, `chat()`, and `stream()` work identically across all providers — swap with a one-line change. - - - One class, every provider — Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, and 90+ more via LiteLLM model strings. - - - Ollama and HuggingFace run fully on-premise — no API key, no data leaves your machine, air-gap compatible. - - - Token-by-token output via `stream()` for responsive agent pipelines and live UI updates. - - - Configurable `max_retries` with exponential backoff. Typed exceptions: `LLMAuthenticationError`, `LLMRateLimitError`, `LLMContextLengthError`. - - - -## Installation - -The base install includes Groq and DeepSeek. Other providers require optional extras: - -| Provider | Install Command | API Key Required | -| -------- | --------------- | ---------------- | -| Groq | `pip install semantica` | Yes — `GROQ_API_KEY` | -| DeepSeek | `pip install semantica` | Yes — `DEEPSEEK_API_KEY` | -| Novita AI | `pip install semantica` | Yes — `NOVITA_API_KEY` | -| OpenAI | `pip install "semantica[llm-openai]"` | Yes — `OPENAI_API_KEY` | -| Anthropic | `pip install "semantica[llm-anthropic]"` | Yes — `ANTHROPIC_API_KEY` | -| Gemini | `pip install "semantica[llm-gemini]"` | Yes — `GOOGLE_API_KEY` | -| Ollama | `pip install "semantica[llm-ollama]"` | No — local server | -| LiteLLM | `pip install "semantica[llm-litellm]"` | Varies by target | -| HuggingFace | `pip install "semantica[llm-huggingface]"` | No — local weights | -| All providers | `pip install "semantica[all]"` | Varies | - -## Quick Start - - - - ```python - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - ``` - - - ```python - from semantica.semantic_extract import NERExtractor - - ner = NERExtractor(method="llm", llm_provider=llm) - ``` - - - ```python - entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") - ``` - - - ```python - from semantica.llms import LiteLLM - from semantica.core import ConfigManager - - config = ConfigManager("config.yaml") - # LiteLLM model strings include the provider prefix: - # "anthropic/claude-opus-4-7", "gemini/gemini-1.5-pro", "ollama/llama3.2" - llm = LiteLLM( - model=config.get("llm_provider.model"), - api_key=config.get("llm_provider.api_key"), - ) - ``` - - +- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application code changes +- **`LiteLLM`** — single class for 100+ providers using model-string routing +- **Local models** — `HuggingFaceLLM` runs fully on-premise, no API key +- **Streaming** — token-by-token output for low-latency UX +- **Custom gateways** — point `OpenAI` at any OpenAI-compatible endpoint via `base_url` ## Providers @@ -94,13 +40,12 @@ from semantica.llms import Groq import os llm = Groq( - model="llama-3.3-70b-versatile", # default and recommended + model="llama-3.3-70b-versatile", # default api_key=os.getenv("GROQ_API_KEY"), - temperature=0.0, max_tokens=64000, - max_retries=3, - timeout=60, + temperature=0.0, ) +# Best for: high-throughput extraction, fast inference at low cost ``` ```python OpenAI @@ -111,267 +56,91 @@ llm = OpenAI( model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY"), temperature=0.0, - max_tokens=4096, - max_retries=3, - timeout=120, - organization=None, # optional org ID ) +# Best for: general purpose, function calling, JSON mode ``` -```python Anthropic (via LiteLLM) +```python LiteLLM (100+ providers) from semantica.llms import LiteLLM import os -# Anthropic Claude is accessed via LiteLLM using the "anthropic/" prefix -llm = LiteLLM( - model="anthropic/claude-opus-4-7", - api_key=os.getenv("ANTHROPIC_API_KEY"), - max_tokens=8192, - temperature=0.0, -) +# pip install "semantica[llm-litellm]" + +# Anthropic Claude +llm = LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTHROPIC_API_KEY")) + +# Google Gemini +llm = LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY")) + +# Ollama (local — no API key) +llm = LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434") + +# DeepSeek +llm = LiteLLM(model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY")) + +# Azure OpenAI +llm = LiteLLM(model="azure/gpt-4o", api_key=os.getenv("AZURE_API_KEY")) + +# AWS Bedrock +llm = LiteLLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0") + +# Novita AI +llm = LiteLLM(model="novita/deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY")) ``` -```python Gemini (via LiteLLM) -from semantica.llms import LiteLLM -import os - -# Google Gemini is accessed via LiteLLM using the "gemini/" prefix -llm = LiteLLM( - model="gemini/gemini-1.5-pro", - api_key=os.getenv("GOOGLE_API_KEY"), - temperature=0.0, - max_tokens=8192, -) -``` - -```python Ollama / Local (via LiteLLM) -from semantica.llms import LiteLLM - -# Ollama local models via LiteLLM using the "ollama/" prefix -llm = LiteLLM( - model="ollama/llama3.2", - base_url="http://localhost:11434", # default Ollama address - temperature=0.0, - timeout=180, # local models can be slower; increase for large models -) -# No API key — model runs entirely on your machine -``` - -```python DeepSeek (via Groq or LiteLLM) -from semantica.llms import Groq # Groq hosts DeepSeek models -import os - -llm = Groq( - model="deepseek-r1-distill-llama-70b", - api_key=os.getenv("GROQ_API_KEY"), - temperature=0.0, - max_tokens=4096, -) - -# Or via LiteLLM for the native DeepSeek endpoint: -from semantica.llms import LiteLLM -llm = LiteLLM( - model="deepseek/deepseek-chat", - api_key=os.getenv("DEEPSEEK_API_KEY"), - temperature=0.0, - max_tokens=4096, -) -``` - -```python Novita AI (via LiteLLM) -from semantica.llms import LiteLLM -import os - -# Novita AI via LiteLLM using the "novita/" prefix -llm = LiteLLM( - model="novita/deepseek/deepseek-v3", - api_key=os.getenv("NOVITA_API_KEY"), - temperature=0.0, - max_tokens=4096, -) -``` - -```python LiteLLM (100+ models) -from semantica.llms import LiteLLM -import os - -llm = LiteLLM( - model="gpt-4o", # any LiteLLM model string - api_key=os.getenv("OPENAI_API_KEY"), - temperature=0.0, - max_tokens=4096, -) -# Supports: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, Together AI, and 90+ more -# Use the LiteLLM model string format: "anthropic/claude-opus-4-7", "bedrock/anthropic.claude-v2" -``` - -```python HuggingFace (Local) +```python HuggingFaceLLM (Local) from semantica.llms import HuggingFaceLLM llm = HuggingFaceLLM( model="mistralai/Mistral-7B-Instruct-v0.3", - device="cuda", # "cpu" | "cuda" | "mps" (Apple Silicon) + device="cuda", # "cpu" | "cuda" | "mps" max_new_tokens=512, temperature=0.1, - load_in_4bit=True, # enable 4-bit quantisation to reduce VRAM ) -# No API key — weights downloaded from Hugging Face Hub (or loaded from local path) +# Bring your own model — full local control, no API key ``` -## Constructor Parameters +## LiteLLM — 100+ Providers -### Common Parameters - -| Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | -| `model` | `str` | Provider default | Model identifier string | -| `api_key` | `str` | `None` | API key — reads from environment if omitted | -| `temperature` | `float` | `0.0` | Sampling temperature: 0 = deterministic, 1 = creative | -| `max_tokens` | `int` | Provider default | Maximum tokens in the response | -| `max_retries` | `int` | `3` | Number of retry attempts on transient failures | -| `timeout` | `int` | `60` | Request timeout in seconds | -| `base_url` | `str` | Provider default | Override the API endpoint — useful for proxies and gateways | - -### Provider-Specific Parameters - -| Provider | Parameter | Description | -| -------- | --------- | ----------- | -| `OpenAI` | `organization` | OpenAI organisation ID | -| `OpenAI` | `project` | OpenAI project ID | -| `LiteLLM` | `model` | Full LiteLLM model string, e.g. `"anthropic/claude-opus-4-7"`, `"gemini/gemini-1.5-pro"`, `"ollama/llama3.2"` | -| `LiteLLM` | `base_url` | Override endpoint — use for Ollama (`http://localhost:11434`) or proxies | -| `HuggingFaceLLM` | `device` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | -| `HuggingFaceLLM` | `load_in_4bit` | Enable 4-bit quantisation (requires `bitsandbytes`) | -| `HuggingFaceLLM` | `max_new_tokens` | Maximum new tokens to generate (replaces `max_tokens`) | - -## Direct API Usage - -Providers can be used directly — not just through Semantica modules: +`LiteLLM` is the recommended way to access any provider not directly exported by `semantica.llms`. Use the `provider/model` string format: ```python -from semantica.llms import Groq +from semantica.llms import LiteLLM import os -llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +# Pattern: LiteLLM(model="/") +providers = { + "Anthropic": LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTHROPIC_API_KEY")), + "Gemini": LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY")), + "Ollama": LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434"), + "DeepSeek": LiteLLM(model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY")), + "Azure": LiteLLM(model="azure/gpt-4o", api_key=os.getenv("AZURE_API_KEY")), + "Bedrock": LiteLLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"), + "Cohere": LiteLLM(model="cohere/command-r-plus", api_key=os.getenv("COHERE_API_KEY")), + "Novita AI": LiteLLM(model="novita/deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY")), +} -# Single completion -response = llm.complete("What is a knowledge graph?") -print(response.text) # answer string -print(response.input_tokens) # tokens consumed by the prompt -print(response.output_tokens) # tokens in the response -print(response.model) # model that served the request - -# Multi-turn chat -messages = [ - {"role": "system", "content": "You are a knowledge graph expert."}, - {"role": "user", "content": "What is the difference between RDF and property graphs?"}, -] -response = llm.chat(messages) -print(response.text) - -# Streaming — token-by-token output -for token in llm.stream("Explain knowledge graph reasoning in 3 sentences."): - print(token, end="", flush=True) +# Every LiteLLM instance implements the same .generate() interface +response = providers["Anthropic"].generate("Explain GraphRAG in one paragraph.") ``` -## LLMResponse Object + + The full list of supported LiteLLM model strings is at [docs.litellm.ai/docs/providers](https://docs.litellm.ai/docs/providers). Use the `provider/model` format shown above. + -All three methods (`complete`, `chat`, `stream`) return a `LLMResponse` dataclass: +## Custom / Enterprise Gateways - - - -```python -@dataclass -class LLMResponse: - text: str # the generated text - model: str # model identifier that served the request - input_tokens: int # tokens consumed by the prompt - output_tokens: int # tokens in the generated response - total_tokens: int # input_tokens + output_tokens - latency_ms: float # wall-clock time for the API call in milliseconds - finish_reason: str # "stop" | "length" | "content_filter" | "tool_calls" -``` - - - - -## Error Handling - - - - -```python -from semantica.llms import Groq -from semantica.utils import SemanticaError -import os - -try: - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - response = llm.complete("Summarise this document.") - print(response.text) - -except LLMAuthenticationError as e: - # Invalid or expired API key - print(f"Authentication failed: {e}") - -except LLMRateLimitError as e: - # Rate limit hit — Semantica retries automatically up to max_retries - print(f"Rate limited after retries: {e}") - -except LLMContextLengthError as e: - # Prompt exceeds the model's context window - print(f"Prompt too long ({e.token_count} tokens, limit {e.context_limit}): {e}") - -except LLMProviderError as e: - # General provider-side error (5xx, model unavailable, etc.) - print(f"Provider error: {e}") - -except SemanticaError as e: - # Catch-all for all Semantica framework errors - print(f"Framework error: {e}") -``` - -| Exception | When Raised | -| --------- | ----------- | -| `LLMAuthenticationError` | Invalid or missing API key | -| `LLMRateLimitError` | Rate limit exceeded after all retries | -| `LLMContextLengthError` | Prompt exceeds the model's context window | -| `LLMProviderError` | Provider-side error (5xx, unavailability) | -| `LLMTimeoutError` | Request exceeded the `timeout` parameter | - - - - -## Custom and Enterprise Gateways - -Any provider that exposes an OpenAI-compatible REST API can be used by passing `base_url`: +Any OpenAI-compatible endpoint — internal routing layers, Qwen proxies, or private LLaMA deployments: ```python from semantica.llms import OpenAI -import os -# Internal LLM routing gateway llm = OpenAI( model="qwen2.5-72b", api_key=os.getenv("GATEWAY_API_KEY"), - base_url="https://llm-gateway.internal.company.com/v1", -) - -# Azure OpenAI Service -llm = OpenAI( - model="gpt-4o", - api_key=os.getenv("AZURE_OPENAI_API_KEY"), - base_url="https://my-resource.openai.azure.com/openai/deployments/gpt-4o", -) - -# Self-hosted vLLM server -llm = OpenAI( - model="meta-llama/Llama-3.1-8B-Instruct", - api_key="not-needed", - base_url="http://localhost:8000/v1", + base_url="https://my-internal-gateway.company.com/v1", ) ``` @@ -379,128 +148,49 @@ llm = OpenAI( `base_url` is validated at construction time. Non-HTTP(S) schemes raise `ValueError` to prevent SSRF attacks (fixed in v0.5.0). -## Using in Semantica Modules +## Using in Extractors -Every module that uses an LLM accepts any provider through `llm_provider=`: +All extractors accept any provider as `llm_provider=`: ```python -from semantica.llms import Groq, LiteLLM from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor -from semantica.ontology import LLMOntologyGenerator -from semantica.reasoning import Reasoner -from semantica.context import AgentContext, ContextGraph -from semantica.vector_store import VectorStore -import os -groq_llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -# Anthropic Claude via LiteLLM using the "anthropic/" prefix -claude_llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -# Extraction — use fast Groq for high-throughput NER -ner = NERExtractor(method="llm", llm_provider=groq_llm) -rel = RelationExtractor(method="llm", llm_provider=groq_llm) -trip = TripletExtractor(method="llm", llm_provider=groq_llm) - -# Complex reasoning — use Claude for accuracy -engine = Reasoner() - -# Ontology generation from natural language -gen = LLMOntologyGenerator(llm_provider=claude_llm) +ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) +rel = RelationExtractor(method="llm", llm_provider=llm) +trip = TripletExtractor(method="llm", llm_provider=llm) ``` ## Provider Comparison -| Provider | Speed | Cost | Local | Max Context | Best For | -| -------- | ----- | ---- | ----- | ----------- | -------- | -| **Groq** | ⚡ Very fast | 💲 Low | No | 128k | High-throughput extraction, fast pipelines | -| **OpenAI** | Fast | 💲💲 Medium | No | 128k | General purpose, function calling, JSON mode | -| **Anthropic** | Fast | 💲💲 Medium | No | 200k | Complex reasoning, long documents, safety | -| **Gemini** | Fast | 💲 Low | No | 1M | Very long context, multimodal (text + image) | -| **Ollama** | Medium | Free | ✅ Yes | Varies | Privacy, air-gapped, no API key | -| **DeepSeek** | Fast | 💲 Very low | No | 64k | Coding tasks, structured analysis | -| **Novita AI** | Fast | 💲 Low | No | Varies | DeepSeek and LLaMA models, cost-effective | -| **LiteLLM** | Varies | Varies | Varies | Varies | Multi-provider routing, vendor abstraction | -| **HuggingFace** | Slow | Free | ✅ Yes | Varies | Custom and fine-tuned models, full local control | - -## Environment Variables - -| Variable | Provider | Notes | -| -------- | -------- | ----- | -| `GROQ_API_KEY` | Groq | Required for Groq cloud | -| `OPENAI_API_KEY` | OpenAI, LiteLLM | Also used for OpenAI-compatible gateways | -| `ANTHROPIC_API_KEY` | Anthropic | Required for Claude | -| `GOOGLE_API_KEY` | Gemini | Required for Gemini | -| `DEEPSEEK_API_KEY` | DeepSeek | Required for DeepSeek cloud | -| `NOVITA_API_KEY` | Novita AI | Required for Novita AI | -| `HUGGINGFACE_HUB_TOKEN` | HuggingFace | Required for gated models (optional for public models) | - -## YAML Configuration - -```yaml -# config.yaml -llm_provider: - name: "groq" - model: "llama-3.3-70b-versatile" - api_key: "${GROQ_API_KEY}" # reads from environment - temperature: 0.0 - max_tokens: 64000 - max_retries: 3 - timeout: 60 -``` - -Load it with `ConfigManager`: - -```python -from semantica.core import ConfigManager -from semantica.llms import LiteLLM - -config = ConfigManager("config.yaml") -# LiteLLM model strings carry the provider prefix — e.g. "anthropic/claude-opus-4-7", -# "gemini/gemini-1.5-pro", "ollama/llama3.2" — so provider selection is config-driven -llm = LiteLLM( - model=config.get("llm_provider.model"), - api_key=config.get("llm_provider.api_key"), - temperature=config.get("llm_provider.temperature", default=0.0), -) -``` - -## Tips and Common Pitfalls - - - **Always set `temperature=0.0` for extraction tasks.** NER, relation extraction, and triplet generation need deterministic output — any temperature above 0 introduces randomness that produces inconsistent entity types or hallucinated relationships. Reserve higher temperatures for creative or summarisation tasks. - +| Provider | Import | Speed | Cost | Local | Context | Best For | +| -------- | ------ | ----- | ---- | ----- | ------- | -------- | +| Groq | `Groq` | Very fast | Low | No | 128k | High-throughput extraction | +| OpenAI | `OpenAI` | Fast | Medium | No | 128k | General purpose, function calling | +| Anthropic | `LiteLLM(model="anthropic/...")` | Fast | Medium | No | 200k | Complex reasoning, safety | +| Gemini | `LiteLLM(model="gemini/...")` | Fast | Low | No | 1M | Long context, multimodal | +| Ollama | `LiteLLM(model="ollama/...")` | Medium | Free | Yes | Varies | Privacy, air-gapped | +| DeepSeek | `LiteLLM(model="deepseek/...")` | Fast | Very low | No | 64k | Coding, analysis | +| Azure OpenAI | `LiteLLM(model="azure/...")` | Fast | Medium | No | 128k | Enterprise, compliance | +| AWS Bedrock | `LiteLLM(model="bedrock/...")` | Fast | Varies | No | Varies | AWS-native workloads | +| HuggingFace | `HuggingFaceLLM` | Slow | Free | Yes | Varies | Custom models, BYOM | - **Set `max_retries=3` in production.** Transient rate limits and 5xx errors are normal at scale. All providers retry automatically up to `max_retries` with exponential backoff. Without retries, a single rate-limit hit fails an entire pipeline step that would have succeeded on the second attempt. - - - - **Use `LiteLLM` for config-driven pipelines.** Hard-coding `Groq(...)` in Python means changing the provider requires a code change and redeploy. `LiteLLM(model=config.get("llm_provider.model"), ...)` lets you switch from Groq to Anthropic by editing `config.yaml` with a model string like `"anthropic/claude-opus-4-7"` — no code changes. - - - - **Use `base_url` for internal gateways and Azure.** Enterprise deployments often route LLM calls through an internal proxy or Azure OpenAI Service. Pass `base_url="https://llm-gateway.internal/v1"` to `OpenAI` — you get the same Semantica integration without changing any module code. Non-HTTP schemes raise `ValueError` (SSRF protection, v0.5.0+). - - - - **Catch `LLMContextLengthError` explicitly.** If your chunking is misconfigured, a document chunk can exceed the model's context window. Catch `LLMContextLengthError` and log `e.token_count` — it tells you exactly how much to reduce your `chunk_size`. Don't let it surface as a generic failure. - - - - **Use Ollama or HuggingFace for air-gapped environments.** When data cannot leave the network, Ollama (local inference) or HuggingFace (local weights) are the only viable options. Both support the same `llm_provider=` interface — no other code changes needed. + For production extraction pipelines, Groq delivers the best throughput-to-cost ratio. For complex multi-hop reasoning, Claude Opus or GPT-4o provide the highest accuracy. - NER, relation extraction, and triplet generation with LLMs. + Use LLMs for NER and relation extraction. + + + LLM providers in Agno multi-agent teams. - LLM-backed deductive, abductive, and Datalog reasoning. - - - Generate ontologies from natural language using LLMs. + LLM-backed deductive and abductive reasoning. - GraphRAG and decision intelligence powered by LLMs. + GraphRAG uses LLMs for reasoning over knowledge graphs. diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index 52383ade..bd7a3f2c 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -1,330 +1,210 @@ --- title: "Ontology Module" -description: "Automated ontology generation, OWL export, SHACL validation, domain ontologies, and modular ontology development." +description: "Automated ontology generation, SHACL validation, OWL/RDF export, namespace management, and LLM-powered ontology generation." icon: "sitemap" --- -`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and OWL export to SHACL validation and modular ontology development. Use it for schema design, data modeling, and semantic web interoperability. +`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and SHACL validation to OWL/RDF export. Use it for schema design, data modeling, semantic web interoperability, and SHACL-based data quality validation. + +## Exported Classes + +```python +from semantica.ontology import ( + OntologyGenerator, # auto-generate from KG data (6-stage pipeline) + LLMOntologyGenerator, # LLM-powered ontology generation + OntologyEngine, # unified orchestration facade + ClassInferrer, # class discovery and hierarchy building + PropertyGenerator, # property inference and XSD type mapping + SHACLGenerator, # generate SHACL shapes from ontology + OntologyValidator, # validate graphs against SHACL shapes + SHACLValidationReport, # validation report with violations list + SHACLViolation, # individual constraint violation + OWLGenerator, # OWL/RDF serialization (Turtle, XML, JSON-LD) + OntologyEvaluator, # quality evaluation: coverage, completeness + NamespaceManager, # IRI generation and namespace prefix management + OntologyAligner, # align and merge ontologies across schemas (use OntologyEngine) + AssociativeClassBuilder, # N-ary relationship intermediate class creation + NamingConventions, # PascalCase/camelCase enforcement + DomainOntologies, # pre-built domain ontologies + ingest_ontology, # load ontology from file +) +``` ## What You Get - - - Auto-generate ontologies from existing graph data using a 6-stage pipeline. - - - Generate SHACL shapes from an ontology and validate ontologies for structural consistency. - - - Capture, manage, and validate competency questions that define ontology requirements. - - - Integrate published ontologies (schema.org, FOAF) instead of generating from scratch. - - - Pre-built domain ontologies for biomedical, finance, legal, supply chain, and more. - - - Measure coverage, completeness, and granularity — validate against competency questions. - - +- **`OntologyGenerator`** — auto-generate ontologies from existing knowledge graph data (6-stage pipeline) +- **`LLMOntologyGenerator`** — LLM-powered ontology generation for complex domains +- **`OntologyEngine`** — unified facade that orchestrates the full ontology lifecycle +- **`SHACLGenerator`** / **`OntologyValidator`** — generate SHACL shapes and validate any graph +- **`OWLGenerator`** — serialize ontologies to Turtle, RDF/XML, JSON-LD +- **`NamespaceManager`** — IRI generation, prefix management, namespace binding +- **`OntologyEvaluator`** — coverage, completeness, and granularity quality metrics +- **`AssociativeClassBuilder`** — model N-ary relationships as intermediate OWL classes -## Quick Start +## OntologyEngine (Unified Facade) - - - ```python - from semantica.ontology import OntologyGenerator +The `OntologyEngine` orchestrates the full ontology lifecycle — generation, validation, export, and versioning: - generator = OntologyGenerator() - ontology = generator.generate_from_graph(kg) - ``` - - - ```python - from semantica.ontology import OntologyValidator +```python +from semantica.ontology import OntologyEngine - validator = OntologyValidator(reasoner="hermit", check_consistency=True) - result = validator.validate(ontology) +engine = OntologyEngine(base_uri="https://example.org/ontology/") - if not result.is_valid: - for issue in result.issues: - print(f"Issue: {issue.message} (severity: {issue.severity})") - ``` - - - ```python - from semantica.ontology import OWLGenerator +# Generate ontology from KG data +ontology = engine.generate_ontology({"entities": entities, "relationships": relationships}) - owl_gen = OWLGenerator() - owl_gen.export_owl(ontology, file_path="ontology.ttl", format="turtle") - owl_gen.export_owl(ontology, file_path="ontology.owl", format="xml") - ``` - - +# Validate a graph against the generated SHACL shapes +report = engine.validate(kg) +if not report.conforms: + for v in report.violations: + print(f"{v.severity}: {v.message} on {v.node}") -## Auto-Generation — 6-Stage Pipeline +# Export to OWL Turtle +engine.export(ontology, "ontology.ttl", format="turtle") +``` -Generate an ontology automatically from your knowledge graph data: +## OntologyGenerator (6-Stage Pipeline) + +Generate a formal ontology automatically from your knowledge graph entities and relationships: ```python from semantica.ontology import OntologyGenerator -generator = OntologyGenerator() -ontology = generator.generate_from_graph(kg) +generator = OntologyGenerator(base_uri="https://example.org/ontology/") +ontology = generator.generate_ontology({ + "entities": entities, + "relationships": relationships, +}) ``` The pipeline runs through these stages in order: - - - Extracts concepts and patterns from entity/relationship data. +1. **Semantic Network Parsing** — extract concepts and patterns from entity/relationship data +2. **YAML-to-Definition** — transform patterns into intermediate class definitions +3. **Definition-to-Types** — map definitions to OWL types (`owl:Class`, `owl:ObjectProperty`) +4. **Hierarchy Generation** — build taxonomy trees using transitive closure and cycle detection +5. **TTL Generation** — serialize to Turtle format using `rdflib` +6. **Quality Evaluation** — assess coverage, completeness, and granularity metrics - ```python - generator = OntologyGenerator() - semantic_network = generator.parse_semantic_network(kg) - ``` - - - Transforms extracted patterns into intermediate class definitions. +## SHACL Validation - ```python - definitions = generator.build_definitions(semantic_network) - ``` - - - Maps definitions to OWL types (`owl:Class`, `owl:ObjectProperty`). - - ```python - from semantica.ontology import ClassInferrer, PropertyGenerator - - class_inferrer = ClassInferrer() - classes = class_inferrer.infer_classes(kg.entities) - - prop_generator = PropertyGenerator() - properties = prop_generator.infer_properties(kg.entities, kg.relationships, classes) - ``` - - - Builds taxonomy trees using transitive closure and cycle detection. - - ```python - hierarchy = generator.build_hierarchy(classes) - ``` - - - Serializes to Turtle format using `rdflib`. - - ```python - from semantica.ontology import OWLGenerator - - owl_gen = OWLGenerator() - ttl_str = owl_gen.generate_owl( - {"classes": classes, "properties": properties, "hierarchy": hierarchy}, - format="turtle", # "turtle" | "xml" | "json-ld" | "n3" - ) - ``` - - - Assesses coverage, completeness, and granularity metrics. - - ```python - from semantica.ontology import OntologyEvaluator - - evaluator = OntologyEvaluator() - report = evaluator.evaluate(ttl_str, kg) - print(f"Coverage: {report.coverage:.2%}") - print(f"Completeness: {report.completeness:.2%}") - print(f"Granularity: {report.granularity:.2%}") - ``` - - - -## Advanced Generation Tools - - - - Capture and manage ontology requirements before generation: - - ```python - from semantica.ontology import RequirementsSpecManager - - spec = RequirementsSpecManager() - spec.add_competency_question("What organizations are headquartered in California?") - spec.add_competency_question("Who founded each organization?") - spec.add_competency_question("What products does each organization sell?") - - spec.set_scope( - domain="Technology industry", - excluded_types=["Event", "Date"], - min_confidence=0.7, - ) - - generator = OntologyGenerator() - ontology = generator.generate_from_graph(kg, requirements=spec) - ``` - - - Generate ontologies directly from natural language: - - ```python - from semantica.ontology import LLMOntologyGenerator - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - generator = LLMOntologyGenerator(llm_provider=llm) - - ontology = generator.generate_from_description( - description="An ontology for tracking pharmaceutical clinical trials, including drugs, patients, dosages, outcomes, and adverse events.", - num_classes=20, - ) - - ontology = generator.generate_from_corpus( - documents=["clinical_trial_protocol.pdf"], - domain_hint="biomedical", - ) - - ontology = generator.refine( - ontology, - questions=["What dosage was given to each patient?", "What adverse events occurred?"], - ) - ``` - - - Integrate published ontologies instead of generating from scratch: - - ```python - from semantica.ontology import ReuseManager - - manager = ReuseManager() - manager.load_ontology("schema.org", source="https://schema.org/version/latest/schemaorg-current-https.ttl") - manager.load_ontology("foaf", source="http://xmlns.com/foaf/spec/index.rdf") - - candidates = manager.find_reusable_classes( - your_classes=["Person", "Organization", "Product"], - loaded_ontologies=["schema.org", "foaf"], - ) - for candidate in candidates: - print(f"{candidate.local_class} → reuse {candidate.external_uri} (similarity: {candidate.similarity:.2f})") - - merged = manager.merge_reused(your_ontology, candidates) - ``` - - - Pre-built domain ontologies for common verticals: - - ```python - from semantica.ontology import DomainOntologies - - catalog = DomainOntologies() - - for domain in catalog.list_domains(): - print(f"{domain.name}: {domain.description} ({domain.class_count} classes)") - - biomedical = catalog.load("biomedical") # SNOMED CT-aligned - finance = catalog.load("finance") # FinancialInstrument, Company, Market - legal = catalog.load("legal") # Contract, Party, Jurisdiction - supply_chain = catalog.load("supply_chain") # Supplier, Product, Shipment - - biomedical.add_class("ClinicalTrial", parent="Study", properties=["phase", "participants"]) - ``` - - Available domains: `biomedical`, `finance`, `legal`, `supply_chain`, `cybersecurity`, `e_commerce`, `hr`. - - - -## ModuleManager - -Build ontologies as composable modules — keep domain logic separated and reusable: +Generate SHACL shapes from an ontology and validate any graph against them: ```python -from semantica.ontology import ModuleManager +from semantica.ontology import SHACLGenerator, OntologyValidator, SHACLValidationReport, SHACLViolation -manager = ModuleManager() +# Generate shapes from ontology +generator = SHACLGenerator() +shapes = generator.generate(ontology) +shapes_ttl = shapes.serialize(format="turtle") -core_module = manager.create_module("core", base_uri="http://example.org/core#") -finance_module = manager.create_module("finance", base_uri="http://example.org/finance#") +# Validate a graph against the shapes +validator = OntologyValidator() +report: SHACLValidationReport = validator.validate(kg, shapes=shapes) -core_module.add_class("Entity", properties=["id", "name"]) -finance_module.add_class("Company", parent="Entity", properties=["ticker", "revenue"]) - -finance_module.import_module(core_module) -unified = manager.merge_modules([core_module, finance_module]) +if not report.conforms: + violation: SHACLViolation + for violation in report.violations: + print(f"{violation.severity}: {violation.message}") + print(f" Node: {violation.node}") + print(f" Path: {violation.path}") ``` -## NamespaceManager +## LLM-Powered Ontology Generation -Manage IRI prefixes and generate consistent URIs for all ontology terms: +For complex or novel domains where schema patterns are hard to infer statistically: + +```python +from semantica.ontology import LLMOntologyGenerator +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +generator = LLMOntologyGenerator(llm_provider=llm) +ontology = generator.generate( + domain_description="A biomedical ontology for clinical trial protocols", + examples=["Patient", "Trial", "Intervention", "Outcome"], +) +``` + +## OWL / RDF Export + +```python +from semantica.ontology import OWLGenerator + +generator = OWLGenerator() +generator.generate(ontology, path="ontology.ttl", format="turtle") +generator.generate(ontology, path="ontology.owl", format="xml") +generator.generate(ontology, path="ontology.json", format="json-ld") +``` + +## Namespace Management ```python from semantica.ontology import NamespaceManager -ns_manager = NamespaceManager(base_uri="http://example.org/") -ns_manager.register("ex", "http://example.org/") -ns_manager.register("schema", "https://schema.org/") +ns = NamespaceManager(base_uri="https://example.org/") +ns.register("ex", "https://example.org/") +ns.register("schema", "https://schema.org/") +ns.register("owl", "http://www.w3.org/2002/07/owl#") -class_iri = ns_manager.generate_iri("Person") # → "http://example.org/Person" -prop_iri = ns_manager.generate_iri("worksFor", prefix="ex") # → "http://example.org/worksFor" -iri = ns_manager.resolve("schema:Organization") # → "https://schema.org/Organization" +# Generate IRIs for classes and properties +class_iri = ns.generate_class_iri("Person") +property_iri = ns.generate_property_iri("worksFor") ``` -## OntologyEvaluator +## Ontology Evaluation -Measure quality across coverage, completeness, and competency questions: +Measure coverage, completeness, and granularity of a generated ontology: ```python from semantica.ontology import OntologyEvaluator evaluator = OntologyEvaluator() -report = evaluator.evaluate(ontology, kg) -print(f"Coverage: {report.coverage:.2%}") -print(f"Completeness: {report.completeness:.2%}") -print(f"Granularity: {report.granularity:.2%}") +result = evaluator.evaluate(ontology, kg) -questions = ["What organizations were founded in California?", "Who are the employees of Apple Inc.?"] -cq_results = evaluator.validate_competency_questions(ontology, kg, questions) -for q, result in zip(questions, cq_results): - print(f"Q: {q} → Answerable: {result.answerable} ({result.reason})") +print(f"Class coverage: {result.class_coverage:.2f}") +print(f"Property coverage: {result.property_coverage:.2f}") +print(f"Completeness: {result.completeness:.2f}") +print(f"Granularity: {result.granularity:.2f}") + +for gap in result.gaps: + print(f"Gap: {gap.description}") +``` + +## Ingest an Existing Ontology + +Load and parse an ontology file for downstream use: + +```python +from semantica.ontology import ingest_ontology + +ontology_data = ingest_ontology("schema.ttl") # Turtle +ontology_data = ingest_ontology("schema.owl") # OWL/XML +ontology_data = ingest_ontology("schema.jsonld") # JSON-LD ``` ## Ontology Hub (v0.5.0) -A visual browser UI for the full ontology lifecycle, served by the Explorer CLI: +A visual browser UI for the full ontology lifecycle. Launch via CLI: ```bash pip install "semantica[explorer]" -semantica explore +semantica-explorer --port 8080 # Navigate to http://localhost:8080 → Ontology Hub tab ``` -Features: visual editor, SHACL Studio, health dashboard, and version control. +Features: -## Tips and Common Pitfalls +- **Visual editor** — create and edit classes, properties, and relationships in the browser +- **SHACL Studio** — author and validate SHACL shapes with live feedback +- **Health dashboard** — coverage, completeness, and constraint violation metrics +- **Version control** — snapshot, diff, and restore ontology versions - - **Define competency questions before generating.** An ontology without competency questions has no measurable success criteria. Write 5–10 natural language questions your ontology must answer before calling `OntologyGenerator`. Then validate them with `OntologyEvaluator.validate_competency_questions()`. - - - - **Reuse before generating.** `DomainOntologies` and `ReuseManager` give you schema.org, FOAF, and domain-specific ontologies that took years to develop. Reusing established classes (`schema:Organization`, `foaf:Person`) also improves interoperability with external data. - - - - **`LLMOntologyGenerator` is great for prototyping, not production.** LLM-generated ontologies are a useful starting point but need expert review. Use `OntologyEvaluator` and manual SHACL authoring to harden the schema before relying on it for production graph validation. - - - - **Always validate after schema changes.** When you add new classes or properties, run `OntologyValidator.validate(ontology)` immediately. Validation issues often surface data quality problems that were silently passing before. - - - - **Use `ModuleManager` for large ontologies.** A monolithic 500-class ontology becomes unmanageable quickly. Split by domain (`core`, `finance`, `legal`) and use `owl:imports` to compose them — changes in one module don't break others. - - - - **Namespace your terms consistently.** All classes and properties need stable IRIs. Use `NamespaceManager` to generate them programmatically — never hardcode IRI strings in code, because base URIs change when projects move. - + + Ontology versioning (`VersionManager`, `OntologyVersion`) has moved to `semantica.change_management`. Import from there: `from semantica.change_management import VersionManager`. + diff --git a/docs/reference/parse.md b/docs/reference/parse.md index 7f8b84a5..6aa9830a 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -6,231 +6,103 @@ icon: "file-lines" `semantica.parse` extracts structured text, layout, tables, and metadata from unstructured documents. `DocumentParser` handles clean machine-readable files; `DoclingParser` handles complex layouts, scanned PDFs, and multi-column documents. +## Exported Classes + +```python +from semantica.parse import ( + DocumentParser, # auto-detect format — delegates to format-specific parser + PDFParser, # PDF text extraction + DOCXParser, # Word .docx documents + HTMLParser, # HTML / web pages + MarkdownParser, # Markdown files + TXTParser, # plain text + JSONParser, # JSON documents + XMLParser, # XML documents + CSVParser, # CSV / TSV files + WebParser, # URL fetch + HTML parsing + EmailParser, # .eml / .msg email files + CodeParser, # source code files + # Data types + ParsedDocument, # {text, sections, tables, metadata, source_id} + DocumentMetadata, # {title, author, created_date, page_count, language, ...} +) + +# Optional — requires: pip install "semantica[docling]" +from semantica.parse import DoclingParser # advanced OCR + layout analysis +``` + ## What You Get - - - Standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — zero config, no extras. - - - Advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR. - - - AST structure extraction — functions, classes, imports, dependencies — for 10+ languages. - - - EXIF metadata extraction and OCR via Tesseract for image files. - - - Technical metadata from audio, video, and image files (duration, codec, resolution). - - - Parse Model Context Protocol responses into structured `ParsedDocument` objects. - - +- **`DocumentParser`** — standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — auto-detects format +- **`DoclingParser`** — advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR (optional dep) +- **`ParsedDocument`** — structured output with `text`, `sections`, `tables`, and `metadata` +- **Format-specific parsers** — `PDFParser`, `DOCXParser`, `HTMLParser`, `WebParser`, `EmailParser`, `CodeParser`, etc. -## Quick Start +## DocumentParser - - - ```python - from semantica.parse import DocumentParser +Standard parser for clean, machine-readable documents: - parser = DocumentParser() - parsed = parser.parse("data/report.pdf") +```python +from semantica.parse import DocumentParser - print(parsed.text) # full clean text - print(parsed.metadata) # title, author, date, page_count, language, etc. - print(parsed.sections) # document structure as a list of Section objects - ``` - - - ```bash - pip install "semantica[docling]" - ``` +parser = DocumentParser() +parsed = parser.parse("data/report.pdf") - ```python - from semantica.parse import DoclingParser +print(parsed.text) # full clean text +print(parsed.metadata) # title, author, date, page_count, language, etc. +print(parsed.sections) # document structure as a list of Section objects +``` - parser = DoclingParser( - extract_tables=True, # structured table extraction with cell type detection - extract_images=True, # extract image regions for downstream OCR - output_format="markdown", # "markdown" | "html" | "json" - ) +Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. - parsed = parser.parse("data/annual_report.pdf") - print(parsed.tables) # structured TableData objects with headers and rows - ``` - - - ```python - from semantica.split import TextSplitter - from semantica.semantic_extract import NERExtractor - from semantica.llms import Groq - import os +## DoclingParser - splitter = TextSplitter(method="structural") - chunks = splitter.split_document(parsed) +Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - extractor = NERExtractor(method="llm", llm_provider=llm) - entities = extractor.extract_batch([c.text for c in chunks]) - ``` - - +```bash +pip install "semantica[docling]" +``` -## Parser Reference +```python +from semantica.parse import DoclingParser - - - Standard parser for clean, machine-readable documents — no extra dependencies required: +parser = DoclingParser( + extract_tables=True, # structured table extraction with cell type detection + extract_images=True, # extract image regions for downstream OCR + output_format="markdown", # "markdown" | "html" | "json" +) - ```python - from semantica.parse import DocumentParser +parsed = parser.parse("data/annual_report.pdf") - parser = DocumentParser() - parsed = parser.parse("data/report.pdf") +print(parsed.text) # full clean text +print(parsed.tables) # structured TableData objects with headers and rows +print(parsed.sections) # document structure with heading hierarchy +``` - print(parsed.text) # full clean text - print(parsed.metadata) # title, author, date, page_count, language, etc. - print(parsed.sections) # document structure as a list of Section objects - ``` +Use `DoclingParser` for: - **Supported formats:** PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. - - - Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: +- Multi-column PDF layouts +- Tables with merged cells or complex headers +- PPTX slides with embedded charts +- XLSX spreadsheets with formulas +- Scanned documents with OCR +- Academic papers and technical reports - ```python - from semantica.parse import DoclingParser +## OCR Support - parser = DoclingParser( - extract_tables=True, # structured table extraction with cell type detection - extract_images=True, # extract image regions for downstream OCR - output_format="markdown", # "markdown" | "html" | "json" - ) +```python +parser = DoclingParser( + ocr=True, + ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents + extract_tables=True, +) - parsed = parser.parse("data/annual_report.pdf") - print(parsed.tables) # structured TableData objects with headers and rows - print(parsed.sections) # document structure with heading hierarchy - ``` +parsed = parser.parse("data/scanned_contract.pdf") +``` - **Use `DoclingParser` for:** - - Multi-column PDF layouts - - Tables with merged cells or complex headers - - PPTX slides with embedded charts - - XLSX spreadsheets with formulas - - Scanned documents with OCR - - Academic papers and technical reports +## Parsed Document Object - **OCR support:** - - ```python - parser = DoclingParser( - ocr=True, - ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents - extract_tables=True, - ) - parsed = parser.parse("data/scanned_contract.pdf") - ``` - - - Parse source code files — extracts AST structure, functions, classes, imports, and comments: - - ```python - from semantica.parse import CodeParser - - parser = CodeParser( - extract_comments=True, # include docstrings and inline comments - extract_dependencies=True, # import/require statements - language="auto", # "auto" | "python" | "javascript" | "java" | "go" | "rust" | "cpp" - ) - - parsed = parser.parse("src/main.py") - - print(parsed.text) # raw source code as text - print(parsed.metadata["language"]) # detected language - print(parsed.metadata["functions"]) # list of function names - print(parsed.metadata["classes"]) # list of class names - print(parsed.metadata["imports"]) # list of import statements - print(parsed.metadata["comments"]) # docstrings and inline comments - ``` - - **Supported languages:** Python, JavaScript/TypeScript, Java, Go, Rust, C/C++, C#, Ruby, PHP, Swift. - - - Extract EXIF metadata and optionally perform OCR on image files: - - ```python - from semantica.parse import ImageParser - - parser = ImageParser( - extract_exif=True, # camera, GPS, timestamps, etc. - ocr=True, # OCR via Tesseract (requires tesseract-ocr installed) - ocr_language="en", # ISO 639-1 language code for OCR - ) - - parsed = parser.parse("photo.jpg") - - print(parsed.text) # OCR-extracted text (if ocr=True) - print(parsed.metadata["width"]) # image dimensions - print(parsed.metadata["height"]) - print(parsed.metadata["format"]) # "JPEG" | "PNG" | "TIFF" | ... - print(parsed.metadata["exif"]["GPS"]) # GPS coordinates if available - print(parsed.metadata["exif"]["DateTime"]) - ``` - - - ### MediaParser - - Extract technical metadata from audio, video, and image files: - - ```python - from semantica.parse import MediaParser - - parser = MediaParser() - - # Video file - parsed = parser.parse("interview.mp4") - print(parsed.metadata["duration_seconds"]) - print(parsed.metadata["codec"]) - print(parsed.metadata["resolution"]) - print(parsed.metadata["fps"]) - - # Audio file - parsed = parser.parse("podcast.mp3") - print(parsed.metadata["duration_seconds"]) - print(parsed.metadata["bitrate"]) - print(parsed.metadata["channels"]) - ``` - - **Supported formats:** MP4, AVI, MOV, MKV, MP3, WAV, FLAC, OGG, JPEG, PNG, TIFF, WebP. - - ### MCPParser - - Parse Model Context Protocol (MCP) responses into structured `ParsedDocument` objects: - - ```python - from semantica.parse import MCPParser - - parser = MCPParser() - - mcp_response = { - "content": [{"type": "text", "text": "Apple Inc. was founded in 1976..."}], - "metadata": {"tool": "web_search", "query": "Apple Inc history"} - } - - parsed = parser.parse(mcp_response) - print(parsed.text) # "Apple Inc. was founded in 1976..." - print(parsed.metadata) # tool name, query, and other MCP metadata - ``` - - - -## Parsed Document Schema - - - +Both parsers return a `ParsedDocument` with the same structure: ```python @dataclass @@ -240,12 +112,7 @@ class ParsedDocument: tables: List[TableData] # structured table data (DoclingParser only) metadata: DocumentMetadata # title, author, dates, page count source_id: str # links back to the original DataSource -``` - - - -```python @dataclass class DocumentMetadata: title: Optional[str] @@ -259,21 +126,6 @@ class DocumentMetadata: format: str # "pdf" | "docx" | "pptx" | ... ``` - - - -## Choosing a Parser - -| Scenario | Parser | -| -------- | ------ | -| Clean PDFs, DOCX, HTML, TXT, CSV, Excel | `DocumentParser` — zero config, no extras | -| Scanned PDFs, OCR required | `DoclingParser(ocr=True)` — requires `pip install "semantica[docling]"` | -| Multi-column PDFs, merged-cell tables | `DoclingParser(extract_tables=True)` | -| Source code files | `CodeParser(language="auto")` | -| Images with embedded text | `ImageParser(ocr=True)` — requires Tesseract | -| Audio/video metadata | `MediaParser()` | -| MCP tool responses | `MCPParser()` | - ## Integration with FileIngestor The most common pattern — ingest a directory then parse each source: @@ -295,28 +147,6 @@ for source in sources: Docling is an optional dependency. If `docling` is not installed, `DoclingParser` raises an `ImportError` with installation instructions. `DocumentParser` is always available and requires no extras. -## Tips and Common Pitfalls - - - **Start with `DocumentParser` and only switch to `DoclingParser` when needed.** `DoclingParser` is significantly more powerful but slower and requires an additional dependency. For clean machine-readable PDFs and Office files, `DocumentParser` is fast and accurate enough. - - - - **OCR requires Tesseract installed on the system.** `ImageParser(ocr=True)` and `DoclingParser(ocr=True)` both call Tesseract under the hood. Install it with `apt-get install tesseract-ocr` (Linux) or `brew install tesseract` (macOS) before enabling OCR. - - - - **`extract_tables=True` is off by default for speed.** Table extraction in `DoclingParser` requires additional layout analysis passes. Only enable it when you actually need structured table data — for text-only extraction, leave it off. - - - - **`CodeParser` outputs AST metadata, not just raw text.** The `parsed.metadata["functions"]` and `parsed.metadata["classes"]` lists are useful for building code-level knowledge graphs — function call graphs, class inheritance hierarchies, dependency graphs. - - - - **Always pass the `ParsedDocument` to `TextSplitter` before extraction.** Raw `parsed.text` is a flat string. Use `TextSplitter` to chunk it into semantically meaningful pieces before running NER — this dramatically reduces context window overflow on large documents. - - Load files before parsing. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index 8cc594c1..d83ddb3f 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -1,291 +1,168 @@ --- title: "Provenance Module" -description: "W3C PROV-O compliant lineage tracking, source attribution, and audit trails across all modules." +description: "W3C PROV-O compliant lineage tracking, source attribution, tamper-evident checksums, and audit trails across all modules." icon: "link" --- `semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, reasoning, and export. Compliant with W3C PROV-O, suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 environments. +## Exported Classes + +```python +from semantica.provenance import ( + ProvenanceManager, # track entities, get lineage, export PROV-O + ProvenanceEntry, # single provenance record (entity_id, source, method, ...) + SourceReference, # rich source pointer (DOI, page, quote, URL) + ProvenanceStorage, # abstract storage backend + InMemoryStorage, # default in-memory backend + SQLiteStorage, # persistent SQLite backend for production + compute_checksum, # compute tamper-evident hash for an entry + verify_checksum, # verify integrity of a stored entry +) + +# GraphBuilderWithProvenance is in semantica.kg, not semantica.provenance +from semantica.kg import GraphBuilderWithProvenance +``` + ## What You Get - - - Track entities, relationships, and activities with full source attribution and confidence scores. - - - Track entity and relationship lineage within a knowledge graph. - - - Full directed lineage from any entity back to its originating source document. - - - Serialize lineage as Turtle RDF or JSON-LD for compliance reporting. - - - Drop-in replacement for GraphBuilder that auto-tracks every node and edge. - - - SHA-256 checksums to detect tampering in HIPAA and FDA 21 CFR Part 11 environments. - - - -## Quick Start - - - - ```python - from semantica.provenance import ProvenanceManager, SQLiteStorage - - # SQLite — persistent across process restarts (recommended for production) - manager = ProvenanceManager( - storage=SQLiteStorage(db_path="provenance.db") - ) - ``` - - - ```python - manager.track_entity( - entity_id="apple_inc", - source="annual_report_2023.pdf", - entity_type="Organization", - extraction_method="llm", - confidence=0.98, - ) - - manager.track_relationship( - rel_id="steve_jobs_founded_apple", - source="annual_report_2023.pdf", - extraction_method="llm", - confidence=0.92, - ) - ``` - - - ```python - lineage = manager.get_lineage("apple_inc") - print(f"Source: {lineage.source}") - print(f"Extracted: {lineage.extracted_at}") - print(f"Method: {lineage.extraction_method}") - print(f"Confidence: {lineage.confidence}") - ``` - - - ```python - # Full provenance graph for all tracked entities - manager.export_all(path="provenance.ttl", format="turtle") - manager.export_all(path="provenance.jsonld", format="json-ld") - ``` - - +- **`ProvenanceManager`** — track entities and relationships with source attribution and lineage retrieval +- **`ProvenanceEntry`** / **`SourceReference`** — structured records with DOI, page, quote, confidence, and timestamp +- **`InMemoryStorage`** / **`SQLiteStorage`** — swappable persistence backends +- **`compute_checksum`** / **`verify_checksum`** — tamper-evident integrity verification +- **Lineage graph** — full upstream lineage from any entity back to its source document +- **W3C PROV-O export** — serialize lineage as Turtle RDF or JSON-LD for compliance reporting +- **`GraphBuilderWithProvenance`** (in `semantica.kg`) — drop-in replacement that auto-tracks every node and edge ## ProvenanceManager ```python -from semantica.provenance import ProvenanceManager +from semantica.provenance import ProvenanceManager, InMemoryStorage, SQLiteStorage -manager = ProvenanceManager() +# In-memory (default) — fast, not persisted across restarts +manager = ProvenanceManager(storage=InMemoryStorage()) -# Track an extracted entity +# SQLite — persisted, production-ready +manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) + +# Track an extracted entity (with rich source reference) manager.track_entity( entity_id="apple_inc", source="annual_report_2023.pdf", - entity_type="Organization", - extraction_method="llm", + source_location="Page 12, Section 3.1", + source_quote="Apple Inc. was incorporated on January 3, 1977.", confidence=0.98, ) # Track an extracted relationship -manager.track_relationship( - rel_id="steve_jobs_founded_apple", +manager.track_entity( + entity_id="steve_jobs_founded_apple", source="annual_report_2023.pdf", - extraction_method="llm", confidence=0.92, ) # Retrieve full lineage for any entity -lineage = manager.get_lineage("apple_inc") -print(f"Source: {lineage.source}") -print(f"Extracted: {lineage.extracted_at}") -print(f"Method: {lineage.extraction_method}") -print(f"Confidence: {lineage.confidence}") +entry = manager.get_lineage("apple_inc") +print(f"Source: {entry.source}") +print(f"Quote: {entry.source_quote}") +print(f"Confidence: {entry.confidence}") +print(f"Tracked at: {entry.tracked_at}") ``` -## Activity Tracking +## SourceReference -Record pipeline activities — what was consumed and what was produced: +`SourceReference` provides a rich, citable pointer to the exact location in a source document: ```python -# Start and end an activity -activity_id = manager.start_activity( - activity_type="ner_extraction", - used=["annual_report_2023.pdf"], - generated=["apple_inc", "steve_jobs"], +from semantica.provenance import SourceReference + +ref = SourceReference( + document_id="annual_report_2023.pdf", + page=12, + section="3.1", + quote="Apple Inc. was incorporated on January 3, 1977.", + url="https://investor.apple.com/sec-filings/annual-reports/", + doi="10.0000/example.doi", ) -manager.end_activity(activity_id) - -# Query activities for an entity -activities = manager.get_activities(entity_id="apple_inc") -for activity in activities: - print(f"{activity.type} at {activity.started_at}") - print(f" Used: {activity.used}") - print(f" Generated: {activity.generated}") +manager.track_entity( + entity_id="apple_inc", + source_reference=ref, + confidence=0.98, +) ``` -## Lineage Graph +## Tamper-Evident Checksums -Retrieve a full directed lineage graph from any entity back to its source: - -```python -lineage_graph = manager.get_lineage_graph("apple_inc") - -for node in lineage_graph.nodes: - print(f"{node.id}: {node.type} — {node.timestamp}") - -for edge in lineage_graph.edges: - print(f"{edge.source} → {edge.target} ({edge.relation})") -``` - -## Storage Backends - - - - Fast, no persistence — default backend. Data is lost on process exit. - - ```python - from semantica.provenance import ProvenanceManager, InMemoryStorage - - manager = ProvenanceManager(storage=InMemoryStorage()) - - manager.track_entity("apple_inc", source="report.pdf", confidence=0.98) - lineage = manager.get_lineage("apple_inc") - ``` - - Best for: development, unit tests, short-lived pipelines. - - - Persistent file-based storage — survives process restarts. The API is identical to `InMemoryStorage`. - - ```python - from semantica.provenance import ProvenanceManager, SQLiteStorage - - manager = ProvenanceManager( - storage=SQLiteStorage(db_path="provenance.db") - ) - - manager.track_entity("apple_inc", source="report.pdf", confidence=0.98) - lineage = manager.get_lineage("apple_inc") - ``` - - Best for: production single-machine deployments, compliance environments. - - - - | Storage | Persistence | Best For | - | ------- | ----------- | -------- | - | `InMemoryStorage` | No | Development, unit tests, short-lived pipelines | - | `SQLiteStorage` | Yes (file) | Production single-machine deployments | - - - - -## Integration with GraphBuilder - -`GraphBuilderWithProvenance` automatically records provenance for every node and edge constructed — no manual `track_entity()` calls needed: - -```python -from semantica.kg import GraphBuilderWithProvenance - -builder = GraphBuilderWithProvenance(provenance=True) -result = builder.build_single_source(graph_data) - -# Every node and edge has a source_id linking back to the originating document -lineage = result.provenance_manager.get_lineage("apple_inc") -print(f"Source document: {lineage.source}") -print(f"Extracted by: {lineage.extraction_method}") -``` - -## Integrity Verification - -Compute and verify checksums for provenance entries to detect tampering: +Verify that provenance records have not been modified after creation: ```python from semantica.provenance import compute_checksum, verify_checksum -# Compute a SHA-256 checksum over an entity's provenance record -entry = manager.get_provenance_entry("apple_inc") -checksum = compute_checksum(entry, algorithm="sha256") -print(f"Checksum: {checksum}") +entry = manager.get_lineage("apple_inc") -# Later — verify the record has not been modified -is_valid = verify_checksum(entry, expected_checksum=checksum, algorithm="sha256") +# Compute and store a checksum on first write +checksum = compute_checksum(entry) + +# Later: verify the entry hasn’t been altered +is_valid = verify_checksum(entry, checksum) if not is_valid: raise RuntimeError("Provenance record has been tampered with!") ``` -Supported algorithms: `"sha256"` (default), `"sha512"`, `"md5"`. - ## W3C PROV-O Export +Export lineage as W3C PROV-O Turtle for compliance reporting: + ```python # Single entity lineage prov_ttl = manager.export_prov_o("apple_inc", format="turtle") # Full provenance graph for all tracked entities -manager.export_all(path="provenance.ttl", format="turtle") +manager.export_all(path="provenance.ttl", format="turtle") + +# Compliance-ready JSON-LD export manager.export_all(path="provenance.jsonld", format="json-ld") ``` -## Schemas +## Integration with GraphBuilder - - +`GraphBuilderWithProvenance` (from `semantica.kg`) automatically records provenance for every node and edge constructed: ```python -@dataclass -class ProvenanceEntry: - entity_id: str - source: str # source document or system - source_location: str # e.g. "page 3, paragraph 2" - source_quote: str # verbatim text from source - extraction_method: str # "llm" | "ml" | "pattern" - confidence: float # extraction confidence 0–1 - timestamp: datetime # when this entry was recorded - entity_type: Optional[str] +from semantica.kg import GraphBuilderWithProvenance +from semantica.provenance import ProvenanceManager, SQLiteStorage + +prov_manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) +builder = GraphBuilderWithProvenance(provenance_manager=prov_manager) +kg = builder.build_single_source(graph_data) + +# Every node and edge now has full source attribution +entry = prov_manager.get_lineage("apple_inc") +print(f"Source document: {entry.source}") +print(f"Confidence: {entry.confidence}") ``` - - +## Enable Provenance in Extractors ```python -@dataclass -class SourceReference: - source_id: str - doi: Optional[str] # academic DOI if available - page: Optional[int] # page number in document - paragraph: Optional[int] - quote: str # verbatim text supporting the fact - url: Optional[str] - accessed_at: Optional[datetime] +from semantica.semantic_extract import NERExtractor +from semantica.provenance import ProvenanceManager + +prov_manager = ProvenanceManager() + +ner = NERExtractor(method="llm", llm_provider=llm, provenance=True) +entities = ner.extract(text) + +# Retrieve lineage for the first extracted entity +entry = prov_manager.get_lineage(entities[0]["id"]) +print(f"Source: {entry.source}") ``` - - - -## W3C PROV-O Mapping - -| Semantica Concept | PROV-O Class / Property | -| ----------------- | ----------------------- | -| Entity (node/fact) | `prov:Entity` | -| Extraction activity | `prov:Activity` | -| Source document | `prov:Entity` | -| `track_entity()` | `prov:wasDerivedFrom` | -| `start_activity()` | `prov:wasGeneratedBy` | -| Extraction method | `prov:wasAssociatedWith` | -| Timestamp | `prov:startedAtTime`, `prov:endedAtTime` | - ## Compliance Standards +Provenance tracking in Semantica is designed to satisfy: + | Standard | Requirement Met | | -------- | --------------- | | **W3C PROV-O** | Full PROV-O compliant serialization (Turtle and JSON-LD) | @@ -294,32 +171,6 @@ class SourceReference: | **GDPR** | Data lineage supporting right-to-erasure impact analysis | | **FDA 21 CFR Part 11** | Electronic records with origination timestamp and extraction method | -## Tips and Common Pitfalls - - - **Use `SQLiteStorage` in production, not `InMemoryStorage`.** The in-memory backend is the default for backwards compatibility, but provenance data is lost on process exit. Switch to `SQLiteStorage(db_path="provenance.db")` before going to production — migrating later means losing all historical lineage. - - - - **Track provenance at ingestion time, not after.** The `source_location` and `source_quote` fields become unavailable once you've moved past the parsing stage. Capture them during ingestion and pass them to `track_entity()` immediately. - - - - **Use `GraphBuilderWithProvenance` instead of plain `GraphBuilder`.** It auto-tracks every node and edge without manual `track_entity()` calls — ensuring nothing is accidentally omitted from the provenance record. - - - - **Verify checksums on high-stakes data.** `compute_checksum()` + `verify_checksum()` detects any modification to a provenance entry since it was recorded — critical for HIPAA and FDA 21 CFR Part 11 environments where tampered records carry legal liability. - - - - **Export PROV-O Turtle for external auditors.** Compliance teams and external auditors often need machine-readable lineage in a standard format. `manager.export_all("provenance.ttl", format="turtle")` produces W3C PROV-O Turtle that any RDF tool can parse. - - - - **Use `get_audit_trail(entity_id=...)` for GDPR subject-access requests.** The GDPR right-of-access requires you to show what personal data you hold and where it came from. Scoped lineage per entity ID makes this a one-line export. - - Version control and snapshot audit trails. diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 63af28e2..203c9f46 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -6,402 +6,283 @@ icon: "microchip" `semantica.reasoning` derives new knowledge from existing facts using logical rules. Every engine produces **explainable inference paths** — traceable chains of rules and facts, not black-box conclusions. -## Why Reasoning? +## Exported Classes -Knowledge graphs encode what you know explicitly. Reasoning lets you derive what must logically follow — without manually asserting every implication: - -- If A `located_in` B and B `located_in` C, then A `located_in` C — without storing that triple -- If Alice `parent_of` Bob and Bob `parent_of` Charlie, then Alice `ancestor_of` Charlie -- If a drug is contraindicated for a condition class, it's also contraindicated for all subclasses — inferred from the ontology hierarchy -- If an employee's CEO tenure ended in 2020, they cannot have signed contracts as CEO in 2021 — caught by temporal reasoning - -Reasoning turns sparse explicit knowledge into a dense, coherent, contradiction-free knowledge base. +```python +from semantica.reasoning import ( + # Engines + Reasoner, # IF/THEN forward-chaining facade + GraphReasoner, # inference over full KG structure + ReteEngine, # high-performance Rete pattern matching + SPARQLReasoner, # SPARQL-based RDF inference + DatalogReasoner, # recursive Horn clause fixpoint evaluation + TemporalReasoningEngine, # Allen interval algebra (13 relations) + ExplanationGenerator, # structured step-by-step explanations + # Data types + Rule, # IF/THEN rule definition + Fact, # base fact (subject, predicate, obj) + RuleType, # enum: FORWARD_CHAIN, BACKWARD_CHAIN, ... + InferenceResult, # result of infer() — contains derived_facts list + DatalogFact, # Datalog base fact (predicate, args tuple) + DatalogRule, # Datalog Horn clause ("head :- body.") + TemporalInterval, # time interval with start/end + IntervalRelation, # enum of 13 Allen relations + # Explanation types + Explanation, # conclusion + confidence + reasoning_path + ReasoningPath, # ordered list of ReasoningSteps + ReasoningStep, # single step: fact + rule_name + depth + Justification, # full justification record +) +``` ## What You Get - - - Main facade — IF/THEN forward-chaining with variable substitution and rule templates. - - - Inference over full knowledge graph structure: transitivity, symmetry, inverses. - - - High-performance pattern matching via the Rete algorithm for large rule sets. - - - Query expansion and property chain inference over RDF graphs. - - - Recursive Horn clause rules with guaranteed fixpoint termination (v0.4.0). - - - All 13 Allen interval algebra relations for time-aware inference. - - +- **`Reasoner`** — main facade for IF/THEN forward-chaining with variable substitution +- **`GraphReasoner`** — inference over full knowledge graph structure (transitivity, symmetry, inverses) +- **`ReteEngine`** — high-performance pattern matching via the Rete algorithm for large rule sets +- **`SPARQLReasoner`** — query expansion and property chain inference over RDF graphs +- **`DatalogReasoner`** — recursive Horn clause rules with guaranteed fixpoint termination (v0.4.0) +- **`TemporalReasoningEngine`** — all 13 Allen interval algebra relations for time-aware inference +- **`ExplanationGenerator`** — structured explanation paths for every derived conclusion + +## Quick Start + +The most common pattern: add facts + rules, run inference, explain a conclusion: + +```python +from semantica.reasoning import Reasoner, Rule, Fact, RuleType, InferenceResult + +reasoner = Reasoner() + +reasoner.add_fact(Fact(subject="Alice", predicate="is_a", obj="Manager")) +reasoner.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[{"subject": "?x", "predicate": "is_a", "object": "Manager"}], + conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"}, +)) + +result: InferenceResult = reasoner.infer() +for fact in result.derived_facts: + print(f"{fact.subject} {fact.predicate} {fact.obj}") + print(f" via: {fact.explanation}") +``` Forward chaining inference: known facts + IF/THEN rules produce derived facts with a full traceable explanation path -## Choosing a Reasoning Engine +## Reasoner (Main Facade) -| Engine | When to Use | -| ------ | ----------- | -| `Reasoner` | Simple IF/THEN rules, transitivity/symmetry templates, one-shot inference | -| `GraphReasoner` | Rules that operate on graph structure (paths, neighborhoods, multi-hop) | -| `ReteEngine` | Large rule sets (100+), rules fire repeatedly, performance is critical | -| `SPARQLReasoner` | Already using RDF/Turtle, need property chains, SPARQL ecosystem tools | -| `DatalogReasoner` | Recursive rules (ancestry, reachability), guaranteed termination required | -| `TemporalReasoningEngine` | Time-aware facts, interval relationships, historical validity | +The unified entry point for rule-based forward-chaining inference: -## Engines +```python +from semantica.reasoning import Reasoner, Rule, Fact, RuleType - - - The unified entry point for rule-based forward-chaining inference. Start here for most use cases. +reasoner = Reasoner() - ```python - from semantica.reasoning import Reasoner, Rule, Fact, RuleType +# Add base facts +reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Manager")) +reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Employee")) - reasoner = Reasoner() +# Add an IF/THEN rule +reasoner.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[ + {"subject": "?x", "predicate": "is_a", "object": "Manager"} + ], + conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} +)) - # Add base facts - reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Manager")) - reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Employee")) +# Run inference +result = reasoner.infer() +for inference in result.derived_facts: + print(f"{inference.subject} {inference.predicate} {inference.obj}") + print(f" Derived via: {inference.explanation}") +``` - # Add an IF/THEN rule - reasoner.add_rule(Rule( - rule_type=RuleType.FORWARD_CHAIN, - conditions=[ - {"subject": "?x", "predicate": "is_a", "object": "Manager"} - ], - conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} - )) +### Built-In Rule Templates - # Run inference — always call explicitly after adding facts/rules - result = reasoner.forward_chain() - for inference in result.derived_facts: - print(f"{inference.subject} {inference.predicate} {inference.obj}") - print(f" Derived via: {inference.explanation}") - ``` +```python +engine = Reasoner() - - Always call `reasoner.forward_chain()` after adding facts and rules. Adding them updates internal state but does **not** trigger inference automatically. - - +# Transitive closure: A→B, B→C ⟹ A→C +engine.apply_transitivity("located_in") - - Inference over the full knowledge graph structure — rules that operate on graph paths, neighborhoods, and multi-hop connections. +# Symmetry: A knows B ⟹ B knows A +engine.apply_symmetry("knows") - ```python - from semantica.reasoning import GraphReasoner +# Inverse: A parent_of B ⟹ B child_of A +engine.apply_inverse("parent_of", "child_of") +``` - graph_reasoner = GraphReasoner() +## GraphReasoner - # Define a transitive ancestor rule - graph_reasoner.add_rule({ - "if": [ - {"subject": "?a", "predicate": "parent_of", "object": "?b"}, - {"subject": "?b", "predicate": "parent_of", "object": "?c"} - ], - "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} - }) +Inference over the full knowledge graph structure: - inferences = graph_reasoner.reason(graph, query="") - for inf in inferences: - print(f"{inf['subject']} {inf['predicate']} {inf['object']}") - ``` - +```python +from semantica.reasoning import GraphReasoner - - High-performance pattern matching using the Rete algorithm — far faster than naive forward chaining for large rule sets because it caches partial matches across iterations. +graph_reasoner = GraphReasoner(kg) - ```python - from semantica.reasoning import ReteEngine +# Define a transitive ancestor rule +graph_reasoner.add_rule({ + "if": [ + {"subject": "?a", "predicate": "parent_of", "object": "?b"}, + {"subject": "?b", "predicate": "parent_of", "object": "?c"} + ], + "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} +}) - engine = ReteEngine() - engine.load_rules("rules/domain_rules.json") - results = engine.run(kg) +inferences = graph_reasoner.infer(kg) +for inf in inferences: + print(f"{inf['subject']} {inf['predicate']} {inf['object']}") +``` - # Inspect the Rete network - root = engine.get_root() - alpha_nodes = engine.get_alpha_nodes() # single-condition filters - beta_nodes = engine.get_beta_nodes() # join nodes - ``` +## ReteEngine - Rule file format (JSON): +High-performance pattern matching using the Rete algorithm — far faster than naive forward chaining for large rule sets because it caches partial matches across iterations: - ```json +```python +from semantica.reasoning import ReteEngine + +engine = ReteEngine() +engine.load_rules("rules/domain_rules.json") +results = engine.run(kg) + +# Inspect the Rete network +root = engine.get_root() +alpha_nodes = engine.get_alpha_nodes() # single-condition filters +beta_nodes = engine.get_beta_nodes() # join nodes +``` + +Rule format (JSON): + +```json +{ + "rules": [ { - "rules": [ - { - "name": "manager_authority", - "conditions": [ - { "subject": "?x", "predicate": "role", "object": "Manager" }, - { "subject": "?x", "predicate": "dept", "object": "?dept" } - ], - "action": { - "subject": "?x", - "predicate": "has_authority_over", - "object": "?dept" - }, - "priority": 10 - } - ] + "name": "manager_authority", + "conditions": [ + { "subject": "?x", "predicate": "role", "object": "Manager" } + ], + "action": { "subject": "?x", "predicate": "has_authority", "object": "true" } } - ``` + ] +} +``` - | Field | Type | Description | - | ----- | ---- | ----------- | - | `name` | `str` | Unique rule identifier — appears in `ExplanationGenerator` output | - | `conditions` | `List[Dict]` | Pattern to match — use `?variable` for wildcards | - | `action` | `Dict` | Fact to derive when all conditions match | - | `priority` | `int` | Higher priority rules fire first | +## SPARQLReasoner - - Use `ReteEngine` when you have more than ~20 rules or when rules can fire repeatedly. `Reasoner` re-evaluates all rules from scratch each cycle; `ReteEngine` caches partial matches and is orders of magnitude faster. - - +Query-based inference over RDF graphs with property chain support: - - Query-based inference over RDF graphs with property chain support. Use this when you're already in the RDF/Turtle ecosystem. +```python +from semantica.reasoning import SPARQLReasoner - ```python - from semantica.reasoning import SPARQLReasoner +reasoner = SPARQLReasoner(graph=rdf_graph) - reasoner = SPARQLReasoner(graph=rdf_graph) +result = reasoner.query(""" + PREFIX ex: + SELECT ?person ?company WHERE { + ?person ex:founded ?company . + ?company ex:located_in ex:SiliconValley . + } +""") - result = reasoner.query(""" - PREFIX ex: - SELECT ?person ?company WHERE { - ?person ex:founded ?company . - ?company ex:located_in ex:SiliconValley . - } - """) +for row in result.bindings: + print(row["person"], row["company"]) - for row in result.bindings: - print(row["person"], row["company"]) +# Property chain inference: A knows B, B colleague_of C ⟹ A knows C +reasoner.add_property_chain("knows", ["knows", "colleague_of"]) +inferences = reasoner.infer_property_chains() +``` - # Property chain inference: A knows B, B colleague_of C ⟹ A knows C - reasoner.add_property_chain("knows", ["knows", "colleague_of"]) - inferences = reasoner.infer_property_chains() - ``` - +## DatalogReasoner (v0.4.0) - - Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops. +Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops: - - Added in **v0.4.0**. Use `DatalogReasoner` whenever your rules can create cycles — it's the only engine with a termination guarantee. - +```python +from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule - ```python - from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule +datalog = DatalogReasoner() - datalog = DatalogReasoner() +# Base facts +datalog.add_fact(DatalogFact("parent", ("alice", "bob"))) +datalog.add_fact(DatalogFact("parent", ("bob", "charlie"))) - # Base facts - datalog.add_fact(DatalogFact("parent", ("alice", "bob"))) - datalog.add_fact(DatalogFact("parent", ("bob", "charlie"))) +# Recursive rules (Horn clauses) +datalog.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) +datalog.add_rule(DatalogRule("ancestor(?X, ?Z) :- parent(?X, ?Y), ancestor(?Y, ?Z).")) - # Recursive rules (Horn clauses) - datalog.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) - datalog.add_rule(DatalogRule("ancestor(?X, ?Z) :- parent(?X, ?Y), ancestor(?Y, ?Z).")) +# Evaluate to fixpoint +datalog.evaluate() - # Evaluate to fixpoint - datalog.evaluate() +# Query +results = datalog.query("ancestor(alice, ?Z)") +# → [{"Z": "bob"}, {"Z": "charlie"}] +``` - # Query - results = datalog.query("ancestor(alice, ?Z)") - # → [{"Z": "bob"}, {"Z": "charlie"}] - ``` +## TemporalReasoningEngine - - `Reasoner` has **no cycle detection** — rules that create cycles (A derives B, B derives C, C re-derives A) will loop infinitely. Use `DatalogReasoner` whenever recursive rules are involved. - - +Reason about time intervals using all 13 Allen interval algebra relations: - - Reason about time intervals using all 13 Allen interval algebra relations. +```python +from semantica.reasoning import TemporalReasoningEngine, TemporalInterval, IntervalRelation - ```python - from semantica.reasoning import TemporalReasoningEngine, TemporalInterval, IntervalRelation +engine = TemporalReasoningEngine() - engine = TemporalReasoningEngine() +ceo_tenure = TemporalInterval(start="1997-09-16", end="2011-08-24") +board_member = TemporalInterval(start="2000-01-01", end="2012-06-01") - ceo_tenure = TemporalInterval(start="1997-09-16", end="2011-08-24") - board_member = TemporalInterval(start="2000-01-01", end="2012-06-01") +relation = engine.get_relation(ceo_tenure, board_member) +# → IntervalRelation.DURING (ceo_tenure is fully inside board_member) +``` - relation = engine.get_relation(ceo_tenure, board_member) - # → IntervalRelation.DURING (ceo_tenure is fully inside board_member) - ``` +All 13 Allen interval algebra relations are supported: - All 13 Allen interval algebra relations: - - | Relation | Meaning | - | -------- | ------- | - | `BEFORE` | A ends before B starts | - | `MEETS` | A ends exactly when B starts | - | `OVERLAPS` | A starts before B, ends inside B | - | `DURING` | A is fully inside B | - | `STARTS` | A and B start together, A ends first | - | `FINISHES` | A and B end together, A starts later | - | `EQUALS` | Identical intervals | - | + 6 inverses | `AFTER`, `MET_BY`, `OVERLAPPED_BY`, `CONTAINS`, `STARTED_BY`, `FINISHED_BY` | - - +| Relation | Meaning | +| -------- | ------- | +| `BEFORE` | A ends before B starts | +| `MEETS` | A ends exactly when B starts | +| `OVERLAPS` | A starts before B, ends inside B | +| `DURING` | A is fully inside B | +| `STARTS` | A and B start together, A ends first | +| `FINISHES` | A and B end together, A starts later | +| `EQUALS` | Identical intervals | +| + 6 inverses | `AFTER`, `MET_BY`, `OVERLAPPED_BY`, `CONTAINS`, `STARTED_BY`, `FINISHED_BY` | ## ExplanationGenerator Generate structured step-by-step explanations for any derived conclusion: ```python -from semantica.reasoning import ExplanationGenerator +from semantica.reasoning import ExplanationGenerator, Explanation, ReasoningStep generator = ExplanationGenerator(reasoner) -explanation = generator.explain( +explanation: Explanation = generator.explain( conclusion={"subject": "John", "predicate": "has_authority", "object": "true"} ) -print(explanation.conclusion) +print(f"Conclusion: {explanation.conclusion}") print(f"Confidence: {explanation.confidence:.2f}") -print(explanation.justification.summary) +step: ReasoningStep for step in explanation.reasoning_path.steps: - indent = " " * step.depth - print(f"{indent}Step {step.depth}: {step.fact}") - print(f"{indent} via rule: '{step.rule_name}'") - print(f"{indent} premises: {step.premises}") + print(f" Step {step.depth}: {step.fact}") + print(f" via rule: '{step.rule_name}'") ``` - - Name every rule with a descriptive string. `ExplanationGenerator` includes the rule name in each derivation step — unnamed rules produce useless explanations like "rule_0 fired." Use names like `"manager_authority"` or `"transitive_location"`. - +## Choosing an Engine - - - -```python -@dataclass -class Explanation: - conclusion: Dict[str, str] # the fact being explained - confidence: float # aggregated rule confidence - reasoning_path: ReasoningPath # full derivation trace - justification: Justification # plain-language summary -``` - - - - -```python -@dataclass -class ReasoningPath: - steps: List[ReasoningStep] # ordered derivation steps - -@dataclass -class ReasoningStep: - depth: int # 0 = base fact, n = nth inference - fact: Dict[str, str] # the fact derived at this step - rule_name: str # name of the rule that fired - premises: List[Dict] # facts that triggered this rule - confidence: float # confidence at this step -``` - - - - -```python -@dataclass -class Justification: - summary: str # one-sentence natural language explanation - evidence: List[str] # list of supporting source facts -``` - - - - -## Combining Multiple Reasoning Engines - -Different engines cover different expressivity levels — compose them for richer inference: - - - - ```python - from semantica.reasoning import Reasoner, Rule, Fact, RuleType - - engine = Reasoner() - engine.add_rule(Rule( - rule_type=RuleType.FORWARD_CHAIN, - conditions=[ - {"subject": "?x", "predicate": "located_in", "object": "?y"}, - {"subject": "?y", "predicate": "located_in", "object": "?z"} - ], - conclusion={"subject": "?x", "predicate": "located_in", "object": "?z"} - )) - structural_result = engine.forward_chain() - ``` - - - ```python - from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule - - datalog = DatalogReasoner() - for fact in structural_result.derived_facts: - datalog.add_fact(DatalogFact(fact.predicate, (fact.subject, fact.obj))) - - datalog.add_rule(DatalogRule("reachable(?X, ?Z) :- located_in(?X, ?Y), reachable(?Y, ?Z).")) - datalog.evaluate() - ``` - - - ```python - from semantica.reasoning import TemporalReasoningEngine - from datetime import datetime - - temporal = TemporalReasoningEngine() - active_facts = [ - f for f in datalog.query("reachable(?X, ?Z)") - if temporal.is_active(f, at=datetime(2024, 1, 1)) - ] - ``` - - - ```python - from semantica.reasoning import ExplanationGenerator - - generator = ExplanationGenerator(engine) - explanation = generator.explain( - {"subject": "london_office", "predicate": "located_in", "object": "UK"} - ) - print(explanation.summary) - ``` - - - -## Tips and Common Pitfalls - - - **Always call `reasoner.forward_chain()` after adding facts and rules.** Adding facts and rules updates internal state but doesn't trigger inference automatically. Inference is a separate, explicit step. - +| Engine | Best For | Termination | Complexity | +| ------ | -------- | ----------- | ---------- | +| `Reasoner` | Simple IF/THEN rules, templates | Always | Low | +| `GraphReasoner` | KG-wide structural inference | Always | Medium | +| `ReteEngine` | Large rule sets (100+ rules) | Always | Low per-match | +| `SPARQLReasoner` | RDF graphs with SPARQL endpoint | Always | Low | +| `DatalogReasoner` | Recursive rules (ancestry, reachability) | Guaranteed fixpoint | Medium | +| `TemporalReasoningEngine` | Time interval relationships | Always | Low | - **Use `ReteEngine` for large rule sets.** If you have more than ~20 rules and rules can fire repeatedly, `Reasoner` re-evaluates all rules from scratch on each cycle. `ReteEngine` caches partial matches and is orders of magnitude faster for complex rule sets. - - - - **`DatalogReasoner` guarantees termination; `Reasoner` does not.** If your rules can create cycles (A derives B, B derives C, C re-derives A), `DatalogReasoner`'s semi-naive fixpoint evaluation will stop when no new facts are added. `Reasoner` has no cycle detection and may loop infinitely. - - - - **Name every rule for readable explanations.** `ExplanationGenerator.explain()` includes the rule name in each derivation step. Unnamed or generic rule names produce useless explanations like "rule_0 fired." Use descriptive names: `"manager_authority"`, `"transitive_location"`. - - - - **Use rule `priority` to control inference order.** When multiple rules could fire on the same facts, higher-priority rules fire first. This matters when a higher-priority rule produces a fact that gates a lower-priority rule's conditions. - - - - **Combine engines for maximum expressivity.** Forward-chain structural rules with `Reasoner`, then pass derived facts to `DatalogReasoner` for recursive closure, then filter by time with `TemporalReasoningEngine`. Each engine covers a different expressivity class — they compose cleanly. + For recursive rules (e.g. ancestor, reachability, transitivity), always use `DatalogReasoner` — it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner` does not handle recursion. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 01cc90c5..c88398f3 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -6,162 +6,76 @@ icon: "magnifying-glass-chart" `semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica. All extractors support three modes: pattern-based (no API key), ML-based, and LLM-based. -## Quick Start +## Exported Classes - - - ```python - from semantica.semantic_extract import CoreferenceResolver +```python +from semantica.semantic_extract import ( + # Primary extractors + NamedEntityRecognizer, # full NER coordinator (confidence_threshold, merge_overlapping) + NERExtractor, # core NER implementation used by NamedEntityRecognizer + RelationExtractor, # typed relationship extraction + TripletExtractor, # (subject, predicate, object) triplet generation + EventDetector, # event detection with participants and temporal context + CoreferenceResolver, # resolve pronouns and aliases to canonical entities + # Data types + Entity, # {id, text, type, confidence, start, end} + Relation, # {subject, predicate, object, confidence} + Event, # {type, participants, temporal, location, confidence} + CoreferenceChain, # list of mentions resolving to the same entity + # Advanced + EntityClassifier, # classify entity candidates by type + CustomEntityDetector, # pattern/dictionary-based custom entity detection + TemporalEventProcessor, # extract temporal information from events +) +``` - resolver = CoreferenceResolver() - resolved_text = resolver.resolve( - "Apple Inc. was founded in 1976. The company is headquartered in Cupertino." - ) - # "Apple Inc." replaces "The company" — consistent downstream extraction - ``` - - - ```python - from semantica.semantic_extract import NERExtractor - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) - entities = ner.extract(resolved_text) - # → [{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98, ...}] - ``` - - - ```python - from semantica.semantic_extract import RelationExtractor - - rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) - relationships = rel.extract(resolved_text, entities=entities) - # → [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", ...}] - ``` - - - ```python - from semantica.semantic_extract import ExtractionValidator - - validator = ExtractionValidator(min_confidence=0.7) - valid_entities, _ = validator.validate_entities(entities) - valid_rels, _ = validator.validate_relations(relationships) - ``` - - + + `NamedEntityRecognizer` is the high-level coordinator with confidence thresholding and overlap merging. `NERExtractor` is the lower-level implementation. For most use cases, start with `NERExtractor` for simplicity or `NamedEntityRecognizer` for fine-grained control. + ## What You Get - - - Named entity recognition: Person, Organization, Location, Date, and custom types. - - - Typed semantic relationships between entities (`founded_by`, `located_in`, etc.). - - - Direct `(subject, predicate, object)` triplet generation for RDF-ready output. - - - Event detection with participants, temporal context, and confidence scores. - - - Resolve "Apple" and "the company" to the same entity across a document. - - - Semantic role labeling, clustering, and entity similarity analysis. - - +- **`NERExtractor`** / **`NamedEntityRecognizer`** — named entity recognition: Person, Organization, Location, Date, and custom types +- **`RelationExtractor`** — typed semantic relationships between entities (`founded_by`, `located_in`, etc.) +- **`TripletExtractor`** — direct `(subject, predicate, object)` triplet generation for RDF-ready output +- **`EventDetector`** — event detection with participants, temporal context, and confidence scores +- **`CoreferenceResolver`** — resolve "Apple" and "the company" to the same entity across a document + +## Quick Start + +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor +from semantica.llms import Groq +import os + +text = "Apple Inc. was founded by Steve Jobs in Cupertino in 1976." +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +entities = NERExtractor(method="llm", llm_provider=llm).extract(text) +relationships = RelationExtractor(method="llm", llm_provider=llm).extract(text, entities=entities) +triplets = TripletExtractor(method="llm", llm_provider=llm).extract(text) +``` Semantic extraction pipeline: raw text fans into NER, Relation, and Coreference extractors, then merges into a Triplet Generator -## Extraction Methods - - - - Uses a language model to extract entities and relationships. Handles complex schemas, novel entity types, and domain-specific language. Requires an API key. - - ```python - from semantica.semantic_extract import NERExtractor - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) - - entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino in 1976.") - ``` - - - **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. - - - Works with every Semantica LLM provider — swap `Groq` for any other provider with a one-line change. Anthropic, Gemini, Ollama, and DeepSeek are accessed via `LiteLLM` using their provider prefix: - - ```python - from semantica.llms import LiteLLM - llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) - ner = NERExtractor(method="llm", llm_provider=llm) - ``` - - - Uses a pre-trained BERT-based NER model. High accuracy for standard entity types (Person, Organization, Location, Date) at zero API cost. - - ```python - ner = NERExtractor(method="ml", model="dslim/bert-large-NER") - entities = ner.extract(text) - ``` - - For relations, the ML backend uses the REBEL model: - - ```python - rel = RelationExtractor(method="ml") - relationships = rel.extract(text, entities=entities) - ``` - - Best for: high-throughput extraction where API cost matters and entity types are standard CoNLL/OntoNotes categories. - - - Dictionary and regex matching — extremely fast, zero API cost, zero model loading. Accuracy depends entirely on your dictionaries. - - ```python - ner = NERExtractor( - method="pattern", - custom_entities={ - "DRUG": ["aspirin", "ibuprofen", "metformin"], - "GENE": ["BRCA1", "TP53", "EGFR"] - } - ) - entities = ner.extract(text) - ``` - - For relations, uses hand-crafted rules: - - ```python - rel = RelationExtractor(method="rule") - ``` - - Best for: known entity sets (drug names, product codes, gene symbols), no-API-key environments, or as a first pass before LLM validation. - - - The pattern matcher is **case-sensitive and whitespace-sensitive**. Normalize text first with `TextNormalizer` so "BRCA1" and "brca1" both match. For fuzzy matching, use `method="ml"`. - - - - -### Method Comparison - -| Method | Speed | Cost | Accuracy | Custom Types | Best For | -| ------ | ----- | ---- | -------- | ------------ | -------- | -| `pattern` | Very fast | Free | Medium | Yes (dictionary) | Known entity sets, no-API environments | -| `ml` | Fast | Free | High | Limited | Standard types at scale, no API budget | -| `llm` | Medium | API cost | Highest | Yes (schema) | Complex schemas, novel types, best accuracy | - ## NERExtractor ```python +from semantica.semantic_extract import NERExtractor +from semantica.llms import Groq +import os + +# Pattern-based — fast, no API key, good for standard entity types +ner = NERExtractor(method="pattern") +entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino.") + +# ML-based — higher accuracy, no API cost +ner = NERExtractor(method="ml", model="dslim/bert-large-NER") +entities = ner.extract(text) + +# LLM-based — best accuracy, handles complex schemas and custom types +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) entities = ner.extract(text) ``` @@ -175,16 +89,28 @@ Output format: ] ``` -Batch processing for large corpora: +### Custom Entity Types ```python -texts = ["Text 1...", "Text 2...", "Text 3..."] -batch_results = ner.extract_batch(texts, batch_size=10) +ner = NERExtractor( + method="pattern", + custom_entities={ + "DRUG": ["aspirin", "ibuprofen", "metformin"], + "GENE": ["BRCA1", "TP53", "EGFR"] + } +) ``` + + **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. + + ## RelationExtractor ```python +from semantica.semantic_extract import RelationExtractor + +rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) relationships = rel.extract(text, entities=entities) ``` @@ -197,9 +123,7 @@ Output format: ] ``` - - Always pass `entities=entities` from your NER output. This anchors relationships to known entity spans — improving accuracy and eliminating hallucinated entity names. - +Available methods: `"rule"` (pattern-based), `"ml"` (REBEL model), `"llm"`. ## TripletExtractor @@ -208,142 +132,84 @@ Generate RDF-ready `(subject, predicate, object)` triplets directly from text: ```python from semantica.semantic_extract import TripletExtractor -trip = TripletExtractor(method="llm", llm_provider=llm) +trip = TripletExtractor(method="llm", llm_provider=llm) triplets = trip.extract(text) # → [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", ...}] ``` -Triplets are suitable for loading directly into a triplet store or knowledge graph without a separate relation extraction step. +Triplets are suitable for loading directly into a triplet store or knowledge graph. ## EventDetector Detect events with participants and temporal context: ```python -from semantica.semantic_extract import EventDetector +from semantica.semantic_extract import EventDetector, Event extractor = EventDetector(method="llm", llm_provider=llm) -events = extractor.extract(text) +events: list[Event] = extractor.extract(text) + +for event in events: + print(f"Event type: {event.type}") + print(f"Participants: {event.participants}") + print(f"Temporal: {event.temporal}") + print(f"Confidence: {event.confidence:.2f}") ``` -Output includes: event type, participants (with roles), temporal information, location, and confidence score. +Output fields per event: `type`, `participants` (with roles), `temporal`, `location`, and `confidence`. -## SemanticAnalyzer +## CoreferenceResolver -Semantic role labeling, clustering, and similarity analysis on extracted content: +Resolve pronoun and alias references to canonical entities before extraction: ```python -from semantica.semantic_extract import SemanticAnalyzer +from semantica.semantic_extract import CoreferenceResolver -analyzer = SemanticAnalyzer() - -# Who did what to whom -roles = analyzer.label_roles(text) -# → [{"agent": "Apple", "action": "acquired", "patient": "Intel's modem unit"}] - -# Group similar entities -clusters = analyzer.cluster_entities(entities, n_clusters=5) -# → [{"cluster_id": 0, "entities": ["Apple Inc.", "Apple", "AAPL"]}, ...] - -# Pairwise semantic similarity -score = analyzer.calculate_similarity(entity_a, entity_b) -# → 0.87 -``` - -| Method | Returns | Description | -| ------ | ------- | ----------- | -| `label_roles(text)` | `List[Dict]` | Semantic role labeling (agent, action, patient) | -| `cluster_entities(entities, n_clusters)` | `List[Cluster]` | Group similar entities | -| `calculate_similarity(a, b)` | `float` | Cosine similarity between entity embeddings | -| `analyze_sentiment(text)` | `Dict` | Sentiment and subjectivity scores | - -## SemanticNetworkExtractor - -Extracts a full semantic network (nodes + typed edges) from text in one pass: - -```python -from semantica.semantic_extract import SemanticNetworkExtractor - -extractor = SemanticNetworkExtractor(method="llm", llm_provider=llm) -network = extractor.extract_network(text) - -print(f"Nodes: {len(network.nodes)}") -print(f"Edges: {len(network.edges)}") - -for edge in network.edges: - print(f" {edge.source} --[{edge.relation}]--> {edge.target} (conf: {edge.confidence:.2f})") -``` - - - -```python -@dataclass -class SemanticNetwork: - nodes: List[NetworkNode] - edges: List[NetworkEdge] - -@dataclass -class NetworkNode: - id: str - label: str # entity type (PERSON, ORG, etc.) - properties: Dict[str, Any] - -@dataclass -class NetworkEdge: - source: str - target: str - relation: str # e.g. "founded_by", "located_in" - confidence: float - metadata: Dict[str, Any] -``` - - - -## ExtractionValidator - -Validates extraction quality and filters low-confidence results: - -```python -from semantica.semantic_extract import ExtractionValidator - -validator = ExtractionValidator( - min_confidence=0.7, # drop entities below this score - require_entity_text=True, # entity text must be non-empty - max_entity_length=100, # discard suspiciously long entities +resolver = CoreferenceResolver() +resolved_text = resolver.resolve( + "Apple Inc. was founded in 1976. The company is headquartered in Cupertino." ) - -valid_entities, rejected = validator.validate_entities(entities) -valid_rels, rejected = validator.validate_relations(relationships) - -report = validator.get_quality_report(entities, relationships) -print(f"Precision estimate: {report['precision']:.2f}") +# "Apple Inc." replaces "The company" for consistent downstream extraction ``` -## Tips and Common Pitfalls +## Batch Processing - - **Run `CoreferenceResolver` before extraction.** If a paragraph says "Apple Inc. was founded in 1976. The company launched..." without resolving "the company" → "Apple Inc.", your extractor may miss the second entity or create a phantom "The company" node. - +All extractors support batch input for efficient large-scale processing: - - **Always pass `entities=` to `RelationExtractor`.** Passing the entity list from NER output anchors relationships to known entity spans — improving accuracy and eliminating hallucinated entity names. - +```python +texts = ["Text 1...", "Text 2...", "Text 3..."] - - **Validate before building the graph.** Use `ExtractionValidator(min_confidence=0.7)` to drop low-confidence entities before they reach `GraphBuilder`. Noisy extractions produce noisy graphs that corrupt analytics and search. - +ner = NERExtractor(method="llm", llm_provider=llm) +batch_results = ner.extract_batch(texts, batch_size=10) +``` - - **Set `max_retries=3` for LLM extractors.** API calls fail transiently. Setting `max_retries=3` prevents pipeline crashes on flaky network conditions without slowing down the happy path. - +## Using All Extractors Together - - **Don't mix extraction methods mid-pipeline.** If you extract entities with `pattern` and relations with `llm`, entity names in the relation output may not match the pattern-extracted entity IDs — causing alignment failures in `GraphBuilder`. Use the same method throughout, or normalize entity names before the relation step. - +The standard extraction pipeline — entities → relationships → triplets: - - **Batch large inputs.** Call `ner.extract_batch(texts, batch_size=10)` rather than looping over individual texts. Batch mode is significantly faster for both ML (GPU batching) and LLM (fewer API round-trips with prompt packing). - +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) +rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) +trip = TripletExtractor(method="llm", llm_provider=llm, max_retries=3) + +entities = ner.extract(text) +relationships = rel.extract(text, entities=entities) +triplets = trip.extract(text) +``` + +## Extraction Method Comparison + +| Method | Speed | Cost | Accuracy | Custom Types | +| ------ | ----- | ---- | -------- | ------------ | +| `pattern` | Very fast | Free | Medium | Yes (dictionary) | +| `ml` | Fast | Free | High | Limited | +| `llm` | Medium | API cost | Highest | Yes (schema) | diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 2da34230..c123bdfd 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -6,6 +6,31 @@ icon: "wrench" `semantica.utils` provides shared infrastructure used throughout Semantica. Most users won't call it directly, but its APIs are available when you need fine-grained control over logging, validation, progress tracking, or error handling. +## Exported Classes + +```python +from semantica.utils import ( + # Logging + setup_logging, # configure root logger — level, format (json/text) + get_logger, # get a named logger instance + log_performance, # @decorator — logs function name, duration, exception + # Validation + validate_entity, # validate entity dict structure, raises ValidationError + validate_config, # validate config dict against schema, raises ValidationError + # Progress tracking + ProgressTracker, # class-based tracker with ETA + track_progress, # wraps any iterable with live progress bar + # Helpers + clean_text, # normalize whitespace, strip control characters + hash_data, # deterministic SHA-256 hash of any serializable object + safe_filename, # sanitize a string for use as a filename + # Exceptions + SemanticaError, # base exception for all Semantica errors + ValidationError, # raised when input fails validation + ProcessingError, # raised during extraction, graph build, or pipeline step +) +``` + ## What You Get From beacc88b02dea1730746bc557616eab1c706fdd9 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 14:46:50 +0530 Subject: [PATCH 10/11] fix(ci): replace list[Event] with List[Event] for Python 3.8 compat --- docs/reference/semantic_extract.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index c88398f3..c1efe0b8 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -144,10 +144,11 @@ Triplets are suitable for loading directly into a triplet store or knowledge gra Detect events with participants and temporal context: ```python +from typing import List from semantica.semantic_extract import EventDetector, Event extractor = EventDetector(method="llm", llm_provider=llm) -events: list[Event] = extractor.extract(text) +events: List[Event] = extractor.extract(text) for event in events: print(f"Event type: {event.type}") From 68fcff5b3ab6b2110c5c15bbd0f63ecf8fa3c137 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sun, 24 May 2026 14:56:11 +0530 Subject: [PATCH 11/11] docs: add Exported Classes blocks to all remaining reference docs Adds ## Exported Classes (or equivalent interface block) to: - change_management.md, conflicts.md, context.md, embeddings.md - graph_store.md, ingest.md, normalize.md, pipeline.md - seed.md, split.md, triplet_store.md, vector_store.md - visualization.md Adds ## Launch Interface to explorer.md (CLI-only module). Adds ## Server Interface to mcp_server.md (stdio process, not importable). All blocks sourced from module __all__ with inline usage hints. evals.md intentionally skipped (placeholder, __all__ = []). --- docs/reference/change_management.md | 23 +++++++++++++++ docs/reference/conflicts.md | 35 +++++++++++++++++++++++ docs/reference/context.md | 41 +++++++++++++++++++++++++++ docs/reference/embeddings.md | 31 ++++++++++++++++++++ docs/reference/explorer.md | 17 +++++++++++ docs/reference/graph_store.md | 32 +++++++++++++++++++++ docs/reference/ingest.md | 41 +++++++++++++++++++++++++++ docs/reference/mcp_server.md | 24 ++++++++++++++++ docs/reference/normalize.md | 44 +++++++++++++++++++++++++++++ docs/reference/pipeline.md | 36 +++++++++++++++++++++++ docs/reference/seed.md | 10 +++++++ docs/reference/split.md | 31 ++++++++++++++++++++ docs/reference/triplet_store.md | 24 ++++++++++++++++ docs/reference/vector_store.md | 36 +++++++++++++++++++++++ docs/reference/visualization.md | 22 +++++++++++++++ 15 files changed, 447 insertions(+) diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 092e55e9..8d45dcc1 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -10,6 +10,29 @@ icon: "clock-rotate-left" Compliance frameworks supported out of the box: **HIPAA**, **SOX**, **GDPR**, and **FDA 21 CFR Part 11**. +## Exported Classes + +```python +from semantica.change_management import ( + # Change metadata + ChangeLogEntry, # snapshot record: version, author, message, checksum, changes + # Storage backends + VersionStorage, # abstract storage interface + InMemoryVersionStorage, # fast in-memory backend (dev/test only) + SQLiteVersionStorage, # persistent SQLite backend (production) + # Integrity utilities + compute_checksum, # SHA-256 checksum of a graph state + verify_checksum, # verify graph against a stored checksum + # Version managers + TemporalVersionManager, # KG version management: snapshot, diff, rollback + OntologyVersionManager, # ontology version management + BaseVersionManager, # base class for custom version managers + # Ontology versioning (moved from ontology module) + VersionManager, # OWL ontology version control + OntologyVersion, # ontology version metadata dataclass +) +``` + ## What You Get diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index 6346acd4..5d75d637 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -18,6 +18,41 @@ Semantica's conflict detection makes disagreements explicit and actionable: - **Logical conflicts** — an entity simultaneously holds two mutually exclusive properties - **Relationship conflicts** — the same relationship has inconsistent cardinality or properties across sources +## Exported Classes + +```python +from semantica.conflicts import ( + # Detection + ConflictDetector, # detect value, type, temporal, logical, relationship conflicts + Conflict, # {id, entity_id, attribute, values, sources, conflict_type, severity} + ConflictType, # enum: VALUE_CONFLICT, TYPE_CONFLICT, TEMPORAL_CONFLICT, ... + # Resolution + ConflictResolver, # resolve conflicts with configurable strategy + ResolutionStrategy, # enum: VOTING, CREDIBILITY_WEIGHTED, MOST_RECENT, FIRST_SEEN, ... + ResolutionResult, # outcome of a resolve_conflicts() call + # Convenience strategy aliases + voting, credibility_weighted, most_recent, first_seen, highest_confidence, + manual_review, expert_review, + # Source tracking + SourceTracker, # track which source contributed each property value + SourceReference, # {source_id, credibility, timestamp} + PropertySource, # per-property source attribution record + # Analysis + ConflictAnalyzer, # analyze patterns, severity distribution, source stats + ConflictPattern, # recurring conflict pattern detected across entities + # Investigation + InvestigationGuideGenerator, # generate step-by-step checklists for manual review + InvestigationGuide, # {title, context, steps} + InvestigationStep, # {order, description, check, priority} + # Convenience functions + detect_conflicts, # quick: detect_conflicts(entities, attribute="name") + resolve_conflicts, # quick: resolve_conflicts(conflicts, strategy=voting) + analyze_conflicts, # quick: analyze_conflicts(conflicts) + track_sources, # quick: track_sources(entities) + generate_investigation_guide,# quick: generate_investigation_guide(conflict) +) +``` + ## What You Get diff --git a/docs/reference/context.md b/docs/reference/context.md index 4ca29248..e448e7e9 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -6,6 +6,47 @@ icon: "brain" `semantica.context` is the memory and decision layer for AI agents. It stores facts with provenance, records decisions as first-class objects with full causal chains, lets agents search their own history to stay consistent across runs, and answers complex queries by traversing the knowledge graph. +## Exported Classes + +```python +from semantica.context import ( + # High-level interfaces + AgentContext, # primary entry point: store, retrieve, record_decision, find_precedents + DecisionContext, # decision-focused facade (wraps AgentContext + DecisionRecorder) + # Graph primitives + ContextGraph, # in-memory graph: add/get entities, record decisions, find precedents + ContextNode, # {id, label, node_type, properties, embedding, confidence} + ContextEdge, # {source, target, edge_type, weight, properties} + # Memory + AgentMemory, # RAG memory: store(text), retrieve(query, max_results) + MemoryItem, # {id, content, timestamp, conversation_id, embedding, metadata} + # Retrieval + ContextRetriever, # retrieve(query, max_results, use_graph, min_score) + RetrievedContext, # {content, score, source, metadata} + TemporalGraphRetriever, # retrieval with temporal decay weighting + # Entity linking + EntityLinker, # link_entity(text, entity_type) -> LinkedEntity with URI + EntityLink, # {entity_id, uri, source_text, confidence} + LinkedEntity, # {canonical_id, uri, aliases, type, properties} + # Decision tracking models + Decision, # {id, category, scenario, reasoning, outcome, confidence, timestamp} + Policy, # {id, name, conditions, action, priority} + PolicyException, # {policy_id, decision_id, reason, override_authority} + Precedent, # {decision_id, scenario, outcome, similarity, timestamp} + ApprovalChain, # ordered list of approvers for escalation + # Decision tracking classes + DecisionRecorder, # record and persist decisions with embeddings + DecisionQuery, # query decisions: by_category, by_outcome, by_date_range + CausalChainAnalyzer, # trace causality: get_causal_chain, analyze_impact + PolicyEngine, # check_compliance, get_applicable_policies, enforce_policy + # Convenience functions + record_decision, # record_decision(category, scenario, reasoning, outcome, confidence) + find_precedents, # find_precedents(scenario, category, limit) + analyze_decision_impact, # analyze_decision_impact(decision_id) + check_decision_compliance, # check_decision_compliance(decision, policies) +) +``` + ## What You Get diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index 5d2cb1d4..7685353b 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -19,6 +19,37 @@ Semantica uses embeddings for: - **Distance Intelligence** — N×N semantic distance matrices across entity sets - **Semantic chunking** — detect topic shift boundaries in `TextSplitter(method="semantic_transformer")` +## Exported Classes + +```python +from semantica.embeddings import ( + # Core generators + EmbeddingGenerator, # main handler: generate_embeddings(text, data_type="text") + TextEmbedder, # text embedding: embed(text), embed_batch(texts) + GraphEmbeddingManager, # embed KG nodes/subgraphs for GraphRAG + VectorEmbeddingManager, # embedding management for vector databases + # Provider stores + OpenAIStore, # OpenAI text-embedding-* API + BGEStore, # BAAI/bge-* via sentence-transformers + FastEmbedStore, # ONNX-accelerated, no CUDA required + LlamaStore, # Ollama local embedding models + ProviderStoreFactory, # create(provider="bge", model="...") factory + # Pooling strategies + MeanPooling, # default — best for retrieval and clustering + MaxPooling, # captures presence of any feature + CLSPooling, # CLS token (BERT-style classification models) + AttentionPooling, # softmax-weighted sum + HierarchicalPooling, # for long documents exceeding context length + PoolingStrategyFactory, # create(strategy="mean") factory + # Convenience functions + embed_text, # embed_text(text, method="sentence_transformers") + generate_embeddings, # generate_embeddings(texts, method="openai") + calculate_similarity, # calculate_similarity(a, b, method="cosine") + pool_embeddings, # pool_embeddings(token_embeddings, strategy="mean") + check_available_providers, # returns {"sentence_transformers": True, ...} +) +``` + ## What You Get diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index c89448a8..3174081b 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -6,6 +6,23 @@ icon: "map" `semantica.explorer` is a browser-based dashboard for exploring knowledge graphs, managing ontologies, and running visual analyses — no code required after launch. +## Launch Interface + +```bash +# Install and launch +pip install semantica[explorer] + +# Start the Explorer dashboard +semantica-explorer --graph my_graph.json --port 8000 + +# Or via Python module +python -m semantica.explorer --graph my_graph.json --port 8000 --host 0.0.0.0 +``` + + + `semantica.explorer` is a **server process**, not a Python library. It exposes no importable classes. Use the CLI or `python -m semantica.explorer` to launch. + + ## What You Get diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index afeb9a68..819fb64a 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -6,6 +6,38 @@ icon: "server" `semantica.graph_store` provides a single API for persisting and querying knowledge graphs in production graph databases. Swap backends with a one-line change — no application code changes needed. +## Exported Classes + +```python +from semantica.graph_store import ( + # Core interface + GraphStore, # unified interface: add_node, add_edge, query, find_paths + GraphManager, # store management and operations + NodeManager, # node CRUD operations + RelationshipManager, # relationship CRUD operations + QueryEngine, # Cypher query execution with caching + GraphAnalytics, # centrality, community detection, shortest path + # Backend stores + Neo4jStore, # Neo4j via Bolt — production workloads + ApacheAgeStore, # PostgreSQL + AGE extension + AmazonNeptuneStore, # AWS Neptune — SPARQL/Gremlin/openCypher + FalkorDBStore, # Redis-based — ultra-low latency + # Convenience functions + create_node, # create_node(labels, properties) + create_nodes, # bulk: create_nodes(entities) + create_relationship, # create_relationship(start_id, end_id, rel_type) + create_relationships, # bulk: create_relationships(rels) + get_nodes, # get_nodes(labels, filters) + get_relationships, # get_relationships(start_id, rel_type) + get_neighbors, # get_neighbors(node_id, direction="both") + update_node, # update_node(node_id, properties) + delete_node, # delete_node(node_id) + execute_query, # execute_query(cypher, parameters) + shortest_path, # shortest_path(source, target) + run_analytics, # run_analytics(graph, algorithm) +) +``` + ## What You Get diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 8f28de6b..eec13563 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -6,6 +6,47 @@ icon: "database" `semantica.ingest` is the entry point for loading data into Semantica. Every ingestor returns a list of `DataSource` objects with normalized content and metadata, regardless of the original format. +## Exported Classes + +```python +from semantica.ingest import ( + # File ingestion (always available) + FileIngestor, # local files and directories: ingest(path, recursive=True) + CloudStorageIngestor, # AWS S3, Google Cloud Storage, Azure Blob Storage + FileObject, # {content, source_id, source_type, metadata, raw_bytes} + FileTypeDetector, # auto-detect file type from extension and magic bytes + ParquetIngestor, # Apache Parquet files and partitioned datasets + XMLIngestor, # XXE-safe lxml XML parsing with optional XSD validation + # Web ingestion (requires beautifulsoup4) + WebIngestor, # web scraping: ingest_url(url), crawl(url, max_pages) + FeedIngestor, # RSS/Atom feeds: ingest_feed(url), monitor_feeds(...) + FeedMonitor, # live feed monitoring with callback on new items + # Stream ingestion + StreamIngestor, # real-time: ingest_kafka/rabbitmq/kinesis/pulsar + KafkaProcessor, # Kafka consumer group processor + RabbitMQProcessor, # AMQP queue processor + KinesisProcessor, # AWS Kinesis stream processor + PulsarProcessor, # Apache Pulsar consumer + # Repository ingestion (requires gitpython) + RepoIngestor, # Git repos: ingest(url_or_path), include_commits=True + # Email ingestion + EmailIngestor, # IMAP/POP3: ingest() with attachment extraction + # Database ingestion + DBIngestor, # SQL: ingest_database(connection_string, include_tables) + SnowflakeIngestor, # Snowflake: ingest_query(sql), ingest_table(name) + OntologyIngestor, # OWL/RDF ontology files: ingest_ontology(path) + # Convenience functions + ingest, # ingest(source, source_type="file") — unified dispatcher + ingest_file, # ingest_file(path, method="directory") + ingest_web, # ingest_web(url, method="url") + ingest_feed, # ingest_feed(url) + ingest_stream, # ingest_stream(topic, ...) + ingest_database, # ingest_database(connection_string, ...) + ingest_parquet, # ingest_parquet(path, columns=[...]) + ingest_xml, # ingest_xml(path, validate_xsd=None) +) +``` + ## What You Get diff --git a/docs/reference/mcp_server.md b/docs/reference/mcp_server.md index 5e17c33a..64e75bb0 100644 --- a/docs/reference/mcp_server.md +++ b/docs/reference/mcp_server.md @@ -10,6 +10,30 @@ Once configured, any connected AI assistant can extract entities, record decisio Compatible with **Claude Desktop**, **Windsurf**, **Cline**, **Continue**, **VS Code**, **Roo Code**, **Cursor**, and any MCP-aware client. +## Server Interface + +```json +// Configure in your MCP client (Claude Desktop, Windsurf, Cursor, VS Code, etc.) +{ + "mcpServers": { + "semantica": { + "command": "semantica-mcp" + } + } +} +``` + +```bash +# Or run directly +semantica-mcp +# or +python -m semantica.mcp_server +``` + + + `semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes — all interaction happens through MCP tool calls from a connected AI client. + + ## What You Get diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index 1143c422..0d1abba2 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -17,6 +17,50 @@ Unstructured data is inconsistent by nature. Without normalization, the same rea Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data — producing cleaner entities, fewer false duplicates, and more reliable downstream results. +## Exported Classes + +```python +from semantica.normalize import ( + # Text normalization + TextNormalizer, # coordinator: strip_html, normalize_unicode, fix_encoding + UnicodeNormalizer, # NFC/NFD/NFKC/NFKD normalization + WhitespaceNormalizer, # collapse spaces, normalize line endings + SpecialCharacterProcessor, # smart quotes, dashes, diacritics + TextCleaner, # general text cleaning utilities + # Entity normalization + EntityNormalizer, # coordinator: normalize_entity(text, entity_type) + AliasResolver, # resolve "ML" -> "Machine Learning" via dictionary + EntityDisambiguator, # disambiguate("Apple", context=...) with confidence + NameVariantHandler, # normalize("Dr. JOHN P. SMITH Jr.") -> "John P. Smith" + # Date/time normalization + DateNormalizer, # normalize_date(str) -> ISO 8601 + TimeZoneNormalizer, # normalize to UTC or target timezone + RelativeDateProcessor, # "3 days ago" -> datetime + TemporalExpressionParser, # "Q2 2023" -> {start, end, type} + # Number normalization + NumberNormalizer, # normalize_number("$1.2B") -> 1200000000.0 + UnitConverter, # convert(100, from_unit="km/h", to_unit="m/s") + CurrencyNormalizer, # normalize("$42.50") -> {amount, currency, raw} + ScientificNotationHandler, # parse scientific notation strings + # Data cleaning + DataCleaner, # remove_duplicates, fill_missing + DataValidator, # validate(records, schema={"name": str, "age": int}) + DuplicateDetector, # detect duplicate records by similarity threshold + MissingValueHandler, # fill missing values: mean/median/mode/constant + # Language & encoding + LanguageDetector, # detect(text) -> {language, confidence} + EncodingHandler, # detect_encoding, to_utf8, remove_bom + # Convenience functions + normalize_text, # normalize_text(text, method="default") + normalize_entity, # normalize_entity(name, entity_type="Person") + normalize_date, # normalize_date("Jan 1st, 2020") + normalize_number, # normalize_number("$1,234.56") + clean_text, # clean_text(text) + detect_language, # detect_language(text) + resolve_aliases, # resolve_aliases(text, aliases_dict) +) +``` + ## What You Get diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index 4b01b691..10194d9f 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -6,6 +6,42 @@ icon: "gear" `semantica.pipeline` lets you chain Semantica components into reproducible, fault-tolerant workflows with parallel execution and configurable error handling. Pipelines are serializable — save them to YAML and reload in any environment. +## Exported Classes + +```python +from semantica.pipeline import ( + # Pipeline construction + PipelineBuilder, # DSL: add_step, connect_steps, build + Pipeline, # pipeline definition dataclass + PipelineStep, # step definition: name, step_type, handler, dependencies + StepStatus, # enum: PENDING, RUNNING, COMPLETED, FAILED, SKIPPED + PipelineSerializer, # serialize/deserialize pipeline to JSON/YAML + # Execution + ExecutionEngine, # execute_pipeline(pipeline, data) -> ExecutionResult + ExecutionResult, # {success, output, metadata, metrics, errors} + PipelineStatus, # enum: RUNNING, PAUSED, STOPPED + ProgressTracker, # get_progress(pipeline_id) -> {completed, total, pct} + # Failure handling + FailureHandler, # configure strategy: skip/retry/abort + RetryHandler, # retry with exponential backoff + FallbackHandler, # fall back to alternative step on failure + RetryPolicy, # {max_retries, backoff, jitter} + RetryStrategy, # enum: FIXED, EXPONENTIAL, LINEAR + ErrorSeverity, # enum: LOW, MEDIUM, HIGH, CRITICAL + # Parallelism + ParallelismManager, # execute_parallel(tasks, timeout) — thread or process pool + ParallelExecutionResult, # {success, result, error, task_id} + # Resource management + ResourceScheduler, # allocate_resources / release_resources + ResourceType, # enum: CPU, MEMORY, GPU, NETWORK, DISK + # Validation + PipelineValidator, # validate_pipeline(pipeline) -> ValidationResult + # Templates + PipelineTemplateManager, # get_template("full-qa") -> pre-wired Pipeline + PipelineTemplate, # template metadata dataclass +) +``` + ## Why Use a Pipeline? You could wire Semantica modules together with plain Python code. Pipelines add: diff --git a/docs/reference/seed.md b/docs/reference/seed.md index 322c9259..47174346 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -6,6 +6,16 @@ icon: "database" `semantica.seed` gives your knowledge graph a reliable starting point. Rather than building from an empty graph and hoping extraction produces consistent reference data, you load verified, structured sources first — ISO codes, employee rosters, product catalogs, domain taxonomies — then merge freshly extracted data on top. +## Exported Classes + +```python +from semantica.seed import ( + SeedDataManager, # coordinator: register_source, create_foundation_graph, integrate_with_extracted + SeedDataSource, # {name, source_type, path, config} — dataclass for a registered source + SeedData, # {entities, relationships, metadata} — loaded seed data container +) +``` + ## What You Get diff --git a/docs/reference/split.md b/docs/reference/split.md index e13f0aa5..fbfda6dc 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -16,6 +16,37 @@ Most LLMs and embedding models have fixed context windows. Documents larger than Semantica's chunking methods are designed to avoid these failure modes. +## Exported Classes + +```python +from semantica.split import ( + # Unified splitter (start here) + TextSplitter, # method=: recursive, sentence, token, semantic_transformer, + # entity_aware, relation_aware, code, structural, markdown + Splitter, # alias for TextSplitter (backward compat) + # Data type + Chunk, # {text, start_char, end_char, token_count, metadata, entities, relationships} + # Specialized chunkers + SemanticChunker, # embedding-based semantic boundary detection + StructuralChunker, # heading/section-based splits from ParsedDocument + SlidingWindowChunker, # fixed-size sliding window with overlap + TableChunker, # table-specific chunking + EntityAwareChunker, # KG: preserves named entities across chunk boundaries + RelationAwareChunker, # KG: keeps subject-predicate-object triplets intact + GraphBasedChunker, # splits based on graph community structure + OntologyAwareChunker, # splits respecting ontology concept boundaries + HierarchicalChunker, # multi-level hierarchical chunking + ProvenanceTracker, # track chunk provenance back to source document + # Convenience split functions + split_recursive, # split_recursive(text, chunk_size, chunk_overlap) + split_by_sentences, # split_by_sentences(text) + split_by_tokens, # split_by_tokens(text, chunk_size, tokenizer) + split_semantic_transformer, # split_semantic_transformer(text, threshold) + split_entity_aware, # split_entity_aware(text, entities) + split_relation_aware, # split_relation_aware(text, relationships) +) +``` + ## What You Get diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index e30717ae..63f9dfe7 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -6,6 +6,30 @@ icon: "table" `semantica.triplet_store` provides W3C-standard RDF storage with full SPARQL query support. Use it when you need semantic web compatibility, OWL reasoning, SPARQL-based queries, or standards-compliant RDF serialization. +## Exported Classes + +```python +from semantica.triplet_store import ( + # Core interface + TripletStore, # unified: add_triplet, get_triplets, execute_query, bulk_load + QueryEngine, # SPARQL execution: execute_query, optimize_query, plan_query + BulkLoader, # high-volume loading with progress tracking and transaction support + # Backend stores + BlazegraphStore, # Blazegraph REST API (HTTP/HTTPS, Named Graphs, SPARQL 1.1) + JenaStore, # Apache Jena Fuseki (SPARQL 1.1, TDB2, GeoSPARQL) + RDF4JStore, # Eclipse RDF4J (SailRepository, in-memory or native) + # Convenience functions + add_triplet, # add_triplet(subject, predicate, obj) + add_triplets, # bulk: add_triplets(triplets) + get_triplets, # get_triplets(subject=None, predicate=None, obj=None) + delete_triplet, # delete_triplet(subject, predicate, obj) + execute_query, # execute_query(sparql, result_format="json") + optimize_query, # optimize_query(sparql) -> optimized SPARQL string + bulk_load, # bulk_load(file_path, format="turtle") + validate_triplets,# validate_triplets(triplets) -> ValidationResult +) +``` + ## What You Get diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index f711db99..0fbd1c68 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -6,6 +6,42 @@ icon: "database" `semantica.vector_store` provides a unified API for storing and searching vector embeddings across all major backends. Swap backends with a one-line change — no application code changes needed. +## Exported Classes + +```python +from semantica.vector_store import ( + # Core interface + VectorStore, # unified: store_vectors, search_vectors, update_vectors, delete_vectors + VectorIndexer, # build/rebuild FAISS/ANN indices + VectorRetriever, # kNN and hybrid search + VectorManager, # store management and CRUD operations + # Backend stores + FAISSStore, # local disk / in-memory (Flat, IVF, HNSW, PQ index types) + WeaviateStore, # cloud/self-hosted, schema-aware, GraphQL queries + QdrantStore, # cloud/self-hosted, payload filtering + MilvusStore, # highly scalable, partitioning and complex queries + PineconeStore, # managed cloud vector database + PgVectorStore, # PostgreSQL with pgvector extension + # Hybrid & metadata search + HybridSearch, # fuse vector + metadata results (RRF or weighted average) + MetadataFilter, # MetadataFilter().eq("category", "science").gt("year", 2020) + SearchRanker, # configurable re-ranking after fusion + MetadataStore, # inverted index for fast metadata filtering + NamespaceManager, # multi-tenant namespace isolation + # Decision-specific helpers + DecisionEmbeddingPipeline, # end-to-end: record + embed + store + retrieve + quick_decision, # quick_decision(text, entities, outcome) — shorthand record + find_precedents, # find_precedents(scenario, k=5) — similarity search + # Convenience functions + store_vectors, # store_vectors(vectors, metadata) + search_vectors, # search_vectors(query_vector, k=10) + hybrid_search, # hybrid_search(query_vector, filter=...) + update_vectors, # update_vectors(ids, new_vectors) + delete_vectors, # delete_vectors(ids) + create_index, # create_index(index_type="hnsw", dimension=768) +) +``` + ## What You Get diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index 39f5a541..136f99b0 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -6,6 +6,28 @@ icon: "chart-bar" `semantica.visualization` renders knowledge graphs, ontologies, embedding spaces, and temporal data as interactive HTML or static images — without launching the full Explorer server. +## Exported Classes + +```python +from semantica.visualization import ( + # Visualizers + KGVisualizer, # visualize_network(graph), visualize_communities(graph, communities) + OntologyVisualizer, # visualize_hierarchy(ontology), visualize_structure(ontology) + EmbeddingVisualizer, # visualize_2d_projection(embeddings, labels, method="umap") + SemanticNetworkVisualizer, # visualize_network(semantic_network) + AnalyticsVisualizer, # visualize_centrality(analytics), visualize_communities(analytics) + TemporalVisualizer, # visualize_timeline(events), visualize_evolution(snapshots) + # D3Visualizer is listed in __all__ but loaded lazily (requires d3js dependency) + # Convenience functions + visualize_kg, # visualize_kg(graph, output="interactive", method="default") + visualize_ontology, # visualize_ontology(ontology, output="interactive") + visualize_embeddings, # visualize_embeddings(embeddings, labels, method="umap") + visualize_semantic_network, # visualize_semantic_network(network) + visualize_analytics, # visualize_analytics(analytics_result) + visualize_temporal, # visualize_temporal(temporal_data) +) +``` + ## What You Get