diff --git a/docs/reference/core.md b/docs/reference/core.md index 548e7151..b8f1d30d 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -6,78 +6,33 @@ icon: "gear" `semantica.core` is the coordination layer for the framework. For most tasks you should use individual modules directly (`semantica.ingest`, `semantica.kg`, etc.). Reach for Core when you need application-level lifecycle management, centralized configuration, or a plugin registry. +## Exported Classes + +```python +from semantica.core import ( + Semantica, # orchestration class — coordinates full KG pipeline + ConfigManager, # YAML config loading, deep-merge, env var overrides + LifecycleManager,# startup/shutdown state machine + health monitoring + PluginRegistry, # plugin discovery, registration, and loading + method_registry, # global MethodRegistry instance for custom dispatch +) + +# For custom build methods: +from semantica.core.methods import build_knowledge_base +``` + ## What You Get - - - Orchestration class for coordinating complex multi-module workflows and full KG construction pipelines. - - - Unified config loading, merging, and validation with environment variable overrides. - - - Startup/shutdown hooks with priority ordering and component health monitoring. - - - Dynamic plugin discovery, registration, loading, and unloading. - - - Register and dispatch custom orchestration methods by name. - - - Live configuration state — dot-notation access, update, validate, and serialize. - - +- **`Semantica`** — orchestration class for coordinating complex multi-module workflows +- **`ConfigManager`** — unified config loading, merging, and validation with environment variable overrides +- **`LifecycleManager`** — startup/shutdown hooks and component health monitoring +- **`PluginRegistry`** — dynamic plugin discovery, registration, and loading +- **`method_registry`** — global `MethodRegistry` instance — register and dispatch custom orchestration methods **Use individual modules directly** for the vast majority of use cases. Use the `Semantica` orchestration class only when you need application-level lifecycle management or a plugin system. -## Quick Start - - - - ```python - from semantica.core import ConfigManager - - manager = ConfigManager() - config = manager.load_from_file("config.yaml") - - # Override one key at runtime - config.set("processing.batch_size", 64) - ``` - - - ```python - from semantica.core import Semantica - - framework = Semantica(config=config) - framework.initialize() - - status = framework.get_status() - print(f"State: {status['state']}") # → "READY" - ``` - - - ```python - result = framework.build_knowledge_base( - sources=["doc1.pdf", "doc2.docx"], - embeddings=True, - graph=True, - ) - ``` - - - ```python - # Always shut down in a finally block - try: - result = framework.build_knowledge_base(sources) - finally: - framework.shutdown(graceful=True) - ``` - - - ## Semantica (Orchestration) High-level entry point that coordinates the full KG construction pipeline: @@ -86,7 +41,7 @@ High-level entry point that coordinates the full KG construction pipeline: from semantica.core import Semantica, ConfigManager config_manager = ConfigManager() -config = config_manager.load_from_file("config.yaml") +config = config_manager.load_from_file("config.yaml") framework = Semantica(config=config) framework.initialize() @@ -103,6 +58,8 @@ finally: framework.shutdown(graceful=True) ``` +### Core Methods + | Method | Description | | ------ | ----------- | | `initialize()` | Initialize all framework components | @@ -119,7 +76,7 @@ Centralized config loading with deep-merge and environment variable overrides: from semantica.core import ConfigManager manager = ConfigManager() -config = manager.load_from_file("config.yaml") +config = manager.load_from_file("config.yaml") # Merge base config with environment-specific overrides merged = manager.merge_configs( @@ -127,45 +84,44 @@ merged = manager.merge_configs( manager.load_from_file("prod.yaml"), ) -# Nested dot-notation access +# Nested key access with dot notation batch_size = config.get("processing.batch_size", default=16) config.set("processing.batch_size", 64) -config.update({"quality": {"min_confidence": 0.75}}, merge=True) config.validate() - -config_dict = config.to_dict() ``` -### Config Section Reference +### YAML Configuration -| Section | Key Fields | Description | -| ------- | ---------- | ----------- | -| `llm_provider` | `name`, `model`, `api_key`, `base_url` | LLM used for extraction and reasoning | -| `embedding_model` | `provider`, `model`, `dimension`, `device` | Embedding provider and model | -| `vector_store` | `backend`, `dimension`, `index_type` | Vector storage backend | -| `graph_db` | `backend`, `uri`, `user`, `password` | Graph database connection | -| `processing` | `batch_size`, `max_workers`, `chunk_size` | Parallelism and batching | -| `pipeline` | `retry_max`, `backoff`, `failure_strategy` | Pipeline retry and failure policy | -| `logging` | `level`, `format`, `file` | Logging configuration | -| `quality` | `min_confidence`, `dedup_threshold` | Quality thresholds | -| `security` | `redact_pii`, `allowed_domains` | Security and compliance settings | -| `custom` | any key | User-defined extension settings | +```yaml +llm_provider: + name: openai + model: gpt-4o + api_key: ${OPENAI_API_KEY} -### Environment Variable Overrides +processing: + batch_size: 32 + max_workers: 4 -Any config key can be overridden with a `SEMANTICA_` prefix using double underscores for nesting: +quality: + min_confidence: 0.7 + +logging: + level: INFO +``` + +Environment variable overrides (prefix `SEMANTICA_`): ```bash -export SEMANTICA_PROCESSING__BATCH_SIZE=64 -export SEMANTICA_LLM_PROVIDER__MODEL=gpt-4o -export SEMANTICA_LOGGING__LEVEL=DEBUG -export SEMANTICA_QUALITY__MIN_CONFIDENCE=0.8 +export SEMANTICA_PROCESSING_BATCH_SIZE=64 +export SEMANTICA_LOG_LEVEL=DEBUG ``` ## LifecycleManager Manages framework state with a defined state machine and ordered startup/shutdown hooks: +**State machine:** `UNINITIALIZED` → `INITIALIZING` → `READY` → `RUNNING` → `STOPPING` → `STOPPED` + ```python from semantica.core import LifecycleManager @@ -198,7 +154,7 @@ manager.shutdown(graceful=True) ## PluginRegistry -Register custom components that participate in the full pipeline: +Register custom components that participate in the full pipeline — provenance tracking, retry policies, and parallel execution included: ```python from semantica.core import PluginRegistry @@ -211,23 +167,13 @@ class MyPlugin: return {"processed": True} registry = PluginRegistry(plugin_paths=["./plugins"]) -registry.register_plugin( - "my_plugin", MyPlugin, - version="1.0.0", - description="Custom domain extractor", - author="team@example.com", - capabilities=["extract"], -) +registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") plugin = registry.load_plugin("my_plugin", api_key="xxx") result = plugin.execute("sample data") -# Inspect registered plugins for info in registry.list_plugins(): - print(f"{info['name']} v{info['version']} — {info['description']}") - -# Unload when done -registry.unload_plugin("my_plugin") + print(f"{info['name']}: {info['version']}") ``` ## MethodRegistry @@ -236,6 +182,7 @@ Register custom orchestration methods and dispatch them by name: ```python from semantica.core import method_registry +from semantica.core.methods import build_knowledge_base def fast_kb_builder(sources, **kwargs): # Custom logic — skip embeddings for speed @@ -243,181 +190,23 @@ def fast_kb_builder(sources, **kwargs): method_registry.register("knowledge_base", "fast", fast_kb_builder) -from semantica.core.methods import build_knowledge_base result = build_knowledge_base(sources=["doc.pdf"], method="fast") ``` -## Schemas +## When to Use Core vs. Individual Modules - - - -```python -from semantica.core import SystemState - -SystemState.UNINITIALIZED # → startup() → -SystemState.INITIALIZING # → hooks complete → -SystemState.READY # → first operation → -SystemState.RUNNING # → shutdown() → -SystemState.STOPPING # → hooks complete → -SystemState.STOPPED -# Any unhandled exception during startup/shutdown → -SystemState.ERROR -``` - -Check current state at any time: - -```python -state = manager.get_state() -if manager.is_ready(): - result = framework.build_knowledge_base(sources) -``` - - - - -```python -@dataclass -class HealthStatus: - component: str # component name - healthy: bool # True = operational - message: str # human-readable status - timestamp: datetime # time of last check - details: Dict[str, Any] # component-specific diagnostics -``` - -```python -health = manager.health_check() -for name, status in health.items(): - icon = "✓" if status.healthy else "✗" - print(f"{icon} {name}: {status.message}") -``` - - - - -```python -@dataclass -class PluginInfo: - name: str - version: str - plugin_class: Type - description: str - author: str - dependencies: List[str] # pip package names required - capabilities: List[str] # e.g. ["ingest", "extract"] - metadata: Dict[str, Any] - -@dataclass -class LoadedPlugin: - info: PluginInfo - instance: Any # the live plugin object - config: Dict[str, Any] # config passed at load time - loaded_at: datetime -``` - -```python -registry = PluginRegistry() -registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") - -if registry.is_plugin_loaded("my_plugin"): - plugin = registry.get_loaded_plugin("my_plugin") -else: - plugin = registry.load_plugin("my_plugin") - -details = registry.get_plugin_info("my_plugin") -``` - - - - -## Complete Configuration Example - -```yaml -# config.yaml -llm_provider: - name: groq - model: llama-3.3-70b-versatile - api_key: ${GROQ_API_KEY} - -embedding_model: - provider: sentence-transformers - model: all-mpnet-base-v2 - dimension: 768 - device: cpu # "cpu" | "cuda" | "mps" - -vector_store: - backend: faiss - dimension: 768 - index_type: hnsw # "flat" | "ivf" | "hnsw" | "pq" - -graph_db: - backend: neo4j - uri: bolt://localhost:7687 - user: neo4j - password: ${NEO4J_PASSWORD} - -processing: - batch_size: 32 - max_workers: 4 - chunk_size: 512 - -pipeline: - retry_max: 3 - backoff: exponential # "fixed" | "linear" | "exponential" - failure_strategy: skip # "skip" | "stop" | "retry" - -quality: - min_confidence: 0.7 - dedup_threshold: 0.85 - -logging: - level: INFO # DEBUG | INFO | WARNING | ERROR - format: "%(asctime)s %(name)s %(levelname)s %(message)s" -``` - -Load and use: - -```python -from semantica.core import ConfigManager, Semantica - -manager = ConfigManager() -config = manager.load_from_file("config.yaml") - -config.set("processing.batch_size", 64) # runtime override - -framework = Semantica(config=config) -framework.initialize() -result = framework.build_knowledge_base(["doc1.pdf", "doc2.docx"]) -framework.shutdown() -``` - -## Tips and Common Pitfalls +| Scenario | Recommended Approach | +| -------- | -------------------- | +| Single extraction task | `from semantica.semantic_extract import NERExtractor` | +| Build a knowledge graph | `from semantica.kg import GraphBuilder` | +| Multi-step pipeline | `from semantica.pipeline import Pipeline` | +| App-level lifecycle + config | `from semantica.core import Semantica, ConfigManager` | +| Custom dispatch / plugins | `from semantica.core import method_registry, PluginRegistry` | - **Use individual modules directly unless you need application lifecycle management.** `Semantica` orchestrates the full pipeline, but for simple scripts and notebooks, using `FileIngestor`, `NERExtractor`, and `GraphBuilder` directly is clearer and more debuggable. + Use `Semantica` and `LifecycleManager` only when building a long-running application (e.g. a FastAPI service) that needs ordered startup, health checks, and graceful shutdown. For scripts and notebooks, use individual modules directly. - - **Always call `framework.shutdown(graceful=True)` in a `finally` block.** Without graceful shutdown, in-flight pipeline steps may leave partial writes in your vector store or graph database. Wrapping in `try/finally` guarantees cleanup even on exceptions. - - - - **Use `ConfigManager.merge_configs()` for environment-specific overrides.** Keep a `base.yaml` with default settings and a `prod.yaml` with overrides. Merge them at startup rather than maintaining separate copies — this prevents configuration drift between environments. - - - - **Environment variable overrides use double underscores for nesting.** `SEMANTICA_PROCESSING__BATCH_SIZE=64` sets `processing.batch_size` — the double underscore (`__`) represents a nesting level. A single underscore is reserved for multi-word keys within the same level. - - - - **Register startup hooks with explicit priorities.** `register_startup_hook(fn, priority=10)` — lower numbers run first. If your database hook (priority 10) must run before your cache hook (priority 20), those numbers guarantee the order. Without explicit priorities, execution order is undefined. - - - - **Check `manager.is_ready()` before running pipelines.** If `Semantica.initialize()` failed partway through (e.g., a database connection refused), the state transitions to `ERROR` rather than `READY`. Always check before submitting work to avoid errors that are hard to trace. - - Pipeline execution and step orchestration. diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index 6246aad8..ccf5a943 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -6,127 +6,41 @@ icon: "copy" `semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control. -## Why Deduplicate? +## Exported Classes -Real-world data sources disagree on names. "Apple Inc.", "Apple Computer", and "Apple International Ltd" can all refer to the same company — but if they land in your knowledge graph as separate nodes, every query that should return one entity returns three. Relationships, analytics, and retrieval all degrade. - -Deduplication solves this before it reaches the graph: - -- **Cross-source ingestion** — Wikipedia calls it "OpenAI", SEC filings call it "OpenAI, Inc.", your CRM calls it "OpenAI LLC" -- **Data entry variation** — "Steve Jobs", "Steven P. Jobs", "S. Jobs" are the same person -- **Transliteration differences** — Cyrillic, Chinese, or Arabic names romanized inconsistently across sources -- **Abbreviation drift** — "US", "U.S.", "United States", "USA" all mean the same thing +```python +from semantica.deduplication import ( + DuplicateDetector, # pairwise + batch duplicate detection + EntityMerger, # merge duplicate groups with per-property policies + SimilarityCalculator, # Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding + ClusterBuilder, # Union-Find + hierarchical clustering + PropertyMergeRule, # enum: KEEP_FIRST, KEEP_LONGEST, UNION, VOTING, ... + MergeStrategyManager, # manage and apply merge strategies + # Convenience functions + detect_duplicates, # quick: detect_duplicates(entities, method="semantic_v2") + merge_entities, # quick: merge_entities(entities, duplicates, method="union") + calculate_similarity, # quick: calculate_similarity(a, b, method="hybrid_v2") + # Registry + method_registry, # register custom similarity functions +) +``` ## What You Get - - - Pairwise and batch duplicate detection with configurable strategies and result filtering. - - - Merge duplicate groups with configurable property-level merge policies. - - - Multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, and embedding. - - - Union-Find and hierarchical clustering for large-scale batch deduplication. - - - Reusable per-property merge rule configurations — define once, apply across operations. - - - `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster than v1 equivalents. - - - -## Quick Start - -```python -from semantica.deduplication import detect_duplicates, merge_entities - -# Detect -duplicates = detect_duplicates(entities, method="hybrid_v2", similarity_threshold=0.85) - -# Merge -merged = merge_entities(entities, duplicates, method="keep_most_complete") -print(f"Reduced {len(entities)} → {len(merged)} entities") -``` - -## Choosing a Strategy - - - - Combines blocking + string similarity + semantic embedding — best default for production: - - ```python - from semantica.deduplication import DuplicateDetector - - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") - ``` - - Best for: general production use — handles 95% of real-world cases without GPU. Catches string variants ("Apple Inc." / "Apple Inc") and semantic aliases ("Machine Learning" / "ML"). - - - Pure embedding-based similarity — highest accuracy for cross-language and abbreviation matching: - - ```python - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="semantic_v2") - ``` - - Best for: entities that use different words for the same concept ("ML" vs "Machine Learning"), cross-language entity matching, and abbreviation expansion. Requires embedding model. - - - Blocking + Jaro-Winkler string similarity only — fastest option for CPU-only environments: - - ```python - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="blocking_v2") - ``` - - Best for: large datasets (500K+ entities) where speed is critical and entities are in the same language. No GPU required. - - - - | Strategy | Algorithm | Speed | Accuracy | Best For | - | -------- | --------- | ----- | -------- | -------- | - | `jaro_winkler` | String similarity only (v1) | Fast | Medium | Small datasets, names, single-source | - | `blocking_v2` | Blocking + Jaro-Winkler | Very fast | Medium | Large datasets, speed-critical, CPU-only | - | `hybrid_v2` | Blocking + string + semantic | Fast | High | General production use — best default | - | `semantic_v2` | Embedding similarity | Medium | Highest | Semantic aliases, cross-language, abbreviations | - - **Rules of thumb:** - - Start with `hybrid_v2` — it handles 95% of real-world cases without GPU - - Use `semantic_v2` when entities use different words for the same concept - - Use `blocking_v2` when processing >500k entities and speed matters most - - Never use v1 strategies on new projects — they exist for backwards compatibility only - - - -### Threshold Tuning - -| Domain | Recommended Threshold | Notes | -| ------ | --------------------- | ----- | -| Person names | 0.85–0.90 | Names vary a lot; too high misses "Steve" / "Steven" | -| Organization names | 0.80–0.88 | Corporate suffixes create variation; lower threshold helps | -| Product names | 0.88–0.95 | Product names are more stable | -| Medical terms | 0.90–0.95 | High precision required; false merges are dangerous | -| General entities | 0.85 | Safe default starting point | - -Start at 0.85, inspect false positives and false negatives, then adjust ±0.05. +- **`DuplicateDetector`** — pairwise and batch duplicate detection with configurable strategies +- **`EntityMerger`** — merge duplicate groups with configurable property-level merge policies +- **`SimilarityCalculator`** — multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding +- **`ClusterBuilder`** — Union-Find and hierarchical clustering for large-scale batch deduplication +- **Convenience functions** — `detect_duplicates`, `merge_entities`, `calculate_similarity` ## DuplicateDetector - - **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. - +Find duplicate entity pairs with configurable strategies and result filtering: ```python from semantica.deduplication import DuplicateDetector -detector = DuplicateDetector(similarity_threshold=0.85) +detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities) for dup in duplicates: @@ -138,23 +52,30 @@ Fine-grained control over strategy, thresholds, and result size: ```python duplicates = detector.detect_duplicates( entities, - strategy="hybrid_v2", - min_similarity=0.85, - top_k_per_entity=3, # max candidates per entity — avoids false-positive floods - max_results=100, - sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" + strategy="semantic_v2", # see strategies table below + min_similarity=0.85, # minimum score to consider a match + top_k_per_entity=3, # max candidates per entity + max_results=100, # total result cap + sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" ) ``` -**Key behaviours:** -- Defaults to `hybrid_v2` when no strategy is specified -- `top_k_per_entity=3` prevents one entity from flooding results by being a near-match to everything -- Pairs are returned once — never both `(A, B)` and `(B, A)` -- `sort_by="similarity"` puts highest-confidence duplicates first for faster manual review +### Detection Strategies + +| Strategy | Algorithm | Speed | Accuracy | +| -------- | --------- | ----- | -------- | +| `jaro_winkler` | String similarity (v1) | Fast | Medium | +| `blocking_v2` | Blocking + Jaro-Winkler (v2) | Very fast | Medium | +| `hybrid_v2` | Blocking + semantic + string (v2) | Fast | High | +| `semantic_v2` | Embedding similarity (v2) | Medium | Highest | + + + **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. + ## EntityMerger -Merge detected duplicate groups into canonical entities, preserving provenance: +Merges detected duplicate groups into canonical entities, preserving provenance: ```python from semantica.deduplication import EntityMerger @@ -162,63 +83,56 @@ from semantica.deduplication import EntityMerger merger = EntityMerger() merged_entities = merger.merge_duplicates( entities, - strategy="keep_most_complete", - preserve_provenance=True, + strategy="keep_most_complete", # see strategies table below + preserve_provenance=True, # keep source references after merge ) - -print(f"Merged to: {len(merged_entities)} canonical entities") ``` ### Merge Strategies -| Strategy | Behavior | When to Use | -| -------- | -------- | ----------- | -| `keep_first` | Keep the first entity in each group | Source order is meaningful (most authoritative first) | -| `keep_last` | Keep the most recently seen entity | Most recent source is most accurate | -| `keep_most_complete` | Keep the entity with the most non-null properties | Default — maximizes data richness | -| `keep_highest_confidence` | Keep the entity with the highest confidence score | Extraction pipelines produce confidence scores | -| `merge_all` | Merge all properties; combine non-conflicting fields | You want every known alias, tag, and label | +| Strategy | Behavior | +| -------- | -------- | +| `keep_first` | Keep the first entity in each duplicate group | +| `keep_last` | Keep the most recently seen entity | +| `keep_most_complete` | Keep the entity with the most non-null properties | +| `union` | Merge all properties — non-conflicting fields combined | +| `voting` | Most common property value wins | -### Per-Property Merge Rules - -Use `MergeStrategyManager` to apply different `MergeStrategy` values per property: +Fine-grained per-property merge rules: ```python -from semantica.deduplication import MergeStrategyManager, MergeStrategy +from semantica.deduplication import EntityMerger, PropertyMergeRule -manager = MergeStrategyManager() -manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) -manager.add_property_rule("confidence", MergeStrategy.KEEP_HIGHEST_CONFIDENCE) -manager.add_property_rule("created_at", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("updated_at", MergeStrategy.KEEP_LAST) - -merged_entity = manager.merge_entities(duplicate_group) +merger = EntityMerger( + property_rules={ + "name": PropertyMergeRule.KEEP_FIRST, + "aliases": PropertyMergeRule.UNION, + "description": PropertyMergeRule.KEEP_LONGEST, + } +) ``` -| Strategy | Behaviour | -| -------- | --------- | -| `KEEP_FIRST` | Value from the first entity in the group | -| `KEEP_LAST` | Value from the last entity | -| `KEEP_MOST_COMPLETE` | Entity with the most non-null fields | -| `KEEP_HIGHEST_CONFIDENCE` | Entity with the highest confidence score | -| `MERGE_ALL` | Combine all properties from every entity in the group | - ## SimilarityCalculator -Compute multi-factor similarity scores — useful for debugging why two entities were (or were not) detected as duplicates: +Compute multi-factor similarity scores between entity pairs: ```python from semantica.deduplication import SimilarityCalculator -calc = SimilarityCalculator() +calc = SimilarityCalculator() + score = calc.calculate_similarity(entity_a, entity_b) +# → SimilarityResult(score=0.91, components={...}) print(score.score) # overall score 0.0–1.0 -print(score.components["label"]) # label similarity contribution -print(score.components["embedding"]) # semantic similarity contribution -print(score.components["property"]) # property overlap contribution +print(score.components["label"]) # label similarity +print(score.components["embedding"]) # semantic similarity +print(score.components["property"]) # property overlap +``` +Individual string and vector metrics: + +```python lev = calc.levenshtein("Apple Inc.", "Apple Inc") jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs") cos = calc.cosine_similarity(embedding_a, embedding_b) @@ -232,200 +146,57 @@ Build entity clusters for large-scale batch deduplication: ```python from semantica.deduplication import ClusterBuilder -builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" -result = builder.build_clusters(entities, similarity_threshold=0.85) - -print(f"Original: {len(entities)} entities") -print(f"Clusters: {len(result.clusters)} groups") -print(f"Singletons: {result.singleton_count}") +builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" +result = builder.build_clusters(entities, similarity_threshold=0.85) +print(f"Clusters: {len(result.clusters)}") for cluster in result.clusters: - print(f" [{cluster.id}] members={cluster.members} cohesion={cluster.cohesion:.2f}") -``` - -### Union-Find vs Hierarchical - -```python -# Union-Find — O(n·α(n)), scales to millions; use for production -builder = ClusterBuilder(algorithm="union_find") - -# Hierarchical — tighter clusters; use when quality matters more than speed -builder = ClusterBuilder(algorithm="hierarchical", linkage="average") -result = builder.build_clusters(entities, similarity_threshold=0.85) -``` - -**Key behaviours:** -- Union-Find groups entities transitively: if A≈B and B≈C, all three land in one cluster even if A and C are only 0.70 similar -- Hierarchical with `linkage="average"` avoids chaining by requiring average similarity across all pairs to meet the threshold - -### Cluster Quality Metrics - -```python -print(f"Silhouette score: {result.quality.silhouette_score:.3f}") -# → -1.0 to 1.0; above 0.5 is good, above 0.7 is excellent - -print(f"Avg cohesion: {result.quality.avg_cohesion:.3f}") -print(f"Avg separation: {result.quality.avg_separation:.3f}") -``` - -## MergeStrategyManager - -Define complex, reusable merge configurations once and apply them across multiple operations: - -```python -from semantica.deduplication import MergeStrategyManager, MergeStrategy - -manager = MergeStrategyManager() -manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) -manager.add_property_rule("description", MergeStrategy.KEEP_MOST_COMPLETE) -manager.add_property_rule("confidence", MergeStrategy.KEEP_HIGHEST_CONFIDENCE) -manager.add_property_rule("sources", MergeStrategy.MERGE_ALL) -manager.add_property_rule("created_at", MergeStrategy.KEEP_FIRST) -manager.add_property_rule("updated_at", MergeStrategy.KEEP_LAST) - -merged_entity = manager.merge_entities(duplicate_group) + print(f" [{cluster.id}] {cluster.members} — cohesion: {cluster.cohesion:.2f}") ``` ## Blocking Strategies +Blocking reduces the O(n²) pairwise comparison problem to a manageable candidate set: + ```python detector = DuplicateDetector( - blocking_strategy="token", # "token" | "phonetic" | "ngram" + blocking_strategy="token", # "token" | "phonetic" | "ngram" blocking_threshold=0.6, similarity_threshold=0.85 ) ``` -| Blocking Strategy | How It Works | Best For | -| ----------------- | ------------ | -------- | -| `token` | Shared token overlap (default) | General entity names | -| `phonetic` | Soundex/Metaphone phonetic codes | Names with spelling variations | -| `ngram` | Character n-gram overlap | Short strings, typos | - ## Custom Similarity Functions +Register domain-specific similarity logic: + ```python from semantica.deduplication import method_registry -def drug_name_similarity(entity_a, entity_b) -> float: - compound_a = entity_a.properties.get("active_compound", "") - compound_b = entity_b.properties.get("active_compound", "") - return 1.0 if compound_a == compound_b else 0.0 +def drug_name_similarity(entity_a, entity_b): + # Match drug names by active compound + return score # 0.0 to 1.0 method_registry.register("similarity", "drug_name", drug_name_similarity) -detector = DuplicateDetector(similarity_method="drug_name", similarity_threshold=0.90) +detector = DuplicateDetector(similarity_method="drug_name") ``` -## Schemas - - - +## Convenience Functions ```python -@dataclass -class Cluster: - id: str - members: List[str] # entity IDs in this duplicate group - cohesion: float # mean pairwise similarity within the cluster (0–1) - centroid: str # member ID closest to the cluster centroid +from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity -@dataclass -class ClusterResult: - clusters: List[Cluster] - singleton_count: int # entities with no duplicates found - merge_candidates: int # clusters with > 1 member - quality: ClusterQuality +# Quick detection +duplicates = detect_duplicates(entities, method="semantic_v2", similarity_threshold=0.85) + +# Quick merge +merged = merge_entities(entities, duplicates, method="keep_most_complete") + +# Quick similarity +score = calculate_similarity(entity_a, entity_b, method="hybrid_v2") ``` - - - -```python -@dataclass -class ClusterQuality: - silhouette_score: float # -1 to 1; above 0.5 is good - avg_cohesion: float # mean within-cluster similarity - avg_separation: float # mean between-cluster distance -``` - - - - -## End-to-End Pipeline - - - - ```python - entities = load_entities_from_sources(["crunchbase", "wikipedia", "internal_db"]) - print(f"Loaded: {len(entities)} raw entities") - ``` - - - ```python - from semantica.deduplication import DuplicateDetector - - detector = DuplicateDetector(similarity_threshold=0.85) - duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") - print(f"Found: {len(duplicates)} duplicate pairs") - ``` - - - ```python - from semantica.deduplication import MergeStrategyManager, MergeStrategy - - manager = MergeStrategyManager() - manager.add_property_rule("name", MergeStrategy.KEEP_FIRST) - manager.add_property_rule("aliases", MergeStrategy.MERGE_ALL) - manager.add_property_rule("description", MergeStrategy.KEEP_MOST_COMPLETE) - ``` - - - ```python - from semantica.deduplication import EntityMerger - - merger = EntityMerger() - merged = merger.merge_duplicates(entities, preserve_provenance=True) - print(f"Result: {len(merged)} canonical entities") - - for entity in merged: - if hasattr(entity, "source_entities"): - print(f"{entity.label} merged from: {entity.source_entities}") - ``` - - - -## Tips and Common Pitfalls - - - **Normalize before deduplicating.** Run `TextNormalizer` and `EntityNormalizer` first. "APPLE INC." and "apple inc." will score 0.50 on string similarity but 1.0 after case normalization. See the [Normalize](normalize) module. - - - - **Too many false positives?** Raise `similarity_threshold` by 0.05 or switch from `blocking_v2` to `hybrid_v2` to add semantic precision on top of string matching. - - - - **Too many missed duplicates?** Lower `similarity_threshold` by 0.05, or switch to `semantic_v2` to catch entities that use different words ("ML" vs "Machine Learning"). - - - - **Union-Find over-merges?** The transitive grouping means weak chains can connect unrelated entities. Switch to `hierarchical` clustering with `linkage="average"` to require that every pair within a cluster meets the threshold — not just a chain of nearby pairs. - - - - **Preserve provenance on merge.** Set `preserve_provenance=True` so you can always trace which source contributed each property to the merged entity. Critical for audit and debugging. - - - - **Inspect similarity components.** When a pair is flagged and you're not sure why, use `SimilarityCalculator.calculate_similarity()` to see the per-component breakdown (`label`, `embedding`, `property`) and identify which factor is driving the match. - - - - **Don't deduplicate after building the graph.** Deduplicate entities *before* `GraphBuilder` ingests them. Merging inside a live graph is possible but requires tracking and rewriting all relationship endpoints. - - Detect value conflicts between non-duplicate entities. @@ -434,9 +205,9 @@ class ClusterQuality: GraphBuilder uses deduplication during construction. - Normalize entity names before deduplication for better accuracy. + Normalize entity names before deduplication. - Track merged entity lineage and source attribution. + Track merged entity lineage. diff --git a/docs/reference/export.md b/docs/reference/export.md index 76b5b341..8c4d67f7 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -6,6 +6,30 @@ icon: "file-export" `semantica.export` serializes knowledge graphs to every downstream format — semantic web standards, analytics pipelines, graph databases, and vector stores. All exporters share a consistent `export(graph, path, format)` interface. +## Exported Classes + +```python +from semantica.export import ( + RDFExporter, # Turtle, JSON-LD, N-Triples, RDF/XML + ParquetExporter, # columnar Parquet (Spark, BigQuery, Databricks, Snowflake) + LPGExporter, # Cypher CREATE/MERGE for Neo4j / Memgraph + ArangoAQLExporter, # AQL INSERT for ArangoDB + GraphExporter, # GraphML, GEXF, Graphviz DOT + OWLExporter, # OWL 2.0 in Turtle, XML, JSON-LD + CSVExporter, # flat CSV nodes + edges + VectorExporter, # embedding vectors as JSON, NumPy, or FAISS + ArrowExporter, # Apache Arrow IPC (zero-copy transfer) + DistanceExporter, # semantic distance matrices and ego-graphs + ReportGenerator, # human-readable analytics reports (HTML, Markdown, JSON) + NamespaceManager, # register and resolve RDF namespace prefixes + SemanticNetworkYAMLExporter, # YAML semantic network export + # Convenience functions + export_rdf, export_parquet, export_csv, export_lpg, + export_arango, export_graph, export_owl, export_vector, + export_arrow, generate_report, +) +``` + ## What You Get diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 6ae284ee..ec928734 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -1,33 +1,47 @@ --- title: "Knowledge Graph Module" -description: "Graph construction, temporal models, analytics, and distance intelligence." +description: "Graph construction, temporal models, analytics, similarity scoring, and structural embeddings." icon: "diagram-project" --- -`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and Distance Intelligence (v0.5.0). +`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and structural similarity scoring. + +## Exported Classes + +```python +from semantica.kg import ( + KnowledgeGraph, # core graph data structure + GraphBuilder, # construct from entities + relationships + GraphBuilderWithProvenance, # auto-tracks provenance for every node/edge + EntityResolver, # entity deduplication during construction + GraphAnalyzer, # temporal evolution, diversity metrics + GraphValidator, # schema and constraint validation + TemporalGraphQuery, # point-in-time snapshots, diffs, interval queries + TemporalPatternDetector, # sequence/cycle/trend detection + TemporalVersionManager, # snapshot creation and version comparison + TemporalNormalizer, # normalize timestamps across granularities + BiTemporalFact, # bi-temporal fact model (transaction + valid time) + CentralityCalculator, # degree, betweenness, closeness, PageRank, eigenvector + CommunityDetector, # Louvain, Leiden, Label Propagation, K-Clique + PathFinder, # Dijkstra, A*, BFS, K-Shortest paths + LinkPredictor, # Preferential Attachment, Jaccard, Adamic-Adar + NodeEmbedder, # Node2Vec, DeepWalk structural embeddings + SimilarityCalculator, # cosine, Euclidean, Manhattan, correlation similarity + ConnectivityAnalyzer, # connected components, bridges, density + ProvenanceTracker, # source tracking and lineage management +) +``` ## What You Get - - - Construct graphs from entities and relationships with automatic entity merging. - - - Time-aware queries — filter by `valid_from`/`valid_until`, range queries, and evolution analysis. - - - Connected components, bridge detection, and edge density analysis. - - - PageRank, degree, betweenness, closeness, and eigenvector centrality. - - - Louvain, Leiden, Label Propagation, and K-Clique community detection. - - - Dijkstra, A\*, BFS, and K-Shortest path algorithms. - - +- **`GraphBuilder`** — construct graphs from entities and relationships with automatic entity merging +- **`TemporalGraphQuery`** — time-aware point-in-time snapshots, diffs, and Allen interval queries (v0.4.0) +- **`SimilarityCalculator`** — cosine, Euclidean, Manhattan, and correlation similarity scoring +- **`CentralityCalculator`** — PageRank, degree, betweenness, closeness, eigenvector centrality +- **`CommunityDetector`** — Louvain, Leiden, Label Propagation, K-Clique community detection +- **`PathFinder`** — Dijkstra, A\*, BFS, K-Shortest path algorithms +- **`LinkPredictor`** — Preferential Attachment, Jaccard, Adamic-Adar link prediction +- **`NodeEmbedder`** — Node2Vec, DeepWalk structural embeddings For conflict detection and advanced entity resolution, use `semantica.conflicts` and `semantica.deduplication` alongside this module. @@ -35,54 +49,6 @@ icon: "diagram-project" Knowledge graph entity and relation structure: Person, Organization, Location, Date nodes with typed labeled edges -## Quick Start - - - - ```python - from semantica.kg import GraphBuilder - - builder = GraphBuilder(merge_entities=True) - kg = builder.build(entities=entities, relationships=relationships) - - print(f"Nodes: {kg.node_count}, Edges: {kg.edge_count}") - ``` - - - ```python - from semantica.kg import CentralityCalculator - - calc = CentralityCalculator() - pagerank = calc.calculate_pagerank(kg, damping_factor=0.85) - top_10 = calc.get_top_nodes(pagerank, top_k=10) - - for node_id, score in top_10: - print(f" {node_id}: {score:.4f}") - ``` - - - ```python - from semantica.kg import CommunityDetector - - detector = CommunityDetector() - communities = detector.detect_communities(kg, algorithm="louvain") - metrics = detector.calculate_community_metrics(kg, communities) - - print(f"Communities: {len(communities)}") - ``` - - - ```python - from semantica.graph_store import GraphStore - - store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", - user="neo4j", password="password") - store.add_nodes_bulk(kg.entities, batch_size=1000) - store.add_edges_bulk(kg.relationships, batch_size=1000) - ``` - - - ## GraphBuilder Constructs knowledge graphs from extracted entities and relationships: @@ -91,7 +57,7 @@ Constructs knowledge graphs from extracted entities and relationships: from semantica.kg import GraphBuilder builder = GraphBuilder(merge_entities=True) -kg = builder.build(entities=entities, relationships=relationships) +kg = builder.build(entities=entities, relationships=relationships) ``` | Method | Description | @@ -100,230 +66,166 @@ kg = builder.build(entities=entities, relationships=relationships) | `build_single_source(data)` | Build graph from a single data source | | `merge_entities()` | Deduplicate and merge entities during construction | - - Always use `merge_entities=True` in production. Without it, "Steve Jobs" extracted from five different documents creates five separate person nodes. `GraphBuilder(merge_entities=True)` uses edit distance matching to consolidate them at build time. - +## Temporal Knowledge Graphs (v0.4.0) -## Temporal Queries - -Use `TemporalGraphQuery` to run time-aware queries against a knowledge graph whose relationships carry `valid_from` / `valid_until` fields: +Use `TemporalGraphQuery` to attach `valid_from`/`valid_until` windows and query time-aware graphs: ```python -from semantica.kg import TemporalGraphQuery +from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalVersionManager from datetime import datetime -query_engine = TemporalGraphQuery( - enable_temporal_reasoning=True, - temporal_granularity="day", -) +# Build a time-aware graph +builder = GraphBuilder() +kg = builder.build(sources=[ + { + "entities": [ + {"id": "alice", "type": "Person"}, + {"id": "acme_corp", "type": "Organization"}, + ], + "relationships": [ + { + "source": "alice", "target": "acme_corp", "type": "ceo_of", + "valid_from": "2020-01-01", + "valid_until": "2023-06-01", + } + ] + } +]) -# Point-in-time query — returns only edges valid at the given time -result_2021 = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15)) -result_2023 = query_engine.query_at_time(kg, query="", at_time=datetime(2023, 1, 1)) +# Point-in-time snapshot +query = TemporalGraphQuery(kg) +snapshot_2021 = query.at_time("2021-06-15") +snapshot_2023 = query.at_time("2023-01-01") -# Compare what changed between two snapshots -added = [ - r for r in result_2023["relationships"] - if r not in result_2021["relationships"] -] -print(f"New edges since 2021: {len(added)}") +# Diff between two snapshots +diff = query.diff("2020-01-01", "2023-01-01") +print(f"New nodes since 2020: {len(diff.get('added_nodes', []))}") -# Range query — edges valid within a time window -range_result = query_engine.query_time_range(kg, query="", start_time=datetime(2020, 1, 1), end_time=datetime(2023, 1, 1)) - -# Evolution analysis -evolution = query_engine.analyze_evolution(kg) +# Versioned snapshots +versioner = TemporalVersionManager() +versioner.create_snapshot(kg, version_label="2024-Q1") ``` - - Relationships added without `valid_from`/`valid_until` are treated as **always-valid**. For historical data, always attach timestamps — otherwise point-in-time queries return misleading results. - +Supports all 13 Allen interval algebra relations (before, after, meets, overlaps, during, starts, finishes, equals, and their inverses). OWL-Time export available. + +## Similarity Scoring + +`SimilarityCalculator` computes cosine, Euclidean, Manhattan, and correlation similarity between node embeddings: + +```python +from semantica.kg import SimilarityCalculator, NodeEmbedder + +# First compute structural embeddings +embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) +embeddings = embedder.compute_embeddings(kg, ["Person", "Organization"], ["RELATED_TO"]) + +# Then compare nodes by embedding similarity +calc = SimilarityCalculator() +score = calc.cosine_similarity(embeddings["Apple Inc."], embeddings["Google"]) +print(f"Apple–Google structural similarity: {score:.3f}") + +# Find structurally similar nodes +similar = embedder.find_similar_nodes(kg, "Apple Inc.", top_k=5) +for node in similar: + print(f"{node['id']}: {node['score']:.3f}") +``` ## Graph Analytics - - - Identify the most structurally important nodes in your graph: +### Centrality Analysis - ```python - from semantica.kg import CentralityCalculator +```python +from semantica.kg import CentralityCalculator - calc = CentralityCalculator() +calculator = CentralityCalculator() - pagerank = calc.calculate_pagerank(kg, damping_factor=0.85) - degree = calc.calculate_degree_centrality(kg) - betweenness = calc.calculate_betweenness_centrality(kg) - closeness = calc.calculate_closeness_centrality(kg) - eigenvector = calc.calculate_eigenvector_centrality(kg) - all_metrics = calc.calculate_all_centrality(kg) +centrality = calculator.calculate_degree_centrality(graph) +pagerank = calculator.calculate_pagerank(graph, damping_factor=0.85) +betweenness = calculator.calculate_betweenness_centrality(graph) +closeness = calculator.calculate_closeness_centrality(graph) +eigenvector = calculator.calculate_eigenvector_centrality(graph) +all_metrics = calculator.calculate_all_centrality(graph) - top_nodes = calc.get_top_nodes(pagerank, top_k=10) - ``` +top_nodes = calculator.get_top_nodes(centrality, top_k=10) +``` - | Measure | Best For | - | ------- | -------- | - | PageRank | Overall importance (link-based) | - | Degree | Most connected nodes | - | Betweenness | Bridge / bottleneck nodes | - | Closeness | Fastest to reach all others | - | Eigenvector | Connected to other important nodes | - - - Partition the graph into thematically dense clusters: +| Method | Algorithm | +| ------ | --------- | +| `calculate_degree_centrality()` | Degree-based importance | +| `calculate_betweenness_centrality()` | Bridge-based importance (bottleneck nodes) | +| `calculate_closeness_centrality()` | Distance-based importance | +| `calculate_eigenvector_centrality()` | Influence-based importance | +| `calculate_pagerank()` | Link-based importance (PageRank) | +| `calculate_all_centrality()` | All measures at once | - ```python - from semantica.kg import CommunityDetector +### Community Detection - detector = CommunityDetector() +```python +from semantica.kg import CommunityDetector - # Louvain — fast, high quality (default) - communities = detector.detect_communities(kg, algorithm="louvain") +detector = CommunityDetector() - # Leiden — higher quality, slower - leiden_communities = detector.detect_communities_leiden(kg, resolution=1.2) +# Louvain (default — fast, high quality) +communities = detector.detect_communities(graph, algorithm="louvain") - metrics = detector.calculate_community_metrics(kg, communities) - print(f"Communities: {len(communities)}") - ``` +# Leiden (higher quality, slower) +leiden_communities = detector.detect_communities_leiden(graph, resolution=1.2) - Algorithms available: **Louvain**, **Leiden**, **Label Propagation**, **K-Clique Communities**. +metrics = detector.calculate_community_metrics(graph, communities) +``` - - Community detection finds thematic clusters — often corresponding to real-world subject groups. Use cluster membership as context boundaries for GraphRAG retrieval. - - - - Find shortest and alternative paths between nodes: +Algorithms: Louvain, Leiden, Label Propagation, K-Clique Communities. - ```python - from semantica.kg import PathFinder +### Path Finding - finder = PathFinder() +```python +from semantica.kg import PathFinder - path = finder.dijkstra_shortest_path(kg, "node_a", "node_b") - paths = finder.all_shortest_paths(kg, "source", "target") - k_paths = finder.find_k_shortest_paths(kg, "source", "target", k=3) - ``` +finder = PathFinder() - Algorithms: **Dijkstra**, **A\***, **BFS**, **All Shortest Paths**, **K-Shortest Paths**. - - - Analyse graph structure — components, bridges, and density: +path = finder.dijkstra_shortest_path(graph, "node_a", "node_b") +paths = finder.all_shortest_paths(graph, "source", "target") +k_paths = finder.find_k_shortest_paths(graph, "source", "target", k=3) +``` - ```python - from semantica.kg import ConnectivityAnalyzer +Algorithms: Dijkstra, A\*, BFS, All Shortest Paths, K-Shortest Paths. - analyzer = ConnectivityAnalyzer() - components = analyzer.find_connected_components(kg) - density = analyzer.calculate_density(kg) - bridges = analyzer.find_bridges(kg) +### Link Prediction - print(f"Components: {len(components)}, Largest: {len(components[0])} nodes") - print(f"Density: {density:.4f}") - print(f"Bridges: {bridges}") - ``` +```python +from semantica.kg import LinkPredictor - | Method | Returns | Description | - | ------ | ------- | ----------- | - | `find_connected_components(kg)` | `List[List[str]]` | Groups of mutually reachable nodes | - | `calculate_density(kg)` | `float` | Edge density (actual / possible edges) | - | `find_bridges(kg)` | `List[str]` | Nodes whose removal disconnects the graph | - - - Predict which edges are likely missing from the graph: +predictor = LinkPredictor(method="preferential_attachment") +links = predictor.predict_links(graph, top_k=20) +score = predictor.score_link(graph, "node_a", "node_b") +``` - ```python - from semantica.kg import LinkPredictor +Algorithms: Preferential Attachment, Common Neighbors, Jaccard, Adamic-Adar, Resource Allocation. - predictor = LinkPredictor(method="preferential_attachment") - links = predictor.predict_links(kg, top_k=20) - score = predictor.score_link(kg, "node_a", "node_b") - ``` +### Node Embeddings - Algorithms: **Preferential Attachment**, **Common Neighbors**, **Jaccard**, **Adamic-Adar**, **Resource Allocation**. - - - Compute structural embeddings for similarity search and downstream ML: +```python +from semantica.kg import NodeEmbedder - ```python - from semantica.kg import NodeEmbedder +embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) +embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) +similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) +``` - embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) - embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) - similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) - ``` - - Algorithms: **Node2Vec**, **DeepWalk**, **Word2Vec**. - - +Algorithms: Node2Vec, DeepWalk, Word2Vec. ## Algorithm Summary | Category | Algorithms | Use Cases | | -------- | ---------- | --------- | | Node Embeddings | Node2Vec, DeepWalk, Word2Vec | Structural similarity, node representation | +| Similarity | Cosine, Euclidean, Manhattan, Correlation | Node matching, recommendation | | Path Finding | Dijkstra, A\*, BFS, K-Shortest | Route planning, network analysis | | Link Prediction | Preferential Attachment, Jaccard, Adamic-Adar | Network completion | | Centrality | Degree, Betweenness, Closeness, PageRank | Influence analysis | | Community Detection | Louvain, Leiden, Label Propagation | Social clustering | | Connectivity | Components, Bridges, Density | Network robustness | -## SeedManager - -Load and inject curated seed data into a knowledge graph: - -```python -from semantica.kg import SeedManager - -manager = SeedManager() -seed_data = manager.load_seed("seeds/domain_entities.json") -normalized = manager.normalize(seed_data, source="manual_curation_v1") - -builder = GraphBuilder(merge_entities=True) -kg = builder.build(normalized + extracted_sources) -``` - -## MethodRegistry - -Register custom KG construction methods and dispatch by name: - -```python -from semantica.kg import method_registry - -def my_kg_builder(entities, relationships, **kwargs): - filtered = [e for e in entities if e["confidence"] >= 0.9] - return {"entities": filtered, "relationships": relationships} - -method_registry.register("build", "high_confidence", my_kg_builder) - -# Dispatch by name via the registry -result = method_registry.execute("build", "high_confidence", entities=entities, relationships=relationships) -``` - -## ProvenanceTracker - -Track entity and relationship lineage within a knowledge graph: - -```python -from semantica.kg import ProvenanceTracker - -tracker = ProvenanceTracker() - -tracker.track_entity( - entity_id="apple_inc", - source="sec_filing_2024q1.pdf", - source_location="page 3, paragraph 2", - source_quote="Apple Inc. reported revenue of...", - confidence=0.98, -) - -lineage = tracker.get_lineage("apple_inc") -for entry in lineage.entries: - print(f" Source: {entry.source} ({entry.timestamp})") -``` - -For full W3C PROV-O compliance and provenance export, see the [Provenance module](provenance). - ## Configuration ```yaml @@ -337,24 +239,6 @@ kg: default_validity: infinite ``` -## Tips and Common Pitfalls - - - **Deduplicate before `GraphBuilder`, not after.** It's far easier to merge entities before they become nodes than to update all relationship endpoints after the fact. Run `DuplicateDetector` on extracted entities before calling `builder.build()`. - - - - **PageRank identifies your most connected, important nodes.** If you're not sure which entities in your graph are the most structurally significant, `CentralityCalculator.calculate_pagerank()` gives you a ranked list — useful for GraphRAG context anchoring. - - - - **Community detection finds thematic clusters.** `CommunityDetector` with Louvain partitions your graph into clusters of densely-connected nodes — often corresponding to real-world thematic groups. Use these clusters for exploratory analysis and to scope GraphRAG retrieval. - - - - **`ProvenanceTracker` links entities back to their source documents.** Use it during graph construction so you can always answer "where did this fact come from?" — critical for compliance and for debugging incorrect graph data. - - Persist graphs in Neo4j, FalkorDB, or Apache AGE. diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 0db513d5..ddd74494 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -1,89 +1,35 @@ --- title: "LLMs Module" -description: "Unified interface for Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace — swap providers with a one-line change." +description: "Unified interface for Groq, OpenAI, LiteLLM (Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, 100+ models), and HuggingFace." icon: "microchip" --- -`semantica.llms` gives every Semantica module a single, consistent interface to 9+ LLM providers. Every extractor, reasoning engine, and context graph accepts any provider through the same `llm_provider=` parameter — swap Groq for Anthropic, or a cloud API for a local Ollama model, by changing one line. +`semantica.llms` provides a single consistent API across every major LLM provider. Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoning engines, and agents. + +## Exported Classes + +```python +from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM +``` + +| Class | Provider | API Key Required | +| ----- | -------- | ---------------- | +| `Groq` | Groq Cloud | `GROQ_API_KEY` | +| `OpenAI` | OpenAI / any OpenAI-compatible gateway | `OPENAI_API_KEY` | +| `LiteLLM` | 100+ providers via LiteLLM routing | Depends on model | +| `HuggingFaceLLM` | Local HuggingFace Transformers | None (local) | + + + **Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, Cohere, and 90+ others** are all available via `LiteLLM` using their model-string prefix. See the [LiteLLM section](#litellm-100-providers) below. + ## What You Get - - - Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace — all behind one interface. - - - `complete()`, `chat()`, and `stream()` work identically across all providers — swap with a one-line change. - - - One class, every provider — Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, and 90+ more via LiteLLM model strings. - - - Ollama and HuggingFace run fully on-premise — no API key, no data leaves your machine, air-gap compatible. - - - Token-by-token output via `stream()` for responsive agent pipelines and live UI updates. - - - Configurable `max_retries` with exponential backoff. Typed exceptions: `LLMAuthenticationError`, `LLMRateLimitError`, `LLMContextLengthError`. - - - -## Installation - -The base install includes Groq and DeepSeek. Other providers require optional extras: - -| Provider | Install Command | API Key Required | -| -------- | --------------- | ---------------- | -| Groq | `pip install semantica` | Yes — `GROQ_API_KEY` | -| DeepSeek | `pip install semantica` | Yes — `DEEPSEEK_API_KEY` | -| Novita AI | `pip install semantica` | Yes — `NOVITA_API_KEY` | -| OpenAI | `pip install "semantica[llm-openai]"` | Yes — `OPENAI_API_KEY` | -| Anthropic | `pip install "semantica[llm-anthropic]"` | Yes — `ANTHROPIC_API_KEY` | -| Gemini | `pip install "semantica[llm-gemini]"` | Yes — `GOOGLE_API_KEY` | -| Ollama | `pip install "semantica[llm-ollama]"` | No — local server | -| LiteLLM | `pip install "semantica[llm-litellm]"` | Varies by target | -| HuggingFace | `pip install "semantica[llm-huggingface]"` | No — local weights | -| All providers | `pip install "semantica[all]"` | Varies | - -## Quick Start - - - - ```python - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - ``` - - - ```python - from semantica.semantic_extract import NERExtractor - - ner = NERExtractor(method="llm", llm_provider=llm) - ``` - - - ```python - entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") - ``` - - - ```python - from semantica.llms import LiteLLM - from semantica.core import ConfigManager - - config = ConfigManager("config.yaml") - # LiteLLM model strings include the provider prefix: - # "anthropic/claude-opus-4-7", "gemini/gemini-1.5-pro", "ollama/llama3.2" - llm = LiteLLM( - model=config.get("llm_provider.model"), - api_key=config.get("llm_provider.api_key"), - ) - ``` - - +- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application code changes +- **`LiteLLM`** — single class for 100+ providers using model-string routing +- **Local models** — `HuggingFaceLLM` runs fully on-premise, no API key +- **Streaming** — token-by-token output for low-latency UX +- **Custom gateways** — point `OpenAI` at any OpenAI-compatible endpoint via `base_url` ## Providers @@ -94,13 +40,12 @@ from semantica.llms import Groq import os llm = Groq( - model="llama-3.3-70b-versatile", # default and recommended + model="llama-3.3-70b-versatile", # default api_key=os.getenv("GROQ_API_KEY"), - temperature=0.0, max_tokens=64000, - max_retries=3, - timeout=60, + temperature=0.0, ) +# Best for: high-throughput extraction, fast inference at low cost ``` ```python OpenAI @@ -111,267 +56,91 @@ llm = OpenAI( model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY"), temperature=0.0, - max_tokens=4096, - max_retries=3, - timeout=120, - organization=None, # optional org ID ) +# Best for: general purpose, function calling, JSON mode ``` -```python Anthropic (via LiteLLM) +```python LiteLLM (100+ providers) from semantica.llms import LiteLLM import os -# Anthropic Claude is accessed via LiteLLM using the "anthropic/" prefix -llm = LiteLLM( - model="anthropic/claude-opus-4-7", - api_key=os.getenv("ANTHROPIC_API_KEY"), - max_tokens=8192, - temperature=0.0, -) +# pip install "semantica[llm-litellm]" + +# Anthropic Claude +llm = LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTHROPIC_API_KEY")) + +# Google Gemini +llm = LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY")) + +# Ollama (local — no API key) +llm = LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434") + +# DeepSeek +llm = LiteLLM(model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY")) + +# Azure OpenAI +llm = LiteLLM(model="azure/gpt-4o", api_key=os.getenv("AZURE_API_KEY")) + +# AWS Bedrock +llm = LiteLLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0") + +# Novita AI +llm = LiteLLM(model="novita/deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY")) ``` -```python Gemini (via LiteLLM) -from semantica.llms import LiteLLM -import os - -# Google Gemini is accessed via LiteLLM using the "gemini/" prefix -llm = LiteLLM( - model="gemini/gemini-1.5-pro", - api_key=os.getenv("GOOGLE_API_KEY"), - temperature=0.0, - max_tokens=8192, -) -``` - -```python Ollama / Local (via LiteLLM) -from semantica.llms import LiteLLM - -# Ollama local models via LiteLLM using the "ollama/" prefix -llm = LiteLLM( - model="ollama/llama3.2", - base_url="http://localhost:11434", # default Ollama address - temperature=0.0, - timeout=180, # local models can be slower; increase for large models -) -# No API key — model runs entirely on your machine -``` - -```python DeepSeek (via Groq or LiteLLM) -from semantica.llms import Groq # Groq hosts DeepSeek models -import os - -llm = Groq( - model="deepseek-r1-distill-llama-70b", - api_key=os.getenv("GROQ_API_KEY"), - temperature=0.0, - max_tokens=4096, -) - -# Or via LiteLLM for the native DeepSeek endpoint: -from semantica.llms import LiteLLM -llm = LiteLLM( - model="deepseek/deepseek-chat", - api_key=os.getenv("DEEPSEEK_API_KEY"), - temperature=0.0, - max_tokens=4096, -) -``` - -```python Novita AI (via LiteLLM) -from semantica.llms import LiteLLM -import os - -# Novita AI via LiteLLM using the "novita/" prefix -llm = LiteLLM( - model="novita/deepseek/deepseek-v3", - api_key=os.getenv("NOVITA_API_KEY"), - temperature=0.0, - max_tokens=4096, -) -``` - -```python LiteLLM (100+ models) -from semantica.llms import LiteLLM -import os - -llm = LiteLLM( - model="gpt-4o", # any LiteLLM model string - api_key=os.getenv("OPENAI_API_KEY"), - temperature=0.0, - max_tokens=4096, -) -# Supports: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, Together AI, and 90+ more -# Use the LiteLLM model string format: "anthropic/claude-opus-4-7", "bedrock/anthropic.claude-v2" -``` - -```python HuggingFace (Local) +```python HuggingFaceLLM (Local) from semantica.llms import HuggingFaceLLM llm = HuggingFaceLLM( model="mistralai/Mistral-7B-Instruct-v0.3", - device="cuda", # "cpu" | "cuda" | "mps" (Apple Silicon) + device="cuda", # "cpu" | "cuda" | "mps" max_new_tokens=512, temperature=0.1, - load_in_4bit=True, # enable 4-bit quantisation to reduce VRAM ) -# No API key — weights downloaded from Hugging Face Hub (or loaded from local path) +# Bring your own model — full local control, no API key ``` -## Constructor Parameters +## LiteLLM — 100+ Providers -### Common Parameters - -| Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | -| `model` | `str` | Provider default | Model identifier string | -| `api_key` | `str` | `None` | API key — reads from environment if omitted | -| `temperature` | `float` | `0.0` | Sampling temperature: 0 = deterministic, 1 = creative | -| `max_tokens` | `int` | Provider default | Maximum tokens in the response | -| `max_retries` | `int` | `3` | Number of retry attempts on transient failures | -| `timeout` | `int` | `60` | Request timeout in seconds | -| `base_url` | `str` | Provider default | Override the API endpoint — useful for proxies and gateways | - -### Provider-Specific Parameters - -| Provider | Parameter | Description | -| -------- | --------- | ----------- | -| `OpenAI` | `organization` | OpenAI organisation ID | -| `OpenAI` | `project` | OpenAI project ID | -| `LiteLLM` | `model` | Full LiteLLM model string, e.g. `"anthropic/claude-opus-4-7"`, `"gemini/gemini-1.5-pro"`, `"ollama/llama3.2"` | -| `LiteLLM` | `base_url` | Override endpoint — use for Ollama (`http://localhost:11434`) or proxies | -| `HuggingFaceLLM` | `device` | Compute device: `"cpu"` / `"cuda"` / `"mps"` | -| `HuggingFaceLLM` | `load_in_4bit` | Enable 4-bit quantisation (requires `bitsandbytes`) | -| `HuggingFaceLLM` | `max_new_tokens` | Maximum new tokens to generate (replaces `max_tokens`) | - -## Direct API Usage - -Providers can be used directly — not just through Semantica modules: +`LiteLLM` is the recommended way to access any provider not directly exported by `semantica.llms`. Use the `provider/model` string format: ```python -from semantica.llms import Groq +from semantica.llms import LiteLLM import os -llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +# Pattern: LiteLLM(model="/") +providers = { + "Anthropic": LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTHROPIC_API_KEY")), + "Gemini": LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY")), + "Ollama": LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434"), + "DeepSeek": LiteLLM(model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY")), + "Azure": LiteLLM(model="azure/gpt-4o", api_key=os.getenv("AZURE_API_KEY")), + "Bedrock": LiteLLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"), + "Cohere": LiteLLM(model="cohere/command-r-plus", api_key=os.getenv("COHERE_API_KEY")), + "Novita AI": LiteLLM(model="novita/deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY")), +} -# Single completion -response = llm.complete("What is a knowledge graph?") -print(response.text) # answer string -print(response.input_tokens) # tokens consumed by the prompt -print(response.output_tokens) # tokens in the response -print(response.model) # model that served the request - -# Multi-turn chat -messages = [ - {"role": "system", "content": "You are a knowledge graph expert."}, - {"role": "user", "content": "What is the difference between RDF and property graphs?"}, -] -response = llm.chat(messages) -print(response.text) - -# Streaming — token-by-token output -for token in llm.stream("Explain knowledge graph reasoning in 3 sentences."): - print(token, end="", flush=True) +# Every LiteLLM instance implements the same .generate() interface +response = providers["Anthropic"].generate("Explain GraphRAG in one paragraph.") ``` -## LLMResponse Object + + The full list of supported LiteLLM model strings is at [docs.litellm.ai/docs/providers](https://docs.litellm.ai/docs/providers). Use the `provider/model` format shown above. + -All three methods (`complete`, `chat`, `stream`) return a `LLMResponse` dataclass: +## Custom / Enterprise Gateways - - - -```python -@dataclass -class LLMResponse: - text: str # the generated text - model: str # model identifier that served the request - input_tokens: int # tokens consumed by the prompt - output_tokens: int # tokens in the generated response - total_tokens: int # input_tokens + output_tokens - latency_ms: float # wall-clock time for the API call in milliseconds - finish_reason: str # "stop" | "length" | "content_filter" | "tool_calls" -``` - - - - -## Error Handling - - - - -```python -from semantica.llms import Groq -from semantica.utils import SemanticaError -import os - -try: - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - response = llm.complete("Summarise this document.") - print(response.text) - -except LLMAuthenticationError as e: - # Invalid or expired API key - print(f"Authentication failed: {e}") - -except LLMRateLimitError as e: - # Rate limit hit — Semantica retries automatically up to max_retries - print(f"Rate limited after retries: {e}") - -except LLMContextLengthError as e: - # Prompt exceeds the model's context window - print(f"Prompt too long ({e.token_count} tokens, limit {e.context_limit}): {e}") - -except LLMProviderError as e: - # General provider-side error (5xx, model unavailable, etc.) - print(f"Provider error: {e}") - -except SemanticaError as e: - # Catch-all for all Semantica framework errors - print(f"Framework error: {e}") -``` - -| Exception | When Raised | -| --------- | ----------- | -| `LLMAuthenticationError` | Invalid or missing API key | -| `LLMRateLimitError` | Rate limit exceeded after all retries | -| `LLMContextLengthError` | Prompt exceeds the model's context window | -| `LLMProviderError` | Provider-side error (5xx, unavailability) | -| `LLMTimeoutError` | Request exceeded the `timeout` parameter | - - - - -## Custom and Enterprise Gateways - -Any provider that exposes an OpenAI-compatible REST API can be used by passing `base_url`: +Any OpenAI-compatible endpoint — internal routing layers, Qwen proxies, or private LLaMA deployments: ```python from semantica.llms import OpenAI -import os -# Internal LLM routing gateway llm = OpenAI( model="qwen2.5-72b", api_key=os.getenv("GATEWAY_API_KEY"), - base_url="https://llm-gateway.internal.company.com/v1", -) - -# Azure OpenAI Service -llm = OpenAI( - model="gpt-4o", - api_key=os.getenv("AZURE_OPENAI_API_KEY"), - base_url="https://my-resource.openai.azure.com/openai/deployments/gpt-4o", -) - -# Self-hosted vLLM server -llm = OpenAI( - model="meta-llama/Llama-3.1-8B-Instruct", - api_key="not-needed", - base_url="http://localhost:8000/v1", + base_url="https://my-internal-gateway.company.com/v1", ) ``` @@ -379,128 +148,49 @@ llm = OpenAI( `base_url` is validated at construction time. Non-HTTP(S) schemes raise `ValueError` to prevent SSRF attacks (fixed in v0.5.0). -## Using in Semantica Modules +## Using in Extractors -Every module that uses an LLM accepts any provider through `llm_provider=`: +All extractors accept any provider as `llm_provider=`: ```python -from semantica.llms import Groq, LiteLLM from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor -from semantica.ontology import LLMOntologyGenerator -from semantica.reasoning import Reasoner -from semantica.context import AgentContext, ContextGraph -from semantica.vector_store import VectorStore -import os -groq_llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -# Anthropic Claude via LiteLLM using the "anthropic/" prefix -claude_llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) -# Extraction — use fast Groq for high-throughput NER -ner = NERExtractor(method="llm", llm_provider=groq_llm) -rel = RelationExtractor(method="llm", llm_provider=groq_llm) -trip = TripletExtractor(method="llm", llm_provider=groq_llm) - -# Complex reasoning — use Claude for accuracy -engine = Reasoner() - -# Ontology generation from natural language -gen = LLMOntologyGenerator(llm_provider=claude_llm) +ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) +rel = RelationExtractor(method="llm", llm_provider=llm) +trip = TripletExtractor(method="llm", llm_provider=llm) ``` ## Provider Comparison -| Provider | Speed | Cost | Local | Max Context | Best For | -| -------- | ----- | ---- | ----- | ----------- | -------- | -| **Groq** | ⚡ Very fast | 💲 Low | No | 128k | High-throughput extraction, fast pipelines | -| **OpenAI** | Fast | 💲💲 Medium | No | 128k | General purpose, function calling, JSON mode | -| **Anthropic** | Fast | 💲💲 Medium | No | 200k | Complex reasoning, long documents, safety | -| **Gemini** | Fast | 💲 Low | No | 1M | Very long context, multimodal (text + image) | -| **Ollama** | Medium | Free | ✅ Yes | Varies | Privacy, air-gapped, no API key | -| **DeepSeek** | Fast | 💲 Very low | No | 64k | Coding tasks, structured analysis | -| **Novita AI** | Fast | 💲 Low | No | Varies | DeepSeek and LLaMA models, cost-effective | -| **LiteLLM** | Varies | Varies | Varies | Varies | Multi-provider routing, vendor abstraction | -| **HuggingFace** | Slow | Free | ✅ Yes | Varies | Custom and fine-tuned models, full local control | - -## Environment Variables - -| Variable | Provider | Notes | -| -------- | -------- | ----- | -| `GROQ_API_KEY` | Groq | Required for Groq cloud | -| `OPENAI_API_KEY` | OpenAI, LiteLLM | Also used for OpenAI-compatible gateways | -| `ANTHROPIC_API_KEY` | Anthropic | Required for Claude | -| `GOOGLE_API_KEY` | Gemini | Required for Gemini | -| `DEEPSEEK_API_KEY` | DeepSeek | Required for DeepSeek cloud | -| `NOVITA_API_KEY` | Novita AI | Required for Novita AI | -| `HUGGINGFACE_HUB_TOKEN` | HuggingFace | Required for gated models (optional for public models) | - -## YAML Configuration - -```yaml -# config.yaml -llm_provider: - name: "groq" - model: "llama-3.3-70b-versatile" - api_key: "${GROQ_API_KEY}" # reads from environment - temperature: 0.0 - max_tokens: 64000 - max_retries: 3 - timeout: 60 -``` - -Load it with `ConfigManager`: - -```python -from semantica.core import ConfigManager -from semantica.llms import LiteLLM - -config = ConfigManager("config.yaml") -# LiteLLM model strings carry the provider prefix — e.g. "anthropic/claude-opus-4-7", -# "gemini/gemini-1.5-pro", "ollama/llama3.2" — so provider selection is config-driven -llm = LiteLLM( - model=config.get("llm_provider.model"), - api_key=config.get("llm_provider.api_key"), - temperature=config.get("llm_provider.temperature", default=0.0), -) -``` - -## Tips and Common Pitfalls - - - **Always set `temperature=0.0` for extraction tasks.** NER, relation extraction, and triplet generation need deterministic output — any temperature above 0 introduces randomness that produces inconsistent entity types or hallucinated relationships. Reserve higher temperatures for creative or summarisation tasks. - +| Provider | Import | Speed | Cost | Local | Context | Best For | +| -------- | ------ | ----- | ---- | ----- | ------- | -------- | +| Groq | `Groq` | Very fast | Low | No | 128k | High-throughput extraction | +| OpenAI | `OpenAI` | Fast | Medium | No | 128k | General purpose, function calling | +| Anthropic | `LiteLLM(model="anthropic/...")` | Fast | Medium | No | 200k | Complex reasoning, safety | +| Gemini | `LiteLLM(model="gemini/...")` | Fast | Low | No | 1M | Long context, multimodal | +| Ollama | `LiteLLM(model="ollama/...")` | Medium | Free | Yes | Varies | Privacy, air-gapped | +| DeepSeek | `LiteLLM(model="deepseek/...")` | Fast | Very low | No | 64k | Coding, analysis | +| Azure OpenAI | `LiteLLM(model="azure/...")` | Fast | Medium | No | 128k | Enterprise, compliance | +| AWS Bedrock | `LiteLLM(model="bedrock/...")` | Fast | Varies | No | Varies | AWS-native workloads | +| HuggingFace | `HuggingFaceLLM` | Slow | Free | Yes | Varies | Custom models, BYOM | - **Set `max_retries=3` in production.** Transient rate limits and 5xx errors are normal at scale. All providers retry automatically up to `max_retries` with exponential backoff. Without retries, a single rate-limit hit fails an entire pipeline step that would have succeeded on the second attempt. - - - - **Use `LiteLLM` for config-driven pipelines.** Hard-coding `Groq(...)` in Python means changing the provider requires a code change and redeploy. `LiteLLM(model=config.get("llm_provider.model"), ...)` lets you switch from Groq to Anthropic by editing `config.yaml` with a model string like `"anthropic/claude-opus-4-7"` — no code changes. - - - - **Use `base_url` for internal gateways and Azure.** Enterprise deployments often route LLM calls through an internal proxy or Azure OpenAI Service. Pass `base_url="https://llm-gateway.internal/v1"` to `OpenAI` — you get the same Semantica integration without changing any module code. Non-HTTP schemes raise `ValueError` (SSRF protection, v0.5.0+). - - - - **Catch `LLMContextLengthError` explicitly.** If your chunking is misconfigured, a document chunk can exceed the model's context window. Catch `LLMContextLengthError` and log `e.token_count` — it tells you exactly how much to reduce your `chunk_size`. Don't let it surface as a generic failure. - - - - **Use Ollama or HuggingFace for air-gapped environments.** When data cannot leave the network, Ollama (local inference) or HuggingFace (local weights) are the only viable options. Both support the same `llm_provider=` interface — no other code changes needed. + For production extraction pipelines, Groq delivers the best throughput-to-cost ratio. For complex multi-hop reasoning, Claude Opus or GPT-4o provide the highest accuracy. - NER, relation extraction, and triplet generation with LLMs. + Use LLMs for NER and relation extraction. + + + LLM providers in Agno multi-agent teams. - LLM-backed deductive, abductive, and Datalog reasoning. - - - Generate ontologies from natural language using LLMs. + LLM-backed deductive and abductive reasoning. - GraphRAG and decision intelligence powered by LLMs. + GraphRAG uses LLMs for reasoning over knowledge graphs. diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index 52383ade..bd7a3f2c 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -1,330 +1,210 @@ --- title: "Ontology Module" -description: "Automated ontology generation, OWL export, SHACL validation, domain ontologies, and modular ontology development." +description: "Automated ontology generation, SHACL validation, OWL/RDF export, namespace management, and LLM-powered ontology generation." icon: "sitemap" --- -`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and OWL export to SHACL validation and modular ontology development. Use it for schema design, data modeling, and semantic web interoperability. +`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and SHACL validation to OWL/RDF export. Use it for schema design, data modeling, semantic web interoperability, and SHACL-based data quality validation. + +## Exported Classes + +```python +from semantica.ontology import ( + OntologyGenerator, # auto-generate from KG data (6-stage pipeline) + LLMOntologyGenerator, # LLM-powered ontology generation + OntologyEngine, # unified orchestration facade + ClassInferrer, # class discovery and hierarchy building + PropertyGenerator, # property inference and XSD type mapping + SHACLGenerator, # generate SHACL shapes from ontology + OntologyValidator, # validate graphs against SHACL shapes + SHACLValidationReport, # validation report with violations list + SHACLViolation, # individual constraint violation + OWLGenerator, # OWL/RDF serialization (Turtle, XML, JSON-LD) + OntologyEvaluator, # quality evaluation: coverage, completeness + NamespaceManager, # IRI generation and namespace prefix management + OntologyAligner, # align and merge ontologies across schemas (use OntologyEngine) + AssociativeClassBuilder, # N-ary relationship intermediate class creation + NamingConventions, # PascalCase/camelCase enforcement + DomainOntologies, # pre-built domain ontologies + ingest_ontology, # load ontology from file +) +``` ## What You Get - - - Auto-generate ontologies from existing graph data using a 6-stage pipeline. - - - Generate SHACL shapes from an ontology and validate ontologies for structural consistency. - - - Capture, manage, and validate competency questions that define ontology requirements. - - - Integrate published ontologies (schema.org, FOAF) instead of generating from scratch. - - - Pre-built domain ontologies for biomedical, finance, legal, supply chain, and more. - - - Measure coverage, completeness, and granularity — validate against competency questions. - - +- **`OntologyGenerator`** — auto-generate ontologies from existing knowledge graph data (6-stage pipeline) +- **`LLMOntologyGenerator`** — LLM-powered ontology generation for complex domains +- **`OntologyEngine`** — unified facade that orchestrates the full ontology lifecycle +- **`SHACLGenerator`** / **`OntologyValidator`** — generate SHACL shapes and validate any graph +- **`OWLGenerator`** — serialize ontologies to Turtle, RDF/XML, JSON-LD +- **`NamespaceManager`** — IRI generation, prefix management, namespace binding +- **`OntologyEvaluator`** — coverage, completeness, and granularity quality metrics +- **`AssociativeClassBuilder`** — model N-ary relationships as intermediate OWL classes -## Quick Start +## OntologyEngine (Unified Facade) - - - ```python - from semantica.ontology import OntologyGenerator +The `OntologyEngine` orchestrates the full ontology lifecycle — generation, validation, export, and versioning: - generator = OntologyGenerator() - ontology = generator.generate_from_graph(kg) - ``` - - - ```python - from semantica.ontology import OntologyValidator +```python +from semantica.ontology import OntologyEngine - validator = OntologyValidator(reasoner="hermit", check_consistency=True) - result = validator.validate(ontology) +engine = OntologyEngine(base_uri="https://example.org/ontology/") - if not result.is_valid: - for issue in result.issues: - print(f"Issue: {issue.message} (severity: {issue.severity})") - ``` - - - ```python - from semantica.ontology import OWLGenerator +# Generate ontology from KG data +ontology = engine.generate_ontology({"entities": entities, "relationships": relationships}) - owl_gen = OWLGenerator() - owl_gen.export_owl(ontology, file_path="ontology.ttl", format="turtle") - owl_gen.export_owl(ontology, file_path="ontology.owl", format="xml") - ``` - - +# Validate a graph against the generated SHACL shapes +report = engine.validate(kg) +if not report.conforms: + for v in report.violations: + print(f"{v.severity}: {v.message} on {v.node}") -## Auto-Generation — 6-Stage Pipeline +# Export to OWL Turtle +engine.export(ontology, "ontology.ttl", format="turtle") +``` -Generate an ontology automatically from your knowledge graph data: +## OntologyGenerator (6-Stage Pipeline) + +Generate a formal ontology automatically from your knowledge graph entities and relationships: ```python from semantica.ontology import OntologyGenerator -generator = OntologyGenerator() -ontology = generator.generate_from_graph(kg) +generator = OntologyGenerator(base_uri="https://example.org/ontology/") +ontology = generator.generate_ontology({ + "entities": entities, + "relationships": relationships, +}) ``` The pipeline runs through these stages in order: - - - Extracts concepts and patterns from entity/relationship data. +1. **Semantic Network Parsing** — extract concepts and patterns from entity/relationship data +2. **YAML-to-Definition** — transform patterns into intermediate class definitions +3. **Definition-to-Types** — map definitions to OWL types (`owl:Class`, `owl:ObjectProperty`) +4. **Hierarchy Generation** — build taxonomy trees using transitive closure and cycle detection +5. **TTL Generation** — serialize to Turtle format using `rdflib` +6. **Quality Evaluation** — assess coverage, completeness, and granularity metrics - ```python - generator = OntologyGenerator() - semantic_network = generator.parse_semantic_network(kg) - ``` - - - Transforms extracted patterns into intermediate class definitions. +## SHACL Validation - ```python - definitions = generator.build_definitions(semantic_network) - ``` - - - Maps definitions to OWL types (`owl:Class`, `owl:ObjectProperty`). - - ```python - from semantica.ontology import ClassInferrer, PropertyGenerator - - class_inferrer = ClassInferrer() - classes = class_inferrer.infer_classes(kg.entities) - - prop_generator = PropertyGenerator() - properties = prop_generator.infer_properties(kg.entities, kg.relationships, classes) - ``` - - - Builds taxonomy trees using transitive closure and cycle detection. - - ```python - hierarchy = generator.build_hierarchy(classes) - ``` - - - Serializes to Turtle format using `rdflib`. - - ```python - from semantica.ontology import OWLGenerator - - owl_gen = OWLGenerator() - ttl_str = owl_gen.generate_owl( - {"classes": classes, "properties": properties, "hierarchy": hierarchy}, - format="turtle", # "turtle" | "xml" | "json-ld" | "n3" - ) - ``` - - - Assesses coverage, completeness, and granularity metrics. - - ```python - from semantica.ontology import OntologyEvaluator - - evaluator = OntologyEvaluator() - report = evaluator.evaluate(ttl_str, kg) - print(f"Coverage: {report.coverage:.2%}") - print(f"Completeness: {report.completeness:.2%}") - print(f"Granularity: {report.granularity:.2%}") - ``` - - - -## Advanced Generation Tools - - - - Capture and manage ontology requirements before generation: - - ```python - from semantica.ontology import RequirementsSpecManager - - spec = RequirementsSpecManager() - spec.add_competency_question("What organizations are headquartered in California?") - spec.add_competency_question("Who founded each organization?") - spec.add_competency_question("What products does each organization sell?") - - spec.set_scope( - domain="Technology industry", - excluded_types=["Event", "Date"], - min_confidence=0.7, - ) - - generator = OntologyGenerator() - ontology = generator.generate_from_graph(kg, requirements=spec) - ``` - - - Generate ontologies directly from natural language: - - ```python - from semantica.ontology import LLMOntologyGenerator - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - generator = LLMOntologyGenerator(llm_provider=llm) - - ontology = generator.generate_from_description( - description="An ontology for tracking pharmaceutical clinical trials, including drugs, patients, dosages, outcomes, and adverse events.", - num_classes=20, - ) - - ontology = generator.generate_from_corpus( - documents=["clinical_trial_protocol.pdf"], - domain_hint="biomedical", - ) - - ontology = generator.refine( - ontology, - questions=["What dosage was given to each patient?", "What adverse events occurred?"], - ) - ``` - - - Integrate published ontologies instead of generating from scratch: - - ```python - from semantica.ontology import ReuseManager - - manager = ReuseManager() - manager.load_ontology("schema.org", source="https://schema.org/version/latest/schemaorg-current-https.ttl") - manager.load_ontology("foaf", source="http://xmlns.com/foaf/spec/index.rdf") - - candidates = manager.find_reusable_classes( - your_classes=["Person", "Organization", "Product"], - loaded_ontologies=["schema.org", "foaf"], - ) - for candidate in candidates: - print(f"{candidate.local_class} → reuse {candidate.external_uri} (similarity: {candidate.similarity:.2f})") - - merged = manager.merge_reused(your_ontology, candidates) - ``` - - - Pre-built domain ontologies for common verticals: - - ```python - from semantica.ontology import DomainOntologies - - catalog = DomainOntologies() - - for domain in catalog.list_domains(): - print(f"{domain.name}: {domain.description} ({domain.class_count} classes)") - - biomedical = catalog.load("biomedical") # SNOMED CT-aligned - finance = catalog.load("finance") # FinancialInstrument, Company, Market - legal = catalog.load("legal") # Contract, Party, Jurisdiction - supply_chain = catalog.load("supply_chain") # Supplier, Product, Shipment - - biomedical.add_class("ClinicalTrial", parent="Study", properties=["phase", "participants"]) - ``` - - Available domains: `biomedical`, `finance`, `legal`, `supply_chain`, `cybersecurity`, `e_commerce`, `hr`. - - - -## ModuleManager - -Build ontologies as composable modules — keep domain logic separated and reusable: +Generate SHACL shapes from an ontology and validate any graph against them: ```python -from semantica.ontology import ModuleManager +from semantica.ontology import SHACLGenerator, OntologyValidator, SHACLValidationReport, SHACLViolation -manager = ModuleManager() +# Generate shapes from ontology +generator = SHACLGenerator() +shapes = generator.generate(ontology) +shapes_ttl = shapes.serialize(format="turtle") -core_module = manager.create_module("core", base_uri="http://example.org/core#") -finance_module = manager.create_module("finance", base_uri="http://example.org/finance#") +# Validate a graph against the shapes +validator = OntologyValidator() +report: SHACLValidationReport = validator.validate(kg, shapes=shapes) -core_module.add_class("Entity", properties=["id", "name"]) -finance_module.add_class("Company", parent="Entity", properties=["ticker", "revenue"]) - -finance_module.import_module(core_module) -unified = manager.merge_modules([core_module, finance_module]) +if not report.conforms: + violation: SHACLViolation + for violation in report.violations: + print(f"{violation.severity}: {violation.message}") + print(f" Node: {violation.node}") + print(f" Path: {violation.path}") ``` -## NamespaceManager +## LLM-Powered Ontology Generation -Manage IRI prefixes and generate consistent URIs for all ontology terms: +For complex or novel domains where schema patterns are hard to infer statistically: + +```python +from semantica.ontology import LLMOntologyGenerator +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +generator = LLMOntologyGenerator(llm_provider=llm) +ontology = generator.generate( + domain_description="A biomedical ontology for clinical trial protocols", + examples=["Patient", "Trial", "Intervention", "Outcome"], +) +``` + +## OWL / RDF Export + +```python +from semantica.ontology import OWLGenerator + +generator = OWLGenerator() +generator.generate(ontology, path="ontology.ttl", format="turtle") +generator.generate(ontology, path="ontology.owl", format="xml") +generator.generate(ontology, path="ontology.json", format="json-ld") +``` + +## Namespace Management ```python from semantica.ontology import NamespaceManager -ns_manager = NamespaceManager(base_uri="http://example.org/") -ns_manager.register("ex", "http://example.org/") -ns_manager.register("schema", "https://schema.org/") +ns = NamespaceManager(base_uri="https://example.org/") +ns.register("ex", "https://example.org/") +ns.register("schema", "https://schema.org/") +ns.register("owl", "http://www.w3.org/2002/07/owl#") -class_iri = ns_manager.generate_iri("Person") # → "http://example.org/Person" -prop_iri = ns_manager.generate_iri("worksFor", prefix="ex") # → "http://example.org/worksFor" -iri = ns_manager.resolve("schema:Organization") # → "https://schema.org/Organization" +# Generate IRIs for classes and properties +class_iri = ns.generate_class_iri("Person") +property_iri = ns.generate_property_iri("worksFor") ``` -## OntologyEvaluator +## Ontology Evaluation -Measure quality across coverage, completeness, and competency questions: +Measure coverage, completeness, and granularity of a generated ontology: ```python from semantica.ontology import OntologyEvaluator evaluator = OntologyEvaluator() -report = evaluator.evaluate(ontology, kg) -print(f"Coverage: {report.coverage:.2%}") -print(f"Completeness: {report.completeness:.2%}") -print(f"Granularity: {report.granularity:.2%}") +result = evaluator.evaluate(ontology, kg) -questions = ["What organizations were founded in California?", "Who are the employees of Apple Inc.?"] -cq_results = evaluator.validate_competency_questions(ontology, kg, questions) -for q, result in zip(questions, cq_results): - print(f"Q: {q} → Answerable: {result.answerable} ({result.reason})") +print(f"Class coverage: {result.class_coverage:.2f}") +print(f"Property coverage: {result.property_coverage:.2f}") +print(f"Completeness: {result.completeness:.2f}") +print(f"Granularity: {result.granularity:.2f}") + +for gap in result.gaps: + print(f"Gap: {gap.description}") +``` + +## Ingest an Existing Ontology + +Load and parse an ontology file for downstream use: + +```python +from semantica.ontology import ingest_ontology + +ontology_data = ingest_ontology("schema.ttl") # Turtle +ontology_data = ingest_ontology("schema.owl") # OWL/XML +ontology_data = ingest_ontology("schema.jsonld") # JSON-LD ``` ## Ontology Hub (v0.5.0) -A visual browser UI for the full ontology lifecycle, served by the Explorer CLI: +A visual browser UI for the full ontology lifecycle. Launch via CLI: ```bash pip install "semantica[explorer]" -semantica explore +semantica-explorer --port 8080 # Navigate to http://localhost:8080 → Ontology Hub tab ``` -Features: visual editor, SHACL Studio, health dashboard, and version control. +Features: -## Tips and Common Pitfalls +- **Visual editor** — create and edit classes, properties, and relationships in the browser +- **SHACL Studio** — author and validate SHACL shapes with live feedback +- **Health dashboard** — coverage, completeness, and constraint violation metrics +- **Version control** — snapshot, diff, and restore ontology versions - - **Define competency questions before generating.** An ontology without competency questions has no measurable success criteria. Write 5–10 natural language questions your ontology must answer before calling `OntologyGenerator`. Then validate them with `OntologyEvaluator.validate_competency_questions()`. - - - - **Reuse before generating.** `DomainOntologies` and `ReuseManager` give you schema.org, FOAF, and domain-specific ontologies that took years to develop. Reusing established classes (`schema:Organization`, `foaf:Person`) also improves interoperability with external data. - - - - **`LLMOntologyGenerator` is great for prototyping, not production.** LLM-generated ontologies are a useful starting point but need expert review. Use `OntologyEvaluator` and manual SHACL authoring to harden the schema before relying on it for production graph validation. - - - - **Always validate after schema changes.** When you add new classes or properties, run `OntologyValidator.validate(ontology)` immediately. Validation issues often surface data quality problems that were silently passing before. - - - - **Use `ModuleManager` for large ontologies.** A monolithic 500-class ontology becomes unmanageable quickly. Split by domain (`core`, `finance`, `legal`) and use `owl:imports` to compose them — changes in one module don't break others. - - - - **Namespace your terms consistently.** All classes and properties need stable IRIs. Use `NamespaceManager` to generate them programmatically — never hardcode IRI strings in code, because base URIs change when projects move. - + + Ontology versioning (`VersionManager`, `OntologyVersion`) has moved to `semantica.change_management`. Import from there: `from semantica.change_management import VersionManager`. + diff --git a/docs/reference/parse.md b/docs/reference/parse.md index 7f8b84a5..6aa9830a 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -6,231 +6,103 @@ icon: "file-lines" `semantica.parse` extracts structured text, layout, tables, and metadata from unstructured documents. `DocumentParser` handles clean machine-readable files; `DoclingParser` handles complex layouts, scanned PDFs, and multi-column documents. +## Exported Classes + +```python +from semantica.parse import ( + DocumentParser, # auto-detect format — delegates to format-specific parser + PDFParser, # PDF text extraction + DOCXParser, # Word .docx documents + HTMLParser, # HTML / web pages + MarkdownParser, # Markdown files + TXTParser, # plain text + JSONParser, # JSON documents + XMLParser, # XML documents + CSVParser, # CSV / TSV files + WebParser, # URL fetch + HTML parsing + EmailParser, # .eml / .msg email files + CodeParser, # source code files + # Data types + ParsedDocument, # {text, sections, tables, metadata, source_id} + DocumentMetadata, # {title, author, created_date, page_count, language, ...} +) + +# Optional — requires: pip install "semantica[docling]" +from semantica.parse import DoclingParser # advanced OCR + layout analysis +``` + ## What You Get - - - Standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — zero config, no extras. - - - Advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR. - - - AST structure extraction — functions, classes, imports, dependencies — for 10+ languages. - - - EXIF metadata extraction and OCR via Tesseract for image files. - - - Technical metadata from audio, video, and image files (duration, codec, resolution). - - - Parse Model Context Protocol responses into structured `ParsedDocument` objects. - - +- **`DocumentParser`** — standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — auto-detects format +- **`DoclingParser`** — advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR (optional dep) +- **`ParsedDocument`** — structured output with `text`, `sections`, `tables`, and `metadata` +- **Format-specific parsers** — `PDFParser`, `DOCXParser`, `HTMLParser`, `WebParser`, `EmailParser`, `CodeParser`, etc. -## Quick Start +## DocumentParser - - - ```python - from semantica.parse import DocumentParser +Standard parser for clean, machine-readable documents: - parser = DocumentParser() - parsed = parser.parse("data/report.pdf") +```python +from semantica.parse import DocumentParser - print(parsed.text) # full clean text - print(parsed.metadata) # title, author, date, page_count, language, etc. - print(parsed.sections) # document structure as a list of Section objects - ``` - - - ```bash - pip install "semantica[docling]" - ``` +parser = DocumentParser() +parsed = parser.parse("data/report.pdf") - ```python - from semantica.parse import DoclingParser +print(parsed.text) # full clean text +print(parsed.metadata) # title, author, date, page_count, language, etc. +print(parsed.sections) # document structure as a list of Section objects +``` - parser = DoclingParser( - extract_tables=True, # structured table extraction with cell type detection - extract_images=True, # extract image regions for downstream OCR - output_format="markdown", # "markdown" | "html" | "json" - ) +Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. - parsed = parser.parse("data/annual_report.pdf") - print(parsed.tables) # structured TableData objects with headers and rows - ``` - - - ```python - from semantica.split import TextSplitter - from semantica.semantic_extract import NERExtractor - from semantica.llms import Groq - import os +## DoclingParser - splitter = TextSplitter(method="structural") - chunks = splitter.split_document(parsed) +Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - extractor = NERExtractor(method="llm", llm_provider=llm) - entities = extractor.extract_batch([c.text for c in chunks]) - ``` - - +```bash +pip install "semantica[docling]" +``` -## Parser Reference +```python +from semantica.parse import DoclingParser - - - Standard parser for clean, machine-readable documents — no extra dependencies required: +parser = DoclingParser( + extract_tables=True, # structured table extraction with cell type detection + extract_images=True, # extract image regions for downstream OCR + output_format="markdown", # "markdown" | "html" | "json" +) - ```python - from semantica.parse import DocumentParser +parsed = parser.parse("data/annual_report.pdf") - parser = DocumentParser() - parsed = parser.parse("data/report.pdf") +print(parsed.text) # full clean text +print(parsed.tables) # structured TableData objects with headers and rows +print(parsed.sections) # document structure with heading hierarchy +``` - print(parsed.text) # full clean text - print(parsed.metadata) # title, author, date, page_count, language, etc. - print(parsed.sections) # document structure as a list of Section objects - ``` +Use `DoclingParser` for: - **Supported formats:** PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. - - - Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: +- Multi-column PDF layouts +- Tables with merged cells or complex headers +- PPTX slides with embedded charts +- XLSX spreadsheets with formulas +- Scanned documents with OCR +- Academic papers and technical reports - ```python - from semantica.parse import DoclingParser +## OCR Support - parser = DoclingParser( - extract_tables=True, # structured table extraction with cell type detection - extract_images=True, # extract image regions for downstream OCR - output_format="markdown", # "markdown" | "html" | "json" - ) +```python +parser = DoclingParser( + ocr=True, + ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents + extract_tables=True, +) - parsed = parser.parse("data/annual_report.pdf") - print(parsed.tables) # structured TableData objects with headers and rows - print(parsed.sections) # document structure with heading hierarchy - ``` +parsed = parser.parse("data/scanned_contract.pdf") +``` - **Use `DoclingParser` for:** - - Multi-column PDF layouts - - Tables with merged cells or complex headers - - PPTX slides with embedded charts - - XLSX spreadsheets with formulas - - Scanned documents with OCR - - Academic papers and technical reports +## Parsed Document Object - **OCR support:** - - ```python - parser = DoclingParser( - ocr=True, - ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents - extract_tables=True, - ) - parsed = parser.parse("data/scanned_contract.pdf") - ``` - - - Parse source code files — extracts AST structure, functions, classes, imports, and comments: - - ```python - from semantica.parse import CodeParser - - parser = CodeParser( - extract_comments=True, # include docstrings and inline comments - extract_dependencies=True, # import/require statements - language="auto", # "auto" | "python" | "javascript" | "java" | "go" | "rust" | "cpp" - ) - - parsed = parser.parse("src/main.py") - - print(parsed.text) # raw source code as text - print(parsed.metadata["language"]) # detected language - print(parsed.metadata["functions"]) # list of function names - print(parsed.metadata["classes"]) # list of class names - print(parsed.metadata["imports"]) # list of import statements - print(parsed.metadata["comments"]) # docstrings and inline comments - ``` - - **Supported languages:** Python, JavaScript/TypeScript, Java, Go, Rust, C/C++, C#, Ruby, PHP, Swift. - - - Extract EXIF metadata and optionally perform OCR on image files: - - ```python - from semantica.parse import ImageParser - - parser = ImageParser( - extract_exif=True, # camera, GPS, timestamps, etc. - ocr=True, # OCR via Tesseract (requires tesseract-ocr installed) - ocr_language="en", # ISO 639-1 language code for OCR - ) - - parsed = parser.parse("photo.jpg") - - print(parsed.text) # OCR-extracted text (if ocr=True) - print(parsed.metadata["width"]) # image dimensions - print(parsed.metadata["height"]) - print(parsed.metadata["format"]) # "JPEG" | "PNG" | "TIFF" | ... - print(parsed.metadata["exif"]["GPS"]) # GPS coordinates if available - print(parsed.metadata["exif"]["DateTime"]) - ``` - - - ### MediaParser - - Extract technical metadata from audio, video, and image files: - - ```python - from semantica.parse import MediaParser - - parser = MediaParser() - - # Video file - parsed = parser.parse("interview.mp4") - print(parsed.metadata["duration_seconds"]) - print(parsed.metadata["codec"]) - print(parsed.metadata["resolution"]) - print(parsed.metadata["fps"]) - - # Audio file - parsed = parser.parse("podcast.mp3") - print(parsed.metadata["duration_seconds"]) - print(parsed.metadata["bitrate"]) - print(parsed.metadata["channels"]) - ``` - - **Supported formats:** MP4, AVI, MOV, MKV, MP3, WAV, FLAC, OGG, JPEG, PNG, TIFF, WebP. - - ### MCPParser - - Parse Model Context Protocol (MCP) responses into structured `ParsedDocument` objects: - - ```python - from semantica.parse import MCPParser - - parser = MCPParser() - - mcp_response = { - "content": [{"type": "text", "text": "Apple Inc. was founded in 1976..."}], - "metadata": {"tool": "web_search", "query": "Apple Inc history"} - } - - parsed = parser.parse(mcp_response) - print(parsed.text) # "Apple Inc. was founded in 1976..." - print(parsed.metadata) # tool name, query, and other MCP metadata - ``` - - - -## Parsed Document Schema - - - +Both parsers return a `ParsedDocument` with the same structure: ```python @dataclass @@ -240,12 +112,7 @@ class ParsedDocument: tables: List[TableData] # structured table data (DoclingParser only) metadata: DocumentMetadata # title, author, dates, page count source_id: str # links back to the original DataSource -``` - - - -```python @dataclass class DocumentMetadata: title: Optional[str] @@ -259,21 +126,6 @@ class DocumentMetadata: format: str # "pdf" | "docx" | "pptx" | ... ``` - - - -## Choosing a Parser - -| Scenario | Parser | -| -------- | ------ | -| Clean PDFs, DOCX, HTML, TXT, CSV, Excel | `DocumentParser` — zero config, no extras | -| Scanned PDFs, OCR required | `DoclingParser(ocr=True)` — requires `pip install "semantica[docling]"` | -| Multi-column PDFs, merged-cell tables | `DoclingParser(extract_tables=True)` | -| Source code files | `CodeParser(language="auto")` | -| Images with embedded text | `ImageParser(ocr=True)` — requires Tesseract | -| Audio/video metadata | `MediaParser()` | -| MCP tool responses | `MCPParser()` | - ## Integration with FileIngestor The most common pattern — ingest a directory then parse each source: @@ -295,28 +147,6 @@ for source in sources: Docling is an optional dependency. If `docling` is not installed, `DoclingParser` raises an `ImportError` with installation instructions. `DocumentParser` is always available and requires no extras. -## Tips and Common Pitfalls - - - **Start with `DocumentParser` and only switch to `DoclingParser` when needed.** `DoclingParser` is significantly more powerful but slower and requires an additional dependency. For clean machine-readable PDFs and Office files, `DocumentParser` is fast and accurate enough. - - - - **OCR requires Tesseract installed on the system.** `ImageParser(ocr=True)` and `DoclingParser(ocr=True)` both call Tesseract under the hood. Install it with `apt-get install tesseract-ocr` (Linux) or `brew install tesseract` (macOS) before enabling OCR. - - - - **`extract_tables=True` is off by default for speed.** Table extraction in `DoclingParser` requires additional layout analysis passes. Only enable it when you actually need structured table data — for text-only extraction, leave it off. - - - - **`CodeParser` outputs AST metadata, not just raw text.** The `parsed.metadata["functions"]` and `parsed.metadata["classes"]` lists are useful for building code-level knowledge graphs — function call graphs, class inheritance hierarchies, dependency graphs. - - - - **Always pass the `ParsedDocument` to `TextSplitter` before extraction.** Raw `parsed.text` is a flat string. Use `TextSplitter` to chunk it into semantically meaningful pieces before running NER — this dramatically reduces context window overflow on large documents. - - Load files before parsing. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index 8cc594c1..d83ddb3f 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -1,291 +1,168 @@ --- title: "Provenance Module" -description: "W3C PROV-O compliant lineage tracking, source attribution, and audit trails across all modules." +description: "W3C PROV-O compliant lineage tracking, source attribution, tamper-evident checksums, and audit trails across all modules." icon: "link" --- `semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, reasoning, and export. Compliant with W3C PROV-O, suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 environments. +## Exported Classes + +```python +from semantica.provenance import ( + ProvenanceManager, # track entities, get lineage, export PROV-O + ProvenanceEntry, # single provenance record (entity_id, source, method, ...) + SourceReference, # rich source pointer (DOI, page, quote, URL) + ProvenanceStorage, # abstract storage backend + InMemoryStorage, # default in-memory backend + SQLiteStorage, # persistent SQLite backend for production + compute_checksum, # compute tamper-evident hash for an entry + verify_checksum, # verify integrity of a stored entry +) + +# GraphBuilderWithProvenance is in semantica.kg, not semantica.provenance +from semantica.kg import GraphBuilderWithProvenance +``` + ## What You Get - - - Track entities, relationships, and activities with full source attribution and confidence scores. - - - Track entity and relationship lineage within a knowledge graph. - - - Full directed lineage from any entity back to its originating source document. - - - Serialize lineage as Turtle RDF or JSON-LD for compliance reporting. - - - Drop-in replacement for GraphBuilder that auto-tracks every node and edge. - - - SHA-256 checksums to detect tampering in HIPAA and FDA 21 CFR Part 11 environments. - - - -## Quick Start - - - - ```python - from semantica.provenance import ProvenanceManager, SQLiteStorage - - # SQLite — persistent across process restarts (recommended for production) - manager = ProvenanceManager( - storage=SQLiteStorage(db_path="provenance.db") - ) - ``` - - - ```python - manager.track_entity( - entity_id="apple_inc", - source="annual_report_2023.pdf", - entity_type="Organization", - extraction_method="llm", - confidence=0.98, - ) - - manager.track_relationship( - rel_id="steve_jobs_founded_apple", - source="annual_report_2023.pdf", - extraction_method="llm", - confidence=0.92, - ) - ``` - - - ```python - lineage = manager.get_lineage("apple_inc") - print(f"Source: {lineage.source}") - print(f"Extracted: {lineage.extracted_at}") - print(f"Method: {lineage.extraction_method}") - print(f"Confidence: {lineage.confidence}") - ``` - - - ```python - # Full provenance graph for all tracked entities - manager.export_all(path="provenance.ttl", format="turtle") - manager.export_all(path="provenance.jsonld", format="json-ld") - ``` - - +- **`ProvenanceManager`** — track entities and relationships with source attribution and lineage retrieval +- **`ProvenanceEntry`** / **`SourceReference`** — structured records with DOI, page, quote, confidence, and timestamp +- **`InMemoryStorage`** / **`SQLiteStorage`** — swappable persistence backends +- **`compute_checksum`** / **`verify_checksum`** — tamper-evident integrity verification +- **Lineage graph** — full upstream lineage from any entity back to its source document +- **W3C PROV-O export** — serialize lineage as Turtle RDF or JSON-LD for compliance reporting +- **`GraphBuilderWithProvenance`** (in `semantica.kg`) — drop-in replacement that auto-tracks every node and edge ## ProvenanceManager ```python -from semantica.provenance import ProvenanceManager +from semantica.provenance import ProvenanceManager, InMemoryStorage, SQLiteStorage -manager = ProvenanceManager() +# In-memory (default) — fast, not persisted across restarts +manager = ProvenanceManager(storage=InMemoryStorage()) -# Track an extracted entity +# SQLite — persisted, production-ready +manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) + +# Track an extracted entity (with rich source reference) manager.track_entity( entity_id="apple_inc", source="annual_report_2023.pdf", - entity_type="Organization", - extraction_method="llm", + source_location="Page 12, Section 3.1", + source_quote="Apple Inc. was incorporated on January 3, 1977.", confidence=0.98, ) # Track an extracted relationship -manager.track_relationship( - rel_id="steve_jobs_founded_apple", +manager.track_entity( + entity_id="steve_jobs_founded_apple", source="annual_report_2023.pdf", - extraction_method="llm", confidence=0.92, ) # Retrieve full lineage for any entity -lineage = manager.get_lineage("apple_inc") -print(f"Source: {lineage.source}") -print(f"Extracted: {lineage.extracted_at}") -print(f"Method: {lineage.extraction_method}") -print(f"Confidence: {lineage.confidence}") +entry = manager.get_lineage("apple_inc") +print(f"Source: {entry.source}") +print(f"Quote: {entry.source_quote}") +print(f"Confidence: {entry.confidence}") +print(f"Tracked at: {entry.tracked_at}") ``` -## Activity Tracking +## SourceReference -Record pipeline activities — what was consumed and what was produced: +`SourceReference` provides a rich, citable pointer to the exact location in a source document: ```python -# Start and end an activity -activity_id = manager.start_activity( - activity_type="ner_extraction", - used=["annual_report_2023.pdf"], - generated=["apple_inc", "steve_jobs"], +from semantica.provenance import SourceReference + +ref = SourceReference( + document_id="annual_report_2023.pdf", + page=12, + section="3.1", + quote="Apple Inc. was incorporated on January 3, 1977.", + url="https://investor.apple.com/sec-filings/annual-reports/", + doi="10.0000/example.doi", ) -manager.end_activity(activity_id) - -# Query activities for an entity -activities = manager.get_activities(entity_id="apple_inc") -for activity in activities: - print(f"{activity.type} at {activity.started_at}") - print(f" Used: {activity.used}") - print(f" Generated: {activity.generated}") +manager.track_entity( + entity_id="apple_inc", + source_reference=ref, + confidence=0.98, +) ``` -## Lineage Graph +## Tamper-Evident Checksums -Retrieve a full directed lineage graph from any entity back to its source: - -```python -lineage_graph = manager.get_lineage_graph("apple_inc") - -for node in lineage_graph.nodes: - print(f"{node.id}: {node.type} — {node.timestamp}") - -for edge in lineage_graph.edges: - print(f"{edge.source} → {edge.target} ({edge.relation})") -``` - -## Storage Backends - - - - Fast, no persistence — default backend. Data is lost on process exit. - - ```python - from semantica.provenance import ProvenanceManager, InMemoryStorage - - manager = ProvenanceManager(storage=InMemoryStorage()) - - manager.track_entity("apple_inc", source="report.pdf", confidence=0.98) - lineage = manager.get_lineage("apple_inc") - ``` - - Best for: development, unit tests, short-lived pipelines. - - - Persistent file-based storage — survives process restarts. The API is identical to `InMemoryStorage`. - - ```python - from semantica.provenance import ProvenanceManager, SQLiteStorage - - manager = ProvenanceManager( - storage=SQLiteStorage(db_path="provenance.db") - ) - - manager.track_entity("apple_inc", source="report.pdf", confidence=0.98) - lineage = manager.get_lineage("apple_inc") - ``` - - Best for: production single-machine deployments, compliance environments. - - - - | Storage | Persistence | Best For | - | ------- | ----------- | -------- | - | `InMemoryStorage` | No | Development, unit tests, short-lived pipelines | - | `SQLiteStorage` | Yes (file) | Production single-machine deployments | - - - - -## Integration with GraphBuilder - -`GraphBuilderWithProvenance` automatically records provenance for every node and edge constructed — no manual `track_entity()` calls needed: - -```python -from semantica.kg import GraphBuilderWithProvenance - -builder = GraphBuilderWithProvenance(provenance=True) -result = builder.build_single_source(graph_data) - -# Every node and edge has a source_id linking back to the originating document -lineage = result.provenance_manager.get_lineage("apple_inc") -print(f"Source document: {lineage.source}") -print(f"Extracted by: {lineage.extraction_method}") -``` - -## Integrity Verification - -Compute and verify checksums for provenance entries to detect tampering: +Verify that provenance records have not been modified after creation: ```python from semantica.provenance import compute_checksum, verify_checksum -# Compute a SHA-256 checksum over an entity's provenance record -entry = manager.get_provenance_entry("apple_inc") -checksum = compute_checksum(entry, algorithm="sha256") -print(f"Checksum: {checksum}") +entry = manager.get_lineage("apple_inc") -# Later — verify the record has not been modified -is_valid = verify_checksum(entry, expected_checksum=checksum, algorithm="sha256") +# Compute and store a checksum on first write +checksum = compute_checksum(entry) + +# Later: verify the entry hasn’t been altered +is_valid = verify_checksum(entry, checksum) if not is_valid: raise RuntimeError("Provenance record has been tampered with!") ``` -Supported algorithms: `"sha256"` (default), `"sha512"`, `"md5"`. - ## W3C PROV-O Export +Export lineage as W3C PROV-O Turtle for compliance reporting: + ```python # Single entity lineage prov_ttl = manager.export_prov_o("apple_inc", format="turtle") # Full provenance graph for all tracked entities -manager.export_all(path="provenance.ttl", format="turtle") +manager.export_all(path="provenance.ttl", format="turtle") + +# Compliance-ready JSON-LD export manager.export_all(path="provenance.jsonld", format="json-ld") ``` -## Schemas +## Integration with GraphBuilder - - +`GraphBuilderWithProvenance` (from `semantica.kg`) automatically records provenance for every node and edge constructed: ```python -@dataclass -class ProvenanceEntry: - entity_id: str - source: str # source document or system - source_location: str # e.g. "page 3, paragraph 2" - source_quote: str # verbatim text from source - extraction_method: str # "llm" | "ml" | "pattern" - confidence: float # extraction confidence 0–1 - timestamp: datetime # when this entry was recorded - entity_type: Optional[str] +from semantica.kg import GraphBuilderWithProvenance +from semantica.provenance import ProvenanceManager, SQLiteStorage + +prov_manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) +builder = GraphBuilderWithProvenance(provenance_manager=prov_manager) +kg = builder.build_single_source(graph_data) + +# Every node and edge now has full source attribution +entry = prov_manager.get_lineage("apple_inc") +print(f"Source document: {entry.source}") +print(f"Confidence: {entry.confidence}") ``` - - +## Enable Provenance in Extractors ```python -@dataclass -class SourceReference: - source_id: str - doi: Optional[str] # academic DOI if available - page: Optional[int] # page number in document - paragraph: Optional[int] - quote: str # verbatim text supporting the fact - url: Optional[str] - accessed_at: Optional[datetime] +from semantica.semantic_extract import NERExtractor +from semantica.provenance import ProvenanceManager + +prov_manager = ProvenanceManager() + +ner = NERExtractor(method="llm", llm_provider=llm, provenance=True) +entities = ner.extract(text) + +# Retrieve lineage for the first extracted entity +entry = prov_manager.get_lineage(entities[0]["id"]) +print(f"Source: {entry.source}") ``` - - - -## W3C PROV-O Mapping - -| Semantica Concept | PROV-O Class / Property | -| ----------------- | ----------------------- | -| Entity (node/fact) | `prov:Entity` | -| Extraction activity | `prov:Activity` | -| Source document | `prov:Entity` | -| `track_entity()` | `prov:wasDerivedFrom` | -| `start_activity()` | `prov:wasGeneratedBy` | -| Extraction method | `prov:wasAssociatedWith` | -| Timestamp | `prov:startedAtTime`, `prov:endedAtTime` | - ## Compliance Standards +Provenance tracking in Semantica is designed to satisfy: + | Standard | Requirement Met | | -------- | --------------- | | **W3C PROV-O** | Full PROV-O compliant serialization (Turtle and JSON-LD) | @@ -294,32 +171,6 @@ class SourceReference: | **GDPR** | Data lineage supporting right-to-erasure impact analysis | | **FDA 21 CFR Part 11** | Electronic records with origination timestamp and extraction method | -## Tips and Common Pitfalls - - - **Use `SQLiteStorage` in production, not `InMemoryStorage`.** The in-memory backend is the default for backwards compatibility, but provenance data is lost on process exit. Switch to `SQLiteStorage(db_path="provenance.db")` before going to production — migrating later means losing all historical lineage. - - - - **Track provenance at ingestion time, not after.** The `source_location` and `source_quote` fields become unavailable once you've moved past the parsing stage. Capture them during ingestion and pass them to `track_entity()` immediately. - - - - **Use `GraphBuilderWithProvenance` instead of plain `GraphBuilder`.** It auto-tracks every node and edge without manual `track_entity()` calls — ensuring nothing is accidentally omitted from the provenance record. - - - - **Verify checksums on high-stakes data.** `compute_checksum()` + `verify_checksum()` detects any modification to a provenance entry since it was recorded — critical for HIPAA and FDA 21 CFR Part 11 environments where tampered records carry legal liability. - - - - **Export PROV-O Turtle for external auditors.** Compliance teams and external auditors often need machine-readable lineage in a standard format. `manager.export_all("provenance.ttl", format="turtle")` produces W3C PROV-O Turtle that any RDF tool can parse. - - - - **Use `get_audit_trail(entity_id=...)` for GDPR subject-access requests.** The GDPR right-of-access requires you to show what personal data you hold and where it came from. Scoped lineage per entity ID makes this a one-line export. - - Version control and snapshot audit trails. diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 63af28e2..203c9f46 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -6,402 +6,283 @@ icon: "microchip" `semantica.reasoning` derives new knowledge from existing facts using logical rules. Every engine produces **explainable inference paths** — traceable chains of rules and facts, not black-box conclusions. -## Why Reasoning? +## Exported Classes -Knowledge graphs encode what you know explicitly. Reasoning lets you derive what must logically follow — without manually asserting every implication: - -- If A `located_in` B and B `located_in` C, then A `located_in` C — without storing that triple -- If Alice `parent_of` Bob and Bob `parent_of` Charlie, then Alice `ancestor_of` Charlie -- If a drug is contraindicated for a condition class, it's also contraindicated for all subclasses — inferred from the ontology hierarchy -- If an employee's CEO tenure ended in 2020, they cannot have signed contracts as CEO in 2021 — caught by temporal reasoning - -Reasoning turns sparse explicit knowledge into a dense, coherent, contradiction-free knowledge base. +```python +from semantica.reasoning import ( + # Engines + Reasoner, # IF/THEN forward-chaining facade + GraphReasoner, # inference over full KG structure + ReteEngine, # high-performance Rete pattern matching + SPARQLReasoner, # SPARQL-based RDF inference + DatalogReasoner, # recursive Horn clause fixpoint evaluation + TemporalReasoningEngine, # Allen interval algebra (13 relations) + ExplanationGenerator, # structured step-by-step explanations + # Data types + Rule, # IF/THEN rule definition + Fact, # base fact (subject, predicate, obj) + RuleType, # enum: FORWARD_CHAIN, BACKWARD_CHAIN, ... + InferenceResult, # result of infer() — contains derived_facts list + DatalogFact, # Datalog base fact (predicate, args tuple) + DatalogRule, # Datalog Horn clause ("head :- body.") + TemporalInterval, # time interval with start/end + IntervalRelation, # enum of 13 Allen relations + # Explanation types + Explanation, # conclusion + confidence + reasoning_path + ReasoningPath, # ordered list of ReasoningSteps + ReasoningStep, # single step: fact + rule_name + depth + Justification, # full justification record +) +``` ## What You Get - - - Main facade — IF/THEN forward-chaining with variable substitution and rule templates. - - - Inference over full knowledge graph structure: transitivity, symmetry, inverses. - - - High-performance pattern matching via the Rete algorithm for large rule sets. - - - Query expansion and property chain inference over RDF graphs. - - - Recursive Horn clause rules with guaranteed fixpoint termination (v0.4.0). - - - All 13 Allen interval algebra relations for time-aware inference. - - +- **`Reasoner`** — main facade for IF/THEN forward-chaining with variable substitution +- **`GraphReasoner`** — inference over full knowledge graph structure (transitivity, symmetry, inverses) +- **`ReteEngine`** — high-performance pattern matching via the Rete algorithm for large rule sets +- **`SPARQLReasoner`** — query expansion and property chain inference over RDF graphs +- **`DatalogReasoner`** — recursive Horn clause rules with guaranteed fixpoint termination (v0.4.0) +- **`TemporalReasoningEngine`** — all 13 Allen interval algebra relations for time-aware inference +- **`ExplanationGenerator`** — structured explanation paths for every derived conclusion + +## Quick Start + +The most common pattern: add facts + rules, run inference, explain a conclusion: + +```python +from semantica.reasoning import Reasoner, Rule, Fact, RuleType, InferenceResult + +reasoner = Reasoner() + +reasoner.add_fact(Fact(subject="Alice", predicate="is_a", obj="Manager")) +reasoner.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[{"subject": "?x", "predicate": "is_a", "object": "Manager"}], + conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"}, +)) + +result: InferenceResult = reasoner.infer() +for fact in result.derived_facts: + print(f"{fact.subject} {fact.predicate} {fact.obj}") + print(f" via: {fact.explanation}") +``` Forward chaining inference: known facts + IF/THEN rules produce derived facts with a full traceable explanation path -## Choosing a Reasoning Engine +## Reasoner (Main Facade) -| Engine | When to Use | -| ------ | ----------- | -| `Reasoner` | Simple IF/THEN rules, transitivity/symmetry templates, one-shot inference | -| `GraphReasoner` | Rules that operate on graph structure (paths, neighborhoods, multi-hop) | -| `ReteEngine` | Large rule sets (100+), rules fire repeatedly, performance is critical | -| `SPARQLReasoner` | Already using RDF/Turtle, need property chains, SPARQL ecosystem tools | -| `DatalogReasoner` | Recursive rules (ancestry, reachability), guaranteed termination required | -| `TemporalReasoningEngine` | Time-aware facts, interval relationships, historical validity | +The unified entry point for rule-based forward-chaining inference: -## Engines +```python +from semantica.reasoning import Reasoner, Rule, Fact, RuleType - - - The unified entry point for rule-based forward-chaining inference. Start here for most use cases. +reasoner = Reasoner() - ```python - from semantica.reasoning import Reasoner, Rule, Fact, RuleType +# Add base facts +reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Manager")) +reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Employee")) - reasoner = Reasoner() +# Add an IF/THEN rule +reasoner.add_rule(Rule( + rule_type=RuleType.FORWARD_CHAIN, + conditions=[ + {"subject": "?x", "predicate": "is_a", "object": "Manager"} + ], + conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} +)) - # Add base facts - reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Manager")) - reasoner.add_fact(Fact(subject="John", predicate="is_a", obj="Employee")) +# Run inference +result = reasoner.infer() +for inference in result.derived_facts: + print(f"{inference.subject} {inference.predicate} {inference.obj}") + print(f" Derived via: {inference.explanation}") +``` - # Add an IF/THEN rule - reasoner.add_rule(Rule( - rule_type=RuleType.FORWARD_CHAIN, - conditions=[ - {"subject": "?x", "predicate": "is_a", "object": "Manager"} - ], - conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"} - )) +### Built-In Rule Templates - # Run inference — always call explicitly after adding facts/rules - result = reasoner.forward_chain() - for inference in result.derived_facts: - print(f"{inference.subject} {inference.predicate} {inference.obj}") - print(f" Derived via: {inference.explanation}") - ``` +```python +engine = Reasoner() - - Always call `reasoner.forward_chain()` after adding facts and rules. Adding them updates internal state but does **not** trigger inference automatically. - - +# Transitive closure: A→B, B→C ⟹ A→C +engine.apply_transitivity("located_in") - - Inference over the full knowledge graph structure — rules that operate on graph paths, neighborhoods, and multi-hop connections. +# Symmetry: A knows B ⟹ B knows A +engine.apply_symmetry("knows") - ```python - from semantica.reasoning import GraphReasoner +# Inverse: A parent_of B ⟹ B child_of A +engine.apply_inverse("parent_of", "child_of") +``` - graph_reasoner = GraphReasoner() +## GraphReasoner - # Define a transitive ancestor rule - graph_reasoner.add_rule({ - "if": [ - {"subject": "?a", "predicate": "parent_of", "object": "?b"}, - {"subject": "?b", "predicate": "parent_of", "object": "?c"} - ], - "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} - }) +Inference over the full knowledge graph structure: - inferences = graph_reasoner.reason(graph, query="") - for inf in inferences: - print(f"{inf['subject']} {inf['predicate']} {inf['object']}") - ``` - +```python +from semantica.reasoning import GraphReasoner - - High-performance pattern matching using the Rete algorithm — far faster than naive forward chaining for large rule sets because it caches partial matches across iterations. +graph_reasoner = GraphReasoner(kg) - ```python - from semantica.reasoning import ReteEngine +# Define a transitive ancestor rule +graph_reasoner.add_rule({ + "if": [ + {"subject": "?a", "predicate": "parent_of", "object": "?b"}, + {"subject": "?b", "predicate": "parent_of", "object": "?c"} + ], + "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?c"} +}) - engine = ReteEngine() - engine.load_rules("rules/domain_rules.json") - results = engine.run(kg) +inferences = graph_reasoner.infer(kg) +for inf in inferences: + print(f"{inf['subject']} {inf['predicate']} {inf['object']}") +``` - # Inspect the Rete network - root = engine.get_root() - alpha_nodes = engine.get_alpha_nodes() # single-condition filters - beta_nodes = engine.get_beta_nodes() # join nodes - ``` +## ReteEngine - Rule file format (JSON): +High-performance pattern matching using the Rete algorithm — far faster than naive forward chaining for large rule sets because it caches partial matches across iterations: - ```json +```python +from semantica.reasoning import ReteEngine + +engine = ReteEngine() +engine.load_rules("rules/domain_rules.json") +results = engine.run(kg) + +# Inspect the Rete network +root = engine.get_root() +alpha_nodes = engine.get_alpha_nodes() # single-condition filters +beta_nodes = engine.get_beta_nodes() # join nodes +``` + +Rule format (JSON): + +```json +{ + "rules": [ { - "rules": [ - { - "name": "manager_authority", - "conditions": [ - { "subject": "?x", "predicate": "role", "object": "Manager" }, - { "subject": "?x", "predicate": "dept", "object": "?dept" } - ], - "action": { - "subject": "?x", - "predicate": "has_authority_over", - "object": "?dept" - }, - "priority": 10 - } - ] + "name": "manager_authority", + "conditions": [ + { "subject": "?x", "predicate": "role", "object": "Manager" } + ], + "action": { "subject": "?x", "predicate": "has_authority", "object": "true" } } - ``` + ] +} +``` - | Field | Type | Description | - | ----- | ---- | ----------- | - | `name` | `str` | Unique rule identifier — appears in `ExplanationGenerator` output | - | `conditions` | `List[Dict]` | Pattern to match — use `?variable` for wildcards | - | `action` | `Dict` | Fact to derive when all conditions match | - | `priority` | `int` | Higher priority rules fire first | +## SPARQLReasoner - - Use `ReteEngine` when you have more than ~20 rules or when rules can fire repeatedly. `Reasoner` re-evaluates all rules from scratch each cycle; `ReteEngine` caches partial matches and is orders of magnitude faster. - - +Query-based inference over RDF graphs with property chain support: - - Query-based inference over RDF graphs with property chain support. Use this when you're already in the RDF/Turtle ecosystem. +```python +from semantica.reasoning import SPARQLReasoner - ```python - from semantica.reasoning import SPARQLReasoner +reasoner = SPARQLReasoner(graph=rdf_graph) - reasoner = SPARQLReasoner(graph=rdf_graph) +result = reasoner.query(""" + PREFIX ex: + SELECT ?person ?company WHERE { + ?person ex:founded ?company . + ?company ex:located_in ex:SiliconValley . + } +""") - result = reasoner.query(""" - PREFIX ex: - SELECT ?person ?company WHERE { - ?person ex:founded ?company . - ?company ex:located_in ex:SiliconValley . - } - """) +for row in result.bindings: + print(row["person"], row["company"]) - for row in result.bindings: - print(row["person"], row["company"]) +# Property chain inference: A knows B, B colleague_of C ⟹ A knows C +reasoner.add_property_chain("knows", ["knows", "colleague_of"]) +inferences = reasoner.infer_property_chains() +``` - # Property chain inference: A knows B, B colleague_of C ⟹ A knows C - reasoner.add_property_chain("knows", ["knows", "colleague_of"]) - inferences = reasoner.infer_property_chains() - ``` - +## DatalogReasoner (v0.4.0) - - Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops. +Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops: - - Added in **v0.4.0**. Use `DatalogReasoner` whenever your rules can create cycles — it's the only engine with a termination guarantee. - +```python +from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule - ```python - from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule +datalog = DatalogReasoner() - datalog = DatalogReasoner() +# Base facts +datalog.add_fact(DatalogFact("parent", ("alice", "bob"))) +datalog.add_fact(DatalogFact("parent", ("bob", "charlie"))) - # Base facts - datalog.add_fact(DatalogFact("parent", ("alice", "bob"))) - datalog.add_fact(DatalogFact("parent", ("bob", "charlie"))) +# Recursive rules (Horn clauses) +datalog.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) +datalog.add_rule(DatalogRule("ancestor(?X, ?Z) :- parent(?X, ?Y), ancestor(?Y, ?Z).")) - # Recursive rules (Horn clauses) - datalog.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y).")) - datalog.add_rule(DatalogRule("ancestor(?X, ?Z) :- parent(?X, ?Y), ancestor(?Y, ?Z).")) +# Evaluate to fixpoint +datalog.evaluate() - # Evaluate to fixpoint - datalog.evaluate() +# Query +results = datalog.query("ancestor(alice, ?Z)") +# → [{"Z": "bob"}, {"Z": "charlie"}] +``` - # Query - results = datalog.query("ancestor(alice, ?Z)") - # → [{"Z": "bob"}, {"Z": "charlie"}] - ``` +## TemporalReasoningEngine - - `Reasoner` has **no cycle detection** — rules that create cycles (A derives B, B derives C, C re-derives A) will loop infinitely. Use `DatalogReasoner` whenever recursive rules are involved. - - +Reason about time intervals using all 13 Allen interval algebra relations: - - Reason about time intervals using all 13 Allen interval algebra relations. +```python +from semantica.reasoning import TemporalReasoningEngine, TemporalInterval, IntervalRelation - ```python - from semantica.reasoning import TemporalReasoningEngine, TemporalInterval, IntervalRelation +engine = TemporalReasoningEngine() - engine = TemporalReasoningEngine() +ceo_tenure = TemporalInterval(start="1997-09-16", end="2011-08-24") +board_member = TemporalInterval(start="2000-01-01", end="2012-06-01") - ceo_tenure = TemporalInterval(start="1997-09-16", end="2011-08-24") - board_member = TemporalInterval(start="2000-01-01", end="2012-06-01") +relation = engine.get_relation(ceo_tenure, board_member) +# → IntervalRelation.DURING (ceo_tenure is fully inside board_member) +``` - relation = engine.get_relation(ceo_tenure, board_member) - # → IntervalRelation.DURING (ceo_tenure is fully inside board_member) - ``` +All 13 Allen interval algebra relations are supported: - All 13 Allen interval algebra relations: - - | Relation | Meaning | - | -------- | ------- | - | `BEFORE` | A ends before B starts | - | `MEETS` | A ends exactly when B starts | - | `OVERLAPS` | A starts before B, ends inside B | - | `DURING` | A is fully inside B | - | `STARTS` | A and B start together, A ends first | - | `FINISHES` | A and B end together, A starts later | - | `EQUALS` | Identical intervals | - | + 6 inverses | `AFTER`, `MET_BY`, `OVERLAPPED_BY`, `CONTAINS`, `STARTED_BY`, `FINISHED_BY` | - - +| Relation | Meaning | +| -------- | ------- | +| `BEFORE` | A ends before B starts | +| `MEETS` | A ends exactly when B starts | +| `OVERLAPS` | A starts before B, ends inside B | +| `DURING` | A is fully inside B | +| `STARTS` | A and B start together, A ends first | +| `FINISHES` | A and B end together, A starts later | +| `EQUALS` | Identical intervals | +| + 6 inverses | `AFTER`, `MET_BY`, `OVERLAPPED_BY`, `CONTAINS`, `STARTED_BY`, `FINISHED_BY` | ## ExplanationGenerator Generate structured step-by-step explanations for any derived conclusion: ```python -from semantica.reasoning import ExplanationGenerator +from semantica.reasoning import ExplanationGenerator, Explanation, ReasoningStep generator = ExplanationGenerator(reasoner) -explanation = generator.explain( +explanation: Explanation = generator.explain( conclusion={"subject": "John", "predicate": "has_authority", "object": "true"} ) -print(explanation.conclusion) +print(f"Conclusion: {explanation.conclusion}") print(f"Confidence: {explanation.confidence:.2f}") -print(explanation.justification.summary) +step: ReasoningStep for step in explanation.reasoning_path.steps: - indent = " " * step.depth - print(f"{indent}Step {step.depth}: {step.fact}") - print(f"{indent} via rule: '{step.rule_name}'") - print(f"{indent} premises: {step.premises}") + print(f" Step {step.depth}: {step.fact}") + print(f" via rule: '{step.rule_name}'") ``` - - Name every rule with a descriptive string. `ExplanationGenerator` includes the rule name in each derivation step — unnamed rules produce useless explanations like "rule_0 fired." Use names like `"manager_authority"` or `"transitive_location"`. - +## Choosing an Engine - - - -```python -@dataclass -class Explanation: - conclusion: Dict[str, str] # the fact being explained - confidence: float # aggregated rule confidence - reasoning_path: ReasoningPath # full derivation trace - justification: Justification # plain-language summary -``` - - - - -```python -@dataclass -class ReasoningPath: - steps: List[ReasoningStep] # ordered derivation steps - -@dataclass -class ReasoningStep: - depth: int # 0 = base fact, n = nth inference - fact: Dict[str, str] # the fact derived at this step - rule_name: str # name of the rule that fired - premises: List[Dict] # facts that triggered this rule - confidence: float # confidence at this step -``` - - - - -```python -@dataclass -class Justification: - summary: str # one-sentence natural language explanation - evidence: List[str] # list of supporting source facts -``` - - - - -## Combining Multiple Reasoning Engines - -Different engines cover different expressivity levels — compose them for richer inference: - - - - ```python - from semantica.reasoning import Reasoner, Rule, Fact, RuleType - - engine = Reasoner() - engine.add_rule(Rule( - rule_type=RuleType.FORWARD_CHAIN, - conditions=[ - {"subject": "?x", "predicate": "located_in", "object": "?y"}, - {"subject": "?y", "predicate": "located_in", "object": "?z"} - ], - conclusion={"subject": "?x", "predicate": "located_in", "object": "?z"} - )) - structural_result = engine.forward_chain() - ``` - - - ```python - from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule - - datalog = DatalogReasoner() - for fact in structural_result.derived_facts: - datalog.add_fact(DatalogFact(fact.predicate, (fact.subject, fact.obj))) - - datalog.add_rule(DatalogRule("reachable(?X, ?Z) :- located_in(?X, ?Y), reachable(?Y, ?Z).")) - datalog.evaluate() - ``` - - - ```python - from semantica.reasoning import TemporalReasoningEngine - from datetime import datetime - - temporal = TemporalReasoningEngine() - active_facts = [ - f for f in datalog.query("reachable(?X, ?Z)") - if temporal.is_active(f, at=datetime(2024, 1, 1)) - ] - ``` - - - ```python - from semantica.reasoning import ExplanationGenerator - - generator = ExplanationGenerator(engine) - explanation = generator.explain( - {"subject": "london_office", "predicate": "located_in", "object": "UK"} - ) - print(explanation.summary) - ``` - - - -## Tips and Common Pitfalls - - - **Always call `reasoner.forward_chain()` after adding facts and rules.** Adding facts and rules updates internal state but doesn't trigger inference automatically. Inference is a separate, explicit step. - +| Engine | Best For | Termination | Complexity | +| ------ | -------- | ----------- | ---------- | +| `Reasoner` | Simple IF/THEN rules, templates | Always | Low | +| `GraphReasoner` | KG-wide structural inference | Always | Medium | +| `ReteEngine` | Large rule sets (100+ rules) | Always | Low per-match | +| `SPARQLReasoner` | RDF graphs with SPARQL endpoint | Always | Low | +| `DatalogReasoner` | Recursive rules (ancestry, reachability) | Guaranteed fixpoint | Medium | +| `TemporalReasoningEngine` | Time interval relationships | Always | Low | - **Use `ReteEngine` for large rule sets.** If you have more than ~20 rules and rules can fire repeatedly, `Reasoner` re-evaluates all rules from scratch on each cycle. `ReteEngine` caches partial matches and is orders of magnitude faster for complex rule sets. - - - - **`DatalogReasoner` guarantees termination; `Reasoner` does not.** If your rules can create cycles (A derives B, B derives C, C re-derives A), `DatalogReasoner`'s semi-naive fixpoint evaluation will stop when no new facts are added. `Reasoner` has no cycle detection and may loop infinitely. - - - - **Name every rule for readable explanations.** `ExplanationGenerator.explain()` includes the rule name in each derivation step. Unnamed or generic rule names produce useless explanations like "rule_0 fired." Use descriptive names: `"manager_authority"`, `"transitive_location"`. - - - - **Use rule `priority` to control inference order.** When multiple rules could fire on the same facts, higher-priority rules fire first. This matters when a higher-priority rule produces a fact that gates a lower-priority rule's conditions. - - - - **Combine engines for maximum expressivity.** Forward-chain structural rules with `Reasoner`, then pass derived facts to `DatalogReasoner` for recursive closure, then filter by time with `TemporalReasoningEngine`. Each engine covers a different expressivity class — they compose cleanly. + For recursive rules (e.g. ancestor, reachability, transitivity), always use `DatalogReasoner` — it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner` does not handle recursion. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 01cc90c5..c88398f3 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -6,162 +6,76 @@ icon: "magnifying-glass-chart" `semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica. All extractors support three modes: pattern-based (no API key), ML-based, and LLM-based. -## Quick Start +## Exported Classes - - - ```python - from semantica.semantic_extract import CoreferenceResolver +```python +from semantica.semantic_extract import ( + # Primary extractors + NamedEntityRecognizer, # full NER coordinator (confidence_threshold, merge_overlapping) + NERExtractor, # core NER implementation used by NamedEntityRecognizer + RelationExtractor, # typed relationship extraction + TripletExtractor, # (subject, predicate, object) triplet generation + EventDetector, # event detection with participants and temporal context + CoreferenceResolver, # resolve pronouns and aliases to canonical entities + # Data types + Entity, # {id, text, type, confidence, start, end} + Relation, # {subject, predicate, object, confidence} + Event, # {type, participants, temporal, location, confidence} + CoreferenceChain, # list of mentions resolving to the same entity + # Advanced + EntityClassifier, # classify entity candidates by type + CustomEntityDetector, # pattern/dictionary-based custom entity detection + TemporalEventProcessor, # extract temporal information from events +) +``` - resolver = CoreferenceResolver() - resolved_text = resolver.resolve( - "Apple Inc. was founded in 1976. The company is headquartered in Cupertino." - ) - # "Apple Inc." replaces "The company" — consistent downstream extraction - ``` - - - ```python - from semantica.semantic_extract import NERExtractor - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) - entities = ner.extract(resolved_text) - # → [{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98, ...}] - ``` - - - ```python - from semantica.semantic_extract import RelationExtractor - - rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) - relationships = rel.extract(resolved_text, entities=entities) - # → [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", ...}] - ``` - - - ```python - from semantica.semantic_extract import ExtractionValidator - - validator = ExtractionValidator(min_confidence=0.7) - valid_entities, _ = validator.validate_entities(entities) - valid_rels, _ = validator.validate_relations(relationships) - ``` - - + + `NamedEntityRecognizer` is the high-level coordinator with confidence thresholding and overlap merging. `NERExtractor` is the lower-level implementation. For most use cases, start with `NERExtractor` for simplicity or `NamedEntityRecognizer` for fine-grained control. + ## What You Get - - - Named entity recognition: Person, Organization, Location, Date, and custom types. - - - Typed semantic relationships between entities (`founded_by`, `located_in`, etc.). - - - Direct `(subject, predicate, object)` triplet generation for RDF-ready output. - - - Event detection with participants, temporal context, and confidence scores. - - - Resolve "Apple" and "the company" to the same entity across a document. - - - Semantic role labeling, clustering, and entity similarity analysis. - - +- **`NERExtractor`** / **`NamedEntityRecognizer`** — named entity recognition: Person, Organization, Location, Date, and custom types +- **`RelationExtractor`** — typed semantic relationships between entities (`founded_by`, `located_in`, etc.) +- **`TripletExtractor`** — direct `(subject, predicate, object)` triplet generation for RDF-ready output +- **`EventDetector`** — event detection with participants, temporal context, and confidence scores +- **`CoreferenceResolver`** — resolve "Apple" and "the company" to the same entity across a document + +## Quick Start + +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor +from semantica.llms import Groq +import os + +text = "Apple Inc. was founded by Steve Jobs in Cupertino in 1976." +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +entities = NERExtractor(method="llm", llm_provider=llm).extract(text) +relationships = RelationExtractor(method="llm", llm_provider=llm).extract(text, entities=entities) +triplets = TripletExtractor(method="llm", llm_provider=llm).extract(text) +``` Semantic extraction pipeline: raw text fans into NER, Relation, and Coreference extractors, then merges into a Triplet Generator -## Extraction Methods - - - - Uses a language model to extract entities and relationships. Handles complex schemas, novel entity types, and domain-specific language. Requires an API key. - - ```python - from semantica.semantic_extract import NERExtractor - from semantica.llms import Groq - import os - - llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) - ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) - - entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino in 1976.") - ``` - - - **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. - - - Works with every Semantica LLM provider — swap `Groq` for any other provider with a one-line change. Anthropic, Gemini, Ollama, and DeepSeek are accessed via `LiteLLM` using their provider prefix: - - ```python - from semantica.llms import LiteLLM - llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")) - ner = NERExtractor(method="llm", llm_provider=llm) - ``` - - - Uses a pre-trained BERT-based NER model. High accuracy for standard entity types (Person, Organization, Location, Date) at zero API cost. - - ```python - ner = NERExtractor(method="ml", model="dslim/bert-large-NER") - entities = ner.extract(text) - ``` - - For relations, the ML backend uses the REBEL model: - - ```python - rel = RelationExtractor(method="ml") - relationships = rel.extract(text, entities=entities) - ``` - - Best for: high-throughput extraction where API cost matters and entity types are standard CoNLL/OntoNotes categories. - - - Dictionary and regex matching — extremely fast, zero API cost, zero model loading. Accuracy depends entirely on your dictionaries. - - ```python - ner = NERExtractor( - method="pattern", - custom_entities={ - "DRUG": ["aspirin", "ibuprofen", "metformin"], - "GENE": ["BRCA1", "TP53", "EGFR"] - } - ) - entities = ner.extract(text) - ``` - - For relations, uses hand-crafted rules: - - ```python - rel = RelationExtractor(method="rule") - ``` - - Best for: known entity sets (drug names, product codes, gene symbols), no-API-key environments, or as a first pass before LLM validation. - - - The pattern matcher is **case-sensitive and whitespace-sensitive**. Normalize text first with `TextNormalizer` so "BRCA1" and "brca1" both match. For fuzzy matching, use `method="ml"`. - - - - -### Method Comparison - -| Method | Speed | Cost | Accuracy | Custom Types | Best For | -| ------ | ----- | ---- | -------- | ------------ | -------- | -| `pattern` | Very fast | Free | Medium | Yes (dictionary) | Known entity sets, no-API environments | -| `ml` | Fast | Free | High | Limited | Standard types at scale, no API budget | -| `llm` | Medium | API cost | Highest | Yes (schema) | Complex schemas, novel types, best accuracy | - ## NERExtractor ```python +from semantica.semantic_extract import NERExtractor +from semantica.llms import Groq +import os + +# Pattern-based — fast, no API key, good for standard entity types +ner = NERExtractor(method="pattern") +entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino.") + +# ML-based — higher accuracy, no API cost +ner = NERExtractor(method="ml", model="dslim/bert-large-NER") +entities = ner.extract(text) + +# LLM-based — best accuracy, handles complex schemas and custom types +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) +ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) entities = ner.extract(text) ``` @@ -175,16 +89,28 @@ Output format: ] ``` -Batch processing for large corpora: +### Custom Entity Types ```python -texts = ["Text 1...", "Text 2...", "Text 3..."] -batch_results = ner.extract_batch(texts, batch_size=10) +ner = NERExtractor( + method="pattern", + custom_entities={ + "DRUG": ["aspirin", "ibuprofen", "metformin"], + "GENE": ["BRCA1", "TP53", "EGFR"] + } +) ``` + + **v0.5.0 fix:** `NERExtractor(method="llm")` no longer silently falls back to pattern extraction on custom gateways. The `response_format=json_object` parameter is now conditionally omitted for incompatible gateways, with a plain `generate()` + JSON parsing fallback applied automatically. + + ## RelationExtractor ```python +from semantica.semantic_extract import RelationExtractor + +rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) relationships = rel.extract(text, entities=entities) ``` @@ -197,9 +123,7 @@ Output format: ] ``` - - Always pass `entities=entities` from your NER output. This anchors relationships to known entity spans — improving accuracy and eliminating hallucinated entity names. - +Available methods: `"rule"` (pattern-based), `"ml"` (REBEL model), `"llm"`. ## TripletExtractor @@ -208,142 +132,84 @@ Generate RDF-ready `(subject, predicate, object)` triplets directly from text: ```python from semantica.semantic_extract import TripletExtractor -trip = TripletExtractor(method="llm", llm_provider=llm) +trip = TripletExtractor(method="llm", llm_provider=llm) triplets = trip.extract(text) # → [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", ...}] ``` -Triplets are suitable for loading directly into a triplet store or knowledge graph without a separate relation extraction step. +Triplets are suitable for loading directly into a triplet store or knowledge graph. ## EventDetector Detect events with participants and temporal context: ```python -from semantica.semantic_extract import EventDetector +from semantica.semantic_extract import EventDetector, Event extractor = EventDetector(method="llm", llm_provider=llm) -events = extractor.extract(text) +events: list[Event] = extractor.extract(text) + +for event in events: + print(f"Event type: {event.type}") + print(f"Participants: {event.participants}") + print(f"Temporal: {event.temporal}") + print(f"Confidence: {event.confidence:.2f}") ``` -Output includes: event type, participants (with roles), temporal information, location, and confidence score. +Output fields per event: `type`, `participants` (with roles), `temporal`, `location`, and `confidence`. -## SemanticAnalyzer +## CoreferenceResolver -Semantic role labeling, clustering, and similarity analysis on extracted content: +Resolve pronoun and alias references to canonical entities before extraction: ```python -from semantica.semantic_extract import SemanticAnalyzer +from semantica.semantic_extract import CoreferenceResolver -analyzer = SemanticAnalyzer() - -# Who did what to whom -roles = analyzer.label_roles(text) -# → [{"agent": "Apple", "action": "acquired", "patient": "Intel's modem unit"}] - -# Group similar entities -clusters = analyzer.cluster_entities(entities, n_clusters=5) -# → [{"cluster_id": 0, "entities": ["Apple Inc.", "Apple", "AAPL"]}, ...] - -# Pairwise semantic similarity -score = analyzer.calculate_similarity(entity_a, entity_b) -# → 0.87 -``` - -| Method | Returns | Description | -| ------ | ------- | ----------- | -| `label_roles(text)` | `List[Dict]` | Semantic role labeling (agent, action, patient) | -| `cluster_entities(entities, n_clusters)` | `List[Cluster]` | Group similar entities | -| `calculate_similarity(a, b)` | `float` | Cosine similarity between entity embeddings | -| `analyze_sentiment(text)` | `Dict` | Sentiment and subjectivity scores | - -## SemanticNetworkExtractor - -Extracts a full semantic network (nodes + typed edges) from text in one pass: - -```python -from semantica.semantic_extract import SemanticNetworkExtractor - -extractor = SemanticNetworkExtractor(method="llm", llm_provider=llm) -network = extractor.extract_network(text) - -print(f"Nodes: {len(network.nodes)}") -print(f"Edges: {len(network.edges)}") - -for edge in network.edges: - print(f" {edge.source} --[{edge.relation}]--> {edge.target} (conf: {edge.confidence:.2f})") -``` - - - -```python -@dataclass -class SemanticNetwork: - nodes: List[NetworkNode] - edges: List[NetworkEdge] - -@dataclass -class NetworkNode: - id: str - label: str # entity type (PERSON, ORG, etc.) - properties: Dict[str, Any] - -@dataclass -class NetworkEdge: - source: str - target: str - relation: str # e.g. "founded_by", "located_in" - confidence: float - metadata: Dict[str, Any] -``` - - - -## ExtractionValidator - -Validates extraction quality and filters low-confidence results: - -```python -from semantica.semantic_extract import ExtractionValidator - -validator = ExtractionValidator( - min_confidence=0.7, # drop entities below this score - require_entity_text=True, # entity text must be non-empty - max_entity_length=100, # discard suspiciously long entities +resolver = CoreferenceResolver() +resolved_text = resolver.resolve( + "Apple Inc. was founded in 1976. The company is headquartered in Cupertino." ) - -valid_entities, rejected = validator.validate_entities(entities) -valid_rels, rejected = validator.validate_relations(relationships) - -report = validator.get_quality_report(entities, relationships) -print(f"Precision estimate: {report['precision']:.2f}") +# "Apple Inc." replaces "The company" for consistent downstream extraction ``` -## Tips and Common Pitfalls +## Batch Processing - - **Run `CoreferenceResolver` before extraction.** If a paragraph says "Apple Inc. was founded in 1976. The company launched..." without resolving "the company" → "Apple Inc.", your extractor may miss the second entity or create a phantom "The company" node. - +All extractors support batch input for efficient large-scale processing: - - **Always pass `entities=` to `RelationExtractor`.** Passing the entity list from NER output anchors relationships to known entity spans — improving accuracy and eliminating hallucinated entity names. - +```python +texts = ["Text 1...", "Text 2...", "Text 3..."] - - **Validate before building the graph.** Use `ExtractionValidator(min_confidence=0.7)` to drop low-confidence entities before they reach `GraphBuilder`. Noisy extractions produce noisy graphs that corrupt analytics and search. - +ner = NERExtractor(method="llm", llm_provider=llm) +batch_results = ner.extract_batch(texts, batch_size=10) +``` - - **Set `max_retries=3` for LLM extractors.** API calls fail transiently. Setting `max_retries=3` prevents pipeline crashes on flaky network conditions without slowing down the happy path. - +## Using All Extractors Together - - **Don't mix extraction methods mid-pipeline.** If you extract entities with `pattern` and relations with `llm`, entity names in the relation output may not match the pattern-extracted entity IDs — causing alignment failures in `GraphBuilder`. Use the same method throughout, or normalize entity names before the relation step. - +The standard extraction pipeline — entities → relationships → triplets: - - **Batch large inputs.** Call `ner.extract_batch(texts, batch_size=10)` rather than looping over individual texts. Batch mode is significantly faster for both ML (GPU batching) and LLM (fewer API round-trips with prompt packing). - +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor +from semantica.llms import Groq +import os + +llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + +ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) +rel = RelationExtractor(method="llm", llm_provider=llm, max_retries=3) +trip = TripletExtractor(method="llm", llm_provider=llm, max_retries=3) + +entities = ner.extract(text) +relationships = rel.extract(text, entities=entities) +triplets = trip.extract(text) +``` + +## Extraction Method Comparison + +| Method | Speed | Cost | Accuracy | Custom Types | +| ------ | ----- | ---- | -------- | ------------ | +| `pattern` | Very fast | Free | Medium | Yes (dictionary) | +| `ml` | Fast | Free | High | Limited | +| `llm` | Medium | API cost | Highest | Yes (schema) | diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 2da34230..c123bdfd 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -6,6 +6,31 @@ icon: "wrench" `semantica.utils` provides shared infrastructure used throughout Semantica. Most users won't call it directly, but its APIs are available when you need fine-grained control over logging, validation, progress tracking, or error handling. +## Exported Classes + +```python +from semantica.utils import ( + # Logging + setup_logging, # configure root logger — level, format (json/text) + get_logger, # get a named logger instance + log_performance, # @decorator — logs function name, duration, exception + # Validation + validate_entity, # validate entity dict structure, raises ValidationError + validate_config, # validate config dict against schema, raises ValidationError + # Progress tracking + ProgressTracker, # class-based tracker with ETA + track_progress, # wraps any iterable with live progress bar + # Helpers + clean_text, # normalize whitespace, strip control characters + hash_data, # deterministic SHA-256 hash of any serializable object + safe_filename, # sanitize a string for use as a filename + # Exceptions + SemanticaError, # base exception for all Semantica errors + ValidationError, # raised when input fails validation + ProcessingError, # raised during extraction, graph build, or pipeline step +) +``` + ## What You Get