diff --git a/docs/assets/custom.css b/docs/assets/custom.css new file mode 100644 index 00000000..97b5e3ee --- /dev/null +++ b/docs/assets/custom.css @@ -0,0 +1,127 @@ +/* ============================================================ + TABLE — row & cell hover highlighting + ============================================================ */ +table tbody tr { + transition: background-color 0.15s ease; + cursor: default; +} + +table tbody tr:hover { + background-color: rgba(16, 185, 129, 0.07) !important; +} + +table tbody tr:hover td { + background-color: transparent !important; +} + +table td, +table th { + transition: background-color 0.15s ease; +} + +/* ============================================================ + CODE BLOCKS — glow on hover + ============================================================ */ +pre, +[class*="codeblock"], +[class*="code-group"], +[class*="CodeBlock"], +[data-rehype-pretty-code-fragment] { + transition: box-shadow 0.2s ease, border-color 0.2s ease !important; +} + +pre:hover, +[class*="codeblock"]:hover, +[class*="CodeBlock"]:hover, +[data-rehype-pretty-code-fragment]:hover { + box-shadow: 0 0 0 1.5px rgba(16, 185, 129, 0.35), + 0 4px 24px rgba(16, 185, 129, 0.09) !important; +} + +/* ============================================================ + CARDS — lift + glow on hover + ============================================================ */ +[class*="card"], +[class*="Card"], +[data-card], +.group\/card { + transition: transform 0.2s ease, + box-shadow 0.2s ease, + border-color 0.2s ease !important; +} + +[class*="card"]:hover, +[class*="Card"]:hover, +[data-card]:hover, +.group\/card:hover { + transform: translateY(-2px) !important; + box-shadow: 0 8px 32px rgba(16, 185, 129, 0.13), + 0 0 0 1px rgba(16, 185, 129, 0.28) !important; +} + +/* ============================================================ + CALLOUTS / NOTES / TIPS — subtle left-border highlight + ============================================================ */ +[class*="callout"], +[class*="Callout"], +[class*="admonition"] { + transition: box-shadow 0.2s ease !important; +} + +[class*="callout"]:hover, +[class*="Callout"]:hover, +[class*="admonition"]:hover { + box-shadow: inset 3px 0 0 rgba(16, 185, 129, 0.55), + 0 2px 12px rgba(16, 185, 129, 0.06) !important; +} + +/* ============================================================ + STEPS — highlight active step on hover + ============================================================ */ +[class*="step"], +[class*="Step"] { + transition: background-color 0.15s ease !important; +} + +[class*="step"]:hover, +[class*="Step"]:hover { + background-color: rgba(16, 185, 129, 0.04) !important; +} + +/* ============================================================ + INLINE CODE — subtle green tint on hover + ============================================================ */ +:not(pre) > code { + transition: background-color 0.15s ease, color 0.15s ease !important; + cursor: text; +} + +:not(pre) > code:hover { + background-color: rgba(16, 185, 129, 0.15) !important; +} + +/* ============================================================ + NAV / SIDEBAR LINKS — subtle hover underline accent + ============================================================ */ +nav a, +[class*="sidebar"] a, +[class*="Sidebar"] a { + transition: color 0.15s ease !important; +} + +/* ============================================================ + HIDE THEME TOGGLE (moon / sun emoji button) + ============================================================ */ +button[aria-label="Switch to light mode"], +button[aria-label="Switch to dark mode"], +button[aria-label*="theme"], +button[title*="theme"], +button[title*="Theme"], +[data-theme-toggle], +[class*="ThemeToggle"], +[class*="theme-toggle"], +[class*="ColorModeToggle"], +[class*="DarkModeToggle"] { + display: none !important; + pointer-events: none !important; +} diff --git a/docs/cli-setup.md b/docs/cli-setup.md index 6edb1987..dea48d68 100644 --- a/docs/cli-setup.md +++ b/docs/cli-setup.md @@ -169,9 +169,11 @@ No other environment variables are read by these commands. ## Troubleshooting -### `command not found` + -The executables are placed in the `bin/` (Linux/Mac) or `Scripts/` (Windows) directory of the active Python environment. If the command is not found, that directory is likely not on `PATH`. + + +The executables land in `bin/` (Linux/Mac) or `Scripts/` (Windows) of the active Python environment. If the command is not found, that directory is likely not on `PATH`. Activate your virtual environment first: @@ -185,23 +187,27 @@ Find where pip placed the scripts: ```bash python -m site --user-scripts # user-level install -pip show -f semantica # shows installed files +pip show -f semantica # shows all installed files ``` -### Command found but crashes on import + + + ```bash pip install --upgrade semantica python -c "import semantica; print(semantica.__version__)" ``` -If you have multiple Python environments, make sure you are installing into the same one the shell resolves: +If you have multiple Python environments, install into the one the shell resolves: ```bash python -m pip install semantica ``` -### `semantica-explorer`: "uvicorn is required" + + + The Explorer extras are not included in the base install: @@ -209,7 +215,9 @@ The Explorer extras are not included in the base install: pip install semantica[explorer] ``` -### `semantica-mcp` silent failure in a MCP client + + + The MCP server communicates over stdio. Test it directly from the shell first: @@ -219,10 +227,16 @@ echo '{"jsonrpc":"2.0","id":1,"method":"ping","params":{}}' | semantica-mcp A response of `{"jsonrpc":"2.0","id":1,"result":{}}` confirms the server is working. If you see nothing, check that the command is on `PATH` and the base package is installed. -### Windows: DLL errors on startup + + + Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency required by PyTorch and related packages, not a Semantica bug. + + + + ## Next Steps diff --git a/docs/concepts.md b/docs/concepts.md index b46f7ca3..cae73133 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -4,9 +4,9 @@ description: "The fundamental ideas behind Semantica: knowledge graphs, reasonin icon: "book-open" --- - + New here? Start with [Getting Started](getting-started) for hands-on examples, then return here for deeper understanding. - + Semantica transforms unstructured data: documents, web pages, reports, databases: into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources. diff --git a/docs/docs.json b/docs/docs.json index f3aa1e25..1059ebc6 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -24,8 +24,10 @@ } }, "appearance": { - "default": "dark" + "default": "dark", + "strict": true }, + "css": "/assets/custom.css", "background": { "color": { "dark": "#080C10", @@ -206,41 +208,29 @@ } ] } - ], - "global": { - "anchors": [ - { - "anchor": "Discord", - "href": "https://discord.gg/sV34vps5hH", - "icon": "discord" - }, - { - "anchor": "GitHub", - "href": "https://github.com/semantica-agi/semantica", - "icon": "github" - }, - { - "anchor": "PyPI", - "href": "https://pypi.org/project/semantica/", - "icon": "python" - }, - { - "anchor": "Follow on X", - "href": "https://x.com/BuildSemantica", - "icon": "x-twitter" - } - ] - } + ] }, "navbar": { "links": [ { - "label": "GitHub", - "href": "https://github.com/semantica-agi/semantica" + "label": "Discord", + "href": "https://discord.gg/sV34vps5hH", + "icon": "discord" }, { - "label": "Discord", - "href": "https://discord.gg/sV34vps5hH" + "label": "GitHub", + "href": "https://github.com/semantica-agi/semantica", + "icon": "github" + }, + { + "label": "PyPI", + "href": "https://pypi.org/project/semantica/", + "icon": "python" + }, + { + "label": "Follow on X", + "href": "https://x.com/BuildSemantica", + "icon": "x-twitter" } ], "primary": { diff --git a/docs/explorer-setup.md b/docs/explorer-setup.md index 63274084..a9610a57 100644 --- a/docs/explorer-setup.md +++ b/docs/explorer-setup.md @@ -190,28 +190,34 @@ python -m semantica.explorer --graph my_graph.json --port 8080 ## Common Startup Errors -**`Error: graph file not found: my_graph.json`** + -The path passed to `--graph` must point to an existing file. The CLI checks with `os.path.isfile()` before attempting to load anything. + + +The path passed to `--graph` must point to an existing file. The CLI checks with `os.path.isfile()` before loading anything. ```bash # Confirm the file exists -ls my_graph.json # Linux / Mac -dir my_graph.json # Windows +ls my_graph.json # Linux / Mac +dir my_graph.json # Windows # Use the full path if needed semantica-explorer --graph /absolute/path/to/my_graph.json ``` -**`Error: uvicorn is required`** + -The `[explorer]` extra was not installed: + + +The `[explorer]` extra was not installed alongside the base package: ```bash pip install semantica[explorer] ``` -**Explorer launches but shows zero nodes** + + + The file loaded but contains no nodes. Verify with Python: @@ -222,9 +228,11 @@ g.load_from_file("my_graph.json") print(g.stats()) # check node_count ``` -A `node_count` of `0` means the file was saved empty or the nodes key is absent. Make sure you called `add_node` before `save_to_file`. +A `node_count` of `0` means the graph was saved before any nodes were added. Make sure you called `add_node()` before `save_to_file()`. -**`Connection refused` from another machine** + + + The default `--host 127.0.0.1` only accepts connections from the same machine. To allow remote access: @@ -232,9 +240,15 @@ The default `--host 127.0.0.1` only accepts connections from the same machine. T semantica-explorer --graph my_graph.json --host 0.0.0.0 ``` -**Browser tab does not open** + -This is expected in headless, SSH, and container environments. Add `--no-browser` to suppress the warning and open `http://127.0.0.1:8000` in a browser that has network access to the server. + + +Expected in headless, SSH, and container environments. Pass `--no-browser` to suppress the warning, then open `http://127.0.0.1:8000` in a browser that has network access to the server. + + + + ## What Explorer Gives You diff --git a/docs/faq.md b/docs/faq.md index dc9b0239..8095a408 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -4,9 +4,9 @@ description: "Common questions about Semantica: installation, features, integrat icon: "circle-question" --- - + Use **Ctrl+F** / **Cmd+F** to search this page. Common jumps: [Installation](#installation) · [Data & Features](#data--features) · [Troubleshooting](#troubleshooting) - + ## Quick Answers diff --git a/docs/glossary.md b/docs/glossary.md index 264889e1..d0776e6a 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -4,9 +4,9 @@ description: "Reference definitions for terms and concepts used throughout Seman icon: "book" --- - + Use Ctrl+F / Cmd+F to search this page for a specific term. - + A quick-reference dictionary of every concept, data structure, algorithm, and standard referenced in Semantica's documentation and codebase. diff --git a/docs/installation.md b/docs/installation.md index 8099dbcd..7aa1f179 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -120,50 +120,66 @@ pip install git+https://github.com/semantica-agi/semantica.git@main ## Troubleshooting -### ModuleNotFoundError + -Make sure you're in the right environment: + + +Make sure you're in the right virtual environment: ```bash pip list | grep semantica pip install --upgrade semantica ``` -### Installation fails with dependency errors + + + ```bash pip install --upgrade pip pip install build wheel -pip install semantica --no-deps # install without optional deps first +pip install semantica --no-deps # install core first, then add extras ``` -### GPU dependencies fail + -Install CPU-only first, then add GPU support: + + +Install CPU-only first, then layer in GPU support: ```bash pip install semantica pip install semantica[gpu] ``` -### Permission denied + + + ```bash pip install --user semantica # or use a virtual environment ``` -### Windows `[all]` install fails + -This was fixed in **v0.5.0**. Upgrade to the latest release: + + +Fixed in **v0.5.0**. Upgrade to the latest release: ```bash pip install --upgrade semantica ``` -### Windows PyTorch DLL errors + + + Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency, not a Semantica bug. + + + + ## Next Steps diff --git a/docs/learning-more.md b/docs/learning-more.md index b82d1258..b9a52e97 100644 --- a/docs/learning-more.md +++ b/docs/learning-more.md @@ -108,7 +108,9 @@ All settings can be overridden with environment variables: no code changes neede ## Troubleshooting -### `ModuleNotFoundError: No module named 'semantica'` + + + Verify installation and that the correct Python environment is active: @@ -124,16 +126,20 @@ pip install "semantica[llm-openai]" # OpenAI provider pip install "semantica[gpu]" # GPU acceleration ``` -### `AuthenticationError` + -Set your API key as an environment variable: never hardcode keys in source files: + + +Set your API key as an environment variable — never hardcode keys in source files: ```bash export OPENAI_API_KEY="sk-..." export GROQ_API_KEY="gsk_..." ``` -### `MemoryError` or OOM crashes + + + Switch from the default in-memory NetworkX backend to a persistent graph database: @@ -147,7 +153,9 @@ builder = GraphBuilder(merge_entities=True, graph_store=store) Also reduce batch sizes and enable streaming ingestion for large corpora. -### Slow processing on large datasets + + + Enable parallel execution and GPU acceleration: @@ -162,7 +170,9 @@ pipeline.run(sources) pip install "semantica[gpu]" # CUDA-backed embeddings ``` -### Windows `[all]` installation fails + + + Fixed in **v0.5.0**. Upgrade: @@ -172,41 +182,66 @@ pip install --upgrade semantica Or install extras individually: `pip install "semantica[core]"`, then add `[llm-openai]`, `[gpu]`, etc. as needed. -### cp1252 encoding crash on Windows + -Fixed in **v0.5.0**. For earlier versions, pass encoding explicitly or set the environment variable: + + +Fixed in **v0.5.0**. For earlier versions, set the encoding environment variable: ```bash set PYTHONIOENCODING=utf-8 ``` + + + + ## Performance Optimization -### Backend Selection + + + | Operation | NetworkX (default) | Neo4j / FalkorDB | | :--------- | :------------------ | :---------------- | | Graph construction | Fast | Moderate | | Query performance | Moderate | Fast | -| Scalability | Low: in-memory only | High: persistent | +| Scalability | In-memory only | Persistent, production-scale | | Recommended for | Development, small graphs | Production, large corpora | Use NetworkX for local development and prototyping. Switch to a persistent backend before deploying to production. -### Batch Processing + + + Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM: a good starting point is 1,000 documents per batch on a 16 GB machine. -### Deduplication v2 +```python +from semantica.pipeline import Pipeline -If deduplication is a bottleneck, switch from v1 strategies to v2: +pipeline = Pipeline(workers=8, batch_size=32) +pipeline.run(sources) +``` + + + + + +If deduplication is a bottleneck, switch from v1 strategies to the v2 engine: ```python resolver = EntityResolver() merged = resolver.resolve(entities, strategy="semantic_v2") # up to 7x faster ``` +The `blocking_v2`, `hybrid_v2`, and `semantic_v2` strategies reduce O(n²) comparisons via candidate blocking before similarity scoring. + + + + + ## Security Best Practices diff --git a/docs/modules.md b/docs/modules.md index f0c18bb9..8ac60fdf 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -4,9 +4,9 @@ description: "Every Semantica module works independently: use only what you need icon: "puzzle-piece" --- - + Looking for a quick reference? Jump to the [Module Index](#module-index) at the bottom. - + Semantica is organized into **27 modules** across six logical layers. Each module is independently importable: you never pay for what you don't use. diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index e31c6fc0..18f98754 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -97,6 +97,10 @@ icon: "clock-rotate-left" + + **Snapshot before every destructive operation.** Call `manager.create_snapshot()` before running deduplication, conflict resolution, or merge operations. `restore_snapshot()` is only possible if a snapshot exists before the change. + + ## TemporalVersionManager Version control for knowledge graphs: snapshot, diff, and rollback. @@ -156,6 +160,10 @@ for item in diff["entities_modified"]: print(" %s: %s -> %s" % (field, change["from"], change["to"])) ``` + + **Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"`: use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes. + + ```python @@ -237,6 +245,10 @@ print("Properties added: ", diff["properties_added"]) The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production: otherwise your entire version history disappears on restart. + + **Use `SQLiteVersionStorage` in production.** The default in-memory storage loses all version history when the process exits. Pass `storage_path="versions.db"` to `TemporalVersionManager` or create `SQLiteVersionStorage(db_path="versions.db")` explicitly. + + ## Integrity Verification SHA-256 checksums detect any unauthorized modification to a graph between snapshots: @@ -311,6 +323,10 @@ print("Added: %d | Removed: %d | Modified: %d" % ( s["entities_added"], s["entities_removed"], s["entities_modified"])) ``` + + **Use `list_versions()` and `diff()` for compliance reviews.** `manager.list_versions()` returns a list of metadata dicts (with `label`, `author`, `timestamp`, `checksum`). Run `verify_checksum(snapshot)` on the dict returned by `get_version()` to confirm integrity before any export. + + Use `verify_checksum()` before any compliance export to confirm snapshot integrity: ```python @@ -348,24 +364,6 @@ for record in history: -## Tips and Common Pitfalls - - - **Use `SQLiteVersionStorage` in production.** The default in-memory storage loses all version history when the process exits. Pass `storage_path="versions.db"` to `TemporalVersionManager` or create `SQLiteVersionStorage(db_path="versions.db")` explicitly. - - - - **Snapshot before every destructive operation.** Call `manager.create_snapshot()` before running deduplication, conflict resolution, or merge operations. `restore_snapshot()` is only possible if a snapshot exists before the change. - - - - **Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"`: use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes. - - - - **Use `list_versions()` and `diff()` for compliance reviews.** `manager.list_versions()` returns a list of metadata dicts (with `label`, `author`, `timestamp`, `checksum`). Run `verify_checksum(snapshot)` on the dict returned by `get_version()` to confirm integrity before any export. - - W3C PROV-O lineage tracking. diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index 37a24eab..da2ea361 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -139,6 +139,10 @@ Semantica's conflict detection makes disagreements explicit and actionable: + + **Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder: you lose the original source attribution. + + ## ConflictDetector ```python @@ -160,6 +164,10 @@ conflicts = detector.detect_value_conflicts(entities, "revenue") | `LOGICAL` | Logically inconsistent property combinations | `is_alive=True` but `death_date` set | | `RELATIONSHIP` | Inconsistent relationship properties across sources | Edge weight 0.9 vs 0.3 from two sources | + + **`TEMPORAL` and `LOGICAL` conflict detection is not implemented on `ConflictDetector` directly.** The `ConflictType` enum includes these types for use in custom pipelines, but the detector class only implements `detect_value_conflicts`, `detect_type_conflicts`, `detect_relationship_conflicts`, and `detect_entity_conflicts`. + + Run targeted detection by type: ```python @@ -199,6 +207,10 @@ for result in results: print(" Strategy: %s Confidence: %.2f" % (result.resolution_strategy, result.confidence)) ``` + + **Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"`: high severity means the disagreement is large and the stakes of getting it wrong are high. + + ### Choosing a Resolution Strategy @@ -314,6 +326,14 @@ chain = tracker.get_traceability_chain("apple_inc") - Credibility scores default to 0.50 for any source not explicitly set - `SourceTracker` stores property-level provenance: so you can trace exactly which source contributed each value + + **Always set credibility scores.** The default credibility is 0.50 for all sources. Without explicit scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. The power of this strategy is in the differentiation. + + + + **Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain. + + ## ConflictAnalyzer ```python @@ -338,6 +358,14 @@ for t in trends: - `analyze_conflicts()["by_source"]` includes `counts` and `top_sources`: sources appearing in many conflicts may have upstream data quality problems - `analyze_trends()` returns a list of per-period dicts (`period`, `conflict_count`, `trend`, `trend_direction`): `trend` is `"increasing"`, `"decreasing"`, or `"stable"` + + **Use `analyze_conflicts()["by_source"]["top_sources"]` to identify bad data feeds.** A single source appearing in many conflicts is a data quality problem upstream, not a conflict to resolve record by record. Flag it and investigate the source pipeline. + + + + **Severity is a string label, not a score.** `ConflictDetector` assigns `"critical"`, `"high"`, or `"medium"` based on property importance and value differences. Critical fields (`id`, `name`, `type`, `revenue`) always yield `"critical"`. Domain context determines what to prioritize. + + ## InvestigationGuideGenerator Auto-generate human-readable investigation checklists for conflicts requiring manual or expert review: @@ -436,36 +464,6 @@ class InvestigationStep: -## Tips and Common Pitfalls - - - **Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder: you lose the original source attribution. - - - - **Always set credibility scores.** The default credibility is 0.50 for all sources. Without explicit scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. The power of this strategy is in the differentiation. - - - - **Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"`: high severity means the disagreement is large and the stakes of getting it wrong are high. - - - - **`TEMPORAL` and `LOGICAL` conflict detection is not implemented on `ConflictDetector` directly.** The `ConflictType` enum includes these types for use in custom pipelines, but the detector class only implements `detect_value_conflicts`, `detect_type_conflicts`, `detect_relationship_conflicts`, and `detect_entity_conflicts`. - - - - **Use `analyze_conflicts()["by_source"]["top_sources"]` to identify bad data feeds.** A single source appearing in many conflicts is a data quality problem upstream, not a conflict to resolve record by record. Flag it and investigate the source pipeline. - - - - **Severity is a string label, not a score.** `ConflictDetector` assigns `"critical"`, `"high"`, or `"medium"` based on property importance and value differences. Critical fields (`id`, `name`, `type`, `revenue`) always yield `"critical"`. Domain context determines what to prioritize. - - - - **Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain. - - Resolve duplicate entities before conflict detection. diff --git a/docs/reference/context.md b/docs/reference/context.md index da311cea..ed5203de 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -73,41 +73,6 @@ icon: "brain" -## Getting Started - -```python -from semantica.context import AgentContext, ContextGraph -from semantica.vector_store import VectorStore - -context = AgentContext( - vector_store=VectorStore(backend="faiss", dimension=768), - knowledge_graph=ContextGraph(advanced_analytics=True), - decision_tracking=True, # requires knowledge_graph to be set -) - -# Store a fact -memory_id = context.store( - "GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%", - metadata={"source": "openai_blog", "date": "2024-01"} -) - -# Retrieve by semantic similarity -results = context.retrieve("LLM benchmark comparisons", max_results=5) -for r in results: - print("{} (score: {:.3f})".format(r["content"], r["score"])) - -# Record a decision -decision_id = context.record_decision( - category="model_selection", - scenario="Choose LLM for production reasoning pipeline", - reasoning="GPT-4 benchmark advantage justifies 3x cost increase", - outcome="selected_gpt4", - confidence=0.91, - entities=["gpt-4", "gpt-3.5"], - decision_maker="pipeline_agent", -) -``` - ## Quick Start @@ -228,9 +193,6 @@ decision_id = context.record_decision( precedents = context.find_precedents("model selection", limit=5) ``` - - `decision_tracking=True` silently no-ops unless `knowledge_graph` is also provided at construction time. - Load a pre-built knowledge graph and answer complex questions with multi-hop graph traversal. @@ -323,9 +285,13 @@ decision_id = context.record_decision( | `advanced_analytics` | `bool` | `True` | Enables PageRank, centrality, and community analysis | | `kg_algorithms` | `bool` | `True` | Adds path-finding and link prediction | - - `decision_tracking=True` has no effect unless `knowledge_graph` is also provided. Both must be set at construction time for decision tracking to be active. - + + **Set `retention_days` to avoid memory bloat.** The default of `30` prunes automatically. Compliance-critical agents may need `retention_days=None` with explicit archival via `export()`. + + + + **Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` so the FAISS index survives process restarts. + ### Memory Methods @@ -344,6 +310,10 @@ decision_id = context.record_decision( | `export(conversation_id, format)` | `str \| Dict` | Export memories as JSON or dict | | `import_data(data, format)` | `int` | Import memories from JSON or dict | + + **`retrieve()` uses `max_results=`, not `top_k=`.** The parameter is `max_results` (default `5`). Pass `use_graph=True` to force GraphRAG or `use_graph=False` to force vector-only retrieval regardless of whether a `knowledge_graph` is configured. + + ### Conversation Methods ```python @@ -396,6 +366,14 @@ print("Sources used: {}".format(result["num_sources"])) | `trace_decision_explainability(decision_id)` | `Dict` | Full explainability: causes, effects, relationship paths | | `get_policy_engine()` | `PolicyEngine` | Access the active `PolicyEngine` instance | + + `decision_tracking=True` requires `knowledge_graph` to also be set. Without it, `record_decision()` raises `RuntimeError`. + + + + **Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. Surface precedents to the LLM as context: "we chose X for similar reasons before." + + ### Checkpoint Methods **Ideal for auditing reasoning loops**: take a snapshot before and after a pass to see exactly what changed: @@ -515,7 +493,7 @@ print("Reachable: {}, hops: {}".format(path["reachable"], path["hop_count"])) ``` -## AgentMemory (Low-Level) +## AgentMemory For fine-grained control over memory storage and retrieval: @@ -636,6 +614,10 @@ print("Entities:", web["statistics"]["total_entities"]) print("Links: ", web["statistics"]["total_links"]) ``` + + **`EntityLinker.link_entities()` links two entity IDs, not a list.** Call `link_entities(entity1_id, entity2_id, link_type)` to create a typed edge between two known IDs. For linking entities extracted from text, use `link(text, entities=[...])` instead. + + `LinkedEntity` fields returned by `link()`: | Field | Type | Description | @@ -907,37 +889,6 @@ class EntityLink: - -## Tips and Common Pitfalls - - - **`decision_tracking=True` silently does nothing without `knowledge_graph`.** Both must be set at construction. Passing only `decision_tracking=True` without a `knowledge_graph` instance leaves the decision backend uninitialised: `record_decision()` will raise `RuntimeError`. - - - - **Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore`: without it the FAISS index lives only in memory and is lost on shutdown. - - - - **Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. Surface precedents to the LLM as context: "we chose X for similar reasons before." - - - - **`retrieve()` uses `max_results=`, not `top_k=`.** The parameter is `max_results` (default `5`). Pass `use_graph=True` to force GraphRAG or `use_graph=False` to force vector-only retrieval regardless of whether a `knowledge_graph` is configured. - - - - **Set `retention_days` to avoid memory bloat.** The default `AgentContext.retention_days=30` prunes automatically. Compliance-critical agents may need `retention_days=None` with explicit archival via `export()`. - - - - **Use `checkpoint()` + `diff_checkpoints()` to audit reasoning loops.** Take a snapshot before and after a reasoning pass to see exactly which decisions and relationships were added. - - - - **`EntityLinker.link_entities()` links two entity IDs, not a list.** Call `link_entities(entity1_id, entity2_id, link_type)` to create a typed edge between two known IDs. For linking entities extracted from text, use `link(text, entities=[...])` instead. - - Embedding storage backend for memory retrieval. @@ -953,7 +904,11 @@ class EntityLink: -### Cookbooks - -- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb): memory and decision tracking · Intermediate -- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb): production FAISS + Neo4j setup · Advanced + + + Memory and decision tracking · Intermediate + + + Production FAISS + Neo4j setup · Advanced + + diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index d165174d..4fde0158 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -87,6 +87,10 @@ for op in operations: )) ``` + + **Normalize entity names before deduplication.** Canonical forms such as `"Apple Inc."` vs `"apple inc"` may score below threshold due to case alone. Run `EntityNormalizer` or `TextNormalizer` first for reliable matching. + + ## DuplicateDetector Find duplicate entity pairs: @@ -124,6 +128,14 @@ new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}] candidates = detector.incremental_detect(new_entities, entities) ``` + + **Tune `similarity_threshold` before `confidence_threshold`.** The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with `similarity_threshold=0.7` and raise it to reduce false positives. + + + + **Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity. + + ### `detect_duplicates()` detection methods The `method=` parameter of the `detect_duplicates()` convenience function controls how @@ -148,6 +160,10 @@ method used internally: | `reasons` | `List[str]` | Why they are considered duplicates | | `metadata` | `Dict` | Additional metadata | + + **`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score`: not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`. + + ### DuplicateGroup fields | Field | Type | Description | @@ -188,6 +204,10 @@ history = merger.get_merge_history() print("Total merges performed:", len(history)) ``` + + **`merge_entities()` and `EntityMerger.merge_duplicates()` return `List[MergeOperation]`, not a list of entity dicts.** Access `.merged_entity` on each operation to get the merged dict. + + ### Merge strategies Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group()`: @@ -229,6 +249,10 @@ merger.merge_strategy_manager.add_property_rule( operations = merger.merge_duplicates(entities) ``` + + **`PropertyMergeRule` is a dataclass, not an Enum.** The merge strategy Enum is `MergeStrategy` (`KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL`). Per-property rules are added via `merger.merge_strategy_manager.add_property_rule(name, strategy)`. + + ### MergeOperation fields | Field | Type | Description | @@ -427,32 +451,6 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name") -## Tips and Common Pitfalls - - - **`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score`: not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`. - - - - **`merge_entities()` and `EntityMerger.merge_duplicates()` return `List[MergeOperation]`, not a list of entity dicts.** Access `.merged_entity` on each operation to get the merged dict. - - - - **`PropertyMergeRule` is a dataclass, not an Enum.** The merge strategy Enum is `MergeStrategy` (`KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL`). Per-property rules are added via `merger.merge_strategy_manager.add_property_rule(name, strategy)`. - - - - **Tune `similarity_threshold` before `confidence_threshold`.** The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with `similarity_threshold=0.7` and raise it to reduce false positives. - - - - **Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity. - - - - **Normalize entity names before deduplication.** Canonical forms such as `"Apple Inc."` vs `"apple inc"` may score below threshold due to case alone. Run `EntityNormalizer` or `TextNormalizer` first for reliable matching. - - Detect value conflicts between non-duplicate entities. diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index b5d1691f..e1719285 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -83,6 +83,10 @@ Semantica uses embeddings for: Default model is `BAAI/bge-small-en-v1.5`. Zero cost, zero GPU, works on any machine. + + + **FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers: passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control. + Broad model selection via HuggingFace. Runs locally, no API key. @@ -103,6 +107,10 @@ Semantica uses embeddings for: ``` Popular models: `all-MiniLM-L6-v2` (fast, small), `all-mpnet-base-v2` (balanced), `BAAI/bge-large-en-v1.5` (high accuracy). + + + **Sequence length limits.** Most sentence-transformers models have a 512-token limit. Text beyond that is silently truncated. Use `TextSplitter(method="hierarchical")` + `HierarchicalPooling` for long documents. + BAAI/bge models via sentence-transformers. State-of-the-art retrieval performance, runs locally. @@ -178,6 +186,10 @@ score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosin print(f"Similarity: {score:.3f}") ``` + + **Always use the same model for indexing and querying.** Vectors from different models are not comparable: they live in different vector spaces. Switching models requires re-embedding your entire corpus. + + To switch provider after construction: ```python @@ -344,6 +356,14 @@ dim = embedder.get_embedding_dimension() - If FastEmbed or sentence-transformers is unavailable, falls back to a 128-dimensional hash-based embedding. Hash embeddings are deterministic but not semantic: do not use in production. - Large batches are chunked internally by the underlying library to avoid OOM. + + **Dimension mismatch.** The dimension you pass to your vector store must exactly match your embedding model's output. `BAAI/bge-small-en-v1.5` → 384, `all-MiniLM-L6-v2` → 384, `all-mpnet-base-v2` → 768, `BAAI/bge-large-en-v1.5` → 1024. Check with `embedder.get_embedding_dimension()` before creating the store. + + + + **Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()`: if it returns `"fallback"`, install your intended provider. + + ## Provider Stores Use provider stores directly when you need fine-grained control over a single backend: @@ -378,6 +398,10 @@ store = ProviderStoreFactory.create(provider="bge", model_name="BAAI/bge-large-e `LlamaStore` exists in the module but is a placeholder: it does not connect to Ollama and always raises `ProcessingError` at embed time. Do not use it in production. + + **LlamaStore is not functional.** `LlamaStore` exists in the module but does not connect to Ollama. It always raises `ProcessingError` at embed time. Use `FastEmbedStore` for local ONNX-based embeddings or `BGEStore` for sentence-transformers-based local embeddings instead. + + ## Pooling Strategies Pooling aggregates a set of embeddings into a single vector: useful when you have multiple chunk embeddings to combine: @@ -609,32 +633,6 @@ providers = check_available_providers() # → {"sentence_transformers": True, "fastembed": True, "openai": False} ``` -## Tips and Common Pitfalls - - - **Dimension mismatch.** The dimension you pass to your vector store must exactly match your embedding model's output. `BAAI/bge-small-en-v1.5` → 384, `all-MiniLM-L6-v2` → 384, `all-mpnet-base-v2` → 768, `BAAI/bge-large-en-v1.5` → 1024. Check with `embedder.get_embedding_dimension()` before creating the store. - - - - **LlamaStore is not functional.** `LlamaStore` exists in the module but does not connect to Ollama. It always raises `ProcessingError` at embed time. Use `FastEmbedStore` for local ONNX-based embeddings or `BGEStore` for sentence-transformers-based local embeddings instead. - - - - **Sequence length limits.** Most sentence-transformers models have a 512-token limit. Text beyond that is silently truncated. Use `TextSplitter(method="hierarchical")` + `HierarchicalPooling` for long documents. - - - - **FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers: passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control. - - - - **Always use the same model for indexing and querying.** Vectors from different models are not comparable: they live in different vector spaces. Switching models requires re-embedding your entire corpus. - - - - **Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()`: if it returns `"fallback"`, install your intended provider. - - Store and search the generated embeddings. diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index 555e5aab..3495eef4 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -103,6 +103,10 @@ The `semantica-explorer` command accepts exactly four flags: There are no flags for authentication, CORS, or log level in the CLI. CORS allowed origins are configured via the `EXPLORER_CORS_ORIGINS` environment variable (comma-separated, default: `http://localhost:5173,http://127.0.0.1:5173`). + + **CORS origins are configured via environment variable.** Set `EXPLORER_CORS_ORIGINS` to a comma-separated list of allowed origins before launching (e.g. `EXPLORER_CORS_ORIGINS="http://myapp.example.com"`). + + ```bash # Full example EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ @@ -144,6 +148,10 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ - **Filter by entity type**: `GET /api/graph/nodes?type=Person` - **Semantic neighborhood**: `GET /api/graph/semantic-neighborhood?node_id=&top_k=20` - **Distance matrix**: `POST /api/graph/distance-matrix` + + + **Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting: the force-directed layout becomes unusable on very large graphs. + Ontology lifecycle management in the browser: @@ -164,6 +172,10 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ - **Enrich: deduplication**: `POST /api/enrich/dedup` - **Enrich: entity extraction**: `POST /api/enrich/extract` - **Temporal**: `GET /api/temporal/snapshot`, `GET /api/temporal/diff`, `GET /api/temporal/bounds` + + + **Use `/api/analytics/validation` to check graph quality.** The validator detects orphaned nodes, missing types, and other structural issues before you expose the graph to downstream pipelines. + Decision tracking and provenance queries: @@ -174,6 +186,10 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ - **Compliance**: `GET /api/decisions/{id}/compliance` - **Provenance**: `GET /api/provenance?node_id=`, `GET /api/provenance/report?node_id=` - **Annotations**: `GET/POST /api/annotations`, `DELETE /api/annotations/{id}` + + + **Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API: pipe them into scripts to automate batch annotation, SPARQL querying, or exports. + @@ -352,6 +368,10 @@ WebSocket message schema: Event types broadcast over the WebSocket include: `connection_ack`, `pong`, and `graph_mutation` (fired when nodes or edges are added/updated/removed via import or enrichment). Send the text `"ping"` to receive a `pong` response. + + **Session state is lost on server restart.** There is no auto-save. Call `POST /api/export` with body `{"format": "json"}` to download the current state before shutting down. + + ## Performance | Scenario | Latency | @@ -392,28 +412,6 @@ Semantic neighborhood requires node embeddings stored in node properties (keys ` **Session state lost after restart** Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot before shutting down. -## Tips and Common Pitfalls - - - **Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting: the force-directed layout becomes unusable on very large graphs. - - - - **Session state is lost on server restart.** There is no auto-save. Call `POST /api/export` with body `{"format": "json"}` to download the current state before shutting down. - - - - **Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API: pipe them into scripts to automate batch annotation, SPARQL querying, or exports. - - - - **CORS origins are configured via environment variable.** Set `EXPLORER_CORS_ORIGINS` to a comma-separated list of allowed origins before launching (e.g. `EXPLORER_CORS_ORIGINS="http://myapp.example.com"`). - - - - **Use `/api/analytics/validation` to check graph quality.** The validator detects orphaned nodes, missing types, and other structural issues before you expose the graph to downstream pipelines. - - Build and save the ContextGraph that Explorer loads. diff --git a/docs/reference/export.md b/docs/reference/export.md index 9c1547d9..39ec7d42 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -116,6 +116,18 @@ export_lpg(graph, "import.cypher", method="cypher") exporter.export_knowledge_graph(graph, "output.ttl", format="turtle") ``` + + **`export_to_rdf()` returns a string: it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk. + + + + **Use `export_to_rdf()` + string for inspection, `export()` for production.** In notebooks or debug sessions, `export_to_rdf()` is handy for quick inspection. For CI pipelines and pipelines writing files, `export()` is a single call. + + + + **Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented: one triple per line: making it safe to stream, concatenate, and process with standard Unix tools. + + **Namespace management:** ```python @@ -166,6 +178,14 @@ export_lpg(graph, "import.cypher", method="cypher") exporter.export(graph, "output_base") ``` + + **`ParquetExporter` and `ArrowExporter` require `pyarrow`.** Both fall back to a no-op stub class if `pyarrow` is not installed. Install with `pip install pyarrow` before using these exporters. + + + + **Use `ParquetExporter` for downstream analytics.** Parquet preserves column types (int, float, datetime) that CSV loses and is natively supported by Spark, BigQuery, Databricks, and Snowflake. Use `compression="snappy"` for a good balance of speed and compression. + + Requires `pyarrow`: `pip install pyarrow`. Schema is explicitly typed. ```python @@ -215,6 +235,10 @@ export_lpg(graph, "import.cypher", method="cypher") ``` Both exporters write to a file and return `None`. + + + **`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string. + ```python @@ -285,6 +309,10 @@ export_lpg(graph, "import.cypher", method="cypher") Available `include` columns: `source_id`, `source_type`, `target_id`, `target_type`, `hop_count`, `weighted_distance`, `semantic_similarity`, `distance_band`, `source_betweenness`, `target_betweenness`. + + **`DistanceExporter` requires a graph at construction.** Instantiate as `DistanceExporter(graph)`, not `DistanceExporter()`. Semantic similarity columns (`semantic_similarity`) require the graph nodes to have embeddings in their properties. + + **ReportGenerator:** ```python @@ -349,36 +377,6 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p | `"faiss"` | `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files | | `"html"` / `"markdown"` / `"json"` / `"text"` |: | `ReportGenerator` | `.html` / `.md` / `.json` / `.txt` | Analytics reports | -## Tips and Common Pitfalls - - - **`export_to_rdf()` returns a string: it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk. - - - - **`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string. - - - - **`DistanceExporter` requires a graph at construction.** Instantiate as `DistanceExporter(graph)`, not `DistanceExporter()`. Semantic similarity columns (`semantic_similarity`) require the graph nodes to have embeddings in their properties. - - - - **`ParquetExporter` and `ArrowExporter` require `pyarrow`.** Both fall back to a no-op stub class if `pyarrow` is not installed. Install with `pip install pyarrow` before using these exporters. - - - - **Use `export_to_rdf()` + string for inspection, `export()` for production.** In notebooks or debug sessions, `export_to_rdf()` is handy for quick inspection. For CI pipelines and pipelines writing files, `export()` is a single call. - - - - **Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented: one triple per line: making it safe to stream, concatenate, and process with standard Unix tools. - - - - **Use `ParquetExporter` for downstream analytics.** Parquet preserves column types (int, float, datetime) that CSV loses and is natively supported by Spark, BigQuery, Databricks, and Snowflake. Use `compression="snappy"` for a good balance of speed and compression. - - **Match your export format to your consumer.** Neo4j → `cypher`; ArangoDB → `aql`; Gephi/yEd → `graphml` or `gexf`; semantic web tools → `turtle` or `json-ld`; analytics pipelines → `parquet`; zero-copy IPC → `arrow`. diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index 44f433ed..709c25aa 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -108,6 +108,10 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass store.create_node(labels=["Person"], properties={"name": "Bob"}) ``` + + **Call `connect()` before any operations.** `GraphStore` does not connect automatically on construction. Either call `store.connect()` explicitly or use the context manager form `with GraphStore(...) as store:`. + + ## Quick Start @@ -226,6 +230,10 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass ``` **Best for:** teams already running PostgreSQL who want graph queries without a separate service. + + + **Apache AGE requires the PostgreSQL extension installed.** `backend="age"` calls the AGE extension functions. If AGE is not installed in your PostgreSQL instance, you'll get a `ProgrammingError`. See the [Apache AGE docs](https://age.apache.org/age-manual/master/intro/setup.html) for setup. + ```bash @@ -249,6 +257,10 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass ``` **Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries: the same query API used for Neo4j. + + + **Amazon Neptune uses `iam_auth=`, not `use_iam_auth=`.** The `AmazonNeptuneStore` and the `GraphStore` Neptune backend both use `iam_auth: bool = True` as the parameter name. + @@ -311,6 +323,10 @@ if path: print(f"Hops: {path['length']}") ``` + + **Use `create_nodes()` for bulk loading.** Individual `create_node()` calls issue one network round-trip each. `create_nodes(list)` is faster for initial graph population. + + ## QueryEngine `QueryEngine` handles query execution and optional caching. Access it via `store.query_engine`: @@ -354,6 +370,14 @@ engine.enable_cache() | `enable_cache()` | `None` | Turn on caching (on by default) | | `disable_cache()` | `None` | Turn off caching | + + **Use `QueryEngine` caching for read-heavy workloads.** Access the engine via `store.query_engine`. Call `engine.execute(query, use_cache=True)` to cache identical queries in-process. Call `engine.clear_cache()` after writes that invalidate results. + + + + **Use parameterized queries, never string interpolation.** `store.query("WHERE n.name = $name", parameters={"name": user_input})` prevents Cypher injection attacks. Never use `f"WHERE n.name = '{user_input}'"`. + + ## GraphAnalytics @@ -422,6 +446,14 @@ store.create_index(label="Organization", property_name="id") stats = store.get_stats() ``` + + **Create indexes before bulk loading.** `store.create_index(label="Person", property_name="name")` makes `MATCH` queries on `name` orders of magnitude faster. Without indexes, every query does a full scan. Create indexes first, then load data. + + + + **`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")`: using `property=` will be silently ignored. + + ## Common Workflows @@ -485,40 +517,6 @@ stats = store.get_stats() -## Tips and Common Pitfalls - - - **Call `connect()` before any operations.** `GraphStore` does not connect automatically on construction. Either call `store.connect()` explicitly or use the context manager form `with GraphStore(...) as store:`. - - - - **Use `create_nodes()` for bulk loading.** Individual `create_node()` calls issue one network round-trip each. `create_nodes(list)` is faster for initial graph population. - - - - **Create indexes before bulk loading.** `store.create_index(label="Person", property_name="name")` makes `MATCH` queries on `name` orders of magnitude faster. Without indexes, every query does a full scan. Create indexes first, then load data. - - - - **Use parameterized queries, never string interpolation.** `store.query("WHERE n.name = $name", parameters={"name": user_input})` prevents Cypher injection attacks. Never use `f"WHERE n.name = '{user_input}'"`. - - - - **`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")`: using `property=` will be silently ignored. - - - - **Use `QueryEngine` caching for read-heavy workloads.** Access the engine via `store.query_engine`. Call `engine.execute(query, use_cache=True)` to cache identical queries in-process. Call `engine.clear_cache()` after writes that invalidate results. - - - - **Apache AGE requires the PostgreSQL extension installed.** `backend="age"` calls the AGE extension functions. If AGE is not installed in your PostgreSQL instance, you'll get a `ProgrammingError`. See the [Apache AGE docs](https://age.apache.org/age-manual/master/intro/setup.html) for setup. - - - - **Amazon Neptune uses `iam_auth=`, not `use_iam_auth=`.** The `AmazonNeptuneStore` and the `GraphStore` Neptune backend both use `iam_auth: bool = True` as the parameter name. - - Build the graph before persisting it. diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 70564bf7..072a318f 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -56,6 +56,10 @@ for f in files: print(f.name, f.file_type, f.size) ``` + + **`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and reads content into `.content` bytes or the `.text` property. Use `read_content=False` when you only need file metadata. + + For web, database, or stream sources, each ingestor exposes its own typed method: ```python @@ -195,6 +199,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} Requires `pyarrow`: `pip install pyarrow`. + + **Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns. + + ### XMLIngestor XXE-safe lxml-based ingestion with optional schema validation: @@ -220,6 +228,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} `XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks. + + + **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML. + ### WebIngestor @@ -248,6 +260,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} Requires `beautifulsoup4`: `pip install beautifulsoup4`. + + **Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting you risk getting blocked by the target server or violating its terms of service. + + ### PublicAPIIngestor Use this for public REST-style APIs that do not require keys or tokens: @@ -403,6 +419,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} Requires `sqlalchemy`: `pip install sqlalchemy` plus your database driver. + + **`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself. + + ### SnowflakeIngestor ```python @@ -467,6 +487,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} ``` Stream processors require the appropriate client library (kafka-python, pika, boto3, pulsar-client). + + + **`StreamIngestor` methods require the target broker's client library to be installed.** `ingest_kafka` needs `kafka-python`, `ingest_rabbitmq` needs `pika`, `ingest_kinesis` needs `boto3`, and `ingest_pulsar` needs `pulsar-client`. Missing dependencies raise `ImportError` at call time, not at import time. + @@ -601,32 +625,6 @@ from semantica.ingest import ingest_file result = ingest_file("source_path", method="my_format") ``` -## Tips and Common Pitfalls - - - **`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself. - - - - **`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and reads content into `.content` bytes or the `.text` property. Use `read_content=False` when you only need file metadata. - - - - **Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns. - - - - **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML. - - - - **Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting you risk getting blocked by the target server or violating its terms of service. - - - - **`StreamIngestor` methods require the target broker's client library to be installed.** `ingest_kafka` needs `kafka-python`, `ingest_rabbitmq` needs `pika`, `ingest_kinesis` needs `boto3`, and `ingest_pulsar` needs `pulsar-client`. Missing dependencies raise `ImportError` at call time, not at import time. - - Parse raw sources into structured text and tables. diff --git a/docs/reference/mcp_server.md b/docs/reference/mcp_server.md index 53bc7ad9..1f213347 100644 --- a/docs/reference/mcp_server.md +++ b/docs/reference/mcp_server.md @@ -10,7 +10,6 @@ icon: "plug" - No Python code required after launch: configure once, use from any MCP-aware client - Compatible with Claude Desktop, Windsurf, Cline, Continue, VS Code, Roo Code, Cursor - ## Server Interface ```json @@ -35,6 +34,10 @@ python -m semantica.mcp_server `semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes: all interaction happens through MCP tool calls from a connected AI client. + + **The server communicates over stdio: don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable. + + ## What You Get @@ -134,6 +137,10 @@ The MCP server is included in the base install: no extras required. + + **Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently: the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works. + + ```bash @@ -156,6 +163,14 @@ The MCP server is included in the base install: no extras required. | `SEMANTICA_KG_PATH` | *(none: in-memory graph)* | Path to a persisted graph file to load on startup | | `SEMANTICA_LOG_LEVEL` | `WARNING` | Log verbosity: `DEBUG`, `INFO`, `WARNING` | + + **The graph starts empty unless you set `SEMANTICA_KG_PATH`.** The MCP server creates a fresh in-memory `ContextGraph` on first use. Set `SEMANTICA_KG_PATH` to a previously saved graph file to restore state across server restarts. Without it, all data is lost when the process exits. + + + + **Enable debug logging for troubleshooting.** Set `SEMANTICA_LOG_LEVEL=DEBUG` in your MCP client's `env` block, or run `python -m semantica.mcp_server` directly and inspect stderr output. + + ## Tools The MCP server exposes 12 tools that any connected AI assistant can call: @@ -294,6 +309,10 @@ Find past decisions similar to a given scenario using hybrid similarity search. `max_results` defaults to `5`, maximum `50`. + + **Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path: it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs. + + @@ -440,28 +459,6 @@ The MCP server exposes three readable resources: | `semantica://decisions/list` | All recorded decisions (up to 50) | | `semantica://schema/info` | Server version and available tools | -## Tips and Common Pitfalls - - - **The graph starts empty unless you set `SEMANTICA_KG_PATH`.** The MCP server creates a fresh in-memory `ContextGraph` on first use. Set `SEMANTICA_KG_PATH` to a previously saved graph file to restore state across server restarts. Without it, all data is lost when the process exits. - - - - **Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path: it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs. - - - - **Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently: the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works. - - - - **The server communicates over stdio: don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable. - - - - **Enable debug logging for troubleshooting.** Set `SEMANTICA_LOG_LEVEL=DEBUG` in your MCP client's `env` block, or run `python -m semantica.mcp_server` directly and inspect stderr output. - - The ContextGraph that the MCP server operates on. diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index 52dad4f6..c8d39403 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -22,7 +22,7 @@ Unstructured data is inconsistent by nature. Without normalization, the same rea - `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."`: multiple nodes, one company - `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"`: three formats, one date - `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"`: three strings, one number -- `"Hello World"` vs `"Hello\u00a0World"`: a non-breaking space that breaks string matching +- `"Hello World"` vs `"Hello World"`: a non-breaking space that breaks string matching Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data. @@ -51,7 +51,7 @@ from semantica.normalize import ( # Text: normalize unicode, collapse whitespace, replace smart quotes normalizer = TextNormalizer() -clean = normalizer.normalize_text(" Hello,\u00a0 World\u2026 ") +clean = normalizer.normalize_text(" Hello,  World… ") # → "Hello, World..." # Date @@ -90,6 +90,10 @@ utf8_text = handler.convert_to_utf8(raw_bytes) # convert_to_utf8 returns a str utf8_text = handler.convert_to_utf8(raw_bytes) ``` + + + **Run encoding repair before anything else.** A single cp1252 character in a UTF-8 stream silently corrupts the surrounding text. Call `handler.convert_to_utf8(raw_bytes)` first, before any other normalizer sees the data. + ```python @@ -103,6 +107,10 @@ utf8_text = handler.convert_to_utf8(raw_bytes) case="preserve", ) ``` + + + **Don't lowercase before NER.** `normalize_text(text, case="lower")` before entity extraction destroys capitalization signals that NER relies on. Apply case normalization only after extraction if needed. + ```python @@ -118,6 +126,10 @@ utf8_text = handler.convert_to_utf8(raw_bytes) ) # → "Apple Inc." (if the alias_map contains it, else title-cased input) ``` + + + **`EntityNormalizer` has no built-in corporate suffix expansion.** There is no automatic mapping of `"Apple Computer Inc."` → `"Apple Inc."`. To canonicalize corporate names, provide an explicit `alias_map` with lowercase keys: `EntityNormalizer(alias_map={"apple computer inc.": "Apple Inc."})`. + ```python @@ -210,13 +222,13 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") # Batch normalization results = normalizer.process_batch( - [" hello ", "WORLD", "caf\u00e9"], + [" hello ", "WORLD", "café"], unicode_form="NFKC", case="lower", ) # normalize() accepts str or List[Dict] (parsed docs from DocumentParser) - docs = [{"content": "Hello\u00a0world"}, {"content": "test text"}] + docs = [{"content": "Hello world"}, {"content": "test text"}] normalized_docs = normalizer.normalize(docs) ``` @@ -244,14 +256,14 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") ) unicode_norm = UnicodeNormalizer() - text = unicode_norm.normalize_unicode("caf\u00e9", form="NFC") + text = unicode_norm.normalize_unicode("café", form="NFC") ws_norm = WhitespaceNormalizer() text = ws_norm.normalize_whitespace("Hello\t\t World\n\n") # → "Hello World\n\n" processor = SpecialCharacterProcessor() - text = processor.normalize_punctuation("\u2018Hello\u2019") + text = processor.normalize_punctuation("‘Hello’") # → "'Hello'" ``` @@ -313,6 +325,10 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") canonical = handler.normalize_name_format("Dr. JOHN P. SMITH Jr.") # → "John P. Smith Jr." (removes leading title) ``` + + + **`AliasResolver` uses lowercase key lookup.** Register aliases with lowercase keys even if the canonical form is title-cased. The resolver converts the input to lowercase before lookup. + `DateNormalizer` takes `config=None, **kwargs`. The `format` and `timezone` @@ -427,6 +443,10 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") `detect()` requires at least 10 characters for reliable detection. On shorter text it returns the `default_language` (default: `"en"`). + + **`LanguageDetector.detect()` returns a `str`, not a dict.** Use `detect_with_confidence()` for `(language_code, confidence)` tuple, or `detect_multiple()` for `List[(code, confidence)]`. + + ### EncodingHandler Detect and repair character encoding issues. Requires `chardet`: `pip install chardet`. @@ -457,6 +477,10 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") then falls back through `latin-1`, `cp1252`, `iso-8859-1` - Always run `EncodingHandler` first: broken bytes cause cascading failures in every downstream normalizer + + + **`EncodingHandler.detect()` returns a `(str, float)` tuple, not a dict.** Unpack with `encoding, confidence = handler.detect(data)`. + @@ -511,6 +535,14 @@ print(f"Warnings: {len(result.warnings)}") | `validate_data(dataset, schema)` | `ValidationResult` | Validate records against a schema dict | | `handle_missing_values(dataset, strategy)` | `List[Dict]` | Remove, fill, or impute missing values | + + **`DataCleaner.remove_duplicates()` does not exist as a standalone method.** Use `detect_duplicates()` to get `DuplicateGroup` objects, or call `clean_data(records, remove_duplicates=True)` to remove them in-place. + + + + **`DataCleaner` operates on flat records, not graph entities.** For entity-level semantic deduplication, use `DuplicateDetector` from the Deduplication module instead. + + ## Pipeline Integration ```python @@ -552,40 +584,6 @@ normalized = normalize_text("Apple Inc.", method="expand_suffixes") # → "Apple Incorporated" ``` -## Tips and Common Pitfalls - - - **Run encoding repair before anything else.** A single cp1252 character in a UTF-8 stream silently corrupts the surrounding text. Call `handler.convert_to_utf8(raw_bytes)` first, before any other normalizer sees the data. - - - - **Don't lowercase before NER.** `normalize_text(text, case="lower")` before entity extraction destroys capitalization signals that NER relies on. Apply case normalization only after extraction if needed. - - - - **`EntityNormalizer` has no built-in corporate suffix expansion.** There is no automatic mapping of `"Apple Computer Inc."` → `"Apple Inc."`. To canonicalize corporate names, provide an explicit `alias_map` with lowercase keys: `EntityNormalizer(alias_map={"apple computer inc.": "Apple Inc."})`. - - - - **`AliasResolver` uses lowercase key lookup.** Register aliases with lowercase keys even if the canonical form is title-cased. The resolver converts the input to lowercase before lookup. - - - - **`LanguageDetector.detect()` returns a `str`, not a dict.** Use `detect_with_confidence()` for `(language_code, confidence)` tuple, or `detect_multiple()` for `List[(code, confidence)]`. - - - - **`EncodingHandler.detect()` returns a `(str, float)` tuple, not a dict.** Unpack with `encoding, confidence = handler.detect(data)`. - - - - **`DataCleaner.remove_duplicates()` does not exist as a standalone method.** Use `detect_duplicates()` to get `DuplicateGroup` objects, or call `clean_data(records, remove_duplicates=True)` to remove them in-place. - - - - **`DataCleaner` operates on flat records, not graph entities.** For entity-level semantic deduplication, use `DuplicateDetector` from the Deduplication module instead. - - Parse documents before normalization. diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index cee512ac..a8763b6a 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -164,7 +164,6 @@ ontology = generator.generate_ontology_from_text( text="A biomedical ontology for clinical trial protocols involving patients, trials, interventions, and outcomes." ) ``` -``` ## OWL / RDF Export diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index 2419ec4e..82d2c7e3 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -97,6 +97,10 @@ You could wire Semantica modules together with plain Python code. Pipelines add: for warning in result.warnings: print(f"Warning: {warning}") ``` + + + **Use `PipelineValidator` before running in production.** It catches dependency cycles, missing step names, and misconfigured connections that would only surface as errors mid-run. Validation is instant; catching them after a 30-minute extraction job is not. + ```python @@ -111,6 +115,10 @@ You could wire Semantica modules together with plain Python code. Pipelines add: print(f"Steps failed: {result.metrics['steps_failed']}") print(f"Duration: {result.metrics['execution_time']:.1f}s") ``` + + + **Inspect `result.metrics` to find bottlenecks.** `result.metrics['steps_executed']` and `result.metrics['execution_time']` give a quick read on overall pipeline health. For per-step timing, check `step.result` on each `PipelineStep` after the run. + @@ -133,6 +141,10 @@ engine = ExecutionEngine(max_workers=4) result = engine.execute_pipeline(pipeline, data="data/") ``` + + **Set `workers=` based on workload type.** Thread workers for I/O-bound steps (web fetching, DB queries), process workers for CPU-bound steps (embedding, OCR, large NER batches). Mixing pool types on the wrong step type wastes resources without speed gains. + + ## Retry and Error Handling @@ -196,6 +208,10 @@ result = engine.execute_pipeline(pipeline, data="data/") In production, configure a `RetryPolicy` with limited retries so a single failing step does not stop the whole run. After execution, inspect `result.errors` to find and reprocess failed documents. + + **Configure retry policies to contain failures in production.** Use `handler.set_retry_policy("step_type", RetryPolicy(max_retries=3))` so transient errors are retried without stopping the pipeline. After the run, inspect `result.errors` to find and reprocess any documents that exhausted retries. + + ## Progress Tracking @@ -349,6 +365,10 @@ The `create_pipeline_from_template(name)` method returns a configured `PipelineB + + **Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing. + + ## ExecutionEngine Fine-grained control over pipeline execution: pause, resume, cancel, and inspect live progress: @@ -563,28 +583,6 @@ StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy -## Tips and Common Pitfalls - - - **Use `PipelineValidator` before running in production.** It catches dependency cycles, missing step names, and misconfigured connections that would only surface as errors mid-run. Validation is instant; catching them after a 30-minute extraction job is not. - - - - **Set `workers=` based on workload type.** Thread workers for I/O-bound steps (web fetching, DB queries), process workers for CPU-bound steps (embedding, OCR, large NER batches). Mixing pool types on the wrong step type wastes resources without speed gains. - - - - **Configure retry policies to contain failures in production.** Use `handler.set_retry_policy("step_type", RetryPolicy(max_retries=3))` so transient errors are retried without stopping the pipeline. After the run, inspect `result.errors` to find and reprocess any documents that exhausted retries. - - - - **Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing. - - - - **Inspect `result.metrics` to find bottlenecks.** `result.metrics['steps_executed']` and `result.metrics['execution_time']` give a quick read on overall pipeline health. For per-step timing, check `step.result` on each `PipelineStep` after the run. - - First step in most pipelines. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index c4002c1d..bc414e92 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -98,6 +98,10 @@ ProvenanceManager( If both `storage` and `storage_path` are omitted, an `InMemoryStorage` is used. + + **`InMemoryStorage` does not persist across restarts.** Pass `storage_path="provenance.db"` or an explicit `SQLiteStorage` instance in any environment where the audit trail must survive process exits. + + ### Tracking Methods ```python @@ -196,6 +200,10 @@ if prov: print(prov["source_document"]) ``` + + `get_lineage()` returns an aggregated **dict**, not a `ProvenanceEntry`. Use `trace_lineage()` to get the raw `ProvenanceEntry` objects when you need field-level access such as `entry.checksum`. + + ### Utility Methods ```python @@ -338,6 +346,10 @@ if not is_valid: The checksum covers `entity_id`, `entity_type`, `activity_id`, `source_document`, `timestamp`, and `confidence`. + + **Run `verify_checksum(entry)` before any compliance export.** Pass the `ProvenanceEntry` object returned by `trace_lineage()` directly. If the stored checksum no longer matches, raise an error before the export proceeds. + + ## Bridge Axiom Translation Chains `BridgeAxiom` and `TranslationChain` are available in `semantica.provenance.bridge_axiom` for tracking multi-layer domain translations with full coefficient attribution: @@ -420,6 +432,10 @@ lineage = manager.get_lineage(entities[0].id) print(lineage["source_documents"]) ``` + + Setting `provenance=True` on `NERExtractor` embeds metadata on the extracted entity objects — it does not automatically call `ProvenanceManager.track_entity()`. You must call `track_entity()` yourself after extraction. + + ## Common Workflows diff --git a/docs/reference/seed.md b/docs/reference/seed.md index a00e8a66..25b4026a 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -61,6 +61,10 @@ icon: "database" manager.register_source("taxonomy", "json", "data/taxonomy.json") manager.register_source("employees", "csv", "data/employees.csv") ``` + + + **Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once. + ```python @@ -82,11 +86,15 @@ icon: "database" else: print(f"Validated {report['metrics']['entity_count']} entities: no issues found") ``` + + + **Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast: always run it first. + ```python from semantica.semantic_extract import NERExtractor - + extractor = NERExtractor(method="ml") new_entities = extractor.extract("Apple Inc. partners with Microsoft Corp.") @@ -97,6 +105,10 @@ icon: "database" merge_strategy="merge" ) ``` + + + **Load seed data before extracted data.** Seed data is your ground truth: normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values. + @@ -230,6 +242,10 @@ Different strategies for resolving conflicts during `integrate_with_extracted()` + + **Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `merge_strategy="seed_first"` ensures those values win over extracted values. Use `merge` only when extracted data may be more current than the seed. + + ## Full Pipeline Example ```python @@ -315,24 +331,6 @@ export SEMANTICA_SEED_DATA_DIR=./data/seed export SEMANTICA_SEED_MERGE_STRATEGY=seed_first ``` -## Tips and Common Pitfalls - - - **Load seed data before extracted data.** Seed data is your ground truth: normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values. - - - - **Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `merge_strategy="seed_first"` ensures those values win over extracted values. Use `merge` only when extracted data may be more current than the seed. - - - - **Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast: always run it first. - - - - **Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once. - - **Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment. diff --git a/docs/reference/split.md b/docs/reference/split.md index 29deeaf2..d64471da 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -179,6 +179,10 @@ splitter = TextSplitter( | `relation_method` | `str` | `"ml"` | Relation extraction method for `relation_aware`: `"ml"` \| `"llm"` \| `"huggingface"` | | `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method: unrecognised names fall back to `cl100k_base` | + + **`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`. + + ## Splitting Method Details @@ -217,6 +221,10 @@ splitter = TextSplitter( - Produces variable-length chunks: some topics are short, others long - Falls back to sentence splitting if `sentence-transformers` is not installed - Slower than `recursive` due to embedding computation; cache embeddings for repeated splits + + + **Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead. + Runs NER internally, then adjusts chunk boundaries so no entity mention is split across two chunks: @@ -346,6 +354,10 @@ The `token` method accepts a `tokenizer=` kwarg that is passed to `tiktoken.enco If `tiktoken` is not installed, the `token` method falls back to splitting by whitespace-separated words. + + **Wrong tokenizer.** The `token` method passes the `tokenizer=` value to `tiktoken.encoding_for_model()`. If the model name is not recognised by tiktoken it silently falls back to `cl100k_base`. Pass a valid tiktoken model name (e.g. `"gpt-4"`, `"gpt-3.5-turbo"`) to get deterministic behaviour. + + ## Pipeline Integration `TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern: parse a file, split the text, then extract entities from each chunk: @@ -373,20 +385,6 @@ for chunk in chunks: For the full pipeline orchestration API, see the [Pipeline reference](pipeline). -## Tips and Common Pitfalls - - - **`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`. - - - - **Wrong tokenizer.** The `token` method passes the `tokenizer=` value to `tiktoken.encoding_for_model()`. If the model name is not recognised by tiktoken it silently falls back to `cl100k_base`. Pass a valid tiktoken model name (e.g. `"gpt-4"`, `"gpt-3.5-turbo"`) to get deterministic behaviour. - - - - **Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead. - - Parse documents before chunking: produces sections and metadata. diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index 02cb3d46..d9a368e1 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -163,7 +163,9 @@ for row in result.bindings: **Best for:** local development with rdflib, SPARQL read queries against a Fuseki endpoint. - **Note on inference:** `JenaStore` accepts `enable_inference=True` in config but OWL reasoning is a placeholder and does not produce inferred triples in the current implementation. + + **`backend="jena"` OWL inference is a placeholder.** `enable_inference=True` is accepted but the inference call returns 0 inferred triples. For production OWL reasoning, use Jena Fuseki directly with its built-in reasoner configuration. + ```bash @@ -191,6 +193,10 @@ for row in result.bindings: + + **Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`. + + ## Triplet Object All store operations use the `Triplet` dataclass from `semantica.semantic_extract.types`: @@ -215,6 +221,10 @@ t = Triplet( | `confidence` | `float` | `1.0` | Confidence score (0–1) | | `metadata` | `dict` | `{}` | Arbitrary metadata | + + **`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object: not `subject=`, `predicate=`, `obj=` to `add_triplet`. + + ## TripletStore Methods | Method | Returns | Description | @@ -277,6 +287,10 @@ store.execute_query(""" | `execution_time` | `float` | Seconds elapsed | | `metadata` | `dict` | Query, graph scope, cache hit flag | + + **`execute_query()` returns `QueryResult`, not a list.** Iterate `result.bindings`, not `result` directly. Each binding is a dict mapping variable name → `{"value": ..., "type": ...}`. + + ## SPARQL Result Pagination For large result sets, paginate with LIMIT and OFFSET: @@ -299,6 +313,10 @@ while True: offset += page_size ``` + + **Paginate large SPARQL result sets.** A `SELECT * WHERE { ?s ?p ?o }` against a large store returns all triples. Always include `LIMIT` and `OFFSET` in exploratory queries. `QueryEngine` adds `LIMIT 1000` automatically unless you specify one. + + ## Named Graph Scoping Blazegraph and RDF4J support named graphs. Scope `execute_query()` to a named graph with the `graph=` parameter: @@ -333,6 +351,10 @@ result = store.execute_query(""" Named graph support is only available for Blazegraph and RDF4J backends. The `graph=` parameter is silently ignored for the Jena backend. + + **Use named graphs to isolate sources.** Pass `graph="http://example.org/source_A"` to `execute_query()` to scope a query to a specific named graph. Blazegraph and RDF4J support named graphs; Jena (rdflib backend) does not. + + ## Bulk Loading `add_triplets()` batches writes via the internal `BulkLoader`. Access `store.bulk_loader` to configure it: @@ -468,32 +490,6 @@ for row in result.bindings: print(row) ``` -## Tips and Common Pitfalls - - - **Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`. - - - - **`execute_query()` returns `QueryResult`, not a list.** Iterate `result.bindings`, not `result` directly. Each binding is a dict mapping variable name → `{"value": ..., "type": ...}`. - - - - **`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object: not `subject=`, `predicate=`, `obj=` to `add_triplet`. - - - - **Paginate large SPARQL result sets.** A `SELECT * WHERE { ?s ?p ?o }` against a large store returns all triples. Always include `LIMIT` and `OFFSET` in exploratory queries. `QueryEngine` adds `LIMIT 1000` automatically unless you specify one. - - - - **Use named graphs to isolate sources.** Pass `graph="http://example.org/source_A"` to `execute_query()` to scope a query to a specific named graph. Blazegraph and RDF4J support named graphs; Jena (rdflib backend) does not. - - - - **`backend="jena"` OWL inference is a placeholder.** `enable_inference=True` is accepted but the inference call returns 0 inferred triples. For production OWL reasoning, use Jena Fuseki directly with its built-in reasoner configuration. - - Export knowledge graphs to RDF formats. diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 60402e99..14ae2106 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -67,6 +67,10 @@ Most users won't call utils directly: it's the **shared foundation** for all mod setup_logging(level="INFO") # "DEBUG" | "INFO" | "WARNING" | "ERROR" logger = get_logger(__name__) ``` + + + **Call `setup_logging(level="INFO")` once at application startup.** Without it, Semantica falls back to Python's root logger, which may be silent or misconfigured. Call it before importing other Semantica modules to capture initialization messages. + ```python @@ -77,6 +81,10 @@ Most users won't call utils directly: it's the **shared foundation** for all mod ... # Logs: "expensive_step completed in 2.34s" ``` + + + **`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected: it is not a decorator. + ```bash @@ -119,10 +127,15 @@ for item in track_progress(items, desc="Processing documents"): ``` Supports: + - **Console**: tqdm progress bar with ETA - **Jupyter**: notebook-compatible widget (auto-detected) - **File**: write progress to a log file + + **`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment: the same call works in both. + + ## Helper Functions ```python @@ -138,6 +151,10 @@ uid = hash_data({"key": "value"}) # -> hex digest string fname = safe_filename("My File?.txt") # -> "My_File.txt" ``` + + **`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string: suitable as a cache key or idempotency token in pipeline steps. + + ## Nested Dict Utilities Helper functions for deep configuration access: used extensively inside `Config` and `ConfigManager`: @@ -196,6 +213,10 @@ except SemanticaError as e: + + **Catch `SemanticaError` as the broadest exception net.** All framework errors inherit from `SemanticaError`, so `except SemanticaError` catches validation failures, processing errors, and everything in between. Use specific subclasses for targeted recovery logic. + + ## File Utilities ```python @@ -205,28 +226,6 @@ from semantica.utils import read_json_file config = read_json_file("config.json") ``` -## Tips and Common Pitfalls - - - **Call `setup_logging(level="INFO")` once at application startup.** Without it, Semantica falls back to Python's root logger, which may be silent or misconfigured. Call it before importing other Semantica modules to capture initialization messages. - - - - **`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected: it is not a decorator. - - - - **`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string: suitable as a cache key or idempotency token in pipeline steps. - - - - **Catch `SemanticaError` as the broadest exception net.** All framework errors inherit from `SemanticaError`, so `except SemanticaError` catches validation failures, processing errors, and everything in between. Use specific subclasses for targeted recovery logic. - - - - **`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment: the same call works in both. - - Framework orchestration that uses Utils internally. diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 69b052d8..8b0e0e60 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -91,6 +91,14 @@ for r in results: print(f"{r['id']}: score: {r['score']:.3f}") ``` + + **Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time. + + + + **Use `add_documents()` for text, `store_vectors()` for pre-computed embeddings.** `add_documents()` auto-embeds in parallel batches. If your embeddings are already computed (e.g. from a fine-tuned model), use `store_vectors()` directly to skip re-embedding. + + ## Quick Start @@ -307,6 +315,10 @@ sources = [ fused = search.multi_source_search(query_vector, sources, k=10) ``` + + **Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter. + + ## Metadata Filtering `MetadataFilter` supports chained conditions: all conditions are ANDed: @@ -394,6 +406,10 @@ ns = ns_manager.get_vector_namespace("vec_0") ns_manager.delete_namespace("tenant_a") ``` + + **Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and risks data leakage if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation). + + ## Batch Operations ```python @@ -436,6 +452,10 @@ store2.load("./vector_store_backup") Cloud backends (Pinecone, Weaviate, Qdrant, Milvus, PgVector) manage persistence themselves. `save()`/`load()` are for the in-memory and FAISS backends only. + + **inmemory and faiss backends lose data on process exit without `save()`.** Call `store.save(path)` after adding vectors. Cloud backends (Pinecone, Qdrant, Weaviate, Milvus, PgVector) persist automatically. + + ## MetadataStore `MetadataStore` indexes structured metadata and lets you query by field values without a vector: @@ -467,6 +487,10 @@ stats = meta_store.get_stats() # {"total_vectors": 2, "indexed_fields": 3, "field_counts": {...}} ``` + + **Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content. + + ## FAISS Index Type Reference FAISS index type is configured by creating a `FAISSStore` directly and calling `create_index()`. Use lowercase type names: @@ -496,6 +520,10 @@ store.create_index(index_type="pq", metric="L2", m=8) | `hnsw` | Medium-High | Very fast | ~97–99% | Low latency, production retrieval | | `pq` | Low | Fast | ~90–95% | Millions of vectors, memory-constrained | + + **FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`. + + When using `VectorStore(backend="faiss")`, the underlying `FAISSStore` is initialised with a flat index by default. To use ivf/hnsw/pq, construct `FAISSStore` directly and call `create_index()` with the desired type. @@ -574,36 +602,6 @@ store.create_index(index_type="pq", metric="L2", m=8) -## Tips and Common Pitfalls - - - **Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time. - - - - **FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`. - - - - **inmemory and faiss backends lose data on process exit without `save()`.** Call `store.save(path)` after adding vectors. Cloud backends (Pinecone, Qdrant, Weaviate, Milvus, PgVector) persist automatically. - - - - **Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter. - - - - **Use `add_documents()` for text, `store_vectors()` for pre-computed embeddings.** `add_documents()` auto-embeds in parallel batches. If your embeddings are already computed (e.g. from a fine-tuned model), use `store_vectors()` directly to skip re-embedding. - - - - **Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and risks data leakage if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation). - - - - **Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content. - - Generate the vectors stored here. diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index bcf32608..5cc23816 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -61,6 +61,10 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o + + **`plotly` is required for all visualizers.** Install before use: `pip install plotly`. All visualizer methods raise `ProcessingError` if Plotly is not installed. + + ## Visualizers @@ -94,6 +98,18 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o viz.visualize_relationship_matrix(graph, output="interactive") ``` + + **Use `max_nodes` for large graphs.** Force-directed layouts become unreadable and slow above ~1,000 nodes. Filter to a subgraph before visualizing large graphs. + + + + **HTML output is always the best starting point.** Interactive HTML lets you zoom, pan, and hover for details. Only export to PNG/SVG/PDF when embedding in a report. + + + + **For interactive dashboards, prefer Explorer.** `KGVisualizer.visualize_network()` generates a self-contained HTML file. The Explorer CLI (`semantica-explorer`) gives a full live web app with search, filtering, path-finding, and REST API. + + **Layout options (`layout=`):** | Layout | Description | Best For | @@ -148,6 +164,10 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o | `umap` | Fast | Global + local structure | Large datasets, cluster discovery | | `tsne` | Medium | Local structure | Tight cluster separation | | `pca` | Very fast | Variance | Quick overview, linear structure | + + + **UMAP is faster than t-SNE at scale.** For embedding spaces with >5,000 points, UMAP completes in seconds; t-SNE may take minutes. Both produce good cluster separation. + Visualize how a knowledge graph changes over time: @@ -231,6 +251,10 @@ viz = KGVisualizer(color_scheme="vibrant") | `light` | White background, thin edges | Publications, print | | `colorblind` | Okabe-Ito safe palette | Accessibility | + + **Use `color_scheme="colorblind"` in publications and dashboards.** The Okabe-Ito palette is readable for everyone, including the ~8% of readers who are red-green colorblind. + + ## Export Formats | Format | Interactive | Scalable | Best For | @@ -266,32 +290,6 @@ semantica-explorer --graph my_graph.json See the [Explorer reference](explorer) for the full feature set and REST API. -## Tips and Common Pitfalls - - - **`plotly` is required for all visualizers.** Install before use: `pip install plotly`. All visualizer methods raise `ProcessingError` if Plotly is not installed. - - - - **Use `max_nodes` for large graphs.** Force-directed layouts become unreadable and slow above ~1,000 nodes. Filter to a subgraph before visualizing large graphs. - - - - **HTML output is always the best starting point.** Interactive HTML lets you zoom, pan, and hover for details. Only export to PNG/SVG/PDF when embedding in a report. - - - - **Use `color_scheme="colorblind"` in publications and dashboards.** The Okabe-Ito palette is readable for everyone, including the ~8% of readers who are red-green colorblind. - - - - **UMAP is faster than t-SNE at scale.** For embedding spaces with >5,000 points, UMAP completes in seconds; t-SNE may take minutes. Both produce good cluster separation. - - - - **For interactive dashboards, prefer Explorer.** `KGVisualizer.visualize_network()` generates a self-contained HTML file. The Explorer CLI (`semantica-explorer`) gives a full live web app with search, filtering, path-finding, and REST API. - - The graph being visualized.