Merge branch 'main' into fix/neo4j-edge-id-space-1136

This commit is contained in:
Mohd Kaif
2026-08-27 15:31:51 +05:30
committed by GitHub
169 changed files with 14498 additions and 1330 deletions
+1 -1
View File
@@ -69,5 +69,5 @@ If you have ideas on how this could be implemented, please share.
---
**Note**: For feature requests that are ready to be implemented, consider creating a [Feature Request issue](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md) instead.
**Note**: For feature requests that are ready to be implemented, consider creating a [Feature Request issue](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md) instead.
+2 -2
View File
@@ -46,8 +46,8 @@ If applicable, paste any error messages or describe unexpected behavior:
## Checklist
- [ ] I have searched existing [discussions](https://github.com/Hawksight-AI/semantica/discussions) and [issues](https://github.com/Hawksight-AI/semantica/issues)
- [ ] I have checked the [documentation](https://github.com/Hawksight-AI/semantica/tree/main/docs) and [FAQ](https://github.com/Hawksight-AI/semantica/blob/main/docs/faq.md)
- [ ] I have searched existing [discussions](https://github.com/semantica-agi/semantica/discussions) and [issues](https://github.com/semantica-agi/semantica/issues)
- [ ] I have checked the [documentation](https://github.com/semantica-agi/semantica/tree/main/docs) and [FAQ](https://github.com/semantica-agi/semantica/blob/main/docs/faq.md)
- [ ] I have provided a minimal code example (if applicable)
- [ ] I have included error messages (if applicable)
- [ ] I have provided environment details
+1 -1
View File
@@ -1,3 +1,3 @@
# Funding options for Semantica
github: Hawksight-AI
github: semantica-agi
+2 -2
View File
@@ -1,8 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: 📚 Documentation
url: https://github.com/Hawksight-AI/semantica/tree/main/docs
url: https://github.com/semantica-agi/semantica/tree/main/docs
about: Browse the documentation
- name: 💬 Discussions
url: https://github.com/Hawksight-AI/semantica/discussions
url: https://github.com/semantica-agi/semantica/discussions
about: Ask questions and discuss with the community
+9 -9
View File
@@ -3,31 +3,31 @@
## Getting Help
### 📚 Documentation
Check the [docs folder](https://github.com/Hawksight-AI/semantica/tree/main/docs) and [README](https://github.com/Hawksight-AI/semantica/blob/main/README.md) for guides and examples.
Check the [docs folder](https://github.com/semantica-agi/semantica/tree/main/docs) and [README](https://github.com/semantica-agi/semantica/blob/main/README.md) for guides and examples.
### 💬 Community Support
- **GitHub Discussions**: [Ask questions](https://github.com/Hawksight-AI/semantica/discussions)
- **GitHub Discussions**: [Ask questions](https://github.com/semantica-agi/semantica/discussions)
- **Discord**: Join our [Discord server](https://discord.gg/sV34vps5hH) for real-time chat
### 💭 Discussions
Join the conversation on [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions):
Join the conversation on [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions):
- **Q&A**: Ask questions and get help from the community
- **Ideas**: Share feature requests and suggestions
- **Show and Tell**: Showcase your projects and use cases
- **General**: General discussions about Semantica
### 🐛 Bug Reports
Found a bug? [Create an issue](https://github.com/Hawksight-AI/semantica/issues/new/choose)
Found a bug? [Create an issue](https://github.com/semantica-agi/semantica/issues/new/choose)
### 📖 Resources
- [Quick Start Guide](https://github.com/Hawksight-AI/semantica/blob/main/docs/quickstart.md)
- [FAQ](https://github.com/Hawksight-AI/semantica/blob/main/docs/faq.md)
- [Cookbook Examples](https://github.com/Hawksight-AI/semantica/tree/main/cookbook)
- [Quick Start Guide](https://github.com/semantica-agi/semantica/blob/main/docs/quickstart.md)
- [FAQ](https://github.com/semantica-agi/semantica/blob/main/docs/faq.md)
- [Cookbook Examples](https://github.com/semantica-agi/semantica/tree/main/cookbook)
## Commercial Support
For enterprise support, custom development, or consulting services:
- Contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- Contact us through [GitHub Issues](https://github.com/semantica-agi/semantica/issues)
- Include "Commercial Support" in the title
## Sponsorship
@@ -35,7 +35,7 @@ For enterprise support, custom development, or consulting services:
### Sponsor this project
Support Semantica development:
- [GitHub Sponsors](https://github.com/sponsors/Hawksight-AI)
- [GitHub Sponsors](https://github.com/sponsors/semantica-agi)
Your sponsorship helps us:
- Maintain and improve the framework
+34 -1
View File
@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **First-class LangChain integration** (closes #963; recreates #969)
- New `pip install semantica[langchain]` extra (`langchain-core>=0.3.0`), included in the `all` bundle
- `integrations/langchain/SemanticaRetriever` — LangChain `BaseRetriever` that seeds from `HybridSearch` then walks graph edges (`hops=2` default) for GraphRAG-style retrieval; falls back to `ContextGraph.query` when hybrid search is unavailable
- `integrations/langchain/SemanticaVectorStore` — LangChain `VectorStore` adapter over `HybridSearch` (`add_texts`, `similarity_search`, `similarity_search_with_score`, `from_texts`)
- `integrations/langchain/SemanticaKGTool` / `SemanticaDecisionTool``BaseTool` subclasses with Pydantic `args_schema` (`semantica_query_graph`, `semantica_query_decisions`); `build()` returns the tool, or `None` when langchain-core is absent
- Retriever and VectorStore read HybridSearch nested `metadata` (`content`, `node_id`, `node_type`) rather than top-level fields that HybridSearch does not set
- All adapters remain importable without langchain-core (`LANGCHAIN_AVAILABLE` flag)
- Docs: `docs/integrations/langchain.md`, README native-integration matrix, and `docs.json` nav entry
## [0.6.6] - 2026-08-20
### Added
@@ -108,6 +119,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **The temporal-evolution `stability` metric was a hardcoded placeholder, not a duration**
- `TemporalGraphQuery.analyze_evolution()` documents `stability` as a "relationship duration/stability measure", but the implementation appended a constant `1` for every relationship with both `valid_from` and `valid_until` set (`durations.append(1) # Placeholder`). The reported stability was therefore always `1.0` when any bounded relationship existed and `0` otherwise — it never reflected how long relationships actually stayed valid, so it could not distinguish a graph of decade-long relationships from one of one-second relationships
- `stability` now computes the mean valid-time duration in seconds (`(valid_until - valid_from).total_seconds()`) across relationships that have both bounds set. Relationships with a missing or open `valid_from`/`valid_until` are skipped (their duration is unbounded), and non-positive intervals are clamped to `0`; an empty set still reports `0`
- New tests in `tests/kg/test_kg.py` assert the mean-duration result, the skipping of unbounded/half-open intervals, and the empty-graph zero case
- **Every timestamp an export or a provenance record wrote was timezone-naive** (closes #1114) by @fabio-rovai
- `semantica/export/` stamped with `datetime.now().isoformat()`, which reads the machine's **local** clock; `semantica/provenance/` stamped with `datetime.utcnow().isoformat()`, which reads **UTC**. Both produce a naive value and both serialize identically, so nothing downstream can tell which zone a given timestamp belongs to — the same string means two different instants depending on which module wrote it
- In RDF the consequence is silent rather than loud. Under XSD 1.1 a value with no timezone compared against one with a timezone is indeterminate whenever the two fall inside the ±14 hour window; SPARQL turns an indeterminate comparison into an error, and `FILTER` discards errors as non-matches. A timezone-qualified query over an Oxigraph store returns an answer with every Semantica-written record quietly absent from it, which is a poor property for `prov:generatedAtTime`, `prov:startedAtTime`, `prov:endedAtTime` and `prov:atTime` to have
@@ -116,6 +132,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- New `tests/export/test_timestamp_timezones.py` and `tests/provenance/test_timestamp_timezones.py`: offset presence on every export and provenance path, PROV-O literals valid as `xsd:dateTimeStamp`, comparison against a timezone-aware instant without `TypeError`, the Oxigraph filter that dropped the naive value (with a bound inside the indeterminate window, so the test cannot pass by accident), and the document `@id` remaining a valid IRI with `+00:00` in it. 11 of the 13 fail on the parent commit
- **Fixed during review** (Qodo): once new entries carry `+00:00` and stored ones do not, `ProvenanceManager.query_recorded_between` and `audit_log` compared ISO timestamps as raw strings, so they ordered by spelling rather than by instant — an inclusive naive bound naming a stored offset-bearing timestamp sorted *below* it and dropped the record, and a bound written in another offset landed wherever its digits fell (`19:45+05:30` is 14:15Z, but sorted after 14:19Z). Both now compare instants through a new `to_utc_datetime()` helper that reads a missing offset as UTC, which is what the values written before this change actually were; a bound that cannot be read as a timestamp keeps the historical string comparison rather than raising on a call that used to work
- The remaining 147 naive call sites are in `context/`, `vector_store/`, `seed/` and elsewhere, where timestamps are compared against values parsed from previously stored naive strings. Converting those without a read-side migration would raise `TypeError: can't compare offset-naive and offset-aware datetimes` on existing data, so they are deliberately left for a separate change
- **`SHACLGenerator` mangles `#`-terminated namespaces into `#/`, so generated shapes target nothing** (#1082) by @changshenhan
- `__init__` normalized `base_uri` with `rstrip("/") + "/"`, which turns `http://example.org/manufacturing#` into `...manufacturing#/` — the most common RDF namespace convention. Every generated URI (`sh:targetClass`, `sh:path`, shape URIs) then landed in a different namespace than the instance data, and SHACL validation silently passed because the shapes targeted nothing
- `__init__` now preserves a namespace already ending in `/` or `#`, matching the `#`-aware normalization `generate()` already applies; `shapes_uri` inherits the fix
- New `test_hash_namespace_base_uri_is_not_mangled` in `tests/ontology/test_ontology_advanced.py` fails on the pre-fix normalization and passes with it; full ontology suite (76 tests) green
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
@@ -395,8 +415,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Entities and relationships round-trip as memory-local provenance only — Markdown import intentionally does not write into `ContextGraph`, matching the MVP scope agreed on in #765
- Documented the file contract and workflow in `docs/reference/context.md`; 43 new tests in `tests/context/test_agent_memory_markdown.py` cover round-trip losslessness, idempotency, validation errors, rollback on failure, and vector-store sync ordering
- **Markdown directory round trips for `ContextGraph`** (#852) by @SaurabhScripts
- `ContextGraph.save_to_file(..., format="markdown")` and `load_from_file(..., format="markdown")` persist a deterministic `graph.md` relationship manifest plus one human-editable Markdown file per node, preserving graph, node, edge, family, temporal, and cross-graph link identities
- Imports validate the complete directory before replacing graph state, rebuild indexes and analytics state atomically, create JSON-compatible stub nodes for dangling edge endpoints, and emit the same granular node/edge audit events as JSON loading
- Existing exports are replaced atomically only after their complete canonical layout is validated; untracked files, renamed node files, symlinks, Windows directory junctions, and other reparse points cause a fail-closed error instead of authorizing directory deletion
- Added 30 focused tests covering deterministic round trips, manual edits, validation rollback, managed-directory identity, publish rollback, audit-manager compatibility, stale-cache clearing, mocked and real Windows junctions, and missing-path behavior
### Fixed
- **Markdown import followed filesystem links even though Markdown export already refused to overwrite them** (#851, follow-up to #765, #786) by @SaurabhScripts
- `AgentMemory._read_markdown_path()` now rejects symlink files, broken symlinks, symlinked directories, Windows directory junctions, and other Windows reparse points supplied directly; linked entries discovered inside an otherwise valid directory are safely skipped, preserving the current directory-import contract
- `_read_markdown_file_content()` re-checks the file and parent directory immediately before and after opening, uses `O_NOFOLLOW` where available, and verifies the resulting descriptor is a regular file via `fstat`/`S_ISREG`, so link swaps are rejected rather than silently followed
- Junction detection uses `os.path.isjunction()` where available and falls back to the Windows reparse-point file attribute on older Python versions; export applies the same link check before replacing a Markdown file
- Documented the import restriction in `docs/reference/context.md`; added 11 tests to `tests/context/test_agent_memory_markdown.py` covering file/directory/broken-symlink rejection, simulated open races, mocked and real Windows junctions, and the reparse-point fallback
- Any additional review follow-up commits land in this same PR/entry rather than as a separate changelog item
- **`PipelineWithProvenance` raised `ModuleNotFoundError` on import and `AttributeError` on `.run()`** (#858, closes #858) by @Karunasagar12
- `from .pipeline import Pipeline` failed because `semantica/pipeline/pipeline.py` does not exist; corrected to `from .pipeline_builder import Pipeline`
- `.run()` called `self._pipeline.run()` on the `Pipeline` dataclass, which has no such method; replaced with `self._engine.execute_pipeline(self._pipeline, ...)` delegating to `ExecutionEngine`
@@ -1513,4 +1546,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
For detailed release notes, see [GitHub Releases](https://github.com/Hawksight-AI/semantica/releases).
For detailed release notes, see [GitHub Releases](https://github.com/semantica-agi/semantica/releases).
+1 -1
View File
@@ -58,7 +58,7 @@ representative at an online or offline event.
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement through
[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[CoC]" prefix.
[GitHub Issues](https://github.com/semantica-agi/semantica/issues) with "[CoC]" prefix.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
+3 -3
View File
@@ -44,7 +44,7 @@ We recognize all types of contributions:
All contributors are recognized in:
- This contributors list
- [GitHub contributors page](https://github.com/Hawksight-AI/semantica/graphs/contributors)
- [GitHub contributors page](https://github.com/semantica-agi/semantica/graphs/contributors)
- Release notes for significant contributions
- Community appreciation
@@ -54,7 +54,7 @@ All contributors are recognized in:
### Automatic Recognition
If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors).
If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/semantica-agi/semantica/graphs/contributors).
### Using All-Contributors Bot
@@ -111,4 +111,4 @@ Every contribution, no matter how small, helps make Semantica better. Thank you
**Want to contribute?**
⭐ Give us a Star • 🍴 [Fork us](https://github.com/Hawksight-AI/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
⭐ Give us a Star • 🍴 [Fork us](https://github.com/semantica-agi/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
+1 -1
View File
@@ -9,7 +9,7 @@ RUN npm ci
COPY explorer/ ./
RUN mkdir -p /app/semantica && npm run build
FROM python:3.14-slim AS runtime
FROM python:3.13-slim AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
+1 -1
View File
@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2026 Hawksight AI
Copyright (c) 2026 Semantica
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+31 -16
View File
@@ -87,7 +87,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
- **Drop-in Integrations:** Native Agno and CrewAI support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
- **Drop-in Integrations:** Native Agno, CrewAI, and LangChain support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
---
@@ -147,6 +147,8 @@ semantica doctor
# Config file pass ~/.semantica/config.yaml
```
**Running in a script or CI?** Progress bars are written only when stdout is an interactive terminal (or a Jupyter notebook), so piping and redirecting stay clean by default. Override with `SEMANTICA_DISABLE_PROGRESS=1` to silence progress everywhere, or `SEMANTICA_FORCE_PROGRESS=1` to keep it when stdout is redirected. `SEMANTICA_DISABLE_PROGRESS` takes precedence.
<div align="center">
If Semantica solves a real problem for you, a star helps others find it.
@@ -1186,7 +1188,7 @@ Start with `semantica`, verify with `doctor`, build a graph, and explore the com
## Integrations
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno and CrewAI support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno, CrewAI, and LangChain support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
@@ -1305,17 +1307,17 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<strong>CrewAI</strong><br/>
<sub>First-class · <code>pip install semantica[crewai]</code></sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langchain"><img src="https://github.com/langchain-ai.png?size=120" alt="LangChain" width="48" height="48" /></a><br/>
<strong>LangChain</strong><br/>
<sub>First-class · <code>pip install semantica[langchain]</code></sub>
</td>
</tr>
<tr>
<th colspan="8" align="left">Already Supported via REST API &amp; MCP</th>
</tr>
<tr>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langchain"><img src="https://github.com/langchain-ai.png?size=120" alt="LangChain" width="48" height="48" /></a><br/>
<strong>LangChain</strong><br/>
<sub>REST API · MCP</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langgraph"><img src="https://github.com/langchain-ai.png?size=120" alt="LangGraph" width="48" height="48" /></a><br/>
<strong>LangGraph</strong><br/>
<sub>REST API · MCP</sub>
@@ -1346,11 +1348,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
</tr>
<tr>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langchain"><img src="https://github.com/langchain-ai.png?size=120" alt="LangChain" width="48" height="48" /></a><br/>
<strong>LangChain</strong><br/>
<sub>Dedicated toolkit</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/run-llama/llama_index"><img src="https://github.com/run-llama.png?size=120" alt="LlamaIndex" width="48" height="48" /></a><br/>
<strong>LlamaIndex</strong><br/>
<sub>Dedicated toolkit</sub>
@@ -1509,6 +1506,7 @@ pip install semantica[all] # everything
```bash
pip install semantica[agno] # Agno multi-agent integration
pip install semantica[crewai] # CrewAI integration
pip install semantica[langchain] # LangChain / LangGraph integration
pip install semantica[llm-litellm] # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
pip install semantica[graph-neo4j] # Neo4j graph store (LPG)
pip install semantica[graph-falkordb] # FalkorDB graph store (LPG)
@@ -1561,11 +1559,11 @@ On-premises deployment · Private cloud · Custom domain implementations · SLA-
## Star History
<a href="https://www.star-history.com/?repos=semantica-agi%2Fsemantica&type=date&legend=top-left">
<a href="https://star-history.dera.page/#semantica-agi/semantica&amp;type=date&amp;legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=semantica-agi/semantica&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=semantica-agi/semantica&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=semantica-agi/semantica&type=date&legend=top-left" />
<source media="(prefers-color-scheme: dark)" srcset="https://star-history.dera.page/svg?repos=semantica-agi/semantica&amp;type=date&amp;theme=dark&amp;legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://star-history.dera.page/svg?repos=semantica-agi/semantica&amp;type=date&amp;legend=top-left" />
<img alt="Star History Chart" src="https://star-history.dera.page/svg?repos=semantica-agi/semantica&amp;type=date&amp;legend=top-left" />
</picture>
</a>
@@ -1594,6 +1592,23 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines.
---
## Cite Us
If you use Semantica in your research or production systems, please cite it as:
```bibtex
@software{semantica2026,
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica}
}
```
All citation formats (APA, MLA, Chicago, IEEE) live on the [Citation](https://docs.getsemantica.ai/citation) page — every format attributes authorship to **Semantica**, not individual contributors.
---
<div align="center">
MIT License · Built by [Semantica](https://github.com/semantica-agi)
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)\n",
"\n",
"# Advanced Extraction\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)\n",
"\n",
"# Complete Visualization Suite\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)\n",
"\n",
"# Advanced Multi-Format Export\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)\n",
"\n",
"# Reasoning and Inference\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/09_Semantic_Layer_Construction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/09_Semantic_Layer_Construction.ipynb)\n",
"\n",
"# Semantic Layer Construction\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)\n",
"\n",
"# Deep Dive: Temporal Knowledge Graphs\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"\n",
"# Unstructured Text to Ontology\n",
"\n",
@@ -18,7 +18,7 @@
"id": "cell-0",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb)\n",
"\n",
"# Manual Ontology + Snowflake Mapping\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"\n",
"# Datalog-Style Reasoning\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"\n",
"# Advanced Vector Store - Made Easy\n",
"\n",
@@ -352,7 +352,7 @@
"- Build a multi-user application\n",
"- Explore the [introduction notebook](../introduction/13_Vector_Store.ipynb) for more basics\n",
"\n",
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/Hawksight-AI/semantica)."
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/semantica-agi/semantica)."
]
}
],
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
"\n",
"Semantica is a **semantic intelligence and knowledge engineering framework**. It helps you:\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
"\n",
"# Data Ingestion - Comprehensive Guide\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/04_Document_Parsing.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)\n",
"\n",
"# Document Parsing\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Data_Normalization.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/04_Data_Normalization.ipynb)\n",
"\n",
"# Data Normalization\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
"\n",
"# Entity Extraction - Comprehensive Guide\n",
"\n",
@@ -622,7 +622,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
"\n",
"# Relation Extraction - Comprehensive Guide\n",
"\n",
@@ -599,7 +599,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Building_Knowledge_Graphs.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)\n",
"\n",
"# Building Knowledge Graphs\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)\n",
"\n",
"# 🚀 Your First Knowledge Graph\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Graph_Analytics.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb)\n",
"\n",
"# Graph Analytics\n",
"\n",
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
"\n",
"# Chunking and Splitting - Comprehensive Guide\n",
"\n",
@@ -817,7 +817,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Embedding_Generation.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/12_Embedding_Generation.ipynb)\n",
"\n",
"# Embedding Generation\n",
"\n",
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"\n",
"# Vector Store - Comprehensive Guide\n",
"\n",
@@ -492,7 +492,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
"\n",
"# Ontology Generation \n",
"\n",
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/15_Export.ipynb)\n",
"\n",
"# Export Module - Comprehensive Guide\n",
"\n",
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/17_Visualization.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)\n",
"\n",
"# Visualization\n",
"\n",
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
"\n",
"# Deduplication in Semantica\n",
"\n",
@@ -5,7 +5,7 @@
"id": "c21e9c8d",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb)\n",
"\n",
"# Context Module — Practical Guide\n",
"\n",
@@ -0,0 +1,253 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Provenance Tracking (W3C PROV-O)\n",
"\n",
"## Overview\n",
"\n",
"In high-stakes domains — healthcare, legal, finance, research — a Knowledge Graph is only as trustworthy as its ability to answer **\"where did this fact come from?\"**. Semantica's `provenance` module provides audit-grade, W3C PROV-O-aligned tracking for every entity, relationship and chunk that flows through your pipeline.\n",
"\n",
"In this cookbook you will learn how to:\n",
"\n",
"- Track entities and relationships with **source details** (DOI, page, verbatim quote, confidence)\n",
"- Walk the full **lineage** of a fact (document → chunk → entity → KG)\n",
"- Audit **revision history** and **all sources** behind an entity\n",
"- **Invalidate** a fact without deleting it (prov:Invalidation) — corrections stay provable\n",
"- Verify **tamper-evidence** with chained SHA-256 checksums\n",
"\n",
"**The Scenario:** a research team ingests findings from two scientific papers (with DOIs) into a Knowledge Graph. A regulator later asks: *\"Which paper, which figure, and which exact sentence supports the claim that fish biomass increased by 463%? And was that fact ever corrected?\"*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"from semantica.provenance import (\n",
" ProvenanceManager,\n",
" compute_checksum,\n",
" verify_checksum,\n",
")\n",
"\n",
"# In-memory storage for this demo; pass storage_path=\"provenance.db\"\n",
"# (or a config with provenance.storage_path) for a persistent SQLite backend.\n",
"prov = ProvenanceManager()\n",
"print(\"ProvenanceManager ready (in-memory storage)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Track Entities with Audit-Grade Source Details\n",
"\n",
"Every fact we ingest carries its evidence with it: the **source identifier** (a DOI here), the **location** inside the source (a figure), the **verbatim quote**, and the extractor's **confidence**."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Finding from paper #1\n",
"entry_biomass = prov.track_entity(\n",
" entity_id=\"claim_biomass_increase\",\n",
" source=\"DOI:10.1371/journal.pone.0023601\",\n",
" confidence=0.92,\n",
" source_location=\"Figure 2\",\n",
" source_quote=\"Total fish biomass increased by 463% ...\",\n",
")\n",
"\n",
"# Supporting entity from paper #2\n",
"entry_reserve = prov.track_entity(\n",
" entity_id=\"marine_reserve_1\",\n",
" source=\"DOI:10.1126/science.1088121\",\n",
" confidence=0.88,\n",
" source_location=\"Table 1\",\n",
" source_quote=\"... no-take marine reserve at Cabo Pulmo ...\",\n",
")\n",
"\n",
"print(\"Tracked:\", entry_biomass.entity_id, \"|\", entry_reserve.entity_id)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Track the Relationship Between Facts\n",
"\n",
"Facts rarely stand alone. The claim about biomass increase is *about* the marine reserve — that relationship is a first-class provenance-tracked object too.\n",
"\n",
"`track_relationship()` has no dedicated subject/object fields, so by convention we record which two entities it connects inside `metadata`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rel = prov.track_relationship(\n",
" relationship_id=\"rel_biomass_about_reserve\",\n",
" source=\"DOI:10.1371/journal.pone.0023601\",\n",
" metadata={\n",
" \"type\": \"measured_at\",\n",
" # No dedicated endpoint fields on track_relationship() yet -- record\n",
" # which entities this relationship connects here by convention.\n",
" \"subject_entity_id\": \"claim_biomass_increase\",\n",
" \"object_entity_id\": \"marine_reserve_1\",\n",
" },\n",
")\n",
"\n",
"print(\"Relationship tracked:\", rel.entity_id, \"|\", rel.metadata[\"subject_entity_id\"], \"->\", rel.metadata[\"object_entity_id\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Walk the Lineage\n",
"\n",
"`get_lineage` reconstructs everything known about a fact; `trace_lineage` returns the ordered chain of `ProvenanceEntry` records — every version, every activity, every agent that touched it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lineage = prov.get_lineage(\"claim_biomass_increase\")\n",
"print(json.dumps(lineage, indent=2, default=str)[:800])\n",
"\n",
"print(\"\\n--- ordered chain ---\")\n",
"for e in prov.trace_lineage(\"claim_biomass_increase\"):\n",
" print(f\"{e.entity_id} | seq#{e.sequence_id} | {e.activity_id}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Audit Sources and Revision History\n",
"\n",
"When the regulator asks *\"has this fact ever been corrected?\"*, `revision_history` answers with the full version chain, and `get_all_sources` lists every source document that ever supported the entity."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"revisions = prov.revision_history(\"claim_biomass_increase\")\n",
"print(f\"{len(revisions)} revision(s) on record\")\n",
"\n",
"for s in prov.get_all_sources(\"claim_biomass_increase\"):\n",
" print(\"source:\", s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Invalidate — Correct Without Deleting\n",
"\n",
"Suppose paper #1 is retracted in part. An audit trail must **not** silently delete the fact: `invalidate` archives the pre-invalidation state and appends a fresh `prov:Invalidation` entry naming **who** retracted it and **why**."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"invalidated = prov.invalidate(\n",
" entity_id=\"claim_biomass_increase\",\n",
" agent_id=\"reviewer_dr_chen\",\n",
" reason=\"Partial retraction: Figure 2 statistics corrected by publisher (see erratum).\",\n",
")\n",
"print(\"Invalidated:\", invalidated.entity_id, \"| invalidated flag:\", getattr(invalidated, \"invalidated\", True))\n",
"\n",
"stats = prov.get_statistics()\n",
"print(\"\\nStorage statistics:\", json.dumps(stats, indent=2, default=str))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Verify Tamper-Evidence\n",
"\n",
"Each entry carries a deterministic SHA-256 checksum chained to the previous entry. Recompute and compare to detect any after-the-fact corruption of the provenance record."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# entry_biomass was returned by track_entity in Step 1\n",
"ok = verify_checksum(entry_biomass)\n",
"print(\"Checksum verified:\", ok)\n",
"\n",
"print(\"Computed:\", compute_checksum(entry_biomass)[:16], \"...\")\n",
"print(\"Stored: \", entry_biomass.checksum[:16] if getattr(entry_biomass, 'checksum', None) else \"(see entry fields)\")\n",
"chain = prov.verify_chain()\n",
"print(\"Chain verification:\", json.dumps(chain, default=str)[:200])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Need | Call |\n",
"|---|---|\n",
"| Record a fact's evidence | `prov.track_entity(entity_id, source, confidence=..., source_location=..., source_quote=...)` |\n",
"| Record a relationship | `prov.track_relationship(relationship_id, source, metadata=...)` |\n",
"| Full lineage of a fact | `prov.get_lineage(entity_id)` / `prov.trace_lineage(entity_id)` |\n",
"| \"Was it ever corrected?\" | `prov.revision_history(entity_id)` |\n",
"| \"Which sources support it?\" | `prov.get_all_sources(entity_id)` |\n",
"| Retract without deleting | `prov.invalidate(entity_id, agent_id, reason=...)` |\n",
"| Tamper check | `verify_checksum(entry)` |\n",
"\n",
"### Where to go next\n",
"\n",
"- **Conflict Detection and Resolution** (notebook 17) — what happens when two sources disagree.\n",
"- **Your First Knowledge Graph** (notebook 08) — plug `provenance=True` into extractors so tracking happens automatically during ingestion.\n",
"- The module docstring (`help(semantica.provenance)`) documents opt-in integration with `kg`, `split` and `conflicts` trackers."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
+383
View File
@@ -0,0 +1,383 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b76a5997",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)\n",
"\n",
"# Reasoning Module — Practical Guide\n",
"\n",
"Semantica's `reasoning` module derives new knowledge from existing facts and knowledge graphs. It ships several strategies behind one facade:\n",
"\n",
"- **`Reasoner`** — unified facade with forward chaining, backward chaining, and one-shot `infer_facts`\n",
"- **`DatalogReasoner`** — semi-naive Datalog fixpoint evaluation with variable queries\n",
"- **`ExplanationGenerator`** — human-readable explanations and reasoning paths for inferred conclusions\n",
"- Plus lower-level engines: `ReteEngine`, `SPARQLReasoner`, `GraphReasoner`, temporal reasoning\n",
"\n",
"This notebook walks through the facade, the Datalog engine, and explanations. All APIs are verified against `semantica/reasoning/`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "52073af7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:45:55.427457Z",
"iopub.status.busy": "2026-08-26T18:45:55.427247Z",
"iopub.status.idle": "2026-08-26T18:45:57.266607Z",
"shell.execute_reply": "2026-08-26T18:45:57.264783Z"
}
},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "markdown",
"id": "06deb916",
"metadata": {},
"source": [
"## 1) Forward chaining with the `Reasoner` facade\n",
"\n",
"Facts are simple `Predicate(args)` strings. Rules use `IF <conditions> THEN <conclusion>` with `?x`-style variables. `forward_chain()` derives everything possible and returns a list of `InferenceResult` objects."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "519ca92d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:45:57.270791Z",
"iopub.status.busy": "2026-08-26T18:45:57.270352Z",
"iopub.status.idle": "2026-08-26T18:45:59.991941Z",
"shell.execute_reply": "2026-08-26T18:45:59.990678Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>Reasoner</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>DatalogReasoner</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>ExplanationGenerator</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"🔄 Semantica is reasoning: Performing forward chaining 🤔 reasoning Reasoner |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Inferred 2 new facts\n",
" Human(Jane) (rule: Rule 1, confidence: 1.0)\n",
" Human(John) (rule: Rule 1, confidence: 1.0)\n"
]
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"\n",
"reasoner.add_fact(\"Person(John)\")\n",
"reasoner.add_fact(\"Person(Jane)\")\n",
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"results = reasoner.forward_chain()\n",
"print(f\"Inferred {len(results)} new facts\")\n",
"for res in results:\n",
" print(f\" {res.conclusion} (rule: {res.rule_used.name}, confidence: {res.confidence})\")"
]
},
{
"cell_type": "markdown",
"id": "c1131c45",
"metadata": {},
"source": [
"## 2) One-shot inference with `infer_facts`\n",
"\n",
"`infer_facts(facts, rules)` **adds** the given facts and rules to this `Reasoner` instance, runs forward chaining to fixpoint, and returns the derived facts as strings. It does not reset the instance's existing state — create a fresh `Reasoner()` first if you need isolation between runs."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "26249990",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:45:59.995447Z",
"iopub.status.busy": "2026-08-26T18:45:59.995069Z",
"iopub.status.idle": "2026-08-26T18:46:00.004107Z",
"shell.execute_reply": "2026-08-26T18:46:00.002873Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['Employee(Jane, Acme)', 'Employee(John, Acme)']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"derived = Reasoner().infer_facts(\n",
" facts=[\"WorksFor(John, Acme)\", \"WorksFor(Jane, Acme)\"],\n",
" rules=[\"IF WorksFor(?x, ?y) THEN Employee(?x, ?y)\"],\n",
")\n",
"derived"
]
},
{
"cell_type": "markdown",
"id": "d5504a38",
"metadata": {},
"source": [
"## 3) Backward chaining: proving a goal\n",
"\n",
"`backward_chain(goal)` works backwards from a conclusion through the rules. It returns the `InferenceResult` that proves the goal, or `None`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c4ef85dd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.007740Z",
"iopub.status.busy": "2026-08-26T18:46:00.007346Z",
"iopub.status.idle": "2026-08-26T18:46:00.015561Z",
"shell.execute_reply": "2026-08-26T18:46:00.014145Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Human(John)\n",
"premises: ['Person(John)']\n"
]
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"proof = reasoner.backward_chain(\"Human(John)\")\n",
"print(proof.conclusion if proof else \"not provable\")\n",
"print(\"premises:\", proof.premises if proof else None)"
]
},
{
"cell_type": "markdown",
"id": "b245581d",
"metadata": {},
"source": [
"## 4) Re-run safety\n",
"\n",
"`add_rule` deduplicates rules with identical conditions and conclusion, so re-executing a setup cell (the common Jupyter re-run) does not duplicate rules — see issue #732."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "fb2aeb39",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.019091Z",
"iopub.status.busy": "2026-08-26T18:46:00.018881Z",
"iopub.status.idle": "2026-08-26T18:46:00.024042Z",
"shell.execute_reply": "2026-08-26T18:46:00.022836Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Skipping duplicate rule (same conditions/conclusion as 'rule_1'): IF Person(?x) THEN Human(?x)\n"
]
},
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"\n",
"# Simulate a Jupyter cell re-run: add the same rule twice\n",
"r1 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"r2 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"len(reasoner.rules)"
]
},
{
"cell_type": "markdown",
"id": "ba2e5c4a",
"metadata": {},
"source": [
"## 5) Datalog reasoning\n",
"\n",
"`DatalogReasoner` uses classic Datalog syntax (`head :- body.`) and semi-naive fixpoint evaluation. Queries return variable bindings as a list of dicts — use uppercase variables to ask *which* facts hold."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9ec5c0c4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.026769Z",
"iopub.status.busy": "2026-08-26T18:46:00.026588Z",
"iopub.status.idle": "2026-08-26T18:46:00.034963Z",
"shell.execute_reply": "2026-08-26T18:46:00.032672Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[{'X': 'tom', 'Z': 'ann'}]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import DatalogReasoner\n",
"\n",
"datalog = DatalogReasoner()\n",
"datalog.add_fact(\"parent(tom, mary)\")\n",
"datalog.add_fact(\"parent(mary, ann)\")\n",
"datalog.add_rule(\"grandparent(X, Z) :- parent(X, Y), parent(Y, Z)\")\n",
"\n",
"datalog.derive_all()\n",
"datalog.query(\"grandparent(X, Z)\")"
]
},
{
"cell_type": "markdown",
"id": "d4f0689b",
"metadata": {},
"source": [
"## 6) Explanations for inferred conclusions\n",
"\n",
"`ExplanationGenerator` turns `InferenceResult` objects into structured `Explanation` and `ReasoningPath` records, so agents can show *why* they believe a derived fact."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "19dcd3a7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.038649Z",
"iopub.status.busy": "2026-08-26T18:46:00.038396Z",
"iopub.status.idle": "2026-08-26T18:46:00.059188Z",
"shell.execute_reply": "2026-08-26T18:46:00.057805Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"('Explanation', 'ReasoningPath')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner, ExplanationGenerator\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"results = reasoner.forward_chain()\n",
"\n",
"gen = ExplanationGenerator()\n",
"explanation = gen.generate_explanation(results[0])\n",
"path = gen.show_reasoning_path(results[0])\n",
"\n",
"type(explanation).__name__, type(path).__name__"
]
},
{
"cell_type": "markdown",
"id": "fb882ee4",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Task | API |\n",
"|---|---|\n",
"| Derive all new facts | `Reasoner.forward_chain()` |\n",
"| One-shot inference | `Reasoner.infer_facts(facts, rules)` |\n",
"| Prove a goal | `Reasoner.backward_chain(goal)` |\n",
"| Datalog fixpoint | `DatalogReasoner.derive_all()` + `query(\"p(X, Y)\")` |\n",
"| Explain a conclusion | `ExplanationGenerator.generate_explanation(result)` |\n",
"\n",
"See also `semantica/reasoning/reasoning_usage.md` and the module docstrings for `ReteEngine`, `SPARQLReasoner`, and temporal reasoning."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
@@ -0,0 +1,299 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "8d7096ea",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)\n",
"\n",
"# Change Management — Practical Guide\n",
"\n",
"Semantica's `change_management` module provides versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies:\n",
"\n",
"- **`ChangeLogEntry`** — standardized change metadata (validated timestamp/author)\n",
"- **`InMemoryVersionStorage` / `SQLiteVersionStorage`** — version snapshot storage with named tags\n",
"- **`compute_checksum` / `verify_checksum`** — SHA-256 integrity verification\n",
"\n",
"This notebook runs a complete save → tag → verify → tamper-detect cycle. All outputs are real executed results verified against the repository's `semantica/change_management/` source at the time of writing (the `pip install` cell may fetch a newer release with slightly different behavior)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "7bdffec1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:37.171333Z",
"iopub.status.busy": "2026-08-26T18:46:37.171183Z",
"iopub.status.idle": "2026-08-26T18:46:39.060860Z",
"shell.execute_reply": "2026-08-26T18:46:39.059594Z"
}
},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "markdown",
"id": "169efee1",
"metadata": {},
"source": [
"## 1) A `ChangeLogEntry` records *who* changed *what*, *when*\n",
"\n",
"`author` must be a valid email — the dataclass validates on construction (`ValidationError` otherwise), which keeps audit trails clean."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5b17acdb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.064077Z",
"iopub.status.busy": "2026-08-26T18:46:39.063818Z",
"iopub.status.idle": "2026-08-26T18:46:39.321881Z",
"shell.execute_reply": "2026-08-26T18:46:39.321036Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"ChangeLogEntry(timestamp='2026-08-15T09:00:00Z', author='demo@example.com', description='initial version', change_id=None, related_changes=[])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.change_management import ChangeLogEntry\n",
"\n",
"entry = ChangeLogEntry(\n",
" timestamp=\"2026-08-15T09:00:00Z\",\n",
" author=\"demo@example.com\",\n",
" description=\"initial version\",\n",
")\n",
"entry"
]
},
{
"cell_type": "markdown",
"id": "53d8df5c",
"metadata": {},
"source": [
"## 2) Save a versioned snapshot\n",
"\n",
"A snapshot is a dict with a required `label` plus your payload. Here we attach the KG data, the change log, and a SHA-256 `checksum` computed over everything except the checksum field itself."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "fec16f24",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.325528Z",
"iopub.status.busy": "2026-08-26T18:46:39.325140Z",
"iopub.status.idle": "2026-08-26T18:46:39.331480Z",
"shell.execute_reply": "2026-08-26T18:46:39.330586Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.change_management import InMemoryVersionStorage, compute_checksum\n",
"\n",
"storage = InMemoryVersionStorage()\n",
"\n",
"snapshot = {\n",
" \"label\": \"v1.0.0\",\n",
" \"data\": {\"entities\": {\"acme\": {\"type\": \"Company\"}}},\n",
" \"change_log\": {\n",
" \"timestamp\": entry.timestamp,\n",
" \"author\": entry.author,\n",
" \"description\": entry.description,\n",
" },\n",
"}\n",
"snapshot[\"checksum\"] = compute_checksum({k: v for k, v in snapshot.items() if k != \"checksum\"})\n",
"\n",
"storage.save(snapshot)\n",
"storage.exists(\"v1.0.0\")"
]
},
{
"cell_type": "markdown",
"id": "0f1c603b",
"metadata": {},
"source": [
"## 3) Named tags pin a version for releases\n",
"\n",
"`save_tag` / `get_tag` map stable names (e.g. `release`) to version labels, decoupling consumers from label churn."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "62f7643e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.335182Z",
"iopub.status.busy": "2026-08-26T18:46:39.334886Z",
"iopub.status.idle": "2026-08-26T18:46:39.339586Z",
"shell.execute_reply": "2026-08-26T18:46:39.338568Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"('v1.0.0', ['v1.0.0'])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"storage.save_tag(\"release\", \"v1.0.0\")\n",
"\n",
"storage.get_tag(\"release\"), [s[\"label\"] for s in storage.list_all()]"
]
},
{
"cell_type": "markdown",
"id": "96df12da",
"metadata": {},
"source": [
"## 4) Verify integrity — and catch tampering\n",
"\n",
"`verify_checksum(snapshot)` recomputes the SHA-256 over the snapshot (minus its `checksum` field) and compares. A single mutated character in the data flips the result to `False`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "26d0de85",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.342653Z",
"iopub.status.busy": "2026-08-26T18:46:39.342466Z",
"iopub.status.idle": "2026-08-26T18:46:39.346714Z",
"shell.execute_reply": "2026-08-26T18:46:39.345623Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"intact: True\n",
"tampered: False\n"
]
}
],
"source": [
"from semantica.change_management import verify_checksum\n",
"\n",
"stored = storage.get(\"v1.0.0\")\n",
"print(\"intact:\", verify_checksum(stored))\n",
"\n",
"tampered = storage.get(\"v1.0.0\")\n",
"tampered[\"data\"][\"entities\"][\"acme\"][\"note\"] = \"mutated after the fact\"\n",
"print(\"tampered:\", verify_checksum(tampered))"
]
},
{
"cell_type": "markdown",
"id": "bd14c3e4",
"metadata": {},
"source": [
"## 5) Retiring a version\n",
"\n",
"`delete(label)` removes a snapshot; tags pointing at it are your responsibility to update."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "de9fe3e5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.349814Z",
"iopub.status.busy": "2026-08-26T18:46:39.349513Z",
"iopub.status.idle": "2026-08-26T18:46:39.354710Z",
"shell.execute_reply": "2026-08-26T18:46:39.353669Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"storage.delete(\"v1.0.0\")\n",
"storage.exists(\"v1.0.0\")"
]
},
{
"cell_type": "markdown",
"id": "ab667b32",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Task | API |\n",
"|---|---|\n",
"| Record audit metadata | `ChangeLogEntry(timestamp, author=email, description)` |\n",
"| Persist a version | `InMemoryVersionStorage().save({\"label\": ..., ...})` |\n",
"| Pin a release name | `save_tag(\"release\", \"v1.0.0\")` / `get_tag(\"release\")` |\n",
"| Integrity check | `compute_checksum(snap)` / `verify_checksum(snap)` |\n",
"| Persistent backend | `SQLiteVersionStorage(path)` — same interface |\n",
"\n",
"See also `semantica/change_management/change_management_usage.md` for the manager classes (`TemporalVersionManager`, `OntologyVersionManager`)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
+314
View File
@@ -0,0 +1,314 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "6eb4dfba",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)\n",
"\n",
"# Seed Data — Practical Guide\n",
"\n",
"The `seed` module bootstraps a knowledge graph from **trusted, pre-known data** (CSV/JSON/database/API sources) before any extraction runs. This gives extraction a foundation to link against instead of starting from an empty graph.\n",
"\n",
"Key pieces:\n",
"\n",
"- **`SeedDataManager`** — registers data sources and builds foundation graphs\n",
"- **`create_foundation_graph()`** — turns registered sources into `entities` + `relationships` + `metadata`\n",
"- **`validate_quality()`** — checks a foundation graph before you commit it\n",
"\n",
"All examples below were executed against `semantica/seed/seed_manager.py`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "32f80cc6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:18.716466Z",
"iopub.status.busy": "2026-08-26T18:51:18.716264Z",
"iopub.status.idle": "2026-08-26T18:51:20.533828Z",
"shell.execute_reply": "2026-08-26T18:51:20.531402Z"
}
},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "markdown",
"id": "75136e5f",
"metadata": {},
"source": [
"## 1) Prepare a seed CSV and register the source\n",
"\n",
"`register_source(name, format, location, entity_type=...)` records where trusted data lives. `verified=True` (the default) marks the source as pre-validated."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a8089e1f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.538772Z",
"iopub.status.busy": "2026-08-26T18:51:20.538323Z",
"iopub.status.idle": "2026-08-26T18:51:20.675403Z",
"shell.execute_reply": "2026-08-26T18:51:20.674060Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import csv\n",
"import tempfile\n",
"from pathlib import Path\n",
"from semantica.seed import SeedDataManager\n",
"\n",
"# Write the sample CSV into a session-scoped temp directory so we never\n",
"# clobber a companies.csv that might exist in the user's working directory.\n",
"seed_csv = Path(tempfile.mkdtemp(prefix=\"semantica-seed-\")) / \"companies.csv\"\n",
"with open(seed_csv, \"w\", newline=\"\") as f:\n",
" writer = csv.DictWriter(f, fieldnames=[\"id\", \"name\", \"type\", \"industry\"])\n",
" writer.writeheader()\n",
" writer.writerow({\"id\": \"c1\", \"name\": \"Acme\", \"type\": \"Company\", \"industry\": \"robotics\"})\n",
" writer.writerow({\"id\": \"c2\", \"name\": \"Globex\", \"type\": \"Company\", \"industry\": \"energy\"})\n",
"\n",
"manager = SeedDataManager()\n",
"manager.register_source(\"companies\", format=\"csv\", location=str(seed_csv), entity_type=\"Company\")\n"
]
},
{
"cell_type": "markdown",
"id": "e87221ba",
"metadata": {},
"source": [
"## 2) Load records from a registered source\n",
"\n",
"`load_source(name)` reads the source and enriches each record with `entity_type` and `source` provenance keys."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "f932e550",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.679424Z",
"iopub.status.busy": "2026-08-26T18:51:20.679156Z",
"iopub.status.idle": "2026-08-26T18:51:20.690659Z",
"shell.execute_reply": "2026-08-26T18:51:20.688812Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is seeding</td><td>🌱 seed</td><td>SeedDataManager</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"🔄 Semantica is seeding: Loading seed data from CSV: /var/folders/7s/bvvstgs10y963tz6_4bbnklr0000gn/T/semantica-seed-eu9__ep1/companies.csv 🌱 seed SeedDataManager |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"loaded 2 records\n"
]
},
{
"data": {
"text/plain": [
"{'id': 'c1',\n",
" 'name': 'Acme',\n",
" 'type': 'Company',\n",
" 'industry': 'robotics',\n",
" 'entity_type': 'Company',\n",
" 'source': 'companies'}"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"records = manager.load_source(\"companies\")\n",
"print(f\"loaded {len(records)} records\")\n",
"records[0]"
]
},
{
"cell_type": "markdown",
"id": "f2ebce64",
"metadata": {},
"source": [
"## 3) Build the foundation graph\n",
"\n",
"`create_foundation_graph()` converts every registered source into graph-ready entities and relationships. Entities carry `confidence: 1.0` — seed data is trusted by definition."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "09388c31",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.695259Z",
"iopub.status.busy": "2026-08-26T18:51:20.694928Z",
"iopub.status.idle": "2026-08-26T18:51:20.708595Z",
"shell.execute_reply": "2026-08-26T18:51:20.707072Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['entities', 'metadata', 'relationships']"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"foundation = manager.create_foundation_graph()\n",
"sorted(foundation.keys())"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4610a59f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.713136Z",
"iopub.status.busy": "2026-08-26T18:51:20.712795Z",
"iopub.status.idle": "2026-08-26T18:51:20.718637Z",
"shell.execute_reply": "2026-08-26T18:51:20.716835Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'c1',\n",
" 'text': 'Acme',\n",
" 'type': 'Company',\n",
" 'confidence': 1.0,\n",
" 'metadata': {'industry': 'robotics', 'source': 'companies'}}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"foundation[\"entities\"][0]"
]
},
{
"cell_type": "markdown",
"id": "f3a52dc7",
"metadata": {},
"source": [
"## 4) Validate quality before committing\n",
"\n",
"`validate_quality(foundation_graph)` returns `valid`, `errors`, `warnings`, and `metrics` so you can gate bad seed data before it pollutes the graph."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "4eb7e664",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.722674Z",
"iopub.status.busy": "2026-08-26T18:51:20.722118Z",
"iopub.status.idle": "2026-08-26T18:51:20.732003Z",
"shell.execute_reply": "2026-08-26T18:51:20.730170Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"quality = manager.validate_quality(foundation)\n",
"quality[\"valid\"]"
]
},
{
"cell_type": "markdown",
"id": "b534be89",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Task | API |\n",
"|---|---|\n",
"| Register a trusted source | `register_source(name, format, location, entity_type=...)` |\n",
"| Load records | `load_source(name)` — adds `entity_type` / `source` keys |\n",
"| Direct file load | `load_from_csv(path)` / `load_from_json(path)` |\n",
"| Build the graph | `create_foundation_graph()` → `entities` / `relationships` / `metadata` |\n",
"| Gate bad data | `validate_quality(graph)` → `valid` / `errors` / `warnings` / `metrics` |\n",
"\n",
"See also `semantica/seed/seed_usage.md` for `load_from_database`, `load_from_api`, and `integrate_with_extracted`."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
+9 -10
View File
@@ -13,26 +13,25 @@ icon: "quote-left"
<Tab title="BibTeX">
```bibtex
@software{semantica2026,
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica},
version = {0.6.6},
doi = {10.5281/zenodo.XXXXXXX}
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica},
doi = {10.5281/zenodo.XXXXXXX}
}
```
</Tab>
<Tab title="APA">
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.6) \[Computer software\]. https://github.com/semantica-agi/semantica
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* \[Computer software\]. https://github.com/semantica-agi/semantica
</Tab>
<Tab title="MLA">
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6, GitHub, 2026, https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. GitHub, 2026, https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="Chicago">
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6. GitHub, 2026. https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. GitHub, 2026. https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="IEEE">
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.6, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
</Tab>
</Tabs>
+5 -1
View File
@@ -35,6 +35,7 @@ Essential guides to master the Semantica framework.
- **[Vector Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)** — Setting up vector stores for similarity search and retrieval. *Intermediate*
- **[Graph Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Graph_Store.ipynb)** — Persisting knowledge graphs in Neo4j or FalkorDB. Topics: Neo4j, Cypher, Persistence · *Intermediate*
- **[Ontology](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)** — Defining domain schemas and ontologies to structure your data. Topics: OWL, RDF, Schema Design · *Intermediate*
- **[Seed Data](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)** — Bootstrapping a knowledge graph from trusted CSV, JSON, database, and API sources before extraction runs. Topics: SeedDataManager, Foundation Graphs · *Intermediate*
## Advanced Concepts
@@ -50,6 +51,9 @@ Deep dive into advanced features, customization, and complex workflows.
- **[Multi-Source Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)** — Merging data from disparate sources into a unified graph. Topics: Entity Resolution, Merging, Fusion · *Advanced*
- **[Reasoning and Inference](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)** — Using logical reasoning to infer new knowledge from existing facts. Topics: Logic Rules, Inference Engines · *Advanced*
- **[Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)** — Modeling and querying data that changes over time. Topics: Time Series, Temporal Logic, Allen Algebra · *Advanced*
- **[Provenance Tracking](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/22_Provenance_Tracking.ipynb)** — Audit-grade, W3C PROV-O-aligned tracking of where every entity, relationship, and chunk came from. Topics: PROV-O, Lineage, Checksums, Invalidation · *Advanced*
- **[Reasoning Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)** — Deriving new knowledge from existing facts with forward chaining, backward chaining, and Datalog strategies. Topics: Reasoner, Datalog, Explanations · *Advanced*
- **[Change Management](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)** — Versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies. Topics: ChangeLogEntry, Version Storage, Data Integrity · *Advanced*
## How to Run
@@ -80,6 +84,6 @@ Deep dive into advanced features, customization, and complex workflows.
You can also run the cookbook using Docker:
```bash
docker run -p 8888:8888 hawksight/semantica-cookbook
docker run -p 8888:8888 semantica/semantica-cookbook
```
</Tip>
+1
View File
@@ -103,6 +103,7 @@
"pages": [
"integrations/agno",
"integrations/crewai",
"integrations/langchain",
"integrations/docling",
"integrations/snowflake",
"integrations/databricks"
+2 -2
View File
@@ -4,12 +4,12 @@ description: "Project governance model: roles, decision process, release cadence
icon: "scale-balanced"
---
> Semantica is maintained by Hawksight AI with community contributions under an open governance model.
> Semantica is maintained by the Semantica team with community contributions under an open governance model.
## Roles
- **Maintainers** — Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards.
- **Maintainers** — Semantica team: review and merge PRs, manage releases and code quality, set project direction and community standards.
- **Contributors** — Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md).
- **Community Members** — Use Semantica, provide feedback, share use cases, and participate in GitHub Discussions and Discord.
+29
View File
@@ -436,6 +436,35 @@ d = graph.to_dict()
# d["statistics"] → {"node_count": int, "edge_count": int}
```
For a human-editable, version-control-friendly representation, save a Markdown
directory instead:
```python
graph.save_to_file("context_graph/", format="markdown")
restored = ContextGraph(advanced_analytics=True)
restored.load_from_file("context_graph/", format="markdown")
```
The directory contains a versioned `graph.md` manifest for graph identity,
relationships, and cross-graph link descriptors, plus one file per node under
`nodes/`. A node's content is its Markdown body; its ID, type, properties,
metadata, and temporal validity are YAML frontmatter. Node, edge, family, graph,
and cross-graph link IDs are preserved across round trips.
Markdown loading uses replacement semantics, like `from_dict()`: it parses and
validates the complete directory before replacing the current graph. Invalid YAML,
duplicate IDs, unsupported versions, and unsafe filesystem links fail without
partially mutating the graph. As with JSON loading, an edge endpoint without a node
file creates an `entity` stub node. Symlinks, Windows directory junctions, and other
Windows reparse points are rejected.
Re-exporting to an existing managed directory atomically replaces it, removing stale
node files. Before replacement, Semantica validates the complete canonical export
layout, not just the manifest header. Untracked files, assets, extra directories, or
renamed node files therefore cause the export to fail closed instead of being deleted.
Keep attachments and hand-written indexes outside the managed export directory.
If the graph had cross-graph links created with `link_graph()`, call `resolve_links()` after loading to restore live navigation — object references cannot be serialized, so they must be reconnected manually:
```python
+4 -4
View File
@@ -222,10 +222,10 @@ builder.register_step_handler("ner_extract", run_ner)
builder.register_step_handler("triplet_extract", run_triplets)
builder.register_step_handler("kg_merge", merge_into_graph)
builder.add_step("ingest", "file_ingest", handler=ingest_stix_bundles, path="./stix_bundles/")
builder.add_step("ner", "ner_extract", handler=run_ner, confidence_threshold=0.75)
builder.add_step("triplets", "triplet_extract", handler=run_triplets, include_temporal=True)
builder.add_step("store", "kg_merge", handler=merge_into_graph, output_path="./cti_output/")
builder.add_step("ingest", "file_ingest", path="./stix_bundles/")
builder.add_step("ner", "ner_extract", confidence_threshold=0.75)
builder.add_step("triplets", "triplet_extract", include_temporal=True)
builder.add_step("store", "kg_merge", output_path="./cti_output/")
# ingest feeds both ner and triplets in parallel
builder.connect_steps("ingest", "ner")
+19 -15
View File
@@ -150,6 +150,12 @@ HighRiskSupplier(DELTA-3) conf=100% rule=Rule 3
DELTA-3 is flagged even though no document described it that way — the system traced: DELTA-3 supplied GAMMA-7, and GAMMA-7 exploits critical CVEs. For rules that need priority ordering or graded confidence, use the `Rule` dataclass:
If a rule has side-effecting actions, one concrete activation runs those
actions at most once on a Reasoner instance. Re-running `forward_chain()` is
therefore safe: already-attempted actions are not repeated. Use
`reasoner.reset_action_history()` when you intentionally want to replay them;
`reasoner.clear()` and `reasoner.reset()` also clear the history.
```python
# Higher priority rules fire first; confidence propagates into InferenceResult.confidence
reasoner.add_rule(Rule(
@@ -269,7 +275,7 @@ print("Loaded {} facts from graph".format(count))
## Step 5 — SPARQL queries over enriched working memory
After forward chaining has derived new facts, `SPARQLReasoner` lets you query the enriched working memory using SPARQL triple-pattern matching with optional inference expansion:
After forward chaining has derived new facts, `SPARQLReasoner` prepares SPARQL queries over the enriched working memory with optional inference expansion:
```python
from semantica.reasoning import SPARQLReasoner
@@ -288,22 +294,13 @@ query = """
}
"""
# execute_query() runs: expansion → inference → deduplication
result = sparql.execute_query(query)
for binding in result.bindings:
print("Actor: {:15s} CVE: {}".format(
binding.get("actor", "?"),
binding.get("cve", "?"),
))
# metadata shows how many results came from inference vs ground facts
print("Original: {} Inferred: {}".format(
result.metadata.get("original_count", 0),
result.metadata.get("inferred_count", 0),
))
# expand_query() applies inference rules to the query text:
expanded = sparql.expand_query(query)
print(expanded)
```
`execute_query()` is not implemented yet: no triplet-store execution path exists, so it raises `NotImplementedError` rather than returning an empty result set that callers would misread as "no matches". Until execution lands, run the expanded query against your RDF store directly (for example with `rdflib`).
Inspect the expanded query before running it:
```python
@@ -369,6 +366,13 @@ engine.reset()
The rule network is compiled once by `build_network()`. Each subsequent `add_fact()` call propagates incrementally through only the nodes whose conditions it satisfies — not the full rule set — which keeps evaluation cost proportional to the number of new activations rather than the total rule count.
With a Reasoner bound, Rete action side effects are attempted once per rule,
bindings, and matched fact identity. Passing the same match to
`execute_matches()` again still returns the same conclusion, but does not repeat
its actions. Call `engine.reset_action_history()` to replay actions without
clearing working memory. `engine.reset()` and `engine.build_network()` also
clear the action history.
## Step 7 — Temporal interval reasoning
`TemporalReasoningEngine` computes Allen interval relations between time windows, letting you identify whether two events overlap, one contains the other, they meet at a boundary, and so on across your graph:
+59 -21
View File
@@ -8,7 +8,7 @@ icon: "shield-check"
SHACL (Shapes Constraint Language) is a standard for validating graph-based data. While an ontology defines the conceptual *schema* (the "what" exists in your domain), SHACL defines the structural *rules and constraints* (the "how" it should be structured).
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and `_run_pyshacl` evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated.
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and the public `run_shacl_validation` function evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated. The historical `_run_pyshacl` name remains available as a compatibility alias.
## Why Use SHACL Validation?
@@ -55,7 +55,7 @@ Let's look at a simple, universally understood example: ensuring every `Employee
```python
from semantica.context import ContextGraph
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
# 1. Prepare your data graph
graph = ContextGraph()
@@ -95,7 +95,7 @@ data_ttl = """
"""
# 5. Run Validation
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
# 6. Analyze the Report
print(f"Graph conforms: {report.conforms}")
@@ -265,10 +265,10 @@ cve_id_shape = NodeShape(
## Step 4 — Run validation and read the report
Serialize the graph to RDF, then run `_run_pyshacl` against the shapes.
Serialize the graph to RDF, then run `run_shacl_validation` against the shapes.
```python
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
# Prepare your RDF data string (since export_rdf primarily exports structural metadata,
# you typically serialize your custom data graph to Turtle using rdflib or similar).
@@ -281,7 +281,7 @@ data_ttl = """
"""
# Run SHACL validation
report = _run_pyshacl(
report = run_shacl_validation(
data_ttl,
shacl_ttl,
data_graph_format="turtle",
@@ -366,8 +366,8 @@ print(f"Malware nodes missing 'family': {len(missing_family)}")
# e.g. graph.update_node(node_id, {"family": "UNKNOWN — requires triage"})
# After remediation, re-run validation to confirm the fix
# (re-export the patched graph to Turtle first, then call _run_pyshacl again)
report2 = _run_pyshacl(patched_data_ttl, shacl_ttl)
# (re-export the patched graph to Turtle first, then call run_shacl_validation again)
report2 = run_shacl_validation(patched_data_ttl, shacl_ttl)
print(f"Violations after remediation: {report2.violation_count}")
# Violations after remediation: 0
```
@@ -377,10 +377,49 @@ print(f"Violations after remediation: {report2.violation_count}")
## Common Pitfalls
- **Assuming the ontology automatically enforces data quality**: `SHACLGenerator` generates shapes based on what it observes in the data. If your data is missing a field, the generator won't know it was mandatory unless you explicitly inject the constraint (as shown in Step 3).
- **Passing `ContextGraph` directly to SHACL validators**: The `_run_pyshacl` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
- **Passing `ContextGraph` directly to SHACL validators**: The `run_shacl_validation` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
- **Forgetting RDF serialization**: You must serialize your graph (often via a temporary file using `export_rdf`) before validating it.
- **Treating validation as a one-time step**: Validation should be integrated as an automated step in your CI/CD pipeline or data ingestion flow, acting as a recurring gatekeeper rather than a one-off script.
- **Ignoring validation reports**: A graph that does not conform must be remediated. Failing to review the `violation_count` and address the issues negates the purpose of SHACL validation.
- **Validating `sh:class`/`sh:node` range checks on a property that declares `rdfs:range` with RDFS entailment on**: RDFS is an entailment rule, not a constraint. When pyshacl runs with `inference="rdfs"`, it infers the range class onto every object of the property, so class-based constraints on that property can never fail — the report says `conforms: True` on data that does not conform:
```python
from pyshacl import validate
from rdflib import Graph
data = Graph()
data.parse(
data="""
@prefix ex: <https://example.org/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
ex:contains rdfs:domain ex:Container ; rdfs:range ex:Item .
ex:box a ex:Container ; ex:contains ex:notAnItem .
ex:notAnItem a ex:Fish .
""",
format="turtle",
)
shapes = Graph()
shapes.parse(
data="""
@prefix ex: <https://example.org/ns#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
ex:ContainerShape a sh:NodeShape ;
sh:targetClass ex:Container ;
sh:property [ sh:path ex:contains ; sh:class ex:Item ] .
""",
format="turtle",
)
for inference in ("none", "rdfs"):
conforms, _, _ = validate(data, shacl_graph=shapes, inference=inference)
print(inference, conforms)
# none False <- correct: notAnItem is a Fish, not an Item
# rdfs True <- the entailment manufactured the type
```
Mitigations: prefer not to declare `rdfs:range` on properties you intend to constrain with `sh:class`; when class membership is the thing under test, run validation without RDFS entailment (`inference="none"`); or express the check as a constraint the entailment cannot satisfy (for example a literal property constraint). Note the trade-off: with entailment off, `sh:targetClass` no longer reaches subclasses, so subclass hierarchies need explicit typing or inference-aware target selection. Semantica's own `run_shacl_validation` wrapper already calls pyshacl with `inference="none"`, so this pitfall only bites when calling `pyshacl.validate` directly with entailment enabled.
- **Trusting `conforms: True` without checking the inference mode**: an inference-enabled run can hide the exact violations the shapes were written to catch (see above). Record which inference mode validation ran under alongside the result, and re-run shape sets that contain `sh:class`/`sh:node` with entailment off before treating a pass as authoritative.
---
@@ -396,7 +435,7 @@ A DoD CTI team enforces STIX-compatible constraints on a threat graph before sha
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
graph = ContextGraph()
ctx = AgentContext(
@@ -448,7 +487,7 @@ data_ttl = """
<http://example.org/hammertoss> a ex:Malware .
"""
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"CTI graph conforms : {report.conforms}")
print(f"Violations : {report.violation_count}")
print(f"Warnings : {report.warning_count}")
@@ -469,7 +508,7 @@ A SOC team validates zero-trust policy nodes before publishing them to the polic
```python
from semantica.context import ContextGraph
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
graph = ContextGraph()
graph.add_node("policy-001", "Policy", "MFA Required for Tier-1 Resources",
@@ -516,7 +555,7 @@ data_ttl = """
<http://example.org/policy-002> a ex:Policy .
"""
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"Policy graph conforms: {report.conforms}")
# Policy graph conforms: False
@@ -534,7 +573,7 @@ A clinical informatics team validates trial ontology nodes before loading them i
```python
from semantica.ontology import LLMOntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
from semantica.export import export_rdf
import tempfile, os
@@ -586,7 +625,7 @@ with open(tmp.name) as f:
data_ttl = f.read()
os.unlink(tmp.name)
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"Trial data conforms: {report.conforms}")
print(f"Warnings : {report.warning_count}")
```
@@ -600,7 +639,7 @@ A credit risk team validates every `LoanApplication` node against Basel III CRE2
```python
from semantica.context import ContextGraph
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
graph = ContextGraph()
graph.add_node("loan-001", "LoanApplication", "Prime mortgage APP-2025-88421",
@@ -645,7 +684,7 @@ data_ttl = """
ex:ltv "0.65" .
"""
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"Loan portfolio conforms: {report.conforms}")
# Loan portfolio conforms: False
@@ -675,14 +714,14 @@ Call this function as a pre-publish gate; exit code 1 blocks the pipeline.
```python
import sys
from semantica.ontology import OntologyGenerator, SHACLGenerator
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
shacl_gen = SHACLGenerator(base_uri="https://example.org/shapes/")
shacl_graph = shacl_gen.generate(ontology)
shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
report = _run_pyshacl(data_graph_str, shacl_ttl)
report = run_shacl_validation(data_graph_str, shacl_ttl)
if not report.conforms:
print(f"Graph validation FAILED — {report.violation_count} violation(s)")
@@ -700,7 +739,6 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
- [Ontology Management](ontology) — generate the OWL ontology that SHACL shapes are derived from
- [Reasoning & Rules](reasoning) — complement SHACL structural constraints with logical inference rules
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `_run_pyshacl` input
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `run_shacl_validation` input
- [Conflict Resolution](conflict-resolution) — detect and resolve data conflicts before SHACL validation
- [Change Management](change-management) — version-gate SHACL shapes alongside ontology versions
+81
View File
@@ -0,0 +1,81 @@
---
title: "LangChain Integration"
description: "Drop Semantica into LangChain / LangGraph pipelines via a GraphRAG retriever, VectorStore adapter, and agent tools."
icon: "link"
---
> Three drop-in adapters that bring Semantica's context graph and hybrid search into LangChain chains and LangGraph agents.
## Installation
```bash
pip install "semantica[langchain]"
```
Requires `langchain-core >= 0.3`. If langchain-core is not installed, the integration still imports — every class carries the full Semantica API and degrades gracefully (`build()` returns `None`; branch on `LANGCHAIN_AVAILABLE`).
## Components at a Glance
- **SemanticaRetriever** — `BaseRetriever`: hybrid-search seeds retrieval, then graph edges are walked `hops` steps (default 2) for GraphRAG-style results.
- **SemanticaVectorStore** — `VectorStore`: `add_texts` / `similarity_search` / `similarity_search_with_score` / `from_texts` over `HybridSearch`.
- **SemanticaKGTool** / **SemanticaDecisionTool**`BaseTool` subclasses: `semantica_query_graph` and `semantica_query_decisions` for LangGraph / tool-calling agents.
## Component Details
<Tabs>
<Tab title="SemanticaRetriever">
Hybrid search seeds retrieval; then graph edges are walked `hops` steps so results go beyond flat vector similarity. If hybrid search is omitted or fails, the retriever falls back to a `ContextGraph.query` keyword scan.
```python
from integrations.langchain import SemanticaRetriever
from semantica.context import ContextGraph
from semantica.vector_store import HybridSearch
graph = ContextGraph()
hybrid = HybridSearch()
retriever = SemanticaRetriever(graph=graph, hybrid=hybrid, hops=2, top_k=10)
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
```
</Tab>
<Tab title="SemanticaVectorStore">
Drop-in `VectorStore` for RetrievalQA / LCEL chains. `from_texts` requires a pre-configured `hybrid` instance.
```python
from integrations.langchain import SemanticaVectorStore
store = SemanticaVectorStore(hybrid=hybrid)
store.add_texts(
["document one", "document two"],
metadatas=[{"source": "a"}, {"source": "b"}],
)
docs = store.similarity_search("document", k=2)
docs, scores = store.similarity_search_with_score("document", k=2)
```
`add_texts` delegates to a Semantica vector store with `add_documents` (pass `vector_store=` to `HybridSearch` or to `SemanticaVectorStore`).
</Tab>
<Tab title="Agent tools">
Instances are LangChain `BaseTool`s and can be passed to an agent directly.
`.build()` returns the tool, or `None` when langchain-core is absent.
```python
from integrations.langchain import SemanticaKGTool, SemanticaDecisionTool
from langgraph.prebuilt import create_react_agent
tools = [
SemanticaKGTool(graph),
SemanticaDecisionTool(graph),
]
agent = create_react_agent(model, tools)
```
| Tool | Description |
| :------ | :------------- |
| `semantica_query_graph` | Keyword / NL query over the shared context graph |
| `semantica_query_decisions` | Search the recorded decision log |
</Tab>
</Tabs>
+1 -1
View File
@@ -12,7 +12,7 @@ icon: "file-contract"
```
MIT License
Copyright (c) 2026 Hawksight AI
Copyright (c) 2026 Semantica
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+6 -4
View File
@@ -435,8 +435,8 @@ print("Nodes: {}, Edges: {}".format(stats["node_count"], stats["edge_count"]))
| `query(query, skip, limit)` | `List[Dict]` | Full-text search over node content |
| `stats()` | `Dict` | Node/edge counts, type breakdowns, graph density |
| `density()` | `float` | Graph density score |
| `save_to_file(path)` | `None` | Persist graph to JSON |
| `load_from_file(path)` | `None` | Load graph from JSON |
| `save_to_file(path, format="json")` | `None` | Persist graph as JSON or a Markdown directory |
| `load_from_file(path, format="json")` | `None` | Replace graph state from JSON or a Markdown directory |
| `build_from_conversations(conversations, link_entities)` | `Dict` | Build graph from conversation data |
| `link_graph(other_graph, source_node_id, target_node_id, link_type)` | `str` | Create cross-graph navigation link; returns `link_id` |
| `navigate_to(link_id)` | `Tuple` | Follow a cross-graph link to `(target_graph, target_node_id)` |
@@ -625,8 +625,10 @@ malformed or duplicate fields before changing memory, and re-importing unchanged
files is idempotent. Memory-local `entities` and `relationships` are preserved as
provenance but are not applied to `ContextGraph` by Markdown import. Use a dedicated
export directory: matching files are overwritten, but unrelated or stale Markdown
files are not deleted automatically. Export refuses to overwrite symbolic links and
uses atomic file replacement. Timestamp offsets are preserved in Markdown and
files are not deleted automatically. Export refuses to overwrite filesystem links and
uses atomic file replacement; import also refuses symlinks, Windows directory
junctions, and other Windows reparse points.
Timestamp offsets are preserved in Markdown and
normalized to UTC only for comparisons, so aware and local-naive records can be
queried together safely. Vector-store writes are deferred until the in-memory import
commits; adapter synchronization remains best-effort and logs failures.
+19 -2
View File
@@ -127,9 +127,19 @@ conclusions = reasoner.infer_facts(
| `forward_chain()` | `List[InferenceResult]` | Derive all possible conclusions iteratively until fixpoint |
| `backward_chain(goal, max_depth)` | `InferenceResult \| None` | Prove a specific goal string, returns `None` if unprovable |
| `infer_facts(facts, rules)` | `List[str]` | Load facts and rules then run `forward_chain()`, returns conclusion strings |
| `clear()` | `None` | Clear all facts and rules |
| `reset_action_history()` | `None` | Allow actions for previously fired activations to run again |
| `clear()` | `None` | Clear all facts, rules, and action activation history |
| `reset()` | `None` | Alias for `clear()` |
Rules with actions use at-most-once attempt semantics per concrete activation
(rule ID, bindings, and matched facts). Calling `forward_chain()` again on the
same instance does not repeat side effects for an activation that was already
attempted, even when an action raised an exception. Call
`reset_action_history()` to deliberately retry without clearing facts or rules;
`clear()` and `reset()` also clear this history. Replacing a rule's actions in
place does not invalidate an existing activation; reset the history explicitly
when the replacement should be replayed.
### Rule and Fact dataclass fields
```python
@@ -230,9 +240,16 @@ engine.reset()
| `add_fact(fact)` | `None` | Add a `Fact` to working memory and propagate through the network |
| `match_patterns(facts)` | `List[Match]` | Match all patterns; optionally add facts before matching |
| `execute_matches(matches)` | `List[Any]` | Execute matched rules and return their conclusion values |
| `reset()` | `None` | Clear facts and all node activation state |
| `reset_action_history()` | `None` | Allow actions for previously executed activations to run again |
| `reset()` | `None` | Clear facts, node activation state, and action activation history |
| `get_network_stats()` | `dict` | Return counts of alpha, beta, terminal nodes and facts |
When a Reasoner is bound, `execute_matches()` deduplicates action side effects
by rule ID, bindings, and matched fact identity. Re-executing a match still
returns its conclusion for compatibility, but its actions are skipped after the
first attempt. `reset_action_history()`, `reset()`, and `build_network()` allow
those actions to run again.
## SPARQLReasoner
+1 -1
View File
@@ -182,7 +182,7 @@ for row in result.bindings:
store = TripletStore(
backend="rdf4j",
endpoint="http://localhost:8080/rdf4j-server",
repository_id="semantica", # passed through **config
repository_id="semantica", # selects the remote repository
)
```
+10
View File
@@ -77,7 +77,17 @@ Most users won't call utils directly: it's the **shared foundation** for all mod
export SEMANTICA_LOG_LEVEL=DEBUG
export SEMANTICA_LOG_FORMAT=json # "json" | "text"
export SEMANTICA_DISABLE_PROGRESS=true
export SEMANTICA_FORCE_PROGRESS=true
```
<Tip>
**Progress bars follow your terminal.** Console progress is written only when
stdout is an interactive terminal (or a Jupyter notebook), so piping or
redirecting output no longer fills logs with progress bars and escape
sequences. Set `SEMANTICA_DISABLE_PROGRESS` to silence progress even in a
terminal, or `SEMANTICA_FORCE_PROGRESS` to keep it when stdout is redirected.
`SEMANTICA_DISABLE_PROGRESS` wins if both are set.
</Tip>
</Step>
</Steps>
+154
View File
@@ -0,0 +1,154 @@
# Graph storage backends and feature matrix
Semantica separates graph modeling from physical storage. LPG backends are accessed through `graph_store` adapters; RDF backends are accessed through `triplet_store` adapters.
This page is intentionally conservative: it distinguishes between an adapter existing, a feature being generally available with that model, and a backend needing user-supplied wiring.
## Status labels
- `built-in`: adapter implementation exists in Semantica core.
- `tested`: covered by automated integration fixtures or tests.
- `example-only`: usable example exists, but support is not asserted by integration tests.
- `interface/BYO`: interface or integration point exists; bring your own backend wiring.
## Adapter inventory
| Backend | Model | Adapter | Status | Reference |
| --- | --- | --- | --- | --- |
| Neo4j | LPG | `semantica.graph_store.Neo4jStore` | built-in | `cookbook/introduction/09_Graph_Store.ipynb` |
| FalkorDB | LPG | `semantica.graph_store.FalkorDBStore` | built-in | `docs/reference/graph_store.md` |
| Amazon Neptune | LPG | `semantica.graph_store.AmazonNeptuneStore` | built-in | `cookbook/introduction/21_Amazon_Neptune_Store.ipynb` |
| Apache AGE | LPG | `semantica.graph_store.ApacheAgeStore` | built-in | `docs/graph_stores/apache_age.md` |
| RDF4J | RDF | `semantica.triplet_store.RDF4JStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Apache Jena | RDF | `semantica.triplet_store.JenaStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Blazegraph | RDF | `semantica.triplet_store.BlazegraphStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Anzo | RDF | `semantica.triplet_store.AnzoStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Oxigraph | RDF | `semantica.triplet_store.OxigraphStore` | built-in | `docs/reference/triplet_store.md` |
## Feature matrix
`Yes` means the capability is expected to work with the adapter and graph model. `Partial` means the capability works with model-specific constraints. `BYO` means the user must supply or validate wiring for the backend.
| Backend | Model | Ingestion | Context graph construction | Reasoning/analytics | Provenance | Known limitations |
| --- | --- | --- | --- | --- | --- | --- |
| Neo4j | LPG | Yes | Yes | Yes | Partial | Provenance and context metadata are stored as node and edge properties; relationship properties and stable node identifiers are required. |
| FalkorDB | LPG | Yes | Yes | Partial | Partial | Redis-based; provenance depends on node/edge properties, and multi-graph isolation depends on the selected graph name. |
| Amazon Neptune | LPG | Yes | Yes | Partial | Partial | Use the property-graph endpoint; AWS auth, VPC, and endpoint configuration can affect local tests. Provenance depends on node/edge properties. |
| Apache AGE | LPG | Yes | Yes | Partial | Partial | Runs through PostgreSQL/AGE; Cypher compatibility and property handling can differ from standalone LPG engines. |
| RDF4J | RDF | Yes | Partial | Partial | Partial | Context separation relies on named graphs; triple-level provenance may require reification or graph-level metadata. |
| Apache Jena | RDF | Yes | Partial | Partial | Partial | Named graphs are needed for context separation; backend configuration and transaction behavior matter. |
| Blazegraph | RDF | Yes | Partial | Partial | Partial | Use quads/named graphs for context; IRI stability and graph naming matter for provenance. |
| Anzo | RDF | Yes | Partial | Partial | Partial | Anzo deployments are environment-specific; validate `dataset_uri`/graphmart naming, named-graph support, and provenance mapping. |
| Oxigraph | RDF | Yes | Partial | Partial | Partial | Embedded, single-process store (in-memory or on-disk); named graphs are supported, but there is no separate server process to scale independently. |
## RDF and LPG differences
- LPG backends store context and provenance as graph elements and properties. If a backend does not support relationship properties, some provenance patterns may be degraded.
- RDF backends rely on IRIs, named graphs, and optional reification. Context graphs and provenance are easiest to preserve when the store supports named graphs/quads.
- Ingestion works across both models, but the physical representation differs: LPG stores nodes/edges directly, while RDF stores subject-predicate-object statements.
- Reasoning and analytics should be validated against the adapter's query capabilities, especially for path traversal, property filters, and named-graph queries.
## Minimal connection examples
Prefer the referenced notebook cells for a working setup. The examples below show the intended adapter entrypoints, not a universal connection DSL.
### Neo4j
```python
import os
from semantica.graph_store import Neo4jStore
store = Neo4jStore(
uri='bolt://localhost:7687',
user='neo4j',
password=os.environ['NEO4J_PASSWORD']
)
```
### FalkorDB
```python
from semantica.graph_store import FalkorDBStore
store = FalkorDBStore(
host='localhost',
port=6379,
graph_name='semantica'
)
```
### Amazon Neptune
```python
from semantica.graph_store import AmazonNeptuneStore
store = AmazonNeptuneStore(
endpoint='your-neptune-cluster-endpoint',
port=8182,
region='us-east-1'
)
```
### Apache AGE
```python
from semantica.graph_store import ApacheAgeStore
store = ApacheAgeStore(
connection_string='host=localhost dbname=agedb user=postgres password=postgres',
graph_name='semantica'
)
```
### RDF4J
```python
from semantica.triplet_store import RDF4JStore
store = RDF4JStore(
endpoint='http://localhost:8080/rdf4j-server',
repository_id='semantica'
)
```
### Apache Jena
```python
from semantica.triplet_store import JenaStore
store = JenaStore(
endpoint='http://localhost:3030/ds'
)
```
### Blazegraph
```python
from semantica.triplet_store import BlazegraphStore
store = BlazegraphStore(
endpoint='http://localhost:9999/blazegraph/sparql'
)
```
### Anzo
```python
from semantica.triplet_store import AnzoStore
store = AnzoStore(
endpoint='http://anzo-host:8080',
dataset_uri='http://cambridgesemantics.com/Graphmart/your-graphmart-id'
)
```
### Oxigraph
```python
from semantica.triplet_store import OxigraphStore
# Omit `path` for an in-memory store; pass a directory for on-disk persistence.
store = OxigraphStore(path='./semantica-oxigraph-data')
```
Replace hostnames, ports, repositories, graphs, and credentials with values from your environment. For regulated or self-hosted deployments, keep credentials in environment variables or secret storage rather than source code.
@@ -1,8 +1,9 @@
import { useState, useRef, useEffect, type CSSProperties } from "react";
import ReactMarkdown from "react-markdown";
import { useState, useRef, useEffect, useMemo, type CSSProperties } from "react";
import ReactMarkdown, { type Components } from "react-markdown";
import remarkGfm from "remark-gfm";
import { Check, Copy, Code2, Eye, ExternalLink, Image as ImageIcon } from "lucide-react";
import { GRAPH_THEME } from "./graphTheme";
import { isSafeUrl } from "./markdownUrlSafety";
export interface MarkdownContentViewerProps {
content?: string | null;
@@ -10,25 +11,6 @@ export interface MarkdownContentViewerProps {
defaultMode?: "preview" | "source";
}
export function isSafeUrl(url?: string): boolean {
if (!url) return false;
const trimmed = url.trim();
// Reject whitespace-only strings — new URL("", base) would resolve to the base
// protocol and produce a false positive. This guards direct callers of the exported
// function; markdown parsers normalise whitespace-only destinations to "" which
// already fails the !url check above.
if (!trimmed) return false;
if (trimmed.startsWith("//")) return false;
if (trimmed.startsWith("#")) return true;
if (trimmed.startsWith("/")) return true;
try {
const parsed = new URL(trimmed, "http://localhost");
return ["http:", "https:", "mailto:"].includes(parsed.protocol);
} catch {
return false;
}
}
export function MarkdownContentViewer({
content,
className,
@@ -65,6 +47,20 @@ export function MarkdownContentViewer({
const rawContent = typeof content === "string" ? content : "";
const hasContent = rawContent.trim().length > 0;
// react-markdown runs the whole remark pipeline synchronously inside its own
// render, so without this memo every unrelated re-render of this component --
// clicking Copy, toggling Preview/Source -- re-parses the entire document.
// Measured at ~364ms per re-render for a 1000-row GFM table (issue #1118).
// Keyed on rawContent so a genuine node change still re-parses exactly once.
const renderedMarkdown = useMemo(
() => (
<ReactMarkdown remarkPlugins={REMARK_PLUGINS} components={MARKDOWN_COMPONENTS}>
{rawContent}
</ReactMarkdown>
),
[rawContent],
);
const handleCopy = async () => {
if (!hasContent) return;
try {
@@ -130,98 +126,103 @@ export function MarkdownContentViewer({
<code style={sourceCodeStyle}>{rawContent}</code>
</pre>
) : (
<div style={previewStyle}>
<ReactMarkdown
remarkPlugins={[remarkGfm]}
components={{
// C-1: react-markdown passes a HAST `node` prop (the raw AST
// Element) to every custom component override via passNode:true.
// In React 19 any unknown prop spreads onto a native element are
// serialised as HTML attributes, producing node="[object Object]"
// on every rendered link. Fix: destructure `node` by name so it
// is explicitly discarded, then spread `...rest` to preserve all
// other legitimate HAST/remark-gfm attributes — e.g. the `id`,
// `aria-describedby`, `aria-label`, `data-footnote-ref`,
// `data-footnote-backref`, and `class` attrs that GFM footnotes
// require for correct in-page navigation and accessibility.
//
// C-2: fragment links (#anchor, GFM footnote backlinks) must
// navigate within the current document. External links continue
// to use target="_blank" with noopener noreferrer.
//
// eslint-disable-next-line @typescript-eslint/no-unused-vars
a: ({ href, children, title, node: _node, ...rest }) => {
if (!isSafeUrl(href)) {
return <span style={{ color: GRAPH_THEME.ui.text.muted, textDecoration: "line-through" }}>{children}</span>;
}
// isSafeUrl returning true guarantees href is a non-empty string.
const safeHref = href ?? "";
// Fragment links (#section, footnote backlinks like
// #user-content-fnref-1) are in-document anchors. Opening them
// in a new tab would break GFM footnote back-navigation.
const isFragment = safeHref.startsWith("#");
if (isFragment) {
return (
<a href={safeHref} title={title} style={linkStyle} {...rest}>
{children}
</a>
);
}
return (
<a href={safeHref} title={title} target="_blank" rel="noopener noreferrer" style={linkStyle} {...rest}>
{children}
<ExternalLink size={10} style={{ marginLeft: 3, verticalAlign: "middle", display: "inline" }} />
</a>
);
},
img: ({ src, alt }) => (
<span style={imageBadgeStyle} title={src || "Image"}>
<ImageIcon size={12} style={{ marginRight: 5 }} />
<span>Image: {alt || src || "unlabeled"}</span>
</span>
),
h1: ({ children }) => <h1 style={h1Style}>{children}</h1>,
h2: ({ children }) => <h2 style={h2Style}>{children}</h2>,
h3: ({ children }) => <h3 style={h3Style}>{children}</h3>,
h4: ({ children }) => <h4 style={h4Style}>{children}</h4>,
p: ({ children }) => <p style={{ margin: "0 0 8px 0" }}>{children}</p>,
ul: ({ children }) => <ul style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ul>,
ol: ({ children }) => <ol style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ol>,
li: ({ children }) => <li style={{ marginBottom: 3 }}>{children}</li>,
blockquote: ({ children }) => <blockquote style={blockquoteStyle}>{children}</blockquote>,
hr: () => <hr style={{ border: "none", borderTop: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`, margin: "10px 0" }} />,
table: ({ children }) => (
<div style={{ width: "100%", overflowX: "auto", margin: "8px 0", borderRadius: 6, border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>
<table style={{ width: "100%", borderCollapse: "collapse", fontSize: 12 }}>{children}</table>
</div>
),
thead: ({ children }) => <thead style={{ background: "rgba(255, 255, 255, 0.04)" }}>{children}</thead>,
tbody: ({ children }) => <tbody>{children}</tbody>,
tr: ({ children }) => <tr style={{ borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</tr>,
th: ({ children }) => <th style={{ padding: "6px 8px", textAlign: "left", fontWeight: 700, color: GRAPH_THEME.ui.text.strong, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</th>,
td: ({ children }) => <td style={{ padding: "6px 8px", color: GRAPH_THEME.ui.text.body, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</td>,
pre: ({ children }) => <pre style={preBlockStyle}>{children}</pre>,
// C-1: discard `node` here too — code elements are custom components
// and would otherwise receive node="[object Object]" in the DOM.
code: ({ className: codeClass, children }) => {
const isInline = !codeClass && typeof children === "string" && !children.includes("\n");
return (
<code style={isInline ? inlineCodeStyle : blockCodeStyle}>
{children}
</code>
);
},
}}
>
{rawContent}
</ReactMarkdown>
</div>
<div style={previewStyle}>{renderedMarkdown}</div>
)}
</div>
</div>
);
}
/* ─── Markdown rendering config ───────────────────────────────────── */
// Both props are hoisted to module scope so they keep a stable identity across
// renders. As inline literals they allocated a fresh plugin array and ~20 fresh
// arrow components on every render, which made React treat every mapped tag as a
// new element type and remount the entire rendered subtree instead of updating
// it (issue #1118). The arrow bodies only read the style constants below at call
// time, so declaring the map before them is safe.
const REMARK_PLUGINS = [remarkGfm];
const MARKDOWN_COMPONENTS: Components = {
// C-1: react-markdown passes a HAST `node` prop (the raw AST
// Element) to every custom component override via passNode:true.
// In React 19 any unknown prop spreads onto a native element are
// serialised as HTML attributes, producing node="[object Object]"
// on every rendered link. Fix: destructure `node` by name so it
// is explicitly discarded, then spread `...rest` to preserve all
// other legitimate HAST/remark-gfm attributes — e.g. the `id`,
// `aria-describedby`, `aria-label`, `data-footnote-ref`,
// `data-footnote-backref`, and `class` attrs that GFM footnotes
// require for correct in-page navigation and accessibility.
//
// C-2: fragment links (#anchor, GFM footnote backlinks) must
// navigate within the current document. External links continue
// to use target="_blank" with noopener noreferrer.
//
// eslint-disable-next-line @typescript-eslint/no-unused-vars
a: ({ href, children, title, node: _node, ...rest }) => {
if (!isSafeUrl(href)) {
return <span style={{ color: GRAPH_THEME.ui.text.muted, textDecoration: "line-through" }}>{children}</span>;
}
// isSafeUrl returning true guarantees href is a non-empty string.
const safeHref = href ?? "";
// Fragment links (#section, footnote backlinks like
// #user-content-fnref-1) are in-document anchors. Opening them
// in a new tab would break GFM footnote back-navigation.
const isFragment = safeHref.startsWith("#");
if (isFragment) {
return (
<a href={safeHref} title={title} style={linkStyle} {...rest}>
{children}
</a>
);
}
return (
<a href={safeHref} title={title} target="_blank" rel="noopener noreferrer" style={linkStyle} {...rest}>
{children}
<ExternalLink size={10} style={{ marginLeft: 3, verticalAlign: "middle", display: "inline" }} />
</a>
);
},
img: ({ src, alt }) => (
<span style={imageBadgeStyle} title={src || "Image"}>
<ImageIcon size={12} style={{ marginRight: 5 }} />
<span>Image: {alt || src || "unlabeled"}</span>
</span>
),
h1: ({ children }) => <h1 style={h1Style}>{children}</h1>,
h2: ({ children }) => <h2 style={h2Style}>{children}</h2>,
h3: ({ children }) => <h3 style={h3Style}>{children}</h3>,
h4: ({ children }) => <h4 style={h4Style}>{children}</h4>,
p: ({ children }) => <p style={{ margin: "0 0 8px 0" }}>{children}</p>,
ul: ({ children }) => <ul style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ul>,
ol: ({ children }) => <ol style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ol>,
li: ({ children }) => <li style={{ marginBottom: 3 }}>{children}</li>,
blockquote: ({ children }) => <blockquote style={blockquoteStyle}>{children}</blockquote>,
hr: () => <hr style={{ border: "none", borderTop: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`, margin: "10px 0" }} />,
table: ({ children }) => (
<div style={{ width: "100%", overflowX: "auto", margin: "8px 0", borderRadius: 6, border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>
<table style={{ width: "100%", borderCollapse: "collapse", fontSize: 12 }}>{children}</table>
</div>
),
thead: ({ children }) => <thead style={{ background: "rgba(255, 255, 255, 0.04)" }}>{children}</thead>,
tbody: ({ children }) => <tbody>{children}</tbody>,
tr: ({ children }) => <tr style={{ borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</tr>,
th: ({ children }) => <th style={{ padding: "6px 8px", textAlign: "left", fontWeight: 700, color: GRAPH_THEME.ui.text.strong, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</th>,
td: ({ children }) => <td style={{ padding: "6px 8px", color: GRAPH_THEME.ui.text.body, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</td>,
pre: ({ children }) => <pre style={preBlockStyle}>{children}</pre>,
// C-1: discard `node` here too — code elements are custom components
// and would otherwise receive node="[object Object]" in the DOM.
code: ({ className: codeClass, children }) => {
const isInline = !codeClass && typeof children === "string" && !children.includes("\n");
return (
<code style={isInline ? inlineCodeStyle : blockCodeStyle}>
{children}
</code>
);
},
};
/* ─── Styles ──────────────────────────────────────────────────────── */
const viewerContainerStyle: CSSProperties = {
@@ -0,0 +1,29 @@
/**
* URL-safety predicate for the Markdown content viewer.
*
* Extracted into a pure module so the check can be unit-tested without
* importing the MarkdownContentViewer React component, and so the component
* module exports only components (react-refresh/only-export-components,
* issue #1119). The behaviour is unchanged from the original in-component
* implementation: only http, https, mailto, in-document fragments, and
* root-relative paths are permitted.
*/
export function isSafeUrl(url?: string): boolean {
if (!url) return false;
const trimmed = url.trim();
// Reject whitespace-only strings — new URL("", base) would resolve to the base
// protocol and produce a false positive. This guards direct callers of the exported
// function; markdown parsers normalise whitespace-only destinations to "" which
// already fails the !url check above.
if (!trimmed) return false;
if (trimmed.startsWith("//")) return false;
if (trimmed.startsWith("#")) return true;
if (trimmed.startsWith("/")) return true;
try {
const parsed = new URL(trimmed, "http://localhost");
return ["http:", "https:", "mailto:"].includes(parsed.protocol);
} catch {
return false;
}
}
+2 -1
View File
@@ -5,7 +5,8 @@ import { renderToString } from "react-dom/server";
(globalThis as any).React = React;
import { isSafeUrl, MarkdownContentViewer } from "../src/workspaces/GraphWorkspace/MarkdownContentViewer.tsx";
import { MarkdownContentViewer } from "../src/workspaces/GraphWorkspace/MarkdownContentViewer.tsx";
import { isSafeUrl } from "../src/workspaces/GraphWorkspace/markdownUrlSafety.ts";
test("isSafeUrl permits safe http, https, and mailto URLs and relative paths", () => {
assert.equal(isSafeUrl("https://example.com"), true);
+1 -1
View File
@@ -1,7 +1,7 @@
"""
Semantica Framework Integrations
Optional integration packages for agentic frameworks (Google ADK, Claude Agent SDK, Agno, etc.).
Optional integration packages for agentic frameworks (Google ADK, Claude Agent SDK, Agno, CrewAI, LangChain, etc.).
Each integration is self-contained, independently installable via extras_require, and maintains
zero impact on core Semantica - keeping the semantic layer lean while maximizing ecosystem reach.
"""
+10 -13
View File
@@ -277,25 +277,22 @@ class AgnoKnowledgeGraph(_KnowledgeBase): # type: ignore[misc]
def load_urls(self, urls: List[str]) -> None:
"""Fetch each URL and ingest the response body.
Only ``http`` and ``https`` schemes are permitted to prevent SSRF.
Uses the shared SSRF guard so that ``http`` and ``https`` are the only
permitted schemes, private/loopback/link-local/cloud-metadata addresses
are blocked by default, DNS resolution is validated, and every redirect
hop is re-checked before being followed.
"""
import urllib.request
from urllib.parse import urlparse
from semantica.ingest.ssrf import request_with_ssrf_guard
from semantica.utils.exceptions import ValidationError
for url in urls:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
logger.warning(
"Skipping URL with disallowed scheme '%s': %s",
parsed.scheme,
url,
)
continue
try:
with urllib.request.urlopen(url, timeout=10) as resp: # noqa: S310
text = resp.read().decode("utf-8", errors="replace")
response = request_with_ssrf_guard("GET", url, timeout=10)
text = response.text
self._ingest_text(text, source=url)
logger.info("Loaded URL: %s", url)
except ValidationError as exc:
logger.warning("Skipping URL (SSRF check failed) %s: %s", url, exc)
except Exception as exc:
logger.warning("Failed to fetch %s: %s", url, exc)
+67
View File
@@ -0,0 +1,67 @@
# Semantica × LangChain
Drop Semantica into existing LangChain / LangGraph pipelines: GraphRAG-style
retrieval, a `VectorStore` adapter, and agent tools.
## Install
```bash
pip install semantica[langchain]
# or just the core adapter dependency:
pip install langchain-core
```
## Retriever (GraphRAG)
```python
from integrations.langchain import SemanticaRetriever
from semantica.context import ContextGraph
from semantica.vector_store import HybridSearch
graph = ContextGraph()
hybrid = HybridSearch()
retriever = SemanticaRetriever(graph=graph, hybrid=hybrid, hops=2, top_k=10)
# Use with any LangChain chain that accepts a retriever:
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
```
Hybrid search seeds retrieval; then graph edges are walked `hops` steps so
results go beyond flat vector similarity.
## VectorStore
```python
from integrations.langchain import SemanticaVectorStore
store = SemanticaVectorStore(hybrid=hybrid)
store.add_texts(["document one", "document two"], metadatas=[{"source": "a"}, {"source": "b"}])
docs = store.similarity_search("document", k=2)
docs, scores = store.similarity_search_with_score("document", k=2)
```
## Agent tools (LangGraph / tool-calling agents)
```python
from integrations.langchain import SemanticaKGTool, SemanticaDecisionTool
from langgraph.prebuilt import create_react_agent
tools = [
SemanticaKGTool(graph),
SemanticaDecisionTool(graph),
]
agent = create_react_agent(model, tools)
```
- `semantica_query_graph` — query the shared context graph (keyword / NL)
- `semantica_query_decisions` — search the recorded decision log
## Compatibility
- Requires `langchain-core >= 0.3`.
- All classes degrade gracefully when `langchain-core` is absent: they remain
importable (carrying the full Semantica API), and `build()` returns `None`,
so agents can branch on `LANGCHAIN_AVAILABLE`.
+48
View File
@@ -0,0 +1,48 @@
"""
Semantica × LangChain Integration
=================================
First-class integration between the Semantica semantic intelligence stack and
the `LangChain <https://github.com/langchain-ai/langchain>`_ / LangGraph
ecosystem.
Public surface
--------------
SemanticaRetriever ``BaseRetriever`` with multi-hop GraphRAG (walks graph
edges from hybrid-search hits)
SemanticaVectorStore ``VectorStore`` adapter over Semantica's hybrid search
(drop-in for RetrievalQA / LCEL chains)
SemanticaKGTool ``BaseTool`` for querying the context graph
SemanticaDecisionTool ``BaseTool`` exposing the recorded decision log
Quick start
-----------
pip install semantica[langchain]
>>> from integrations.langchain import (
... SemanticaRetriever,
... SemanticaVectorStore,
... SemanticaKGTool,
... SemanticaDecisionTool,
... )
Compatibility
-------------
Requires ``langchain-core >= 0.3``. All classes degrade gracefully when
``langchain-core`` is not installed they are still importable and carry the
full Semantica API, but cannot be bound to LangChain chains/agents.
"""
from .retriever import LANGCHAIN_AVAILABLE, SemanticaRetriever
from .tools import SemanticaDecisionTool, SemanticaKGTool
from .vectorstore import SemanticaVectorStore
__all__ = [
"SemanticaRetriever",
"SemanticaVectorStore",
"SemanticaKGTool",
"SemanticaDecisionTool",
"LANGCHAIN_AVAILABLE",
]
__version__ = "0.1.0"
+216
View File
@@ -0,0 +1,216 @@
"""
SemanticaRetriever LangChain ``BaseRetriever`` with multi-hop GraphRAG.
Hybrid search seeds the retrieval, then graph edges are walked for ``hops``
steps so results go beyond flat vector similarity.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Tuple
from semantica.utils.logging import get_logger
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: LangChain core
# ---------------------------------------------------------------------------
LANGCHAIN_AVAILABLE = False
LANGCHAIN_IMPORT_ERROR: Optional[str] = None
_BaseRetriever: Any = object
_Document: Any = None
def _get_document(**kwargs: Any) -> Any:
"""Instantiate a langchain Document lazily (keeps the import optional)."""
if _Document is None: # pragma: no cover - exercised only with langchain
raise RuntimeError(LANGCHAIN_IMPORT_ERROR or "langchain-core not installed")
return _Document(**kwargs)
try:
from langchain_core.documents import Document as _Document # type: ignore
from langchain_core.retrievers import (
BaseRetriever as _BaseRetriever, # type: ignore
)
LANGCHAIN_AVAILABLE = True
except ImportError: # pragma: no cover - exercised only without langchain
LANGCHAIN_IMPORT_ERROR = (
"langchain-core is not installed. Install with: pip install langchain-core"
)
logger.debug(LANGCHAIN_IMPORT_ERROR)
def _hit_layers(hit: Dict[str, Any]) -> Tuple[Dict[str, Any], Dict[str, Any]]:
"""Nested HybridSearch metadata and ContextGraph.query node, if present."""
metadata = hit.get("metadata") if isinstance(hit.get("metadata"), dict) else {}
node = hit.get("node") if isinstance(hit.get("node"), dict) else {}
return metadata, node
def _hit_id(hit: Dict[str, Any]) -> Optional[str]:
"""Graph node id, preferring metadata over a HybridSearch vector id."""
metadata, node = _hit_layers(hit)
return (
hit.get("node_id")
or metadata.get("node_id")
or node.get("id")
or node.get("node_id")
or hit.get("id")
)
def _hit_content(hit: Dict[str, Any], fallback: str = "") -> str:
metadata, node = _hit_layers(hit)
props = node.get("properties") if isinstance(node.get("properties"), dict) else {}
return (
hit.get("content")
or hit.get("text")
or metadata.get("content")
or metadata.get("text")
or props.get("content")
or fallback
)
def _hit_type(hit: Dict[str, Any]) -> str:
metadata, node = _hit_layers(hit)
return (
hit.get("node_type")
or hit.get("type")
or metadata.get("node_type")
or metadata.get("type")
or node.get("type")
or node.get("node_type")
or "node"
)
def _hit_score(hit: Dict[str, Any], default: float = 1.0) -> float:
return float(hit.get("score") if hit.get("score") is not None else hit.get("distance") or default)
class SemanticaRetriever(_BaseRetriever): # type: ignore[misc]
"""GraphRAG-style retriever over a Semantica ``ContextGraph``.
Args:
graph: A semantica.context.ContextGraph instance.
hybrid: A semantica.vector_store.HybridSearch instance used to seed
retrieval. If omitted, a best-effort keyword search on the graph
is used.
hops: Number of graph-edge expansion hops (default 2).
top_k: Number of seed hits (default 10).
"""
graph: Any
hybrid: Any = None
hops: int = 2
top_k: int = 10
def __init__(
self,
graph: Any,
hybrid: Any = None,
hops: int = 2,
top_k: int = 10,
**kwargs: Any,
) -> None:
"""Explicit init so the retriever works with and without langchain."""
if LANGCHAIN_AVAILABLE:
# BaseRetriever is a Pydantic model: pass the declared fields
# through so validation succeeds.
super().__init__(
graph=graph,
hybrid=hybrid,
hops=hops,
top_k=top_k,
**kwargs,
)
else:
# Without langchain-core, BaseRetriever is a plain object
super().__init__() # type: ignore[call-arg]
self.graph = graph
self.hybrid = hybrid
self.hops = hops
self.top_k = top_k
def _get_relevant_documents(self, query: str, **kwargs: Any) -> List[Any]:
"""LangChain BaseRetriever entry point."""
seed = self._seed_results(query)
if not seed:
return []
# Expand each seed node through the graph
expanded: Dict[str, Dict[str, Any]] = {}
for hit in seed:
node_id = _hit_id(hit)
if not node_id:
continue
metadata, _ = _hit_layers(hit)
expanded[node_id] = {
"content": _hit_content(hit, fallback=str(node_id)),
"node_type": _hit_type(hit),
"score": _hit_score(hit),
"metadata": metadata,
}
try:
neighbors = self.graph.get_neighbors(node_id, hops=self.hops)
for neighbor in neighbors:
nid = neighbor.get("node_id") or neighbor.get("id")
if nid and nid not in expanded:
expanded[nid] = {
"content": neighbor.get("content")
or neighbor.get("text")
or neighbor.get("name")
or str(nid),
"node_type": neighbor.get("node_type")
or neighbor.get("type")
or "node",
"score": float(neighbor.get("weight") or 0.5),
"metadata": {},
}
except Exception as exc: # graph expansion is best-effort
logger.debug("graph expansion failed for %s: %s", node_id, exc)
# Order: seed hits first (they have real scores), then neighbors.
# Keep a deterministic id->payload list (sets are unordered — see Qodo).
ordered_pairs: List[tuple] = []
seen_ids = set()
for hit in seed:
nid = _hit_id(hit)
if nid and nid in expanded and nid not in seen_ids:
ordered_pairs.append((nid, expanded[nid]))
seen_ids.add(nid)
for nid, item in expanded.items():
if nid not in seen_ids:
ordered_pairs.append((nid, item))
seen_ids.add(nid)
return [
_get_document(
page_content=item["content"],
metadata={
**item["metadata"],
"node_id": nid,
"node_type": item["node_type"],
"score": item["score"],
},
)
for nid, item in ordered_pairs
]
def _seed_results(self, query: str) -> List[Dict[str, Any]]:
"""Get seed results from hybrid search or a graph keyword scan."""
if self.hybrid is not None:
try:
return self.hybrid.search(query, k=self.top_k)
except Exception as exc:
logger.debug("hybrid search failed, falling back: %s", exc)
# Best-effort keyword scan over graph nodes (ContextGraph.query)
try:
return self.graph.query(query, limit=self.top_k)
except Exception:
return []
+133
View File
@@ -0,0 +1,133 @@
"""
SemanticaKGTool / SemanticaDecisionTool LangChain ``BaseTool`` adapters
for LangChain / LangGraph agents.
"""
from __future__ import annotations
import json
from typing import Any, Optional, Type
from pydantic import BaseModel, ConfigDict, Field
from semantica.utils.logging import get_logger
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: LangChain core
# ---------------------------------------------------------------------------
LANGCHAIN_AVAILABLE = False
LANGCHAIN_IMPORT_ERROR: Optional[str] = None
_BaseTool: Any = object
try:
from langchain_core.tools import BaseTool as _BaseTool # type: ignore
LANGCHAIN_AVAILABLE = True
except ImportError: # pragma: no cover
LANGCHAIN_IMPORT_ERROR = (
"langchain-core is not installed. Install with: pip install langchain-core"
)
logger.debug(LANGCHAIN_IMPORT_ERROR)
def _json(payload: Any) -> str:
return json.dumps(payload, default=str, ensure_ascii=False)
class QueryGraphInput(BaseModel):
query: str = Field(..., description="Natural-language or keyword graph query")
limit: int = Field(10, description="Maximum matching nodes to return")
class QueryDecisionsInput(BaseModel):
category: str = Field(
"",
description="Keyword to search recorded decisions; empty returns insights",
)
limit: int = Field(10, description="Maximum results when searching by keyword")
class SemanticaKGTool(_BaseTool): # type: ignore[misc]
"""LangChain tool for querying a Semantica ``ContextGraph``.
Args:
graph: A semantica.context.ContextGraph instance.
Example:
>>> tool = SemanticaKGTool(graph)
>>> agent = create_react_agent(model, tools=[tool])
"""
model_config = ConfigDict(arbitrary_types_allowed=True)
name: str = "semantica_query_graph"
description: str = (
"Query Semantica's shared context graph with a natural-language "
"keyword query. Returns matching entities and relationships."
)
args_schema: Type[BaseModel] = QueryGraphInput
graph: Any = None
def __init__(self, graph: Any = None, **kwargs: Any) -> None:
if LANGCHAIN_AVAILABLE:
super().__init__(graph=graph, **kwargs)
else:
super().__init__()
self.graph = graph
def build(self) -> Any:
"""Return this tool, or None if langchain-core is missing."""
return self if LANGCHAIN_AVAILABLE else None
def _run(self, query: str, limit: int = 10, **kwargs: Any) -> str:
try:
return _json(self.graph.query(query, limit=limit))
except Exception as exc:
return _json({"error": str(exc)})
async def _arun(self, query: str, limit: int = 10, **kwargs: Any) -> str:
return self._run(query, limit=limit)
class SemanticaDecisionTool(_BaseTool): # type: ignore[misc]
"""LangChain tool for searching Semantica's recorded decision log.
Args:
graph: A semantica.context.ContextGraph instance.
"""
model_config = ConfigDict(arbitrary_types_allowed=True)
name: str = "semantica_query_decisions"
description: str = (
"Search Semantica's recorded decision log with a keyword query. "
"Returns decisions, rationale, and context."
)
args_schema: Type[BaseModel] = QueryDecisionsInput
graph: Any = None
def __init__(self, graph: Any = None, **kwargs: Any) -> None:
if LANGCHAIN_AVAILABLE:
super().__init__(graph=graph, **kwargs)
else:
super().__init__()
self.graph = graph
def build(self) -> Any:
"""Return this tool, or None if langchain-core is missing."""
return self if LANGCHAIN_AVAILABLE else None
def _run(self, category: str = "", limit: int = 10, **kwargs: Any) -> str:
try:
if category:
return _json(self.graph.query(category, limit=limit))
return _json(self.graph.get_decision_insights())
except Exception as exc:
return _json({"error": str(exc)})
async def _arun(self, category: str = "", limit: int = 10, **kwargs: Any) -> str:
return self._run(category=category, limit=limit)
+143
View File
@@ -0,0 +1,143 @@
"""
SemanticaVectorStore LangChain ``VectorStore`` adapter over Semantica's
hybrid search (``semantica.vector_store.HybridSearch``).
"""
from __future__ import annotations
from typing import Any, Dict, Iterable, List, Optional
from semantica.utils.logging import get_logger
from .retriever import _hit_content, _hit_id, _hit_score, _hit_type, _hit_layers
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: LangChain core
# ---------------------------------------------------------------------------
LANGCHAIN_AVAILABLE = False
LANGCHAIN_IMPORT_ERROR: Optional[str] = None
_VectorStoreBase: Any = object
_Document: Any = None
def _make_document(**kwargs: Any) -> Any:
if _Document is None: # pragma: no cover
raise RuntimeError(LANGCHAIN_IMPORT_ERROR or "langchain-core not installed")
return _Document(**kwargs)
try:
from langchain_core.documents import Document as _Document # type: ignore
from langchain_core.vectorstores import (
VectorStore as _VectorStoreBase, # type: ignore
)
LANGCHAIN_AVAILABLE = True
except ImportError: # pragma: no cover
LANGCHAIN_IMPORT_ERROR = (
"langchain-core is not installed. Install with: pip install langchain-core"
)
logger.debug(LANGCHAIN_IMPORT_ERROR)
def _document_from_hit(hit: Dict[str, Any], include_score: bool = True) -> Any:
metadata, _ = _hit_layers(hit)
node_id = _hit_id(hit)
doc_meta = {
**metadata,
"node_id": node_id,
"node_type": _hit_type(hit),
}
if include_score:
doc_meta["score"] = _hit_score(hit, default=0.0)
return _make_document(
page_content=_hit_content(hit),
metadata=doc_meta,
)
class SemanticaVectorStore(_VectorStoreBase): # type: ignore[misc]
"""Wrap Semantica hybrid search as a LangChain ``VectorStore``.
Args:
hybrid: A semantica.vector_store.HybridSearch instance.
vector_store: Optional Semantica vector store passed through to
``HybridSearch.add_texts``.
"""
hybrid: Any
vector_store: Any = None
def __init__(self, hybrid: Any, vector_store: Any = None, **kwargs: Any) -> None:
if LANGCHAIN_AVAILABLE:
super().__init__(**kwargs)
else:
super().__init__()
self.hybrid = hybrid
self.vector_store = vector_store
# -- required VectorStore API ------------------------------------------
def add_texts(
self,
texts: Iterable[str],
metadatas: Optional[List[Dict[str, Any]]] = None,
**kwargs: Any,
) -> List[str]:
"""Embed and store texts; return the generated IDs.
Delegates to the Semantica ``VectorStore.add_documents`` backing the
HybridSearch instance (or to ``hybrid.vector_store`` if provided).
"""
if self.vector_store is not None:
return self.vector_store.add_documents(
list(texts), metadata=metadatas, **kwargs
)
vs = getattr(self.hybrid, "vector_store", None)
if vs is not None and hasattr(vs, "add_documents"):
return vs.add_documents(list(texts), metadata=metadatas, **kwargs)
raise ValueError(
"SemanticaVectorStore requires a Semantica vector store with "
"add_documents (pass vector_store=... to the HybridSearch or to "
"SemanticaVectorStore)"
)
def similarity_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Any]:
"""Return documents most similar to the query."""
return [_document_from_hit(hit) for hit in self.hybrid.search(query, k=k)]
def similarity_search_with_score(
self, query: str, k: int = 4, **kwargs: Any
) -> List[Any]:
"""Return (document, score) pairs."""
return [
(
_document_from_hit(hit, include_score=False),
_hit_score(hit, default=0.0),
)
for hit in self.hybrid.search(query, k=k)
]
@classmethod
def from_texts(
cls,
texts: List[str],
embedding: Any = None,
metadatas: Optional[List[Dict[str, Any]]] = None,
**kwargs: Any,
) -> "SemanticaVectorStore":
"""Build a store from a list of texts (LangChain convention).
Requires a pre-configured ``hybrid`` instance passed via kwargs.
"""
hybrid = kwargs.pop("hybrid", None)
if hybrid is None:
raise ValueError(
"SemanticaVectorStore.from_texts requires a 'hybrid' "
"HybridSearch instance as a keyword argument"
)
store = cls(hybrid=hybrid, **kwargs)
store.add_texts(texts, metadatas=metadatas)
return store
+35 -1
View File
@@ -116,7 +116,41 @@ class OpenClawKGTool:
)
def __init__(self, base_url: str = "http://localhost:8000", timeout: int = 30) -> None:
self.base_url = base_url.rstrip("/")
# Validate base_url at construction time so callers get an immediate,
# actionable error rather than a cryptic failure on the first request.
# allow_private_ips=True because the documented default (localhost:8000)
# is intentionally a local Semantica server; the scheme check and
# URL-structure check still apply unconditionally.
try:
from semantica.ingest.ssrf import validate_url_for_request
validate_url_for_request(base_url, allow_private_ips=True)
except ImportError:
# semantica.ingest not installed in minimal openclaw-only environments;
# mirror the structural checks that validate_url_for_request performs
# unconditionally (before allow_private_ips is consulted), so the
# guarantee in the comment above — "scheme check and URL-structure check
# still apply unconditionally" — holds in this path too.
from urllib.parse import urlparse as _urlparse
if not isinstance(base_url, str) or not base_url.strip():
raise ValueError("OpenClawKGTool base_url must be a non-empty string.")
_parsed = _urlparse(base_url.strip())
_scheme = (_parsed.scheme or "").lower()
if _scheme not in ("http", "https"):
raise ValueError(
f"OpenClawKGTool base_url scheme '{_parsed.scheme}' is not permitted. "
"Only http and https are allowed."
)
if not _parsed.netloc:
raise ValueError(
f"Invalid OpenClawKGTool base_url '{base_url}': "
"URL must include a netloc (domain or host)."
)
if not _parsed.hostname:
raise ValueError(
f"Invalid OpenClawKGTool base_url '{base_url}': "
"URL must include a hostname."
)
self.base_url = base_url.strip().rstrip("/")
self.timeout = timeout
self._session: Any = None
+11
View File
@@ -21,6 +21,17 @@ Configure in Claude Desktop, Windsurf, Cline, Continue, VS Code:
}
"""
import os
# MCP stdio framing IS stdout: any progress bar or console renderer that writes
# to stdout would interleave with the JSON-RPC stream and corrupt framing for
# every client. This package is always used as an MCP stdio server, so force
# progress tracking off for the entire process. Set before importing server /
# tools so the Semantica progress-tracker singleton is never created with
# output enabled (the singleton reads this variable at construction time and
# the enabled.setter re-checks it, so later re-enable attempts are also blocked).
os.environ["SEMANTICA_DISABLE_PROGRESS"] = "1"
# `semantica.__version__` is the authoritative package version — see
# semantica/mcp_server/__init__.py for why it is used directly rather than
# importlib.metadata.version("semantica").
+6 -1
View File
@@ -80,7 +80,12 @@ def handle_export_graph(args: dict) -> dict:
if rdf_fmt:
try:
from semantica.export import RDFExporter
rdf_str = RDFExporter().export_to_rdf(graph, format=rdf_fmt)
# RDFExporter.export_to_rdf() expects the canonical kg dict
# {"entities": [...], "relationships": [...]}, not a ContextGraph
# object. Convert before handing off; passing the raw graph
# caused AttributeError: 'ContextGraph' object has no attribute
# 'get' on every RDF format.
rdf_str = RDFExporter().export_to_rdf(graph.to_kg_dict(), format=rdf_fmt)
return {"format": rdf_fmt, "data": rdf_str}
except Exception as exc:
return {"error": f"RDF export failed: {exc}"}
+1 -1
View File
@@ -53,7 +53,7 @@ plugins/
## Prerequisites
```bash
git clone https://github.com/Hawksight-AI/semantica.git
git clone https://github.com/semantica-agi/semantica.git
cd semantica
pip install semantica # Python 3.10+
```
+2 -2
View File
@@ -1,8 +1,8 @@
{
"name": "semantica-local",
"owner": {
"name": "Hawksight AI",
"url": "https://github.com/Hawksight-AI/semantica"
"name": "Semantica",
"url": "https://github.com/semantica-agi/semantica"
},
"plugins": [
{
+2 -2
View File
@@ -5,8 +5,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -5,8 +5,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
-9
View File
@@ -187,15 +187,6 @@ def poc_vuln3():
})
return nodes
# Simulate the CSV parser — mirrors export_import.py lines 131-133
def parse_import_csv_row(row: dict) -> dict:
"""Mirrors export_import.py CSV node ID extraction (no sanitization)."""
node_id = row.get("id") or row.get("node_id") or row.get(":ID") or row.get("_id")
return {
"id": str(node_id), # ← UNSANITIZED
"type": row.get("type", "entity"),
}
# Attack payloads
payloads = [
# Header injection payload (chained with VULN-1)
+4 -2
View File
@@ -85,7 +85,8 @@ dependencies = [
"loguru>=0.7.3",
"structlog>=22.1.0",
"gensim>=4.4.0",
"httpx<0.29.0"
"httpx<0.29.0",
"pyarrow>=14.0.0"
]
[project.urls]
@@ -205,6 +206,7 @@ agno = ["agno>=1.0.0"]
# needed (it pulls vulnerable transitive deps like chromadb) and would only
# duplicate the prebuilt tooling users can install separately.
crewai = ["crewai>=0.80.0"]
langchain = ["langchain-core>=0.3.0"]
# ---- File Watching ----
watch = ["watchdog>=6.0.0"]
@@ -252,7 +254,7 @@ explorer-lite = [
# dependency-audit/security gates. Install it explicitly via ``semantica[crewai]``.
all = [
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno,langchain]"
]
# ---------------- ENTRYPOINTS ----------------
+797 -320
View File
File diff suppressed because it is too large Load Diff
@@ -1039,6 +1039,6 @@ manager = TemporalVersionManager(storage_path="large_data.db")
## Support
For questions or issues:
- GitHub Issues: https://github.com/Hawksight-AI/semantica/issues
- GitHub Issues: https://github.com/semantica-agi/semantica/issues
- Documentation: https://semantica.readthedocs.io
- Community: https://discord.gg/sV34vps5hH
@@ -31,6 +31,7 @@ import hashlib
import json
import sqlite3
import threading
import warnings
from abc import ABC, abstractmethod
from datetime import datetime
from pathlib import Path
@@ -62,6 +63,12 @@ def create_graph_snapshot_record(
"""
Creates a standardized snapshot metadata record for a named graph.
.. deprecated::
``create_graph_snapshot_record()`` is deprecated and will be removed in
a future major version. It has no callers inside Semantica; build the
record inline and checksum it with
:func:`semantica.change_management.compute_checksum` instead.
Args:
version_id: Unique identifier for this snapshot
graph_uri: The underlying named graph URI in the triplet store
@@ -69,6 +76,13 @@ def create_graph_snapshot_record(
description: Purpose or context of the snapshot
metadata: Additional tags or pipeline context
"""
warnings.warn(
"create_graph_snapshot_record() is deprecated and will be removed in a "
"future major version. Build the snapshot record inline and use "
"semantica.change_management.compute_checksum() instead.",
DeprecationWarning,
stacklevel=2,
)
record = {
"label": version_id,
+193 -4
View File
@@ -773,10 +773,22 @@ def changelog(cli_ctx: CLIContext, local_json: bool) -> None:
_run_with_error_handling(_action)
class _DeepEmbeddingFailure(Exception):
"""A deep-probe failure from doctor's embedding checks.
Marks failures that happened AFTER the backend imported cleanly model
load, probe, or runtime problems so the check's hint can point at the
real remediation instead of `pip install`.
"""
@main.command()
@click.option("--json", "local_json", is_flag=True, default=False)
@click.option("--deep-embeddings", "deep_embeddings", is_flag=True, default=False,
help="Also instantiate the local embedding backends and embed a probe "
"text (catches backends that import cleanly but cannot load).")
@click.pass_obj
def doctor(cli_ctx: CLIContext, local_json: bool) -> None:
def doctor(cli_ctx: CLIContext, local_json: bool, deep_embeddings: bool) -> None:
"""Run a health check on all Semantica components and backends."""
import importlib.metadata
cli_ctx = _require_ctx(cli_ctx)
@@ -787,6 +799,16 @@ def doctor(cli_ctx: CLIContext, local_json: bool) -> None:
try:
note = fn()
return label, "ok", note, None
except _DeepEmbeddingFailure as exc:
# A deep-probe failure means the package IMPORTED fine: the pip
# hint would be the wrong remediation for what is actually a
# runtime/model-load problem (broken torch, failed model
# download, missing shared libs).
return label, "fail", str(exc), (
"runtime/model-load failure — reinstalling the package usually "
"does not help; check the warnings above (torch install, model "
"download, disk space)"
)
except Exception as exc:
return label, "fail", str(exc), hint
@@ -827,6 +849,50 @@ def doctor(cli_ctx: CLIContext, local_json: bool) -> None:
return f"{backend} importable"
checks.append(_check("Vector store", _vector, hint="pip install semantica[vectorstore-…]"))
# Embedding backends (#994): `doctor` used to report all green while
# every local embedding backend was non-functional — import success
# says nothing about model loading. Default checks stay cheap
# (import + version); --deep-embeddings (or
# SEMANTICA_DOCTOR_DEEP_EMBEDDINGS=1) instantiates the backend through
# TextEmbedder and embeds a probe, which is the only level that
# catches a backend that imports cleanly but cannot actually load.
deep = deep_embeddings or os.environ.get("SEMANTICA_DOCTOR_DEEP_EMBEDDINGS", "").strip().lower() in ("1", "true", "yes", "on")
def _embedding_backend(method: str) -> str:
if method == "sentence_transformers":
import sentence_transformers # noqa: F401
note = f"importable ({importlib.metadata.version('sentence-transformers')})"
else:
import fastembed # noqa: F401
note = f"importable ({importlib.metadata.version('fastembed')})"
if not deep:
return note
try:
from .embeddings import TextEmbedder
embedder = TextEmbedder(method=method)
if embedder.model is None and embedder.fastembed_model is None:
raise RuntimeError(
"model failed to load — the hash fallback is active "
"(see warnings above); embedding quality is degraded"
)
probe = embedder.embed_text("semantica doctor embedding probe")
except _DeepEmbeddingFailure:
raise
except Exception as exc:
raise _DeepEmbeddingFailure(str(exc)) from exc
return f"{note}; deep probe ok ({len(probe)}-dim)"
checks.append(_check(
"Embeddings (sentence-transformers)",
lambda: _embedding_backend("sentence_transformers"),
hint="pip install sentence-transformers",
))
checks.append(_check(
"Embeddings (fastembed)",
lambda: _embedding_backend("fastembed"),
hint="pip install fastembed",
))
# LLM provider keys
for provider, var in [("OpenAI", "OPENAI_API_KEY"), ("Anthropic", "ANTHROPIC_API_KEY"),
("Groq", "GROQ_API_KEY")]:
@@ -1666,6 +1732,93 @@ def embed(ctx: click.Context) -> None:
click.echo(ctx.get_help())
def _json_default(obj) -> object:
"""JSON serialiser that converts NumPy scalars/arrays to native Python types.
Falls back to ``str()`` for everything else so the writer never crashes on
unexpected types (e.g. ``datetime``, custom domain objects).
"""
try:
import numpy as np # local import — only needed when result contains numpy
if isinstance(obj, np.ndarray):
return obj.tolist()
if isinstance(obj, np.generic):
return obj.item()
except ImportError:
pass
return str(obj)
def _write_result_output(out_path: Path, result) -> None:
"""Serialize a structured CLI result (dict or list) for ``--output``.
Domain commands like ``deduplicate`` and ``ontology align`` produce dicts
and lists, not numeric matrices routing them through the embeddings
writer rejected their shapes and extensions (.csv is documented for
deduplicate). JSON-family formats serialize anything; CSV serializes a
list of dicts (or a single dict as one row).
Accepted extensions: .json, .jsonl, .csv (no-extension and .txt are
rejected so the path reported to the caller always matches the file
actually created, consistent with every other --output in the CLI).
"""
import json as _json
suffix = out_path.suffix.lower()
# ── JSON ────────────────────────────────────────────────────────────────
if suffix == ".json":
with open(out_path, "w", encoding="utf-8") as fh:
_json.dump(result, fh, indent=2, default=_json_default)
return
# ── JSON Lines ──────────────────────────────────────────────────────────
# Every record must occupy exactly one line. Wrap a bare dict in a list
# so callers never need to know whether their result is singular or plural.
if suffix == ".jsonl":
items = result if isinstance(result, list) else [result]
with open(out_path, "w", encoding="utf-8") as fh:
for item in items:
fh.write(_json.dumps(item, default=_json_default) + "\n")
return
# ── CSV ─────────────────────────────────────────────────────────────────
if suffix == ".csv":
import pandas as pd
rows = result if isinstance(result, list) else [result]
if not rows:
raise click.ClickException(
"No results to write — output file not created."
)
# Normalise numpy scalars/arrays to Python natives so to_csv() does
# not fall back to repr() strings for array-valued cells.
def _normalise(row):
if not isinstance(row, dict):
return row
out = {}
for k, v in row.items():
try:
import numpy as np
if isinstance(v, np.ndarray):
v = v.tolist()
elif isinstance(v, np.generic):
v = v.item()
except ImportError:
pass
out[k] = v
return out
pd.DataFrame([_normalise(r) for r in rows]).to_csv(out_path, index=False)
return
# ── unsupported ─────────────────────────────────────────────────────────
display = suffix if suffix else "(no extension)"
raise click.ClickException(
f"Unsupported output format '{display}'. Use .json, .jsonl, or .csv"
)
@embed.command("generate")
@click.argument("input_path")
@click.option("--model",
@@ -1708,7 +1861,43 @@ def embed_generate(cli_ctx: CLIContext, input_path: str, model: str,
except ImportError as exc:
raise click.ClickException(f"Embeddings module not available: {exc}") from exc
if output:
Path(output).write_text(json.dumps(result, default=str), encoding="utf-8")
output_path = Path(output)
suffix = output_path.suffix.lower()
try:
import numpy as np
import pandas as pd
arr = np.asarray(result)
if arr.ndim == 1:
arr = arr[np.newaxis, :]
if arr.ndim != 2:
raise click.ClickException(
f"embed generate --output expects a 1-D or 2-D array, "
f"got {arr.ndim}-D (shape {arr.shape})"
)
rows = [list(row) for row in arr]
if suffix == ".parquet":
# Schema: single 'embedding' column (list[float] per row).
# embed index detects vector columns via
# isinstance(df[c].iloc[0], (list, np.ndarray)).
df = pd.DataFrame({"embedding": rows})
df.to_parquet(output_path, index=False)
elif suffix in (".json", ".jsonl"):
df = pd.DataFrame({"embedding": rows})
df.to_json(
output_path,
orient="records",
lines=(suffix == ".jsonl"),
)
else:
raise click.ClickException(
f"Unsupported output format '{suffix}'. "
"Use .parquet, .json, or .jsonl"
)
except ImportError as exc:
raise click.ClickException(
f"Missing dependency for --output: {exc}. "
"Install pyarrow with: pip install pyarrow"
) from exc
_ok(cli_ctx, f"Wrote {output}")
elif _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, dict) else {"status": "ok"})
@@ -1930,7 +2119,7 @@ def deduplicate(
except ImportError as exc:
raise click.ClickException(f"Deduplication module not available: {exc}") from exc
if output:
Path(output).write_text(json.dumps(result, default=str), encoding="utf-8")
_write_result_output(Path(output), result)
_ok(cli_ctx, f"Wrote {output}")
elif _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, (dict, list)) else {"result": str(result)})
@@ -3054,7 +3243,7 @@ def ontology_align(cli_ctx: CLIContext, source: str, target: str, strategy: str,
except ImportError as exc:
raise click.ClickException(f"Ontology module not available: {exc}") from exc
if output:
Path(output).write_text(json.dumps(result, default=str), encoding="utf-8")
_write_result_output(Path(output), result)
_ok(cli_ctx, f"Wrote {output}")
elif _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, dict) else {"alignments": str(result)})
+32
View File
@@ -0,0 +1,32 @@
"""Filesystem safety helpers for human-editable Markdown persistence."""
import os
import stat
from pathlib import Path
from typing import Optional
def is_filesystem_link(path: Path) -> bool:
"""Return whether *path* is a symlink, junction, or Windows reparse point."""
if path.is_symlink():
return True
isjunction = getattr(os.path, "isjunction", None)
if isjunction is not None and isjunction(path):
return True
try:
attributes = getattr(os.lstat(path), "st_file_attributes", 0)
except (FileNotFoundError, NotADirectoryError):
return False
reparse_point = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400)
return bool(attributes & reparse_point)
def find_filesystem_link(path: Path) -> Optional[Path]:
"""Return the first linked component in *path*, including its ancestors."""
for candidate in (path, *path.parents):
if is_filesystem_link(candidate):
return candidate
return None
+49 -37
View File
@@ -77,6 +77,7 @@ import yaml
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ..utils.types import EntityDict, RelationshipDict
from ._markdown_filesystem import find_filesystem_link
class _UniqueKeySafeLoader(yaml.SafeLoader):
@@ -1742,9 +1743,10 @@ class AgentMemory:
@staticmethod
def _write_markdown_file(file_path: Path, document: str) -> None:
"""Atomically replace a Markdown file without following output symlinks."""
if file_path.is_symlink():
if find_filesystem_link(file_path) is not None:
raise ValueError(
f"Refusing to overwrite Markdown symbolic link: {file_path}"
"Refusing to overwrite Markdown symbolic link or junction: "
f"{file_path}"
)
temporary_path = None
@@ -1867,7 +1869,8 @@ class AgentMemory:
if "\n" not in data and "\r" not in data:
candidate = Path(data)
try:
candidate_exists = candidate.exists()
candidate_is_link = find_filesystem_link(candidate) is not None
candidate_exists = candidate_is_link or candidate.exists()
except OSError as exc:
error_message = (
"Failed to inspect possible Markdown import "
@@ -1909,62 +1912,71 @@ class AgentMemory:
return memories
def _read_markdown_file_content(self, file_path: Path) -> str:
if file_path.is_symlink():
raise ValueError(f"Symlink Markdown import paths are rejected: {file_path}")
if find_filesystem_link(file_path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links and "
f"junctions are unsafe: {file_path}"
)
flags = os.O_RDONLY
if hasattr(os, "O_NOFOLLOW"):
# On POSIX, O_NOFOLLOW makes os.open() fail with ELOOP if the
# final path component is a symlink, atomically closing the TOCTOU
# window between the is_symlink() check above and the open call.
# On Windows, O_NOFOLLOW is not available; the is_symlink() pre-check
# above is the only symlink defense and remains vulnerable to a narrow
# race. The fstat()/S_ISREG guard below still rejects special files
# (FIFOs, devices) on both platforms.
flags |= os.O_NOFOLLOW
nofollow_flag = getattr(os, "O_NOFOLLOW", 0)
flags |= nofollow_flag
try:
fd = os.open(str(file_path), flags)
except OSError as exc:
if exc.errno == getattr(errno, "ELOOP", None):
if (
(nofollow_flag and exc.errno == errno.ELOOP)
or find_filesystem_link(file_path) is not None
):
raise ValueError(
f"Symlink Markdown import paths are rejected: {file_path}"
"Symlink Markdown import paths are rejected; symbolic links "
f"and junctions are unsafe: {file_path}"
) from exc
raise
try:
stat_res = os.fstat(fd)
if not stat.S_ISREG(stat_res.st_mode):
if find_filesystem_link(file_path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links "
f"and junctions are unsafe: {file_path}"
)
if not stat.S_ISREG(os.fstat(fd).st_mode):
raise ValueError(
f"Markdown import path is not a regular file: {file_path}"
)
with open(fd, "r", encoding="utf-8", closefd=True) as f:
return f.read()
except Exception:
try:
with os.fdopen(fd, mode="r", encoding="utf-8") as source:
fd = -1
return source.read()
finally:
if fd >= 0:
os.close(fd)
except OSError:
pass
raise
def _read_markdown_path(self, path: Path) -> List[Tuple[str, str]]:
if path.is_symlink():
raise ValueError(f"Symlink Markdown import paths are rejected: {path}")
if find_filesystem_link(path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links and "
f"junctions are unsafe: {path}"
)
if not path.exists():
raise FileNotFoundError(f"Markdown import path does not exist: {path}")
if path.is_dir():
file_paths = sorted(
(
file_path
for file_path in path.iterdir()
if file_path.is_file()
and not file_path.is_symlink()
and file_path.suffix.lower() in self._MARKDOWN_EXTENSIONS
),
key=lambda file_path: (file_path.name.casefold(), file_path.name),
)
file_paths = []
for file_path in path.iterdir():
if file_path.suffix.lower() not in self._MARKDOWN_EXTENSIONS:
continue
if find_filesystem_link(file_path) is not None:
continue
if file_path.is_file():
file_paths.append(file_path)
if find_filesystem_link(path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links "
f"and junctions are unsafe: {path}"
)
file_paths.sort(key=lambda item: (item.name.casefold(), item.name))
elif path.is_file():
file_paths = [path]
else:
File diff suppressed because it is too large Load Diff
+15
View File
@@ -6,6 +6,7 @@ including node labels, relationship types, and indexes for graph databases.
"""
import json
import warnings
from typing import Dict, Any, List
from ..graph_store import GraphStore
@@ -460,11 +461,25 @@ def drop_decision_schema(graph_store: GraphStore) -> None:
"""
Drop decision tracking schema (for cleanup/testing).
.. deprecated::
``drop_decision_schema()`` is deprecated and will be removed in a future
major version. It has no callers inside Semantica; issue the DROP
CONSTRAINT / DROP INDEX / DETACH DELETE statements directly against your
:class:`~semantica.graph_store.GraphStore` instead.
Args:
graph_store: Graph database instance
"""
logger = get_logger(__name__)
warnings.warn(
"drop_decision_schema() is deprecated and will be removed in a future "
"major version. Issue the DROP CONSTRAINT / DROP INDEX / DETACH DELETE "
"statements directly against your GraphStore instead.",
DeprecationWarning,
stacklevel=2,
)
try:
# Drop constraints
constraints = [
+36 -8
View File
@@ -278,8 +278,12 @@ class DuplicateDetector:
for i, (entity1, entity2, score) in enumerate(similarities):
candidate = self._create_duplicate_candidate(entity1, entity2, score)
# Filter by confidence threshold
if candidate.confidence >= self.confidence_threshold:
# Filter by confidence threshold; type mismatches are excluded
# structurally so no threshold value can admit them.
if (
candidate.confidence >= self.confidence_threshold
and "type_mismatch" not in candidate.reasons
):
candidates.append(candidate)
remaining = total_similarities - (i + 1)
@@ -624,8 +628,12 @@ class DuplicateDetector:
new_entity, existing_entity, similarity.score
)
# Filter by confidence threshold
if candidate.confidence >= self.confidence_threshold:
# Filter by confidence threshold; type mismatches are
# excluded structurally regardless of the threshold.
if (
candidate.confidence >= self.confidence_threshold
and "type_mismatch" not in candidate.reasons
):
candidates.append(candidate)
processed += 1
@@ -723,7 +731,9 @@ class DuplicateDetector:
if key == "name":
return getattr(entity, "text", default)
if key == "type":
return getattr(entity, "label", default)
# Entity objects store the type on .type; extraction entities
# may expose .label. Missing .label never means "no type".
return getattr(entity, "type", default) or getattr(entity, "label", default)
if key == "properties":
# Check metadata for properties
metadata = getattr(entity, "metadata", {})
@@ -757,6 +767,25 @@ class DuplicateDetector:
reasons = []
confidence = similarity_score
# Check entity type mismatch first: two entities with different
# explicit types are not duplicates, whatever their similarity.
entity_type1 = self._get_entity_value(entity1, "type")
entity_type2 = self._get_entity_value(entity2, "type")
if entity_type1 and entity_type2 and entity_type1 != entity_type2:
return DuplicateCandidate(
entity1=entity1,
entity2=entity2,
similarity_score=similarity_score,
confidence=0.0,
reasons=["type_mismatch"],
metadata={
"name_match": False,
"common_properties": 0,
"type_match": False,
"type_mismatch": True,
},
)
# Check for exact name match (strong indicator)
name1 = str(self._get_entity_value(entity1, "name", "")).lower().strip()
name2 = str(self._get_entity_value(entity2, "name", "")).lower().strip()
@@ -779,9 +808,8 @@ class DuplicateDetector:
# Boost confidence for each matching property
confidence += 0.05 * prop_matches
# Check entity type match
entity_type1 = self._get_entity_value(entity1, "type")
entity_type2 = self._get_entity_value(entity2, "type")
# Check entity type match (only boosts when types are equal; mismatch
# is handled above)
if entity_type1 and entity_type2 and entity_type1 == entity_type2:
reasons.append("same_type")
confidence += 0.05
+29 -31
View File
@@ -147,9 +147,12 @@ def calculate_similarity(
>>> result = calculate_similarity(entity1, entity2, method="levenshtein")
>>> print(f"Similarity: {result.score:.2f}")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-referential wrappers.
# _multi_factor_similarity is registered under "multi_factor" and calls back
# into calculate_similarity(method="multi_factor"), creating indirect
# infinite recursion. The identity guard short-circuits that loop.
custom_method = method_registry.get("similarity", method)
if custom_method:
if custom_method and custom_method is not calculate_similarity:
return custom_method(entity1, entity2, **kwargs)
# Use default SimilarityCalculator
@@ -235,9 +238,11 @@ def detect_duplicates(
>>> duplicates = detect_duplicates(entities, method="pairwise", similarity_threshold=0.8)
>>> print(f"Found {len(duplicates)} duplicate candidates")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-referential wrappers.
# _pairwise_detection is registered under "pairwise" and calls back into
# detect_duplicates(method="pairwise"), creating indirect infinite recursion.
custom_method = method_registry.get("detection", method)
if custom_method:
if custom_method and custom_method is not detect_duplicates:
return custom_method(
entities, similarity_threshold=similarity_threshold, **kwargs
)
@@ -282,9 +287,10 @@ def dedup_triplets(
List of duplicate relationship piars (rel1, rel2).
"""
# Check for custom method in registry (but not ourself)
# Check for custom method in registry (but not ourself — identity guard
# consistent with the other dispatch functions in this module).
custom_method = method_registry.get("detection", "triplets")
if custom_method and custom_method.__name__ != "dedup_triplets":
if custom_method and custom_method is not dedup_triplets:
return custom_method(relationships, mode=mode, threshold=threshold, **kwargs)
detector = DuplicateDetector(**kwargs)
@@ -328,9 +334,11 @@ def merge_entities(
>>> operations = merge_entities(duplicate_entities, method="keep_most_complete")
>>> print(f"Performed {len(operations)} merge operations")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-referential registration.
# merge_entities is now registered directly under its default method name;
# the identity guard prevents a direct recursion loop.
custom_method = method_registry.get("merging", method)
if custom_method:
if custom_method and custom_method is not merge_entities:
return custom_method(
entities, preserve_provenance=preserve_provenance, **kwargs
)
@@ -374,9 +382,12 @@ def build_clusters(
>>> result = build_clusters(entities, method="graph_based", similarity_threshold=0.8)
>>> print(f"Found {len(result.clusters)} clusters")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-referential wrappers.
# _graph_based_clustering is registered under "graph_based" and calls back
# into build_clusters(method="graph_based"), creating indirect infinite
# recursion.
custom_method = method_registry.get("clustering", method)
if custom_method:
if custom_method and custom_method is not build_clusters:
return custom_method(
entities, similarity_threshold=similarity_threshold, **kwargs
)
@@ -546,25 +557,12 @@ def list_available_methods(task: Optional[str] = None) -> Dict[str, List[str]]:
return result
# Register default methods with registry
def _multi_factor_similarity(e1, e2, **kw):
return calculate_similarity(e1, e2, method="multi_factor", **kw)
def _pairwise_detection(entities, **kw):
return detect_duplicates(entities, method="pairwise", **kw)
def _keep_most_complete_merging(entities, **kw):
return merge_entities(entities, method="keep_most_complete", **kw)
def _graph_based_clustering(entities, **kw):
return build_clusters(entities, method="graph_based", **kw)
method_registry.register("similarity", "multi_factor", _multi_factor_similarity)
method_registry.register("detection", "pairwise", _pairwise_detection)
method_registry.register("merging", "keep_most_complete", _keep_most_complete_merging)
method_registry.register("clustering", "graph_based", _graph_based_clustering)
# Register default methods with registry.
# The public dispatch functions are registered directly so the identity guard
# in each function short-circuits the self-reference rather than going through
# an intermediate wrapper that re-enters the same dispatch path.
method_registry.register("similarity", "multi_factor", calculate_similarity)
method_registry.register("detection", "pairwise", detect_duplicates)
method_registry.register("merging", "keep_most_complete", merge_entities)
method_registry.register("clustering", "graph_based", build_clusters)
method_registry.register("detection", "triplets", dedup_triplets)
@@ -69,6 +69,15 @@ class EmbeddingGeneratorWithProvenance:
return embeddings
def __getattr__(self, name):
# __getattr__ only runs when normal lookup fails. Accessing
# self._generator by attribute syntax HERE would re-enter
# __getattr__ for ever when _generator itself is missing — the shape
# pickle/copy protocol probes hit when __init__ never completed
# (#994's RecursionError family). Fail fast on private probes.
if name.startswith("_"):
raise AttributeError(
f"{type(self).__name__!r} object has no attribute {name!r}"
)
return getattr(self._generator, name)
+8 -8
View File
@@ -117,9 +117,9 @@ def generate_embeddings(
>>> emb = generate_embeddings("Hello world", method="default")
>>> embs = generate_embeddings(["text1", "text2"], method="text")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("generation", method)
if custom_method:
if custom_method and custom_method is not generate_embeddings:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, data_type=data_type, fallback_on_custom_error=fallback, **kwargs
@@ -165,9 +165,9 @@ def embed_text(
>>> emb = embed_text("Hello world", method="sentence_transformers")
>>> embs = embed_text(["text1", "text2"], method="sentence_transformers")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("text", method)
if custom_method:
if custom_method and custom_method is not embed_text:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, text, fallback_on_custom_error=fallback, **kwargs
@@ -225,9 +225,9 @@ def calculate_similarity(
>>> similarity = calculate_similarity(emb1, emb2, method="cosine")
>>> print(f"Similarity: {similarity:.3f}")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("similarity", method)
if custom_method:
if custom_method and custom_method is not calculate_similarity:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, embedding1, embedding2, fallback_on_custom_error=fallback, **kwargs
@@ -272,9 +272,9 @@ def pool_embeddings(
>>> pooled = pool_embeddings(embeddings, method="mean")
>>> attention_pooled = pool_embeddings(embeddings, method="attention")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("pooling", method)
if custom_method:
if custom_method and custom_method is not pool_embeddings:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, embeddings, fallback_on_custom_error=fallback, **kwargs
+3 -13
View File
@@ -2,8 +2,9 @@
Semantica Explorer : FastAPI Dependencies
Provides ``Depends()``-compatible callables for injecting the
current ``GraphSession`` and ``ConnectionManager`` into route handlers,
and for enforcing API-key authentication on protected routes.
current ``GraphSession`` into route handlers, and for enforcing API-key
authentication on protected routes. WebSocket manager access is handled
directly via ``app.state.ws_manager``.
"""
import hmac
@@ -14,7 +15,6 @@ from fastapi import Request, HTTPException, Security, status
from fastapi.security.api_key import APIKeyHeader
from .session import GraphSession
from .ws import ConnectionManager
_api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)
@@ -80,13 +80,3 @@ def get_session(request: Request) -> GraphSession:
detail="GraphSession not initialized."
)
return request.app.state.session
def get_ws_manager(request: Request) -> ConnectionManager:
"""Retrieve the ConnectionManager stored on ``app.state``."""
if not hasattr(request.app.state, "ws_manager") or request.app.state.ws_manager is None:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="WebSocket manager not initialized.",
)
return request.app.state.ws_manager
-60
View File
@@ -78,66 +78,6 @@ def _parse_bbox(raw_bbox: Optional[str]) -> Optional[tuple[float, float, float,
return min_x, min_y, max_x, max_y
def _coerce_embedding_vector(value: object) -> Optional[List[float]]:
if isinstance(value, dict):
# Probe keys in priority order: generic first, then framework-specific.
# Must stay aligned with the top-level keys in _extract_node_embeddings.
for key in ("embedding", "embeddings", "vector", "values", "node2vec", "semantic"):
nested = _coerce_embedding_vector(value.get(key))
if nested is not None:
return nested
return None
if not isinstance(value, (list, tuple)):
return None
vector: List[float] = []
for item in value:
try:
vector.append(float(item))
except (TypeError, ValueError):
return None
return vector if vector else None
def _extract_node_embeddings(graph_dict: dict) -> dict[str, List[float]]:
"""Extract embeddings from graph dictionary."""
# Top-level keys to probe on each entity (and its metadata/properties dicts).
# Priority: generic names first, then KG-extras-specific names.
# Must stay aligned with the inner probe list in _coerce_embedding_vector.
embedding_keys = (
"embedding",
"embeddings",
"vector",
"node_embedding",
"node2vec_embedding",
"semantic_embedding",
"reasoning_embedding",
)
embeddings: dict[str, List[float]] = {}
for entity in graph_dict.get("entities") or graph_dict.get("nodes") or []:
if not isinstance(entity, dict):
continue
node_id = entity.get("id") or entity.get("node_id")
if not node_id:
continue
metadata = entity.get("metadata") if isinstance(entity.get("metadata"), dict) else {}
properties = entity.get("properties") if isinstance(entity.get("properties"), dict) else {}
for key in embedding_keys:
vector = _coerce_embedding_vector(
entity.get(key, metadata.get(key, properties.get(key)))
)
if vector is not None:
embeddings[str(node_id)] = vector
break
return embeddings
def _get_cached_embeddings(session: GraphSession) -> dict[str, List[float]]:
"""Get embeddings from session cache for optimal performance."""
return session.get_cached_embeddings()
-4
View File
@@ -456,10 +456,6 @@ class DraftResponse(BaseModel):
updated_at: str
class ProposalState(BaseModel):
state: Literal["draft", "proposed", "approved", "published", "rejected"]
class ProposalRequest(BaseModel):
draft_id: str
ontology_uri: str
-23
View File
@@ -8,11 +8,6 @@ from typing import Any, Dict, List, Literal, Optional, Tuple
from pydantic import BaseModel, Field, field_validator
class ErrorResponse(BaseModel):
detail: str
status_code: int = 500
class NodeResponse(BaseModel):
id: str
type: str
@@ -187,12 +182,6 @@ class ComplianceResponse(BaseModel):
violations: List[Dict[str, Any]] = Field(default_factory=list)
class TemporalSnapshotResponse(BaseModel):
timestamp: str
active_nodes: List[NodeResponse]
active_node_count: int
class TemporalDiffResponse(BaseModel):
from_time: str
to_time: str
@@ -256,13 +245,6 @@ class ExportRequest(BaseModel):
node_ids: Optional[List[str]] = None
class ExportResponse(BaseModel):
format: str
content_type: str
filename: str
size_bytes: int = 0
class ImportResponse(BaseModel):
status: str = "success"
message: str = "Import successful"
@@ -272,11 +254,6 @@ class ImportResponse(BaseModel):
edges_imported: Optional[int] = None
class StandardMessageResponse(BaseModel):
status: str
message: str
class AnnotationCreate(BaseModel):
node_id: str
content: str
+52 -8
View File
@@ -28,12 +28,35 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory, utc_now_iso, write_json_file
from ..utils.helpers import ensure_directory, hash_data, utc_now_iso, write_json_file
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri
def _content_iri(prefix: str, payload: Any) -> str:
"""Mint a document IRI from what was exported, not when.
Minting from ``utc_now_iso()`` gave every export of the same graph a new
identity a few microseconds apart, so re-exporting an unchanged graph was
never idempotent and merging exports duplicated every node (#1147). This
mirrors ``mint_entity_iri`` (#1109): identical content hashes to the same
IRI, and any change to the content changes it too. ``default=str`` keeps
the hash defined for values ``json.dumps`` would otherwise reject, such as
``datetime`` objects a caller may have left in the graph.
Args:
prefix: IRI prefix the digest is appended to
payload: JSON-serializable value whose content determines the digest
Returns:
A stable IRI of the form ``{prefix}{16-hex-char digest}``
"""
canonical = json.dumps(payload, sort_keys=True, default=str)
digest = hash_data(canonical)[:16]
return f"{prefix}{digest}"
def _is_jsonld_document(data: Dict[str, Any]) -> bool:
"""
Report whether a dictionary is already a JSON-LD document.
@@ -230,7 +253,10 @@ class JSONExporter:
- statistics: Statistics dictionary (optional)
file_path: Output JSON file path
format: Export format - 'json' or 'json-ld' (default: self.format)
**options: Additional options passed to conversion methods
**options: Additional options passed to conversion methods:
- graph_uri: Caller-supplied IRI for the graph node when
format='json-ld', overriding the default content-derived
IRI (see #1147)
Example:
>>> kg = {
@@ -401,7 +427,9 @@ class JSONExporter:
data: Data to convert (dict, list, or any value)
include_metadata: Whether to include metadata (default: True)
include_provenance: Whether to include provenance (default: True)
**options: Additional options passed to knowledge graph conversion
**options: Additional options passed to knowledge graph conversion:
- document_uri: Caller-supplied IRI for the document node,
overriding the default content-derived IRI (see #1147)
Returns:
Dictionary in JSON-LD format with @context, @graph/@value, and metadata
@@ -451,13 +479,17 @@ class JSONExporter:
# Add metadata and provenance if requested
if include_metadata:
self._attach_document_metadata(jsonld, include_provenance)
self._attach_document_metadata(
jsonld, include_provenance, options.get("document_uri")
)
return jsonld
@staticmethod
def _attach_document_metadata(
jsonld: Dict[str, Any], include_provenance: bool
jsonld: Dict[str, Any],
include_provenance: bool,
document_uri: Optional[str] = None,
) -> None:
"""
Attach the export's own metadata without naming the graph.
@@ -473,6 +505,9 @@ class JSONExporter:
Args:
jsonld: Document being built, modified in place
include_provenance: Whether to record how and when it was exported
document_uri: Caller-supplied IRI for the document node. Falls back
to a content-derived IRI (#1147) so re-exporting unchanged data
is idempotent instead of minting a new identity every time.
"""
# A caller may hand us a document that is deliberately a named graph.
# That name is theirs to keep, but our own statements must not end up
@@ -483,7 +518,10 @@ class JSONExporter:
# Do not overwrite an identifier the payload already carries: the
# knowledge-graph conversion names its own document node.
if "@id" not in jsonld or payload_is_named_graph:
document["@id"] = f"https://semantica.dev/data/{utc_now_iso()}"
content = {key: value for key, value in jsonld.items() if key != "@context"}
document["@id"] = document_uri or _content_iri(
"https://semantica.dev/data/", content
)
if include_provenance:
document["semantica:exportedAt"] = utc_now_iso()
document["semantica:format"] = "json-ld"
@@ -553,7 +591,9 @@ class JSONExporter:
- entities: List of entity dictionaries
- relationships: List of relationship dictionaries
- metadata: Metadata dictionary (optional)
**options: Additional options (unused)
**options: Additional options:
- graph_uri: Caller-supplied IRI for the graph node,
overriding the default content-derived IRI (see #1147)
Returns:
Dictionary in JSON-LD format with @context, @id, @type, and graph data
@@ -566,7 +606,11 @@ class JSONExporter:
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
},
"@id": f"https://semantica.dev/graph/{utc_now_iso()}",
# Minted from the graph's own content rather than the wall clock
# (#1147): re-exporting an unchanged graph must produce the same
# subject, or merging repeated exports duplicates every node.
"@id": options.get("graph_uri")
or _content_iri("https://semantica.dev/graph/", kg),
"@type": "semantica:KnowledgeGraph",
}
+547 -42
View File
@@ -29,9 +29,12 @@ Author: Semantica Contributors
License: MIT
"""
import re
from pathlib import Path
from decimal import Decimal, InvalidOperation
from html import escape as xml_escape
from typing import Any, Dict, List, Optional, Set, Union
from urllib.parse import quote, urlsplit
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory, hash_data
@@ -104,7 +107,11 @@ def normalize_confidence(value: Any) -> Optional[str]:
# "1e100000000" is eleven characters that expand to a hundred million, and
# the export path continues past validation errors, so a single malformed
# field could exhaust memory. Nothing near this magnitude is a confidence.
if not -MAX_CONFIDENCE_EXPONENT <= decimal_value.adjusted() <= MAX_CONFIDENCE_EXPONENT:
if (
not -MAX_CONFIDENCE_EXPONENT
<= decimal_value.adjusted()
<= MAX_CONFIDENCE_EXPONENT
):
return None
# `str(Decimal("0.00001"))` gives "0.00001", but a float that has already
@@ -139,6 +146,294 @@ def mint_relationship_iri(index: int, source: Any, target: Any) -> str:
return f"{SEMANTICA_NS}rel_{index}_{digest}"
#: The metadata keys Semantica itself produces, and the terms they are written
#: as. GraphBuilder.build_graph writes the first five, create_snapshot writes
#: snapshot_time, and load_from_neo4j writes source / uri / database. These are
#: Semantica's own vocabulary, so they are minted in the declared namespace and
#: declared in semantica-ns.ttl.
#:
#: A key the caller supplied is a different matter. Which namespace an
#: arbitrary metadata key belongs in is issue #1146, and until that is settled
#: the exporter refuses to guess: it warns and skips, and a caller who already
#: knows the answer passes ``metadata_terms``.
#:
#: The map is key -> term rather than key -> namespace because two of the keys
#: cannot keep their own name. ``source`` on a graph loaded from Neo4j is the
#: system it came from, while sem:source is already the ObjectProperty holding
#: the subject of a reified relationship; reusing it would put a string where
#: an entity belongs.
DEFAULT_METADATA_TERMS: Dict[str, str] = {
"num_entities": f"{SEMANTICA_NS}numEntities",
"num_relationships": f"{SEMANTICA_NS}numRelationships",
"temporal_enabled": f"{SEMANTICA_NS}temporalEnabled",
"entity_resolution_applied": f"{SEMANTICA_NS}entityResolutionApplied",
"timestamp": f"{SEMANTICA_NS}builtAt",
"snapshot_time": f"{SEMANTICA_NS}snapshotAt",
"source": f"{SEMANTICA_NS}sourceSystem",
"uri": f"{SEMANTICA_NS}sourceUri",
"database": f"{SEMANTICA_NS}sourceDatabase",
}
#: Terms whose value is a node rather than a string. Everything else stays a
#: literal: a metadata value that merely looks like a URL is not thereby a
#: reference to one.
IRI_VALUED_METADATA_TERMS: Set[str] = {f"{SEMANTICA_NS}sourceUri"}
_XSD_NS = "http://www.w3.org/2001/XMLSchema#"
def _escape_literal(value: str) -> str:
"""Escape a string for a Turtle or N-Triples quoted literal."""
return (
value.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\r", "\\r")
.replace("\t", "\\t")
)
def _escape_temporal_literal(value: Any) -> str:
"""Escape a temporal bound for a Turtle ``dateTimeStamp`` literal.
Bounds are normally strings, but callers may hand us a ``datetime`` or
``None``. ``_escape_literal`` is str-only, so stringify non-str values
first instead of calling ``.replace()`` on them; ``None`` yields an empty
bound rather than crashing. Datetimes must use ISO 8601 so the
``xsd:dateTimeStamp`` ``T`` separator is preserved ``str()`` yields a
space ("00:00:00+00:00"), which is a lexically invalid timestamp.
"""
if value is None:
return ""
if isinstance(value, str):
return _escape_literal(value)
if hasattr(value, "isoformat"):
return value.isoformat()
return str(value)
#: Turtle/N-Triples IRIREF grammar excludes these unescaped between `<` and
#: `>`: control characters, space, and <>"{}|^`\. An IRI-valued metadata
#: value (currently only sem:sourceUri, from the caller-controlled "uri"
#: metadata key) is written as `<{value}>` with no other quoting, so a value
#: containing one of these characters — a ">" followed by a full triple, for
#: instance — closes the IRIREF early and lets the rest of the string be
#: parsed as further RDF statements. This is the same shape of defect the
#: entity/relationship IRIs were hardened against; that hardening resolves
#: prefixes as well, which a metadata value never needs, so this stays a
#: narrower, dedicated guard rather than reusing _as_turtle_iri.
_IRI_REF_UNSAFE_RE = re.compile(r'[\x00-\x20<>"{}|^`\\]')
def _safe_iri_ref(value: str) -> str:
"""Percent-encode the characters an IRIREF may not contain unescaped."""
return _IRI_REF_UNSAFE_RE.sub(lambda m: quote(m.group(0), safe=""), value)
def _escape_xml(value: str) -> str:
"""Escape a string for either XML element text or an attribute value.
The quotes matter. This helper feeds `rdf:about`, `rdf:resource` and
`xmlns:` attribute values, which are delimited by double quotes, so a value
carrying one would close the attribute early and produce a document that
does not parse. Escaping them in element text as well is harmless and
means one helper cannot be used in the wrong place.
"""
return (
value.replace("&", "&amp;")
.replace("<", "&lt;")
.replace(">", "&gt;")
.replace('"', "&quot;")
.replace("'", "&apos;")
)
def _is_ncname(value: str) -> bool:
"""Whether a string can be an XML NCName, which is what RDF/XML requires.
Checked over the ASCII range rather than the full XML production: the
grammar also admits combining characters and extenders, so this is
deliberately conservative. It refuses names it could have accepted, and it
never accepts one that would produce a document a parser rejects. The
earlier check tested only that the first character was not a digit, which
let through every other way a local name can fail to be a name.
"""
if not value:
return False
if not (value[0].isascii() and (value[0].isalpha() or value[0] == "_")):
return False
return all(c.isascii() and (c.isalnum() or c in "._-") for c in value[1:])
def _split_iri(iri: str) -> Optional[tuple]:
"""Split an IRI into (namespace, local name) for RDF/XML's QName syntax.
Returns None when no split yields a usable local name. RDF/XML is the only
serialization here that cannot write an arbitrary predicate IRI, so this is
the one place a term can be unrepresentable, and the caller reports it
rather than dropping it quietly.
"""
for sep in ("#", "/"):
index = iri.rfind(sep)
if index != -1 and index + 1 < len(iri):
local = iri[index + 1 :]
if _is_ncname(local):
return iri[: index + 1], local
return None
def _metadata_statements(
metadata: Any,
terms: Dict[str, str],
logger: Any,
) -> List[tuple]:
"""Resolve a metadata mapping to a list of (term IRI, value) pairs.
A key with no term is skipped and reported. Silence is the defect this
fixes, so an unmapped key must be louder than a mapped one, not quieter.
"""
if not isinstance(metadata, dict):
return []
statements: List[tuple] = []
for key, value in metadata.items():
term = terms.get(key)
if term is None:
logger.warning(
"Metadata key %r has no term and was not exported. Which "
"namespace a caller-supplied key belongs in is issue #1146; "
"pass metadata_terms={%r: '<iri>'} to export it now.",
key,
key,
)
continue
if value is None:
continue
if isinstance(value, (dict, list, tuple, set)):
logger.warning(
"Metadata key %r holds a %s, which has no modelled RDF shape "
"yet, and was not exported.",
key,
type(value).__name__,
)
continue
statements.append((term, value))
return statements
def _resolve_metadata_terms(overrides: Optional[Dict[str, str]]) -> Dict[str, str]:
if not overrides:
return DEFAULT_METADATA_TERMS
return {**DEFAULT_METADATA_TERMS, **overrides}
def _typed_literal_parts(term: str, value: Any) -> tuple:
"""Return (kind, lexical, datatype) for one metadata value.
kind is "iri" or "literal". The lexical form and datatype are chosen once,
here, so that the four serializers cannot disagree about them the way they
disagreed about confidence in #1100.
"""
if term in IRI_VALUED_METADATA_TERMS and isinstance(value, str):
return "iri", value, None
if isinstance(value, bool):
return "literal", "true" if value else "false", f"{_XSD_NS}boolean"
if isinstance(value, int):
return "literal", str(value), f"{_XSD_NS}integer"
if isinstance(value, float):
# xsd:double, not xsd:decimal. `repr(1e-05)` is "1e-05" and
# `repr(float("nan"))` is "nan", and xsd:decimal admits neither the
# exponent form nor the special values, so typing a float as decimal
# produced lexicals a strict parser rejects. A Python float is an IEEE
# 754 double; xsd:double has legal lexicals for all of them, and it is
# also the honest claim, since nothing that arrived as a float was ever
# exact. `normalize_confidence` keeps xsd:decimal for confidence
# deliberately: that is a bounded score where exactness is meaningful
# and NaN is not a confidence at all.
if value != value:
lexical = "NaN"
elif value == float("inf"):
lexical = "INF"
elif value == float("-inf"):
lexical = "-INF"
else:
lexical = repr(value)
return "literal", lexical, f"{_XSD_NS}double"
return "literal", str(value), None
def _turtle_object(term: str, value: Any) -> str:
kind, lexical, datatype = _typed_literal_parts(term, value)
if kind == "iri":
return f"<{_safe_iri_ref(lexical)}>"
if datatype is None:
return f'"{_escape_literal(lexical)}"'
return f'"{lexical}"^^<{datatype}>'
def _turtle_metadata_clauses(statements: List[tuple]) -> List[str]:
return [f"<{term}> {_turtle_object(term, value)}" for term, value in statements]
def _ntriples_metadata_lines(subject: str, statements: List[tuple]) -> List[str]:
return [
f"<{subject}> <{term}> {_turtle_object(term, value)} ."
for term, value in statements
]
def _rdfxml_metadata_lines(
statements: List[tuple], indent: str, logger: Any = None
) -> List[str]:
"""RDF/XML needs a QName, so an unprefixed term declares its own prefix.
A term with no QName form has no RDF/XML representation at all, and this is
the only serialization with that restriction. Skipping it quietly would
reintroduce, in one format, exactly the silent metadata loss this module
was changed to stop, so it is reported and the other three formats still
carry the statement in full.
"""
lines: List[str] = []
for position, (term, value) in enumerate(statements):
split = _split_iri(term)
if split is None:
if logger is not None:
logger.warning(
"Term %r has no QName form, so it cannot be written in "
"RDF/XML and was omitted from that serialization only. "
"Turtle, N-Triples and JSON-LD carry it in full.",
term,
)
continue
namespace, local = split
kind, lexical, datatype = _typed_literal_parts(term, value)
prefix = f"md{position}"
opening = f'{indent}<{prefix}:{local} xmlns:{prefix}="{_escape_xml(namespace)}"'
if kind == "iri":
lines.append(f'{opening} rdf:resource="{_escape_xml(lexical)}"/>')
continue
if datatype is not None:
opening += f' rdf:datatype="{_escape_xml(datatype)}"'
lines.append(f"{opening}>{_escape_xml(lexical)}</{prefix}:{local}>")
return lines
def _jsonld_metadata_entries(statements: List[tuple]) -> Dict[str, Any]:
"""Absolute IRIs as keys, and explicit @value/@type rather than JSON's own
types: JSON's number is xsd:double, which would make the JSON-LD export
disagree with the other three about the datatype of an integer."""
entries: Dict[str, Any] = {}
for term, value in statements:
kind, lexical, datatype = _typed_literal_parts(term, value)
if kind == "iri":
entries[term] = {"@id": lexical}
elif datatype is None:
entries[term] = lexical
else:
entries[term] = {"@value": lexical, "@type": datatype}
return entries
class NamespaceManager:
"""
RDF namespace management engine.
@@ -395,6 +690,66 @@ class RDFSerializer:
# OWL-Time namespace URI
_OWL_TIME_NS = "http://www.w3.org/2006/time#"
_SEMANTICA_NS = "https://semantica.dev/ns#"
# Matches an already-valid percent-escape so it can be passed through
# unchanged instead of being re-encoded into e.g. %2520.
_PERCENT_ESCAPE_RE = re.compile(r"%[0-9A-Fa-f]{2}")
@classmethod
def _quote_preserving_escapes(cls, value: str, safe: str) -> str:
"""quote() that leaves existing valid %XX escapes untouched.
Blanket-quoting an absolute IRI double-encodes any percent-escape it
already carries (%20 -> %2520), which changes the identity of every
previously-valid IRI containing one. Only the spans between existing
valid escapes are quoted; a bare '%' that isn't part of a valid
escape (e.g. "%zz") still gets encoded to %25, keeping the malformed
case handled.
"""
parts = []
pos = 0
for match in cls._PERCENT_ESCAPE_RE.finditer(value):
parts.append(quote(value[pos : match.start()], safe=safe))
parts.append(match.group(0))
pos = match.end()
parts.append(quote(value[pos:], safe=safe))
return "".join(parts)
def _as_turtle_iri(
self, value: Any, namespaces: Optional[Dict[str, str]] = None
) -> str:
"""Return an absolute, safely encoded IRI for a Turtle resource."""
value = str(value)
try:
parsed = urlsplit(value)
except ValueError:
parsed = urlsplit("")
if parsed.scheme:
prefix, separator, local_name = value.partition(":")
# Built-in namespaces (semantica:, rdf:, rdfs:, owl:, ...) must
# always be resolvable, not only when the caller passes no
# namespaces of its own — otherwise a value like "semantica:Foo"
# resolves fine with no @context but stops resolving the moment
# any @context is present, since callers pass extract_namespaces()
# (context-only) here without merging in the built-ins.
effective_namespaces = {
**self.namespace_manager.namespaces,
**(namespaces or {}),
}
namespace = effective_namespaces.get(prefix)
if namespace and separator:
return self._quote_preserving_escapes(
namespace + local_name, safe=":/?#[]@!$&'()*+,;="
)
# A scheme with at least two characters is an absolute IRI,
# including opaque forms such as mailto:foo and isbn:0451450523.
# Keep one-character schemes as the existing Windows drive-path case.
if len(prefix) >= 2:
return self._quote_preserving_escapes(
value, safe=":/?#[]@!$&'()*+,;="
)
return self._SEMANTICA_NS + quote(value, safe="")
# Design decision — TemporalBound.OPEN in RDF:
# OWL-Time has no standard predicate for "no known end date." We use
@@ -434,6 +789,8 @@ class RDFSerializer:
"""
include_temporal: bool = options.pop("include_temporal", False)
time_axis: str = options.pop("time_axis", "valid")
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
lines = []
@@ -467,18 +824,32 @@ class RDFSerializer:
text = entity.get("text") or entity.get("label", "")
confidence = normalize_confidence(entity.get("confidence", 1.0))
lines.append(f"<{entity_id}> a <{entity_type}> ;")
clauses = [
f"a <{self._as_turtle_iri(entity_type, merged_namespaces)}>",
f'semantica:text "{_escape_literal(text)}"',
]
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
lines.append(f' semantica:text "{text}" .')
else:
lines.append(f' semantica:text "{text}" ;')
lines.append(
f' semantica:confidence "{confidence}"^^<{CONFIDENCE_DATATYPE}> .'
clauses.append(
f'semantica:confidence "{confidence}"^^<{CONFIDENCE_DATATYPE}>'
)
clauses.extend(
_turtle_metadata_clauses(
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
)
)
)
entity_iri = self._as_turtle_iri(entity_id, merged_namespaces)
lines.append(f"<{entity_iri}> {clauses[0]} ;")
for clause in clauses[1:-1]:
lines.append(f" {clause} ;")
lines.append(f" {clauses[-1]} .")
lines.append("")
# Convert relationships to RDF triplets
@@ -488,10 +859,16 @@ class RDFSerializer:
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", DEFAULT_RELATION_TYPE)
lines.append(f"<{source_id}> <{rel_type}> <{target_id}> .")
lines.append(
f"<{self._as_turtle_iri(source_id, merged_namespaces)}> "
f"<{self._as_turtle_iri(rel_type, merged_namespaces)}> "
f"<{self._as_turtle_iri(target_id, merged_namespaces)}> ."
)
if include_temporal:
owl_lines = self._owl_time_triples_for_rel(rel, idx, time_axis)
owl_lines = self._owl_time_triples_for_rel(
rel, idx, time_axis, merged_namespaces
)
if owl_lines:
# The interval hangs off the relationship's own IRI, and a
# relationship written as a single triple has no such node
@@ -501,11 +878,36 @@ class RDFSerializer:
# export, and every term is declared in the vocabulary.
lines.extend(
self._reified_relationship_triples(
rel, idx, source_id, target_id, rel_type
rel, idx, source_id, target_id, rel_type, merged_namespaces
)
)
lines.extend(owl_lines)
# Graph-level metadata needs a subject, and this serializer has never
# minted a document node. Rather than invent one here, it is written
# only when the caller names the graph; issue #1147 is where the
# default subject comes from once that lands.
graph_clauses = (
_turtle_metadata_clauses(
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
)
)
if graph_uri
else []
)
if graph_clauses:
graph_iri = self._as_turtle_iri(graph_uri, merged_namespaces)
lines.append("")
lines.append(
f"<{graph_iri}> {graph_clauses[0]} "
+ (";" if len(graph_clauses) > 1 else ".")
)
for clause in graph_clauses[1:-1]:
lines.append(f" {clause} ;")
if len(graph_clauses) > 1:
lines.append(f" {graph_clauses[-1]} .")
return "\n".join(lines)
def _reified_relationship_triples(
@@ -515,6 +917,7 @@ class RDFSerializer:
source_id: str,
target_id: str,
rel_type: str,
namespaces: Optional[Dict[str, str]] = None,
) -> List[str]:
"""
Emit the reified relationship node that OWL-Time triples hang off.
@@ -523,7 +926,11 @@ class RDFSerializer:
to, using the same sem:Relationship shape the JSON-LD export already
writes, so the two serializations describe relationships the same way.
"""
rel_id = rel.get("id") or mint_relationship_iri(idx, source_id or "", target_id or "")
rel_id = self._as_turtle_iri(
rel.get("id")
or mint_relationship_iri(idx, source_id or "", target_id or ""),
namespaces,
)
# The full predicate, not its local name. Truncating to the fragment
# made https://a.example/ns#employs and https://b.example/ns#employs the
@@ -537,17 +944,25 @@ class RDFSerializer:
.replace("\r", "\\r")
)
predicates = [f"a semantica:Relationship"]
predicates = ["a semantica:Relationship"]
if source_id:
predicates.append(f"semantica:source <{source_id}>")
predicates.append(
f"semantica:source <{self._as_turtle_iri(source_id, namespaces)}>"
)
if target_id:
predicates.append(f"semantica:target <{target_id}>")
predicates.append(
f"semantica:target <{self._as_turtle_iri(target_id, namespaces)}>"
)
predicates.append(f'semantica:type "{escaped}"')
return ["", f"<{rel_id}> " + " ;\n ".join(predicates) + " ."]
def _owl_time_triples_for_rel(
self, rel: Dict[str, Any], idx: int, time_axis: str
self,
rel: Dict[str, Any],
idx: int,
time_axis: str,
namespaces: Optional[Dict[str, str]] = None,
) -> List[str]:
"""
Emit OWL-Time Turtle triples for a relationship that carries temporal metadata.
@@ -562,7 +977,7 @@ class RDFSerializer:
def _is_open(v: Any) -> bool:
if v is None:
return False
if hasattr(v, "value"): # TemporalBound enum
if hasattr(v, "value"): # TemporalBound enum
return v.value == _OPEN_SENTINEL
return str(v).strip().upper() == _OPEN_SENTINEL
@@ -579,7 +994,10 @@ class RDFSerializer:
# deterministic IRI.
source_id = rel.get("source_id") or rel.get("source") or ""
target_id = rel.get("target_id") or rel.get("target") or ""
rel_base_id = rel.get("id") or mint_relationship_iri(idx, source_id, target_id)
rel_base_id = self._as_turtle_iri(
rel.get("id") or mint_relationship_iri(idx, source_id, target_id),
namespaces,
)
lines = [""] # blank separator
for axis_name, from_val, until_val in axes:
@@ -594,22 +1012,22 @@ class RDFSerializer:
lines.append(f" time:hasBeginning <{begin_id}> ;")
if _is_open(until_val):
lines.append(
' semantica:openEndedInterval "true"^^xsd:boolean .'
)
lines.append(' semantica:openEndedInterval "true"^^xsd:boolean .')
elif until_val is not None:
end_id = f"{rel_base_id}__{axis_name}_end"
lines.append(f" time:hasEnd <{end_id}> .")
lines.append(f"<{end_id}> a time:Instant ;")
lines.append(
f' time:inXSDDateTimeStamp "{until_val}"^^xsd:dateTimeStamp .'
f' time:inXSDDateTimeStamp "{_escape_temporal_literal(until_val)}"^^xsd:dateTimeStamp .'
)
else:
lines[-1] = lines[-1].rstrip(" ;") + " ." # close interval without hasEnd
lines[-1] = (
lines[-1].rstrip(" ;") + " ."
) # close interval without hasEnd
lines.append(f"<{begin_id}> a time:Instant ;")
lines.append(
f' time:inXSDDateTimeStamp "{from_val}"^^xsd:dateTimeStamp .'
f' time:inXSDDateTimeStamp "{_escape_temporal_literal(from_val)}"^^xsd:dateTimeStamp .'
)
lines.append("")
@@ -638,12 +1056,17 @@ class RDFSerializer:
... }
>>> rdfxml = serializer.serialize_to_rdfxml(rdf_data)
"""
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
lines = ['<?xml version="1.0" encoding="UTF-8"?>']
lines.append('<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"')
lines.append(' xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"')
lines.append(' xmlns:semantica="https://semantica.dev/ns#">')
lines.append("")
namespaces = self.namespace_manager.extract_namespaces(rdf_data)
# Convert entities to RDF/XML
entities = rdf_data.get("entities", [])
for entity in entities:
@@ -653,13 +1076,22 @@ class RDFSerializer:
entity_text = entity.get("text", "")
entity_id = mint_entity_iri(entity_text)
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
entity_type = entity.get("type") or DEFAULT_ENTITY_TYPE
text = entity.get("text") or entity.get("label", "")
confidence = normalize_confidence(entity.get("confidence", 1.0))
# RDF/XML syntax: rdf:Description with rdf:about
lines.append(f' <rdf:Description rdf:about="{entity_id}">')
lines.append(f' <rdf:type rdf:resource="{entity_type}"/>')
# Attribute values are delimited by quotes, and both of these
# are caller input. Element text is left alone deliberately: that
# is #1098, and it is being fixed on its own path.
entity_iri = xml_escape(
self._as_turtle_iri(entity_id, namespaces), quote=True
)
entity_type_iri = xml_escape(
self._as_turtle_iri(entity_type, namespaces), quote=True
)
lines.append(f' <rdf:Description rdf:about="{entity_iri}">')
lines.append(f' <rdf:type rdf:resource="{entity_type_iri}"/>')
lines.append(f" <semantica:text>{text}</semantica:text>")
if confidence is None:
self.logger.warning(
@@ -671,6 +1103,15 @@ class RDFSerializer:
f' <semantica:confidence rdf:datatype="{CONFIDENCE_DATATYPE}">'
f"{confidence}</semantica:confidence>"
)
lines.extend(
_rdfxml_metadata_lines(
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
),
" ",
self.logger,
)
)
lines.append(" </rdf:Description>")
lines.append("")
@@ -679,11 +1120,39 @@ class RDFSerializer:
for rel in relationships:
source_id = rel.get("source_id") or rel.get("source")
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", "semantica:related_to")
# RDF/XML predicates are emitted as QNames, unlike resource
# attributes which use the shared absolute-IRI normalizer.
rel_type = rel.get("type") or "semantica:related_to"
# Relationship as property on source entity
lines.append(f' <rdf:Description rdf:about="{source_id}">')
lines.append(f' <{rel_type} rdf:resource="{target_id}"/>')
source_iri = xml_escape(
self._as_turtle_iri(source_id, namespaces), quote=True
)
target_iri = xml_escape(
self._as_turtle_iri(target_id, namespaces), quote=True
)
lines.append(f' <rdf:Description rdf:about="{source_iri}">')
lines.append(f' <{rel_type} rdf:resource="{target_iri}"/>')
lines.append(" </rdf:Description>")
lines.append("")
graph_lines = (
_rdfxml_metadata_lines(
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
),
" ",
self.logger,
)
if graph_uri
else []
)
if graph_lines:
graph_iri = xml_escape(
self._as_turtle_iri(graph_uri, namespaces), quote=True
)
lines.append(f' <rdf:Description rdf:about="{graph_iri}">')
lines.extend(graph_lines)
lines.append(" </rdf:Description>")
lines.append("")
@@ -717,6 +1186,9 @@ class RDFSerializer:
"""
import json
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
# Initialize JSON-LD structure with context
jsonld = {
"@context": {
@@ -761,6 +1233,13 @@ class RDFSerializer:
"@value": confidence,
"@type": CONFIDENCE_DATATYPE,
}
node.update(
_jsonld_metadata_entries(
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
)
)
)
jsonld["@graph"].append(node)
# Convert relationships to JSON-LD
@@ -785,6 +1264,18 @@ class RDFSerializer:
}
)
graph_entries = (
_jsonld_metadata_entries(
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
)
)
if graph_uri
else {}
)
if graph_entries:
jsonld["@graph"].append({"@id": graph_uri, **graph_entries})
return json.dumps(jsonld, indent=2, ensure_ascii=False)
def serialize_to_ntriples(self, rdf_data: Dict[str, Any], **options) -> str:
@@ -801,22 +1292,17 @@ class RDFSerializer:
Returns:
String containing N-Triples serialization
"""
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
lines = []
namespaces = self.namespace_manager.extract_namespaces(rdf_data)
def expand_uri(uri: str) -> str:
if not uri:
return ""
if uri.startswith("http"):
return f"<{uri}>"
if uri.startswith("semantica:"):
return f"<https://semantica.dev/ns#{uri.split(':', 1)[1]}>"
if uri.startswith("rdf:"):
return f"<http://www.w3.org/1999/02/22-rdf-syntax-ns#{uri.split(':', 1)[1]}>"
if uri.startswith("rdfs:"):
return f"<http://www.w3.org/2000/01/rdf-schema#{uri.split(':', 1)[1]}>"
if ":" in uri:
return f"<{uri}>"
return f"<https://semantica.dev/ns#{uri}>"
return f"<{self._as_turtle_iri(uri, namespaces)}>"
# Convert entities
entities = rdf_data.get("entities", [])
@@ -830,7 +1316,7 @@ class RDFSerializer:
subject = expand_uri(entity_id)
# Type triple
entity_type = entity.get("type", "semantica:Entity")
entity_type = entity.get("type") or DEFAULT_ENTITY_TYPE
lines.append(
f"{subject} <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> {expand_uri(entity_type)} ."
)
@@ -838,7 +1324,7 @@ class RDFSerializer:
# Text property
text = entity.get("text") or entity.get("label", "")
if text:
safe_text = text.replace('"', '\\"').replace("\n", "\\n")
safe_text = _escape_literal(text)
lines.append(
f'{subject} {expand_uri("semantica:text")} "{safe_text}" .'
)
@@ -859,18 +1345,37 @@ class RDFSerializer:
f'"{confidence}"^^<{CONFIDENCE_DATATYPE}> .'
)
lines.extend(
_ntriples_metadata_lines(
subject.strip("<>"),
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
),
)
)
# Convert relationships
relationships = rdf_data.get("relationships", [])
for rel in relationships:
source_id = rel.get("source_id") or rel.get("source")
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", "semantica:related_to")
rel_type = rel.get("type") or DEFAULT_RELATION_TYPE
if source_id and target_id:
lines.append(
f"{expand_uri(source_id)} {expand_uri(rel_type)} {expand_uri(target_id)} ."
)
if graph_uri:
lines.extend(
_ntriples_metadata_lines(
graph_uri,
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
),
)
)
return "\n".join(lines)

Some files were not shown because too many files have changed in this diff Show More