mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into cookbook/prov-o-provenance

Browse Source
This commit is contained in:
Mohd Kaif
2026-08-26 19:50:39 +05:30
committed by GitHub
parent 1ce76055f5 8cc5d364db
commit 2de029ac8d
176 changed files with 16684 additions and 1883 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/DISCUSSION_TEMPLATE/ideas.md
+1 -1
View File
@@ -69,5 +69,5 @@ If you have ideas on how this could be implemented, please share.
---
**Note**: For feature requests that are ready to be implemented, consider creating a [Feature Request issue](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md) instead.
**Note**: For feature requests that are ready to be implemented, consider creating a [Feature Request issue](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md) instead.
.github/DISCUSSION_TEMPLATE/qa.md
+2 -2
View File
@@ -46,8 +46,8 @@ If applicable, paste any error messages or describe unexpected behavior:
## Checklist
- [ ] I have searched existing [discussions](https://github.com/Hawksight-AI/semantica/discussions) and [issues](https://github.com/Hawksight-AI/semantica/issues)
- [ ] I have checked the [documentation](https://github.com/Hawksight-AI/semantica/tree/main/docs) and [FAQ](https://github.com/Hawksight-AI/semantica/blob/main/docs/faq.md)
- [ ] I have searched existing [discussions](https://github.com/semantica-agi/semantica/discussions) and [issues](https://github.com/semantica-agi/semantica/issues)
- [ ] I have checked the [documentation](https://github.com/semantica-agi/semantica/tree/main/docs) and [FAQ](https://github.com/semantica-agi/semantica/blob/main/docs/faq.md)
- [ ] I have provided a minimal code example (if applicable)
- [ ] I have included error messages (if applicable)
- [ ] I have provided environment details
.github/FUNDING.yml
+1 -1
View File
@@ -1,3 +1,3 @@
# Funding options for Semantica
github: Hawksight-AI
github: semantica-agi
.github/ISSUE_TEMPLATE/config.yml
+2 -2
View File
@@ -1,8 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: 📚 Documentation
url: https://github.com/Hawksight-AI/semantica/tree/main/docs
url: https://github.com/semantica-agi/semantica/tree/main/docs
about: Browse the documentation
- name: 💬 Discussions
url: https://github.com/Hawksight-AI/semantica/discussions
url: https://github.com/semantica-agi/semantica/discussions
about: Ask questions and discuss with the community
.github/SUPPORT.md
+9 -9
View File
@@ -3,31 +3,31 @@
## Getting Help
### 📚 Documentation
Check the [docs folder](https://github.com/Hawksight-AI/semantica/tree/main/docs) and [README](https://github.com/Hawksight-AI/semantica/blob/main/README.md) for guides and examples.
Check the [docs folder](https://github.com/semantica-agi/semantica/tree/main/docs) and [README](https://github.com/semantica-agi/semantica/blob/main/README.md) for guides and examples.
### 💬 Community Support
- **GitHub Discussions**: [Ask questions](https://github.com/Hawksight-AI/semantica/discussions)
- **GitHub Discussions**: [Ask questions](https://github.com/semantica-agi/semantica/discussions)
- **Discord**: Join our [Discord server](https://discord.gg/sV34vps5hH) for real-time chat
### 💭 Discussions
Join the conversation on [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions):
Join the conversation on [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions):
- **Q&A**: Ask questions and get help from the community
- **Ideas**: Share feature requests and suggestions
- **Show and Tell**: Showcase your projects and use cases
- **General**: General discussions about Semantica
### 🐛 Bug Reports
Found a bug? [Create an issue](https://github.com/Hawksight-AI/semantica/issues/new/choose)
Found a bug? [Create an issue](https://github.com/semantica-agi/semantica/issues/new/choose)
### 📖 Resources
- [Quick Start Guide](https://github.com/Hawksight-AI/semantica/blob/main/docs/quickstart.md)
- [FAQ](https://github.com/Hawksight-AI/semantica/blob/main/docs/faq.md)
- [Cookbook Examples](https://github.com/Hawksight-AI/semantica/tree/main/cookbook)
- [Quick Start Guide](https://github.com/semantica-agi/semantica/blob/main/docs/quickstart.md)
- [FAQ](https://github.com/semantica-agi/semantica/blob/main/docs/faq.md)
- [Cookbook Examples](https://github.com/semantica-agi/semantica/tree/main/cookbook)
## Commercial Support
For enterprise support, custom development, or consulting services:
- Contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- Contact us through [GitHub Issues](https://github.com/semantica-agi/semantica/issues)
- Include "Commercial Support" in the title
## Sponsorship
@@ -35,7 +35,7 @@ For enterprise support, custom development, or consulting services:
### Sponsor this project
Support Semantica development:
- [GitHub Sponsors](https://github.com/sponsors/Hawksight-AI)
- [GitHub Sponsors](https://github.com/sponsors/semantica-agi)
Your sponsorship helps us:
- Maintain and improve the framework
CHANGELOG.md
+30 -1
View File
@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **First-class LangChain integration** (closes #963; recreates #969)
- New `pip install semantica[langchain]` extra (`langchain-core>=0.3.0`), included in the `all` bundle
- `integrations/langchain/SemanticaRetriever` — LangChain `BaseRetriever` that seeds from `HybridSearch` then walks graph edges (`hops=2` default) for GraphRAG-style retrieval; falls back to `ContextGraph.query` when hybrid search is unavailable
- `integrations/langchain/SemanticaVectorStore` — LangChain `VectorStore` adapter over `HybridSearch` (`add_texts`, `similarity_search`, `similarity_search_with_score`, `from_texts`)
- `integrations/langchain/SemanticaKGTool` / `SemanticaDecisionTool` — `BaseTool` subclasses with Pydantic `args_schema` (`semantica_query_graph`, `semantica_query_decisions`); `build()` returns the tool, or `None` when langchain-core is absent
- Retriever and VectorStore read HybridSearch nested `metadata` (`content`, `node_id`, `node_type`) rather than top-level fields that HybridSearch does not set
- All adapters remain importable without langchain-core (`LANGCHAIN_AVAILABLE` flag)
- Docs: `docs/integrations/langchain.md`, README native-integration matrix, and `docs.json` nav entry
## [0.6.6] - 2026-08-20
### Added
@@ -108,6 +119,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **The temporal-evolution `stability` metric was a hardcoded placeholder, not a duration**
- `TemporalGraphQuery.analyze_evolution()` documents `stability` as a "relationship duration/stability measure", but the implementation appended a constant `1` for every relationship with both `valid_from` and `valid_until` set (`durations.append(1) # Placeholder`). The reported stability was therefore always `1.0` when any bounded relationship existed and `0` otherwise — it never reflected how long relationships actually stayed valid, so it could not distinguish a graph of decade-long relationships from one of one-second relationships
- `stability` now computes the mean valid-time duration in seconds (`(valid_until - valid_from).total_seconds()`) across relationships that have both bounds set. Relationships with a missing or open `valid_from`/`valid_until` are skipped (their duration is unbounded), and non-positive intervals are clamped to `0`; an empty set still reports `0`
- New tests in `tests/kg/test_kg.py` assert the mean-duration result, the skipping of unbounded/half-open intervals, and the empty-graph zero case
- **Every timestamp an export or a provenance record wrote was timezone-naive** (closes #1114) by @fabio-rovai
- `semantica/export/` stamped with `datetime.now().isoformat()`, which reads the machine's **local** clock; `semantica/provenance/` stamped with `datetime.utcnow().isoformat()`, which reads **UTC**. Both produce a naive value and both serialize identically, so nothing downstream can tell which zone a given timestamp belongs to — the same string means two different instants depending on which module wrote it
- In RDF the consequence is silent rather than loud. Under XSD 1.1 a value with no timezone compared against one with a timezone is indeterminate whenever the two fall inside the ±14 hour window; SPARQL turns an indeterminate comparison into an error, and `FILTER` discards errors as non-matches. A timezone-qualified query over an Oxigraph store returns an answer with every Semantica-written record quietly absent from it, which is a poor property for `prov:generatedAtTime`, `prov:startedAtTime`, `prov:endedAtTime` and `prov:atTime` to have
@@ -395,8 +411,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Entities and relationships round-trip as memory-local provenance only — Markdown import intentionally does not write into `ContextGraph`, matching the MVP scope agreed on in #765
- Documented the file contract and workflow in `docs/reference/context.md`; 43 new tests in `tests/context/test_agent_memory_markdown.py` cover round-trip losslessness, idempotency, validation errors, rollback on failure, and vector-store sync ordering
- **Markdown directory round trips for `ContextGraph`** (#852) by @SaurabhScripts
- `ContextGraph.save_to_file(..., format="markdown")` and `load_from_file(..., format="markdown")` persist a deterministic `graph.md` relationship manifest plus one human-editable Markdown file per node, preserving graph, node, edge, family, temporal, and cross-graph link identities
- Imports validate the complete directory before replacing graph state, rebuild indexes and analytics state atomically, create JSON-compatible stub nodes for dangling edge endpoints, and emit the same granular node/edge audit events as JSON loading
- Existing exports are replaced atomically only after their complete canonical layout is validated; untracked files, renamed node files, symlinks, Windows directory junctions, and other reparse points cause a fail-closed error instead of authorizing directory deletion
- Added 30 focused tests covering deterministic round trips, manual edits, validation rollback, managed-directory identity, publish rollback, audit-manager compatibility, stale-cache clearing, mocked and real Windows junctions, and missing-path behavior
### Fixed
- **Markdown import followed filesystem links even though Markdown export already refused to overwrite them** (#851, follow-up to #765, #786) by @SaurabhScripts
- `AgentMemory._read_markdown_path()` now rejects symlink files, broken symlinks, symlinked directories, Windows directory junctions, and other Windows reparse points supplied directly; linked entries discovered inside an otherwise valid directory are safely skipped, preserving the current directory-import contract
- `_read_markdown_file_content()` re-checks the file and parent directory immediately before and after opening, uses `O_NOFOLLOW` where available, and verifies the resulting descriptor is a regular file via `fstat`/`S_ISREG`, so link swaps are rejected rather than silently followed
- Junction detection uses `os.path.isjunction()` where available and falls back to the Windows reparse-point file attribute on older Python versions; export applies the same link check before replacing a Markdown file
- Documented the import restriction in `docs/reference/context.md`; added 11 tests to `tests/context/test_agent_memory_markdown.py` covering file/directory/broken-symlink rejection, simulated open races, mocked and real Windows junctions, and the reparse-point fallback
- Any additional review follow-up commits land in this same PR/entry rather than as a separate changelog item
- **`PipelineWithProvenance` raised `ModuleNotFoundError` on import and `AttributeError` on `.run()`** (#858, closes #858) by @Karunasagar12
- `from .pipeline import Pipeline` failed because `semantica/pipeline/pipeline.py` does not exist; corrected to `from .pipeline_builder import Pipeline`
- `.run()` called `self._pipeline.run()` on the `Pipeline` dataclass, which has no such method; replaced with `self._engine.execute_pipeline(self._pipeline, ...)` delegating to `ExecutionEngine`
@@ -1513,4 +1542,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
For detailed release notes, see [GitHub Releases](https://github.com/Hawksight-AI/semantica/releases).
For detailed release notes, see [GitHub Releases](https://github.com/semantica-agi/semantica/releases).
CODE_OF_CONDUCT.md
+1 -1
View File
@@ -58,7 +58,7 @@ representative at an online or offline event.
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement through
[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[CoC]" prefix.
[GitHub Issues](https://github.com/semantica-agi/semantica/issues) with "[CoC]" prefix.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
CONTRIBUTORS.md
+3 -3
View File
@@ -44,7 +44,7 @@ We recognize all types of contributions:
All contributors are recognized in:
- This contributors list
- [GitHub contributors page](https://github.com/Hawksight-AI/semantica/graphs/contributors)
- [GitHub contributors page](https://github.com/semantica-agi/semantica/graphs/contributors)
- Release notes for significant contributions
- Community appreciation
@@ -54,7 +54,7 @@ All contributors are recognized in:
### Automatic Recognition
If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors).
If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/semantica-agi/semantica/graphs/contributors).
### Using All-Contributors Bot
@@ -111,4 +111,4 @@ Every contribution, no matter how small, helps make Semantica better. Thank you
**Want to contribute?**
⭐ Give us a Star • 🍴 [Fork us](https://github.com/Hawksight-AI/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
⭐ Give us a Star • 🍴 [Fork us](https://github.com/semantica-agi/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
Dockerfile
+1 -1
View File
@@ -9,7 +9,7 @@ RUN npm ci
COPY explorer/ ./
RUN mkdir -p /app/semantica && npm run build
FROM python:3.14-slim AS runtime
FROM python:3.13-slim AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
LICENSE
+1 -1
View File
@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2026 Hawksight AI
Copyright (c) 2026 Semantica
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
README.md
+31 -16
View File
@@ -87,7 +87,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
- **Drop-in Integrations:** Native Agno and CrewAI support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
- **Drop-in Integrations:** Native Agno, CrewAI, and LangChain support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
---
@@ -147,6 +147,8 @@ semantica doctor
# Config file pass ~/.semantica/config.yaml
```
**Running in a script or CI?** Progress bars are written only when stdout is an interactive terminal (or a Jupyter notebook), so piping and redirecting stay clean by default. Override with `SEMANTICA_DISABLE_PROGRESS=1` to silence progress everywhere, or `SEMANTICA_FORCE_PROGRESS=1` to keep it when stdout is redirected. `SEMANTICA_DISABLE_PROGRESS` takes precedence.
<div align="center">
If Semantica solves a real problem for you, a star helps others find it.
@@ -1186,7 +1188,7 @@ Start with `semantica`, verify with `doctor`, build a graph, and explore the com
## Integrations
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno and CrewAI support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno, CrewAI, and LangChain support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
@@ -1305,17 +1307,17 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<strong>CrewAI</strong><br/>
<sub>First-class · <code>pip install semantica[crewai]</code></sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langchain"><img src="https://github.com/langchain-ai.png?size=120" alt="LangChain" width="48" height="48" /></a><br/>
<strong>LangChain</strong><br/>
<sub>First-class · <code>pip install semantica[langchain]</code></sub>
</td>
</tr>
<tr>
<th colspan="8" align="left">Already Supported via REST API &amp; MCP</th>
</tr>
<tr>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langchain"><img src="https://github.com/langchain-ai.png?size=120" alt="LangChain" width="48" height="48" /></a><br/>
<strong>LangChain</strong><br/>
<sub>REST API · MCP</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langgraph"><img src="https://github.com/langchain-ai.png?size=120" alt="LangGraph" width="48" height="48" /></a><br/>
<strong>LangGraph</strong><br/>
<sub>REST API · MCP</sub>
@@ -1346,11 +1348,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
</tr>
<tr>
<td align="center" width="12.5%">
<a href="https://github.com/langchain-ai/langchain"><img src="https://github.com/langchain-ai.png?size=120" alt="LangChain" width="48" height="48" /></a><br/>
<strong>LangChain</strong><br/>
<sub>Dedicated toolkit</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/run-llama/llama_index"><img src="https://github.com/run-llama.png?size=120" alt="LlamaIndex" width="48" height="48" /></a><br/>
<strong>LlamaIndex</strong><br/>
<sub>Dedicated toolkit</sub>
@@ -1509,6 +1506,7 @@ pip install semantica[all] # everything
```bash
pip install semantica[agno] # Agno multi-agent integration
pip install semantica[crewai] # CrewAI integration
pip install semantica[langchain] # LangChain / LangGraph integration
pip install semantica[llm-litellm] # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
pip install semantica[graph-neo4j] # Neo4j graph store (LPG)
pip install semantica[graph-falkordb] # FalkorDB graph store (LPG)
@@ -1561,11 +1559,11 @@ On-premises deployment · Private cloud · Custom domain implementations · SLA-
## Star History
<a href="https://www.star-history.com/?repos=semantica-agi%2Fsemantica&type=date&legend=top-left">
<a href="https://star-history.dera.page/#semantica-agi/semantica&amp;type=date&amp;legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=semantica-agi/semantica&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=semantica-agi/semantica&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=semantica-agi/semantica&type=date&legend=top-left" />
<source media="(prefers-color-scheme: dark)" srcset="https://star-history.dera.page/svg?repos=semantica-agi/semantica&amp;type=date&amp;theme=dark&amp;legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://star-history.dera.page/svg?repos=semantica-agi/semantica&amp;type=date&amp;legend=top-left" />
<img alt="Star History Chart" src="https://star-history.dera.page/svg?repos=semantica-agi/semantica&amp;type=date&amp;legend=top-left" />
</picture>
</a>
@@ -1594,6 +1592,23 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines.
---
## Cite Us
If you use Semantica in your research or production systems, please cite it as:
```bibtex
@software{semantica2026,
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica}
}
```
All citation formats (APA, MLA, Chicago, IEEE) live on the [Citation](https://docs.getsemantica.ai/citation) page — every format attributes authorship to **Semantica**, not individual contributors.
---
<div align="center">
MIT License · Built by [Semantica](https://github.com/semantica-agi)
cookbook/advanced/01_Advanced_Extraction.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)\n",
"\n",
"# Advanced Extraction\n",
"\n",
cookbook/advanced/03_Complete_Visualization_Suite.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)\n",
"\n",
"# Complete Visualization Suite\n",
"\n",
cookbook/advanced/05_Multi_Format_Export.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)\n",
"\n",
"# Advanced Multi-Format Export\n",
"\n",
cookbook/advanced/08_Reasoning_and_Inference.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)\n",
"\n",
"# Reasoning and Inference\n",
"\n",
cookbook/advanced/09_Semantic_Layer_Construction.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/09_Semantic_Layer_Construction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/09_Semantic_Layer_Construction.ipynb)\n",
"\n",
"# Semantic Layer Construction\n",
"\n",
cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)\n",
"\n",
"# Deep Dive: Temporal Knowledge Graphs\n",
"\n",
cookbook/advanced/12_Unstructured_to_Ontology.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"\n",
"# Unstructured Text to Ontology\n",
"\n",
cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb
+1 -1
View File
@@ -18,7 +18,7 @@
"id": "cell-0",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb)\n",
"\n",
"# Manual Ontology + Snowflake Mapping\n",
"\n",
cookbook/advanced/14_Datalog_Style_Reasoning.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"\n",
"# Datalog-Style Reasoning\n",
"\n",
cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"\n",
"# Advanced Vector Store - Made Easy\n",
"\n",
@@ -352,7 +352,7 @@
"- Build a multi-user application\n",
"- Explore the [introduction notebook](../introduction/13_Vector_Store.ipynb) for more basics\n",
"\n",
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/Hawksight-AI/semantica)."
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/semantica-agi/semantica)."
]
}
],
cookbook/introduction/01_Welcome_to_Semantica.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
"\n",
"Semantica is a **semantic intelligence and knowledge engineering framework**. It helps you:\n",
"\n",
cookbook/introduction/02_Data_Ingestion.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
"\n",
"# Data Ingestion - Comprehensive Guide\n",
"\n",
cookbook/introduction/03_Document_Parsing.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/04_Document_Parsing.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)\n",
"\n",
"# Document Parsing\n",
"\n",
cookbook/introduction/04_Data_Normalization.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Data_Normalization.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/04_Data_Normalization.ipynb)\n",
"\n",
"# Data Normalization\n",
"\n",
cookbook/introduction/05_Entity_Extraction.ipynb
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
"\n",
"# Entity Extraction - Comprehensive Guide\n",
"\n",
@@ -622,7 +622,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
cookbook/introduction/06_Relation_Extraction.ipynb
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
"\n",
"# Relation Extraction - Comprehensive Guide\n",
"\n",
@@ -599,7 +599,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
cookbook/introduction/07_Building_Knowledge_Graphs.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Building_Knowledge_Graphs.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)\n",
"\n",
"# Building Knowledge Graphs\n",
"\n",
cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)\n",
"\n",
"# 🚀 Your First Knowledge Graph\n",
"\n",
cookbook/introduction/10_Graph_Analytics.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Graph_Analytics.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb)\n",
"\n",
"# Graph Analytics\n",
"\n",
cookbook/introduction/11_Chunking_and_Splitting.ipynb
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
"\n",
"# Chunking and Splitting - Comprehensive Guide\n",
"\n",
@@ -817,7 +817,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
cookbook/introduction/12_Embedding_Generation.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Embedding_Generation.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/12_Embedding_Generation.ipynb)\n",
"\n",
"# Embedding Generation\n",
"\n",
cookbook/introduction/13_Vector_Store.ipynb
+2 -2
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"\n",
"# Vector Store - Comprehensive Guide\n",
"\n",
@@ -492,7 +492,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
cookbook/introduction/14_Ontology.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
"\n",
"# Ontology Generation \n",
"\n",
cookbook/introduction/15_Export.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/15_Export.ipynb)\n",
"\n",
"# Export Module - Comprehensive Guide\n",
"\n",
cookbook/introduction/16_Visualization.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/17_Visualization.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)\n",
"\n",
"# Visualization\n",
"\n",
cookbook/introduction/18_Deduplication.ipynb
+1 -1
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
"\n",
"# Deduplication in Semantica\n",
"\n",
cookbook/introduction/19_Context_Module.ipynb
+1 -1
View File
@@ -5,7 +5,7 @@
"id": "c21e9c8d",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb)\n",
"\n",
"# Context Module — Practical Guide\n",
"\n",
docs/citation.md
+9 -10
View File
@@ -13,26 +13,25 @@ icon: "quote-left"
<Tab title="BibTeX">
```bibtex
@software{semantica2026,
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica},
version = {0.6.6},
doi = {10.5281/zenodo.XXXXXXX}
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica},
doi = {10.5281/zenodo.XXXXXXX}
}
```
</Tab>
<Tab title="APA">
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.6) \[Computer software\]. https://github.com/semantica-agi/semantica
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* \[Computer software\]. https://github.com/semantica-agi/semantica
</Tab>
<Tab title="MLA">
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6, GitHub, 2026, https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. GitHub, 2026, https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="Chicago">
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6. GitHub, 2026. https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. GitHub, 2026. https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="IEEE">
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.6, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
</Tab>
</Tabs>
docs/cookbook.md
+1 -1
View File
@@ -80,6 +80,6 @@ Deep dive into advanced features, customization, and complex workflows.
You can also run the cookbook using Docker:
```bash
docker run -p 8888:8888 hawksight/semantica-cookbook
docker run -p 8888:8888 semantica/semantica-cookbook
```
</Tip>
docs/docs.json
+1
View File
@@ -103,6 +103,7 @@
"pages": [
"integrations/agno",
"integrations/crewai",
"integrations/langchain",
"integrations/docling",
"integrations/snowflake",
"integrations/databricks"
docs/governance.md
+2 -2
View File
@@ -4,12 +4,12 @@ description: "Project governance model: roles, decision process, release cadence
icon: "scale-balanced"
---
> Semantica is maintained by Hawksight AI with community contributions under an open governance model.
> Semantica is maintained by the Semantica team with community contributions under an open governance model.
## Roles
- **Maintainers** — Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards.
- **Maintainers** — Semantica team: review and merge PRs, manage releases and code quality, set project direction and community standards.
- **Contributors** — Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md).
- **Community Members** — Use Semantica, provide feedback, share use cases, and participate in GitHub Discussions and Discord.
docs/guides/context-graphs.md
+29
View File
@@ -436,6 +436,35 @@ d = graph.to_dict()
# d["statistics"] → {"node_count": int, "edge_count": int}
```
For a human-editable, version-control-friendly representation, save a Markdown
directory instead:
```python
graph.save_to_file("context_graph/", format="markdown")
restored = ContextGraph(advanced_analytics=True)
restored.load_from_file("context_graph/", format="markdown")
```
The directory contains a versioned `graph.md` manifest for graph identity,
relationships, and cross-graph link descriptors, plus one file per node under
`nodes/`. A node's content is its Markdown body; its ID, type, properties,
metadata, and temporal validity are YAML frontmatter. Node, edge, family, graph,
and cross-graph link IDs are preserved across round trips.
Markdown loading uses replacement semantics, like `from_dict()`: it parses and
validates the complete directory before replacing the current graph. Invalid YAML,
duplicate IDs, unsupported versions, and unsafe filesystem links fail without
partially mutating the graph. As with JSON loading, an edge endpoint without a node
file creates an `entity` stub node. Symlinks, Windows directory junctions, and other
Windows reparse points are rejected.
Re-exporting to an existing managed directory atomically replaces it, removing stale
node files. Before replacement, Semantica validates the complete canonical export
layout, not just the manifest header. Untracked files, assets, extra directories, or
renamed node files therefore cause the export to fail closed instead of being deleted.
Keep attachments and hand-written indexes outside the managed export directory.
If the graph had cross-graph links created with `link_graph()`, call `resolve_links()` after loading to restore live navigation — object references cannot be serialized, so they must be reconnected manually:
```python
docs/guides/reasoning.md
+6 -15
View File
@@ -269,7 +269,7 @@ print("Loaded {} facts from graph".format(count))
## Step 5 — SPARQL queries over enriched working memory
After forward chaining has derived new facts, `SPARQLReasoner` lets you query the enriched working memory using SPARQL triple-pattern matching with optional inference expansion:
After forward chaining has derived new facts, `SPARQLReasoner` prepares SPARQL queries over the enriched working memory with optional inference expansion:
```python
from semantica.reasoning import SPARQLReasoner
@@ -288,22 +288,13 @@ query = """
}
"""
# execute_query() runs: expansion → inference → deduplication
result = sparql.execute_query(query)
for binding in result.bindings:
print("Actor: {:15s} CVE: {}".format(
binding.get("actor", "?"),
binding.get("cve", "?"),
))
# metadata shows how many results came from inference vs ground facts
print("Original: {} Inferred: {}".format(
result.metadata.get("original_count", 0),
result.metadata.get("inferred_count", 0),
))
# expand_query() applies inference rules to the query text:
expanded = sparql.expand_query(query)
print(expanded)
```
`execute_query()` is not implemented yet: no triplet-store execution path exists, so it raises `NotImplementedError` rather than returning an empty result set that callers would misread as "no matches". Until execution lands, run the expanded query against your RDF store directly (for example with `rdflib`).
Inspect the expanded query before running it:
```python
docs/guides/shacl-validation.md
+59 -21
View File
@@ -8,7 +8,7 @@ icon: "shield-check"
SHACL (Shapes Constraint Language) is a standard for validating graph-based data. While an ontology defines the conceptual *schema* (the "what" exists in your domain), SHACL defines the structural *rules and constraints* (the "how" it should be structured).
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and `_run_pyshacl` evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated.
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and the public `run_shacl_validation` function evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated. The historical `_run_pyshacl` name remains available as a compatibility alias.
## Why Use SHACL Validation?
@@ -55,7 +55,7 @@ Let's look at a simple, universally understood example: ensuring every `Employee
```python
from semantica.context import ContextGraph
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
# 1. Prepare your data graph
graph = ContextGraph()
@@ -95,7 +95,7 @@ data_ttl = """
"""
# 5. Run Validation
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
# 6. Analyze the Report
print(f"Graph conforms: {report.conforms}")
@@ -265,10 +265,10 @@ cve_id_shape = NodeShape(
## Step 4 — Run validation and read the report
Serialize the graph to RDF, then run `_run_pyshacl` against the shapes.
Serialize the graph to RDF, then run `run_shacl_validation` against the shapes.
```python
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
# Prepare your RDF data string (since export_rdf primarily exports structural metadata,
# you typically serialize your custom data graph to Turtle using rdflib or similar).
@@ -281,7 +281,7 @@ data_ttl = """
"""
# Run SHACL validation
report = _run_pyshacl(
report = run_shacl_validation(
data_ttl,
shacl_ttl,
data_graph_format="turtle",
@@ -366,8 +366,8 @@ print(f"Malware nodes missing 'family': {len(missing_family)}")
# e.g. graph.update_node(node_id, {"family": "UNKNOWN — requires triage"})
# After remediation, re-run validation to confirm the fix
# (re-export the patched graph to Turtle first, then call _run_pyshacl again)
report2 = _run_pyshacl(patched_data_ttl, shacl_ttl)
# (re-export the patched graph to Turtle first, then call run_shacl_validation again)
report2 = run_shacl_validation(patched_data_ttl, shacl_ttl)
print(f"Violations after remediation: {report2.violation_count}")
# Violations after remediation: 0
```
@@ -377,10 +377,49 @@ print(f"Violations after remediation: {report2.violation_count}")
## Common Pitfalls
- **Assuming the ontology automatically enforces data quality**: `SHACLGenerator` generates shapes based on what it observes in the data. If your data is missing a field, the generator won't know it was mandatory unless you explicitly inject the constraint (as shown in Step 3).
- **Passing `ContextGraph` directly to SHACL validators**: The `_run_pyshacl` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
- **Passing `ContextGraph` directly to SHACL validators**: The `run_shacl_validation` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
- **Forgetting RDF serialization**: You must serialize your graph (often via a temporary file using `export_rdf`) before validating it.
- **Treating validation as a one-time step**: Validation should be integrated as an automated step in your CI/CD pipeline or data ingestion flow, acting as a recurring gatekeeper rather than a one-off script.
- **Ignoring validation reports**: A graph that does not conform must be remediated. Failing to review the `violation_count` and address the issues negates the purpose of SHACL validation.
- **Validating `sh:class`/`sh:node` range checks on a property that declares `rdfs:range` with RDFS entailment on**: RDFS is an entailment rule, not a constraint. When pyshacl runs with `inference="rdfs"`, it infers the range class onto every object of the property, so class-based constraints on that property can never fail — the report says `conforms: True` on data that does not conform:
```python
from pyshacl import validate
from rdflib import Graph
data = Graph()
data.parse(
data="""
@prefix ex: <https://example.org/ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
ex:contains rdfs:domain ex:Container ; rdfs:range ex:Item .
ex:box a ex:Container ; ex:contains ex:notAnItem .
ex:notAnItem a ex:Fish .
""",
format="turtle",
)
shapes = Graph()
shapes.parse(
data="""
@prefix ex: <https://example.org/ns#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
ex:ContainerShape a sh:NodeShape ;
sh:targetClass ex:Container ;
sh:property [ sh:path ex:contains ; sh:class ex:Item ] .
""",
format="turtle",
)
for inference in ("none", "rdfs"):
conforms, _, _ = validate(data, shacl_graph=shapes, inference=inference)
print(inference, conforms)
# none False <- correct: notAnItem is a Fish, not an Item
# rdfs True <- the entailment manufactured the type
```
Mitigations: prefer not to declare `rdfs:range` on properties you intend to constrain with `sh:class`; when class membership is the thing under test, run validation without RDFS entailment (`inference="none"`); or express the check as a constraint the entailment cannot satisfy (for example a literal property constraint). Note the trade-off: with entailment off, `sh:targetClass` no longer reaches subclasses, so subclass hierarchies need explicit typing or inference-aware target selection. Semantica's own `run_shacl_validation` wrapper already calls pyshacl with `inference="none"`, so this pitfall only bites when calling `pyshacl.validate` directly with entailment enabled.
- **Trusting `conforms: True` without checking the inference mode**: an inference-enabled run can hide the exact violations the shapes were written to catch (see above). Record which inference mode validation ran under alongside the result, and re-run shape sets that contain `sh:class`/`sh:node` with entailment off before treating a pass as authoritative.
---
@@ -396,7 +435,7 @@ A DoD CTI team enforces STIX-compatible constraints on a threat graph before sha
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
graph = ContextGraph()
ctx = AgentContext(
@@ -448,7 +487,7 @@ data_ttl = """
<http://example.org/hammertoss> a ex:Malware .
"""
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"CTI graph conforms : {report.conforms}")
print(f"Violations : {report.violation_count}")
print(f"Warnings : {report.warning_count}")
@@ -469,7 +508,7 @@ A SOC team validates zero-trust policy nodes before publishing them to the polic
```python
from semantica.context import ContextGraph
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
graph = ContextGraph()
graph.add_node("policy-001", "Policy", "MFA Required for Tier-1 Resources",
@@ -516,7 +555,7 @@ data_ttl = """
<http://example.org/policy-002> a ex:Policy .
"""
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"Policy graph conforms: {report.conforms}")
# Policy graph conforms: False
@@ -534,7 +573,7 @@ A clinical informatics team validates trial ontology nodes before loading them i
```python
from semantica.ontology import LLMOntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
from semantica.export import export_rdf
import tempfile, os
@@ -586,7 +625,7 @@ with open(tmp.name) as f:
data_ttl = f.read()
os.unlink(tmp.name)
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"Trial data conforms: {report.conforms}")
print(f"Warnings : {report.warning_count}")
```
@@ -600,7 +639,7 @@ A credit risk team validates every `LoanApplication` node against Basel III CRE2
```python
from semantica.context import ContextGraph
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
graph = ContextGraph()
graph.add_node("loan-001", "LoanApplication", "Prime mortgage APP-2025-88421",
@@ -645,7 +684,7 @@ data_ttl = """
ex:ltv "0.65" .
"""
report = _run_pyshacl(data_ttl, shacl_ttl)
report = run_shacl_validation(data_ttl, shacl_ttl)
print(f"Loan portfolio conforms: {report.conforms}")
# Loan portfolio conforms: False
@@ -675,14 +714,14 @@ Call this function as a pre-publish gate; exit code 1 blocks the pipeline.
```python
import sys
from semantica.ontology import OntologyGenerator, SHACLGenerator
from semantica.ontology.ontology_validator import _run_pyshacl
from semantica.ontology import run_shacl_validation
def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
shacl_gen = SHACLGenerator(base_uri="https://example.org/shapes/")
shacl_graph = shacl_gen.generate(ontology)
shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
report = _run_pyshacl(data_graph_str, shacl_ttl)
report = run_shacl_validation(data_graph_str, shacl_ttl)
if not report.conforms:
print(f"Graph validation FAILED — {report.violation_count} violation(s)")
@@ -700,7 +739,6 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
- [Ontology Management](ontology) — generate the OWL ontology that SHACL shapes are derived from
- [Reasoning & Rules](reasoning) — complement SHACL structural constraints with logical inference rules
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `_run_pyshacl` input
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `run_shacl_validation` input
- [Conflict Resolution](conflict-resolution) — detect and resolve data conflicts before SHACL validation
- [Change Management](change-management) — version-gate SHACL shapes alongside ontology versions
docs/integrations/langchain.md
+81
View File
@@ -0,0 +1,81 @@
---
title: "LangChain Integration"
description: "Drop Semantica into LangChain / LangGraph pipelines via a GraphRAG retriever, VectorStore adapter, and agent tools."
icon: "link"
---
> Three drop-in adapters that bring Semantica's context graph and hybrid search into LangChain chains and LangGraph agents.
## Installation
```bash
pip install "semantica[langchain]"
```
Requires `langchain-core >= 0.3`. If langchain-core is not installed, the integration still imports — every class carries the full Semantica API and degrades gracefully (`build()` returns `None`; branch on `LANGCHAIN_AVAILABLE`).
## Components at a Glance
- **SemanticaRetriever** — `BaseRetriever`: hybrid-search seeds retrieval, then graph edges are walked `hops` steps (default 2) for GraphRAG-style results.
- **SemanticaVectorStore** — `VectorStore`: `add_texts` / `similarity_search` / `similarity_search_with_score` / `from_texts` over `HybridSearch`.
- **SemanticaKGTool** / **SemanticaDecisionTool** — `BaseTool` subclasses: `semantica_query_graph` and `semantica_query_decisions` for LangGraph / tool-calling agents.
## Component Details
<Tabs>
<Tab title="SemanticaRetriever">
Hybrid search seeds retrieval; then graph edges are walked `hops` steps so results go beyond flat vector similarity. If hybrid search is omitted or fails, the retriever falls back to a `ContextGraph.query` keyword scan.
```python
from integrations.langchain import SemanticaRetriever
from semantica.context import ContextGraph
from semantica.vector_store import HybridSearch
graph = ContextGraph()
hybrid = HybridSearch()
retriever = SemanticaRetriever(graph=graph, hybrid=hybrid, hops=2, top_k=10)
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
```
</Tab>
<Tab title="SemanticaVectorStore">
Drop-in `VectorStore` for RetrievalQA / LCEL chains. `from_texts` requires a pre-configured `hybrid` instance.
```python
from integrations.langchain import SemanticaVectorStore
store = SemanticaVectorStore(hybrid=hybrid)
store.add_texts(
["document one", "document two"],
metadatas=[{"source": "a"}, {"source": "b"}],
)
docs = store.similarity_search("document", k=2)
docs, scores = store.similarity_search_with_score("document", k=2)
```
`add_texts` delegates to a Semantica vector store with `add_documents` (pass `vector_store=` to `HybridSearch` or to `SemanticaVectorStore`).
</Tab>
<Tab title="Agent tools">
Instances are LangChain `BaseTool`s and can be passed to an agent directly.
`.build()` returns the tool, or `None` when langchain-core is absent.
```python
from integrations.langchain import SemanticaKGTool, SemanticaDecisionTool
from langgraph.prebuilt import create_react_agent
tools = [
SemanticaKGTool(graph),
SemanticaDecisionTool(graph),
]
agent = create_react_agent(model, tools)
```
| Tool | Description |
| :------ | :------------- |
| `semantica_query_graph` | Keyword / NL query over the shared context graph |
| `semantica_query_decisions` | Search the recorded decision log |
</Tab>
</Tabs>
docs/project-license.md
+1 -1
View File
@@ -12,7 +12,7 @@ icon: "file-contract"
```
MIT License
Copyright (c) 2026 Hawksight AI
Copyright (c) 2026 Semantica
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
docs/reference/context.md
+6 -4
View File
@@ -435,8 +435,8 @@ print("Nodes: {}, Edges: {}".format(stats["node_count"], stats["edge_count"]))
| `query(query, skip, limit)` | `List[Dict]` | Full-text search over node content |
| `stats()` | `Dict` | Node/edge counts, type breakdowns, graph density |
| `density()` | `float` | Graph density score |
| `save_to_file(path)` | `None` | Persist graph to JSON |
| `load_from_file(path)` | `None` | Load graph from JSON |
| `save_to_file(path, format="json")` | `None` | Persist graph as JSON or a Markdown directory |
| `load_from_file(path, format="json")` | `None` | Replace graph state from JSON or a Markdown directory |
| `build_from_conversations(conversations, link_entities)` | `Dict` | Build graph from conversation data |
| `link_graph(other_graph, source_node_id, target_node_id, link_type)` | `str` | Create cross-graph navigation link; returns `link_id` |
| `navigate_to(link_id)` | `Tuple` | Follow a cross-graph link to `(target_graph, target_node_id)` |
@@ -625,8 +625,10 @@ malformed or duplicate fields before changing memory, and re-importing unchanged
files is idempotent. Memory-local `entities` and `relationships` are preserved as
provenance but are not applied to `ContextGraph` by Markdown import. Use a dedicated
export directory: matching files are overwritten, but unrelated or stale Markdown
files are not deleted automatically. Export refuses to overwrite symbolic links and
uses atomic file replacement. Timestamp offsets are preserved in Markdown and
files are not deleted automatically. Export refuses to overwrite filesystem links and
uses atomic file replacement; import also refuses symlinks, Windows directory
junctions, and other Windows reparse points.
Timestamp offsets are preserved in Markdown and
normalized to UTC only for comparisons, so aware and local-naive records can be
queried together safely. Vector-store writes are deferred until the in-memory import
commits; adapter synchronization remains best-effort and logs failures.
docs/reference/triplet_store.md
+1 -1
View File
@@ -182,7 +182,7 @@ for row in result.bindings:
store = TripletStore(
backend="rdf4j",
endpoint="http://localhost:8080/rdf4j-server",
repository_id="semantica", # passed through **config
repository_id="semantica", # selects the remote repository
)
```
docs/reference/utils.md
+10
View File
@@ -77,7 +77,17 @@ Most users won't call utils directly: it's the **shared foundation** for all mod
export SEMANTICA_LOG_LEVEL=DEBUG
export SEMANTICA_LOG_FORMAT=json # "json" | "text"
export SEMANTICA_DISABLE_PROGRESS=true
export SEMANTICA_FORCE_PROGRESS=true
```
<Tip>
**Progress bars follow your terminal.** Console progress is written only when
stdout is an interactive terminal (or a Jupyter notebook), so piping or
redirecting output no longer fills logs with progress bars and escape
sequences. Set `SEMANTICA_DISABLE_PROGRESS` to silence progress even in a
terminal, or `SEMANTICA_FORCE_PROGRESS` to keep it when stdout is redirected.
`SEMANTICA_DISABLE_PROGRESS` wins if both are set.
</Tip>
</Step>
</Steps>
docs/storage-backends.md
+154
View File
@@ -0,0 +1,154 @@
# Graph storage backends and feature matrix
Semantica separates graph modeling from physical storage. LPG backends are accessed through `graph_store` adapters; RDF backends are accessed through `triplet_store` adapters.
This page is intentionally conservative: it distinguishes between an adapter existing, a feature being generally available with that model, and a backend needing user-supplied wiring.
## Status labels
- `built-in`: adapter implementation exists in Semantica core.
- `tested`: covered by automated integration fixtures or tests.
- `example-only`: usable example exists, but support is not asserted by integration tests.
- `interface/BYO`: interface or integration point exists; bring your own backend wiring.
## Adapter inventory
| Backend | Model | Adapter | Status | Reference |
| --- | --- | --- | --- | --- |
| Neo4j | LPG | `semantica.graph_store.Neo4jStore` | built-in | `cookbook/introduction/09_Graph_Store.ipynb` |
| FalkorDB | LPG | `semantica.graph_store.FalkorDBStore` | built-in | `docs/reference/graph_store.md` |
| Amazon Neptune | LPG | `semantica.graph_store.AmazonNeptuneStore` | built-in | `cookbook/introduction/21_Amazon_Neptune_Store.ipynb` |
| Apache AGE | LPG | `semantica.graph_store.ApacheAgeStore` | built-in | `docs/graph_stores/apache_age.md` |
| RDF4J | RDF | `semantica.triplet_store.RDF4JStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Apache Jena | RDF | `semantica.triplet_store.JenaStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Blazegraph | RDF | `semantica.triplet_store.BlazegraphStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Anzo | RDF | `semantica.triplet_store.AnzoStore` | built-in | `cookbook/introduction/20_Triplet_Store.ipynb` |
| Oxigraph | RDF | `semantica.triplet_store.OxigraphStore` | built-in | `docs/reference/triplet_store.md` |
## Feature matrix
`Yes` means the capability is expected to work with the adapter and graph model. `Partial` means the capability works with model-specific constraints. `BYO` means the user must supply or validate wiring for the backend.
| Backend | Model | Ingestion | Context graph construction | Reasoning/analytics | Provenance | Known limitations |
| --- | --- | --- | --- | --- | --- | --- |
| Neo4j | LPG | Yes | Yes | Yes | Partial | Provenance and context metadata are stored as node and edge properties; relationship properties and stable node identifiers are required. |
| FalkorDB | LPG | Yes | Yes | Partial | Partial | Redis-based; provenance depends on node/edge properties, and multi-graph isolation depends on the selected graph name. |
| Amazon Neptune | LPG | Yes | Yes | Partial | Partial | Use the property-graph endpoint; AWS auth, VPC, and endpoint configuration can affect local tests. Provenance depends on node/edge properties. |
| Apache AGE | LPG | Yes | Yes | Partial | Partial | Runs through PostgreSQL/AGE; Cypher compatibility and property handling can differ from standalone LPG engines. |
| RDF4J | RDF | Yes | Partial | Partial | Partial | Context separation relies on named graphs; triple-level provenance may require reification or graph-level metadata. |
| Apache Jena | RDF | Yes | Partial | Partial | Partial | Named graphs are needed for context separation; backend configuration and transaction behavior matter. |
| Blazegraph | RDF | Yes | Partial | Partial | Partial | Use quads/named graphs for context; IRI stability and graph naming matter for provenance. |
| Anzo | RDF | Yes | Partial | Partial | Partial | Anzo deployments are environment-specific; validate `dataset_uri`/graphmart naming, named-graph support, and provenance mapping. |
| Oxigraph | RDF | Yes | Partial | Partial | Partial | Embedded, single-process store (in-memory or on-disk); named graphs are supported, but there is no separate server process to scale independently. |
## RDF and LPG differences
- LPG backends store context and provenance as graph elements and properties. If a backend does not support relationship properties, some provenance patterns may be degraded.
- RDF backends rely on IRIs, named graphs, and optional reification. Context graphs and provenance are easiest to preserve when the store supports named graphs/quads.
- Ingestion works across both models, but the physical representation differs: LPG stores nodes/edges directly, while RDF stores subject-predicate-object statements.
- Reasoning and analytics should be validated against the adapter's query capabilities, especially for path traversal, property filters, and named-graph queries.
## Minimal connection examples
Prefer the referenced notebook cells for a working setup. The examples below show the intended adapter entrypoints, not a universal connection DSL.
### Neo4j
```python
import os
from semantica.graph_store import Neo4jStore
store = Neo4jStore(
uri='bolt://localhost:7687',
user='neo4j',
password=os.environ['NEO4J_PASSWORD']
)
```
### FalkorDB
```python
from semantica.graph_store import FalkorDBStore
store = FalkorDBStore(
host='localhost',
port=6379,
graph_name='semantica'
)
```
### Amazon Neptune
```python
from semantica.graph_store import AmazonNeptuneStore
store = AmazonNeptuneStore(
endpoint='your-neptune-cluster-endpoint',
port=8182,
region='us-east-1'
)
```
### Apache AGE
```python
from semantica.graph_store import ApacheAgeStore
store = ApacheAgeStore(
connection_string='host=localhost dbname=agedb user=postgres password=postgres',
graph_name='semantica'
)
```
### RDF4J
```python
from semantica.triplet_store import RDF4JStore
store = RDF4JStore(
endpoint='http://localhost:8080/rdf4j-server',
repository_id='semantica'
)
```
### Apache Jena
```python
from semantica.triplet_store import JenaStore
store = JenaStore(
endpoint='http://localhost:3030/ds'
)
```
### Blazegraph
```python
from semantica.triplet_store import BlazegraphStore
store = BlazegraphStore(
endpoint='http://localhost:9999/blazegraph/sparql'
)
```
### Anzo
```python
from semantica.triplet_store import AnzoStore
store = AnzoStore(
endpoint='http://anzo-host:8080',
dataset_uri='http://cambridgesemantics.com/Graphmart/your-graphmart-id'
)
```
### Oxigraph
```python
from semantica.triplet_store import OxigraphStore
# Omit `path` for an in-memory store; pass a directory for on-disk persistence.
store = OxigraphStore(path='./semantica-oxigraph-data')
```
Replace hostnames, ports, repositories, graphs, and credentials with values from your environment. For regulated or self-hosted deployments, keep credentials in environment variables or secret storage rather than source code.
explorer/package-lock.json
Generated
+1489 -14
View File
File diff suppressed because it is too large Load Diff
explorer/package.json
+3 -1
View File
@@ -9,7 +9,7 @@
"lint": "eslint .",
"preview": "vite preview",
"test:graph-store": "node --test tests/graphStore.multi-edge.test.mjs",
"test:graph-workspace": "node --import tsx --test tests/graphSceneState.display.test.ts tests/temporalLifecycle.test.ts",
"test:graph-workspace": "node --import tsx --test tests/markdownContentViewer.test.ts tests/graphSceneState.display.test.ts tests/temporalLifecycle.test.ts",
"test:plugin-registry": "node --import tsx --test tests/pluginRegistry.temporal.test.mjs"
},
"dependencies": {
@@ -29,6 +29,8 @@
"react-arborist": "^3.4.3",
"react-dom": "^19.2.4",
"react-dropzone": "^15.0.0",
"react-markdown": "^10.1.0",
"remark-gfm": "^4.0.1",
"sigma": "^3.0.2",
"vis-data": "^8.0.3",
"vis-timeline": "^8.5.0"
explorer/src/workspaces/GraphWorkspace/GraphInspectorPanel.tsx
+20
View File
@@ -3,6 +3,7 @@ import { Loader2 } from "lucide-react";
import { graph } from "../../store/graphStore";
import { GRAPH_THEME, withAlpha } from "./graphTheme";
import type { GraphSelectedNodeKind } from "./types";
import { MarkdownContentViewer } from "./MarkdownContentViewer";
export type LinkPrediction = {
target: string;
@@ -364,6 +365,11 @@ export function GraphInspectorPanel({
([key]) =>
!["x","y","valid_from","valid_until","content","source","source_url","pmid","pmids","evidence","provenance","confidence"].includes(key),
);
const nodeContent = (typeof attributes?.content === "string" && attributes.content)
? attributes.content
: (typeof properties.content === "string" && properties.content)
? properties.content
: "";
return (
<aside style={{ padding: 24, display: "flex", flexDirection: "column", gap: 18 }}>
@@ -408,6 +414,20 @@ export function GraphInspectorPanel({
</div>
) : null}
{/* Content Section — only rendered when the node carries actual content.
This matches the existing inspector convention: sections that have no
data for the current node are either hidden (temporal bounds) or closed
by default (Source Attribution, Properties). Always showing an open
empty panel would add noise for every relationship/predicate node. */}
{nodeContent && (
<details className="node-panel-collapse" open>
<summary className="node-panel-summary">Content</summary>
<div className="node-panel-body" style={{ marginTop: 8 }}>
<MarkdownContentViewer content={nodeContent} />
</div>
</details>
)}
{/* Actions */}
<section style={sectionStyle}>
<div style={sectionTitleStyle}>Actions</div>
explorer/src/workspaces/GraphWorkspace/MarkdownContentViewer.tsx
+404
View File
@@ -0,0 +1,404 @@
import { useState, useRef, useEffect, useMemo, type CSSProperties } from "react";
import ReactMarkdown, { type Components } from "react-markdown";
import remarkGfm from "remark-gfm";
import { Check, Copy, Code2, Eye, ExternalLink, Image as ImageIcon } from "lucide-react";
import { GRAPH_THEME } from "./graphTheme";
import { isSafeUrl } from "./markdownUrlSafety";
export interface MarkdownContentViewerProps {
content?: string | null;
className?: string;
defaultMode?: "preview" | "source";
}
export function MarkdownContentViewer({
content,
className,
defaultMode = "preview",
}: MarkdownContentViewerProps) {
const [activeMode, setActiveMode] = useState<"preview" | "source">(defaultMode);
const [copied, setCopied] = useState(false);
// Track the content value for which the copied indicator is valid.
// When content changes (i.e. the user selects a different node), reset the
// copied indicator inline during render rather than in a useEffect — this
// avoids a cascading-render lint error and is the React-recommended pattern
// for resetting derived visual state on prop changes.
const [copiedForContent, setCopiedForContent] = useState<string | null | undefined>(content);
if (copiedForContent !== content) {
setCopiedForContent(content);
if (copied) {
// Clear the stale indicator synchronously so the new node's copy button
// never shows "Copied" from the previous selection.
setCopied(false);
}
}
const copyTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// Clean up any outstanding timeout on unmount.
useEffect(() => {
return () => {
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current);
}
};
}, []);
const rawContent = typeof content === "string" ? content : "";
const hasContent = rawContent.trim().length > 0;
// react-markdown runs the whole remark pipeline synchronously inside its own
// render, so without this memo every unrelated re-render of this component --
// clicking Copy, toggling Preview/Source -- re-parses the entire document.
// Measured at ~364ms per re-render for a 1000-row GFM table (issue #1118).
// Keyed on rawContent so a genuine node change still re-parses exactly once.
const renderedMarkdown = useMemo(
() => (
<ReactMarkdown remarkPlugins={REMARK_PLUGINS} components={MARKDOWN_COMPONENTS}>
{rawContent}
</ReactMarkdown>
),
[rawContent],
);
const handleCopy = async () => {
if (!hasContent) return;
try {
await navigator.clipboard.writeText(rawContent);
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current);
}
setCopied(true);
copyTimeoutRef.current = setTimeout(() => setCopied(false), 1500);
} catch {
// Clipboard write unavailable
}
};
return (
<div className={className} style={viewerContainerStyle}>
<div style={viewerHeaderStyle}>
<div style={{ display: "flex", gap: 4 }} role="tablist">
<button
type="button"
role="tab"
aria-selected={activeMode === "preview"}
onClick={() => setActiveMode("preview")}
style={{ ...tabBtnStyle, ...(activeMode === "preview" ? activeTabBtnStyle : {}) }}
>
<Eye size={12} style={{ marginRight: 5 }} />
Preview
</button>
<button
type="button"
role="tab"
aria-selected={activeMode === "source"}
onClick={() => setActiveMode("source")}
style={{ ...tabBtnStyle, ...(activeMode === "source" ? activeTabBtnStyle : {}) }}
>
<Code2 size={12} style={{ marginRight: 5 }} />
Source
</button>
</div>
{hasContent && (
<button type="button" onClick={() => void handleCopy()} style={copyBtnStyle} title="Copy raw content">
{copied ? (
<>
<Check size={12} color="#3fb950" style={{ marginRight: 4 }} />
<span style={{ color: "#3fb950", fontSize: 11 }}>Copied</span>
</>
) : (
<>
<Copy size={12} style={{ marginRight: 4 }} />
<span style={{ fontSize: 11 }}>Copy</span>
</>
)}
</button>
)}
</div>
<div style={viewerBodyStyle}>
{!hasContent ? (
<div style={emptyTextStyle}>No content available for this node.</div>
) : activeMode === "source" ? (
<pre style={sourcePreStyle}>
<code style={sourceCodeStyle}>{rawContent}</code>
</pre>
) : (
<div style={previewStyle}>{renderedMarkdown}</div>
)}
</div>
</div>
);
}
/* ─── Markdown rendering config ───────────────────────────────────── */
// Both props are hoisted to module scope so they keep a stable identity across
// renders. As inline literals they allocated a fresh plugin array and ~20 fresh
// arrow components on every render, which made React treat every mapped tag as a
// new element type and remount the entire rendered subtree instead of updating
// it (issue #1118). The arrow bodies only read the style constants below at call
// time, so declaring the map before them is safe.
const REMARK_PLUGINS = [remarkGfm];
const MARKDOWN_COMPONENTS: Components = {
// C-1: react-markdown passes a HAST `node` prop (the raw AST
// Element) to every custom component override via passNode:true.
// In React 19 any unknown prop spreads onto a native element are
// serialised as HTML attributes, producing node="[object Object]"
// on every rendered link. Fix: destructure `node` by name so it
// is explicitly discarded, then spread `...rest` to preserve all
// other legitimate HAST/remark-gfm attributes — e.g. the `id`,
// `aria-describedby`, `aria-label`, `data-footnote-ref`,
// `data-footnote-backref`, and `class` attrs that GFM footnotes
// require for correct in-page navigation and accessibility.
//
// C-2: fragment links (#anchor, GFM footnote backlinks) must
// navigate within the current document. External links continue
// to use target="_blank" with noopener noreferrer.
//
// eslint-disable-next-line @typescript-eslint/no-unused-vars
a: ({ href, children, title, node: _node, ...rest }) => {
if (!isSafeUrl(href)) {
return <span style={{ color: GRAPH_THEME.ui.text.muted, textDecoration: "line-through" }}>{children}</span>;
}
// isSafeUrl returning true guarantees href is a non-empty string.
const safeHref = href ?? "";
// Fragment links (#section, footnote backlinks like
// #user-content-fnref-1) are in-document anchors. Opening them
// in a new tab would break GFM footnote back-navigation.
const isFragment = safeHref.startsWith("#");
if (isFragment) {
return (
<a href={safeHref} title={title} style={linkStyle} {...rest}>
{children}
</a>
);
}
return (
<a href={safeHref} title={title} target="_blank" rel="noopener noreferrer" style={linkStyle} {...rest}>
{children}
<ExternalLink size={10} style={{ marginLeft: 3, verticalAlign: "middle", display: "inline" }} />
</a>
);
},
img: ({ src, alt }) => (
<span style={imageBadgeStyle} title={src || "Image"}>
<ImageIcon size={12} style={{ marginRight: 5 }} />
<span>Image: {alt || src || "unlabeled"}</span>
</span>
),
h1: ({ children }) => <h1 style={h1Style}>{children}</h1>,
h2: ({ children }) => <h2 style={h2Style}>{children}</h2>,
h3: ({ children }) => <h3 style={h3Style}>{children}</h3>,
h4: ({ children }) => <h4 style={h4Style}>{children}</h4>,
p: ({ children }) => <p style={{ margin: "0 0 8px 0" }}>{children}</p>,
ul: ({ children }) => <ul style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ul>,
ol: ({ children }) => <ol style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ol>,
li: ({ children }) => <li style={{ marginBottom: 3 }}>{children}</li>,
blockquote: ({ children }) => <blockquote style={blockquoteStyle}>{children}</blockquote>,
hr: () => <hr style={{ border: "none", borderTop: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`, margin: "10px 0" }} />,
table: ({ children }) => (
<div style={{ width: "100%", overflowX: "auto", margin: "8px 0", borderRadius: 6, border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>
<table style={{ width: "100%", borderCollapse: "collapse", fontSize: 12 }}>{children}</table>
</div>
),
thead: ({ children }) => <thead style={{ background: "rgba(255, 255, 255, 0.04)" }}>{children}</thead>,
tbody: ({ children }) => <tbody>{children}</tbody>,
tr: ({ children }) => <tr style={{ borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</tr>,
th: ({ children }) => <th style={{ padding: "6px 8px", textAlign: "left", fontWeight: 700, color: GRAPH_THEME.ui.text.strong, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</th>,
td: ({ children }) => <td style={{ padding: "6px 8px", color: GRAPH_THEME.ui.text.body, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</td>,
pre: ({ children }) => <pre style={preBlockStyle}>{children}</pre>,
// C-1: discard `node` here too — code elements are custom components
// and would otherwise receive node="[object Object]" in the DOM.
code: ({ className: codeClass, children }) => {
const isInline = !codeClass && typeof children === "string" && !children.includes("\n");
return (
<code style={isInline ? inlineCodeStyle : blockCodeStyle}>
{children}
</code>
);
},
};
/* ─── Styles ──────────────────────────────────────────────────────── */
const viewerContainerStyle: CSSProperties = {
display: "flex",
flexDirection: "column",
background: "rgba(255, 255, 255, 0.025)",
border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
borderRadius: 12,
overflow: "hidden",
};
const viewerHeaderStyle: CSSProperties = {
display: "flex",
alignItems: "center",
justifyContent: "space-between",
padding: "6px 10px",
background: "rgba(0, 0, 0, 0.2)",
borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
};
const tabBtnStyle: CSSProperties = {
display: "inline-flex",
alignItems: "center",
padding: "4px 9px",
borderRadius: 6,
border: "1px solid transparent",
background: "transparent",
color: GRAPH_THEME.ui.text.muted,
fontSize: 12,
fontWeight: 600,
cursor: "pointer",
transition: "all 150ms ease",
};
const activeTabBtnStyle: CSSProperties = {
background: GRAPH_THEME.ui.timeline.playheadSoft,
border: `1px solid ${GRAPH_THEME.ui.control.activeBorder}`,
color: GRAPH_THEME.ui.timeline.playhead,
};
const copyBtnStyle: CSSProperties = {
display: "inline-flex",
alignItems: "center",
padding: "3px 8px",
borderRadius: 6,
border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
background: "rgba(255, 255, 255, 0.04)",
color: GRAPH_THEME.ui.text.subtle,
fontSize: 11,
cursor: "pointer",
};
const viewerBodyStyle: CSSProperties = {
padding: 12,
maxHeight: 380,
overflowY: "auto",
};
const emptyTextStyle: CSSProperties = {
color: GRAPH_THEME.ui.text.muted,
fontSize: 12,
lineHeight: 1.5,
fontStyle: "italic",
};
const sourcePreStyle: CSSProperties = {
margin: 0,
padding: 10,
borderRadius: 8,
background: "rgba(0, 0, 0, 0.3)",
border: "1px solid rgba(255, 255, 255, 0.05)",
overflowX: "auto",
};
const sourceCodeStyle: CSSProperties = {
fontFamily: "'JetBrains Mono', 'Fira Code', monospace",
fontSize: 12,
lineHeight: 1.6,
color: GRAPH_THEME.ui.text.strong,
whiteSpace: "pre-wrap",
wordBreak: "break-word",
userSelect: "text",
};
const previewStyle: CSSProperties = {
color: GRAPH_THEME.ui.text.body,
fontSize: 13,
lineHeight: 1.6,
wordBreak: "break-word",
};
const h1Style: CSSProperties = {
fontSize: 16,
fontWeight: 700,
color: GRAPH_THEME.ui.text.strong,
marginTop: 8,
marginBottom: 6,
paddingBottom: 3,
borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
};
const h2Style: CSSProperties = {
fontSize: 14,
fontWeight: 700,
color: GRAPH_THEME.ui.text.strong,
marginTop: 8,
marginBottom: 4,
};
const h3Style: CSSProperties = {
fontSize: 13,
fontWeight: 600,
color: GRAPH_THEME.ui.text.strong,
marginTop: 6,
marginBottom: 4,
};
const h4Style: CSSProperties = {
fontSize: 12,
fontWeight: 600,
color: GRAPH_THEME.ui.text.strong,
marginTop: 4,
marginBottom: 2,
};
const blockquoteStyle: CSSProperties = {
margin: "8px 0",
padding: "6px 12px",
borderLeft: `3px solid ${GRAPH_THEME.ui.timeline.playhead}`,
background: "rgba(98, 226, 205, 0.05)",
borderRadius: "0 6px 6px 0",
color: GRAPH_THEME.ui.text.body,
fontStyle: "italic",
};
const linkStyle: CSSProperties = {
color: "#79c0ff",
textDecoration: "underline",
textUnderlineOffset: "3px",
wordBreak: "break-all",
};
const imageBadgeStyle: CSSProperties = {
display: "inline-flex",
alignItems: "center",
padding: "3px 7px",
background: "rgba(255, 255, 255, 0.04)",
border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
borderRadius: 6,
color: GRAPH_THEME.ui.text.muted,
fontSize: 11,
margin: "3px 0",
};
const inlineCodeStyle: CSSProperties = {
fontFamily: "'JetBrains Mono', monospace",
fontSize: 12,
padding: "2px 5px",
borderRadius: 4,
background: "rgba(255, 255, 255, 0.07)",
color: "#e6edf3",
border: "1px solid rgba(255, 255, 255, 0.08)",
};
const preBlockStyle: CSSProperties = {
margin: "8px 0",
padding: 10,
borderRadius: 8,
background: "rgba(0, 0, 0, 0.35)",
border: "1px solid rgba(255, 255, 255, 0.08)",
overflowX: "auto",
};
const blockCodeStyle: CSSProperties = {
fontFamily: "'JetBrains Mono', monospace",
fontSize: 12,
lineHeight: 1.5,
color: "#e6edf3",
};
explorer/src/workspaces/GraphWorkspace/markdownUrlSafety.ts
+29
View File
@@ -0,0 +1,29 @@
/**
* URL-safety predicate for the Markdown content viewer.
*
* Extracted into a pure module so the check can be unit-tested without
* importing the MarkdownContentViewer React component, and so the component
* module exports only components (react-refresh/only-export-components,
* issue #1119). The behaviour is unchanged from the original in-component
* implementation: only http, https, mailto, in-document fragments, and
* root-relative paths are permitted.
*/
export function isSafeUrl(url?: string): boolean {
if (!url) return false;
const trimmed = url.trim();
// Reject whitespace-only strings — new URL("", base) would resolve to the base
// protocol and produce a false positive. This guards direct callers of the exported
// function; markdown parsers normalise whitespace-only destinations to "" which
// already fails the !url check above.
if (!trimmed) return false;
if (trimmed.startsWith("//")) return false;
if (trimmed.startsWith("#")) return true;
if (trimmed.startsWith("/")) return true;
try {
const parsed = new URL(trimmed, "http://localhost");
return ["http:", "https:", "mailto:"].includes(parsed.protocol);
} catch {
return false;
}
}
explorer/tests/markdownContentViewer.test.ts
+265
View File
@@ -0,0 +1,265 @@
import test from "node:test";
import assert from "node:assert/strict";
import React from "react";
import { renderToString } from "react-dom/server";
(globalThis as any).React = React;
import { MarkdownContentViewer } from "../src/workspaces/GraphWorkspace/MarkdownContentViewer.tsx";
import { isSafeUrl } from "../src/workspaces/GraphWorkspace/markdownUrlSafety.ts";
test("isSafeUrl permits safe http, https, and mailto URLs and relative paths", () => {
assert.equal(isSafeUrl("https://example.com"), true);
assert.equal(isSafeUrl("http://localhost:8000"), true);
assert.equal(isSafeUrl("mailto:user@example.com"), true);
assert.equal(isSafeUrl("#section-1"), true);
assert.equal(isSafeUrl("/relative/path"), true);
});
test("isSafeUrl rejects protocol-relative URLs and dangerous schemes", () => {
// Protocol-relative URLs (must be blocked)
assert.equal(isSafeUrl("//evil.com"), false);
assert.equal(isSafeUrl("//localhost:8000"), false);
assert.equal(isSafeUrl("//"), false);
// Dangerous schemes
assert.equal(isSafeUrl("javascript:alert('xss')"), false);
assert.equal(isSafeUrl("JAVASCRIPT:alert(1)"), false);
assert.equal(isSafeUrl("data:text/html;base64,PHNjcmlwdD4="), false);
assert.equal(isSafeUrl("vbscript:MsgBox(1)"), false);
assert.equal(isSafeUrl(""), false);
assert.equal(isSafeUrl(undefined), false);
});
// ─── C URL contract: whitespace-only strings ────────────────────────────────
// The CommonMark parser normalises whitespace-only link destinations to "" so
// these values are unreachable through normal markdown rendering. However, the
// function is exported and its direct-call contract must be correct.
test("isSafeUrl rejects whitespace-only strings (contract correctness)", () => {
assert.equal(isSafeUrl(" "), false, "single space must be rejected");
assert.equal(isSafeUrl("\t"), false, "tab must be rejected");
assert.equal(isSafeUrl("\n"), false, "newline must be rejected");
assert.equal(isSafeUrl(" "), false, "multiple spaces must be rejected");
assert.equal(isSafeUrl(" \t\n "), false, "mixed whitespace must be rejected");
});
test("renders Preview mode with formatted Markdown elements and tabs", () => {
const markdown = `# Main Title\n\n**Bold Statement**\n\n* Item A\n* Item B`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content: markdown, defaultMode: "preview" }));
// Tab buttons are present
assert.equal(html.includes("Preview"), true);
assert.equal(html.includes("Source"), true);
assert.equal(html.includes("Copy"), true);
// Formatted preview elements
assert.equal(html.includes("Main Title"), true);
assert.equal(html.includes("Bold Statement"), true);
assert.equal(html.includes("<strong>Bold Statement</strong>"), true);
assert.equal(html.includes("Item A"), true);
assert.equal(html.includes("Item B"), true);
});
test("renders Source mode with exact unmodified text inside pre/code", () => {
const markdown = `# Title 🚀\n\n * Indented item\n\n\`\`\`python\ndef test():\n return "α + β"\n\`\`\``;
const html = renderToString(React.createElement(MarkdownContentViewer, { content: markdown, defaultMode: "source" }));
assert.equal(html.includes("<pre"), true);
assert.equal(html.includes("<code"), true);
assert.equal(html.includes("# Title 🚀"), true);
assert.equal(html.includes(" * Indented item"), true);
assert.equal(html.includes('return &quot;α + β&quot;'), true);
});
test("renders raw HTML safely as escaped text without executing elements", () => {
const dangerousHtml = `<script>alert("XSS")</script><iframe src="https://evil.com"></iframe>`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content: dangerousHtml, defaultMode: "preview" }));
// Script and iframe tags must NOT be rendered as active DOM tags
assert.equal(html.includes("<script>"), false);
assert.equal(html.includes("<iframe"), false);
// Content is escaped as text
assert.equal(html.includes("&lt;script&gt;"), true);
});
// ─── C-1: HAST node prop must not reach the DOM ─────────────────────────────
// react-markdown passes a HAST `node` (Element) object to custom component
// overrides. Before this fix, ...props spread caused React 19 to serialise it
// as node="[object Object]" on every <a> and <code> element.
test("rendered links do not expose the HAST node object as a DOM attribute", () => {
const content = `[Example](https://example.com)\n\nInline \`code\` here.`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// The rendered HTML must not contain the serialised HAST object
assert.equal(html.includes("node="), false, "node= attribute must not appear in rendered HTML");
assert.equal(html.includes("[object Object]"), false, "serialised HAST object must not appear in rendered HTML");
// The link must still render correctly with the right href
assert.equal(html.includes('href="https://example.com"'), true, "href must be present");
});
// ─── C-2: Fragment links must not open in a new tab ─────────────────────────
// Links to in-document anchors such as #section or GFM footnote backlinks like
// #user-content-fn-1 must stay in the current document. Only external links
// use target="_blank".
test("fragment links render in the current document without target blank", () => {
const content = `[Jump to section](#introduction)\n\n[External](https://example.com)`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// Fragment link must have the href
assert.equal(html.includes('href="#introduction"'), true, "fragment href must be present");
// Confirm no target=_blank attribute appears anywhere near the fragment link.
// We check that the output contains a fragment href WITHOUT target="_blank"
// by verifying the two strings are not both present (the external link has
// target blank; the fragment link must not).
const fragmentLinkIdx = html.indexOf('href="#introduction"');
assert.notEqual(fragmentLinkIdx, -1, "fragment link must be rendered");
// Inspect the 80 chars around the fragment href — should not contain target
const fragmentContext = html.slice(Math.max(0, fragmentLinkIdx - 10), fragmentLinkIdx + 90);
assert.equal(fragmentContext.includes('target="_blank"'), false, "fragment link must not have target=_blank");
// External link must still have target blank
assert.equal(html.includes('href="https://example.com"'), true, "external href must be present");
assert.equal(html.includes('target="_blank"'), true, "external link must have target=_blank");
assert.equal(html.includes('rel="noopener noreferrer"'), true, "external link must have rel");
});
test("GFM footnote backlinks render without target blank", () => {
// GFM footnote syntax: footnote ref in text + definition below
const content = `See the note[^1] for more.\n\n[^1]: This is the footnote text.`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// The footnote reference link (#user-content-fn-1) and backlink
// (#user-content-fnref-1) are fragment links and must not open in a new tab.
// We verify no fragment href is paired with target=_blank.
// Extract all href="#..." occurrences and confirm none is adjacent to target=_blank.
const anchorMatches = [...html.matchAll(/href="#[^"]*"/g)];
assert.ok(anchorMatches.length > 0, "GFM footnotes must produce fragment links");
for (const match of anchorMatches) {
const start = match.index ?? 0;
const context = html.slice(Math.max(0, start - 10), start + 120);
assert.equal(
context.includes('target="_blank"'),
false,
`fragment link ${match[0]} must not have target=_blank`,
);
}
});
// ─── C-1-R: GFM footnote attributes must be preserved (regression test) ─────
// The C-1 fix (removing the HAST `node` prop) must NOT silently drop other
// legitimate HAST attributes. remark-gfm generates the following on footnote
// links that are required for correct in-page navigation and accessibility:
//
// Footnote reference anchor:
// id="user-content-fnref-1" ← backlink target
// data-footnote-ref="true"
// aria-describedby="footnote-label"
//
// Footnote back-link anchor:
// data-footnote-backref=""
// aria-label="Back to reference 1" ← screen-reader label
// class="data-footnote-backref"
//
// If these are absent, clicking the ↩ back-link cannot scroll back to the
// in-text reference, and screen readers cannot announce the backlink purpose.
test("GFM footnote links preserve generated id, aria, and class attributes", () => {
const content = `See the note[^1] for more.\n\n[^1]: This is the footnote text.`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// The HAST `node` object must not appear serialised as a DOM attribute.
assert.equal(html.includes("node="), false, "node= attribute must not appear in HTML");
assert.equal(html.includes("[object Object]"), false, "serialised HAST object must not appear in HTML");
// Footnote reference anchor must retain its id so the backlink can navigate to it.
assert.equal(
html.includes('id="user-content-fnref-1"'),
true,
"footnote reference anchor must retain id for back-navigation",
);
// Footnote backlink must retain its aria-label for screen-reader accessibility.
assert.equal(
html.includes('aria-label="Back to reference 1"'),
true,
"footnote backlink must retain aria-label for accessibility",
);
// Footnote backlink must retain its class attribute.
assert.equal(
html.includes('class="data-footnote-backref"'),
true,
"footnote backlink must retain class attribute",
);
});
test("renders safe links as <a> with target blank and unclickable span for unsafe links", () => {
const content = `[Safe Link](https://getsemantica.ai)\n\n[Unsafe Scheme](javascript:alert(1))\n\n[Protocol Relative](//evil.com)`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// Safe link renders as <a> with security attributes
assert.equal(html.includes('href="https://getsemantica.ai"'), true);
assert.equal(html.includes('target="_blank"'), true);
assert.equal(html.includes('rel="noopener noreferrer"'), true);
// Unsafe links do NOT render as <a> tags
assert.equal(html.includes('href="javascript:alert(1)"'), false);
assert.equal(html.includes('href="//evil.com"'), false);
assert.equal(html.includes("Unsafe Scheme"), true);
assert.equal(html.includes("Protocol Relative"), true);
});
test("renders remote images as safe placeholder badges instead of <img> tags", () => {
const content = `![System Diagram](https://example.com/diagram.png)`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// No <img> tag rendered
assert.equal(html.includes("<img"), false);
// Image placeholder badge rendered
assert.equal(html.includes("Image:"), true);
assert.equal(html.includes("System Diagram"), true);
});
test("renders clear empty-state message when content is empty or null", () => {
const emptyHtml = renderToString(React.createElement(MarkdownContentViewer, { content: "" }));
assert.equal(emptyHtml.includes("No content available for this node."), true);
const nullHtml = renderToString(React.createElement(MarkdownContentViewer, { content: null }));
assert.equal(nullHtml.includes("No content available for this node."), true);
});
test("renders plain text cleanly without requiring Markdown formatting", () => {
const plainText = "Plain entity summary text without markdown formatting.";
const html = renderToString(React.createElement(MarkdownContentViewer, { content: plainText, defaultMode: "preview" }));
assert.equal(html.includes(plainText), true);
});
test("handles very large Markdown content without failure", () => {
const largeContent = `# Large Knowledge Node\n\n` + "Structured observation paragraph. ".repeat(400);
assert.equal(largeContent.length > 10000, true);
const html = renderToString(React.createElement(MarkdownContentViewer, { content: largeContent, defaultMode: "preview" }));
assert.equal(html.includes("Large Knowledge Node"), true);
});
// ─── H-2: Stale copied state lifecycle (SSR-compatible portion) ─────────────
// Full state-transition testing (Node A → copy → Node B) requires an interactive
// framework. The lifecycle correctness is guaranteed by the render-phase
// previous-prop synchronisation pattern: a `copiedForContent` state value tracks
// the content for which the copied indicator was set; when `content` changes, the
// mismatch is detected during render and `copied` is reset to false in the same
// React batch, before the new node's UI is painted. What we CAN verify in SSR
// is that the initial render for any content value shows the Copy button (not the
// Copied indicator), which confirms the initial state is always clean.
test("copy button always starts in un-copied state on initial render", () => {
const html = renderToString(React.createElement(MarkdownContentViewer, {
content: "# Some Node\n\nDescription text.",
defaultMode: "preview",
}));
// Initial render must show 'Copy', never 'Copied'
assert.equal(html.includes("Copy"), true, "Copy button must be present on initial render");
assert.equal(html.includes("Copied"), false, "Copied indicator must NOT be present on initial render");
});
integrations/__init__.py
+1 -1
View File
@@ -1,7 +1,7 @@
"""
Semantica Framework Integrations
Optional integration packages for agentic frameworks (Google ADK, Claude Agent SDK, Agno, etc.).
Optional integration packages for agentic frameworks (Google ADK, Claude Agent SDK, Agno, CrewAI, LangChain, etc.).
Each integration is self-contained, independently installable via extras_require, and maintains
zero impact on core Semantica - keeping the semantic layer lean while maximizing ecosystem reach.
"""
integrations/agno/knowledge_graph.py
+10 -13
View File
@@ -277,25 +277,22 @@ class AgnoKnowledgeGraph(_KnowledgeBase): # type: ignore[misc]
def load_urls(self, urls: List[str]) -> None:
"""Fetch each URL and ingest the response body.
Only ``http`` and ``https`` schemes are permitted to prevent SSRF.
Uses the shared SSRF guard so that ``http`` and ``https`` are the only
permitted schemes, private/loopback/link-local/cloud-metadata addresses
are blocked by default, DNS resolution is validated, and every redirect
hop is re-checked before being followed.
"""
import urllib.request
from urllib.parse import urlparse
from semantica.ingest.ssrf import request_with_ssrf_guard
from semantica.utils.exceptions import ValidationError
for url in urls:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
logger.warning(
"Skipping URL with disallowed scheme '%s': %s",
parsed.scheme,
url,
)
continue
try:
with urllib.request.urlopen(url, timeout=10) as resp: # noqa: S310
text = resp.read().decode("utf-8", errors="replace")
response = request_with_ssrf_guard("GET", url, timeout=10)
text = response.text
self._ingest_text(text, source=url)
logger.info("Loaded URL: %s", url)
except ValidationError as exc:
logger.warning("Skipping URL (SSRF check failed) %s: %s", url, exc)
except Exception as exc:
logger.warning("Failed to fetch %s: %s", url, exc)
integrations/langchain/README.md
+67
View File
@@ -0,0 +1,67 @@
# Semantica × LangChain
Drop Semantica into existing LangChain / LangGraph pipelines: GraphRAG-style
retrieval, a `VectorStore` adapter, and agent tools.
## Install
```bash
pip install semantica[langchain]
# or just the core adapter dependency:
pip install langchain-core
```
## Retriever (GraphRAG)
```python
from integrations.langchain import SemanticaRetriever
from semantica.context import ContextGraph
from semantica.vector_store import HybridSearch
graph = ContextGraph()
hybrid = HybridSearch()
retriever = SemanticaRetriever(graph=graph, hybrid=hybrid, hops=2, top_k=10)
# Use with any LangChain chain that accepts a retriever:
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
```
Hybrid search seeds retrieval; then graph edges are walked `hops` steps so
results go beyond flat vector similarity.
## VectorStore
```python
from integrations.langchain import SemanticaVectorStore
store = SemanticaVectorStore(hybrid=hybrid)
store.add_texts(["document one", "document two"], metadatas=[{"source": "a"}, {"source": "b"}])
docs = store.similarity_search("document", k=2)
docs, scores = store.similarity_search_with_score("document", k=2)
```
## Agent tools (LangGraph / tool-calling agents)
```python
from integrations.langchain import SemanticaKGTool, SemanticaDecisionTool
from langgraph.prebuilt import create_react_agent
tools = [
SemanticaKGTool(graph),
SemanticaDecisionTool(graph),
]
agent = create_react_agent(model, tools)
```
- `semantica_query_graph` — query the shared context graph (keyword / NL)
- `semantica_query_decisions` — search the recorded decision log
## Compatibility
- Requires `langchain-core >= 0.3`.
- All classes degrade gracefully when `langchain-core` is absent: they remain
importable (carrying the full Semantica API), and `build()` returns `None`,
so agents can branch on `LANGCHAIN_AVAILABLE`.
integrations/langchain/__init__.py
+48
View File
@@ -0,0 +1,48 @@
"""
Semantica × LangChain Integration
=================================
First-class integration between the Semantica semantic intelligence stack and
the `LangChain <https://github.com/langchain-ai/langchain>`_ / LangGraph
ecosystem.
Public surface
--------------
SemanticaRetriever — ``BaseRetriever`` with multi-hop GraphRAG (walks graph
edges from hybrid-search hits)
SemanticaVectorStore — ``VectorStore`` adapter over Semantica's hybrid search
(drop-in for RetrievalQA / LCEL chains)
SemanticaKGTool — ``BaseTool`` for querying the context graph
SemanticaDecisionTool — ``BaseTool`` exposing the recorded decision log
Quick start
-----------
pip install semantica[langchain]
>>> from integrations.langchain import (
... SemanticaRetriever,
... SemanticaVectorStore,
... SemanticaKGTool,
... SemanticaDecisionTool,
... )
Compatibility
-------------
Requires ``langchain-core >= 0.3``. All classes degrade gracefully when
``langchain-core`` is not installed — they are still importable and carry the
full Semantica API, but cannot be bound to LangChain chains/agents.
"""
from .retriever import LANGCHAIN_AVAILABLE, SemanticaRetriever
from .tools import SemanticaDecisionTool, SemanticaKGTool
from .vectorstore import SemanticaVectorStore
__all__ = [
"SemanticaRetriever",
"SemanticaVectorStore",
"SemanticaKGTool",
"SemanticaDecisionTool",
"LANGCHAIN_AVAILABLE",
]
__version__ = "0.1.0"
integrations/langchain/retriever.py
+216
View File
@@ -0,0 +1,216 @@
"""
SemanticaRetriever — LangChain ``BaseRetriever`` with multi-hop GraphRAG.
Hybrid search seeds the retrieval, then graph edges are walked for ``hops``
steps so results go beyond flat vector similarity.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Tuple
from semantica.utils.logging import get_logger
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: LangChain core
# ---------------------------------------------------------------------------
LANGCHAIN_AVAILABLE = False
LANGCHAIN_IMPORT_ERROR: Optional[str] = None
_BaseRetriever: Any = object
_Document: Any = None
def _get_document(**kwargs: Any) -> Any:
"""Instantiate a langchain Document lazily (keeps the import optional)."""
if _Document is None: # pragma: no cover - exercised only with langchain
raise RuntimeError(LANGCHAIN_IMPORT_ERROR or "langchain-core not installed")
return _Document(**kwargs)
try:
from langchain_core.documents import Document as _Document # type: ignore
from langchain_core.retrievers import (
BaseRetriever as _BaseRetriever, # type: ignore
)
LANGCHAIN_AVAILABLE = True
except ImportError: # pragma: no cover - exercised only without langchain
LANGCHAIN_IMPORT_ERROR = (
"langchain-core is not installed. Install with: pip install langchain-core"
)
logger.debug(LANGCHAIN_IMPORT_ERROR)
def _hit_layers(hit: Dict[str, Any]) -> Tuple[Dict[str, Any], Dict[str, Any]]:
"""Nested HybridSearch metadata and ContextGraph.query node, if present."""
metadata = hit.get("metadata") if isinstance(hit.get("metadata"), dict) else {}
node = hit.get("node") if isinstance(hit.get("node"), dict) else {}
return metadata, node
def _hit_id(hit: Dict[str, Any]) -> Optional[str]:
"""Graph node id, preferring metadata over a HybridSearch vector id."""
metadata, node = _hit_layers(hit)
return (
hit.get("node_id")
or metadata.get("node_id")
or node.get("id")
or node.get("node_id")
or hit.get("id")
)
def _hit_content(hit: Dict[str, Any], fallback: str = "") -> str:
metadata, node = _hit_layers(hit)
props = node.get("properties") if isinstance(node.get("properties"), dict) else {}
return (
hit.get("content")
or hit.get("text")
or metadata.get("content")
or metadata.get("text")
or props.get("content")
or fallback
)
def _hit_type(hit: Dict[str, Any]) -> str:
metadata, node = _hit_layers(hit)
return (
hit.get("node_type")
or hit.get("type")
or metadata.get("node_type")
or metadata.get("type")
or node.get("type")
or node.get("node_type")
or "node"
)
def _hit_score(hit: Dict[str, Any], default: float = 1.0) -> float:
return float(hit.get("score") if hit.get("score") is not None else hit.get("distance") or default)
class SemanticaRetriever(_BaseRetriever): # type: ignore[misc]
"""GraphRAG-style retriever over a Semantica ``ContextGraph``.
Args:
graph: A semantica.context.ContextGraph instance.
hybrid: A semantica.vector_store.HybridSearch instance used to seed
retrieval. If omitted, a best-effort keyword search on the graph
is used.
hops: Number of graph-edge expansion hops (default 2).
top_k: Number of seed hits (default 10).
"""
graph: Any
hybrid: Any = None
hops: int = 2
top_k: int = 10
def __init__(
self,
graph: Any,
hybrid: Any = None,
hops: int = 2,
top_k: int = 10,
**kwargs: Any,
) -> None:
"""Explicit init so the retriever works with and without langchain."""
if LANGCHAIN_AVAILABLE:
# BaseRetriever is a Pydantic model: pass the declared fields
# through so validation succeeds.
super().__init__(
graph=graph,
hybrid=hybrid,
hops=hops,
top_k=top_k,
**kwargs,
)
else:
# Without langchain-core, BaseRetriever is a plain object
super().__init__() # type: ignore[call-arg]
self.graph = graph
self.hybrid = hybrid
self.hops = hops
self.top_k = top_k
def _get_relevant_documents(self, query: str, **kwargs: Any) -> List[Any]:
"""LangChain BaseRetriever entry point."""
seed = self._seed_results(query)
if not seed:
return []
# Expand each seed node through the graph
expanded: Dict[str, Dict[str, Any]] = {}
for hit in seed:
node_id = _hit_id(hit)
if not node_id:
continue
metadata, _ = _hit_layers(hit)
expanded[node_id] = {
"content": _hit_content(hit, fallback=str(node_id)),
"node_type": _hit_type(hit),
"score": _hit_score(hit),
"metadata": metadata,
}
try:
neighbors = self.graph.get_neighbors(node_id, hops=self.hops)
for neighbor in neighbors:
nid = neighbor.get("node_id") or neighbor.get("id")
if nid and nid not in expanded:
expanded[nid] = {
"content": neighbor.get("content")
or neighbor.get("text")
or neighbor.get("name")
or str(nid),
"node_type": neighbor.get("node_type")
or neighbor.get("type")
or "node",
"score": float(neighbor.get("weight") or 0.5),
"metadata": {},
}
except Exception as exc: # graph expansion is best-effort
logger.debug("graph expansion failed for %s: %s", node_id, exc)
# Order: seed hits first (they have real scores), then neighbors.
# Keep a deterministic id->payload list (sets are unordered — see Qodo).
ordered_pairs: List[tuple] = []
seen_ids = set()
for hit in seed:
nid = _hit_id(hit)
if nid and nid in expanded and nid not in seen_ids:
ordered_pairs.append((nid, expanded[nid]))
seen_ids.add(nid)
for nid, item in expanded.items():
if nid not in seen_ids:
ordered_pairs.append((nid, item))
seen_ids.add(nid)
return [
_get_document(
page_content=item["content"],
metadata={
**item["metadata"],
"node_id": nid,
"node_type": item["node_type"],
"score": item["score"],
},
)
for nid, item in ordered_pairs
]
def _seed_results(self, query: str) -> List[Dict[str, Any]]:
"""Get seed results from hybrid search or a graph keyword scan."""
if self.hybrid is not None:
try:
return self.hybrid.search(query, k=self.top_k)
except Exception as exc:
logger.debug("hybrid search failed, falling back: %s", exc)
# Best-effort keyword scan over graph nodes (ContextGraph.query)
try:
return self.graph.query(query, limit=self.top_k)
except Exception:
return []
integrations/langchain/tools.py
+133
View File
@@ -0,0 +1,133 @@
"""
SemanticaKGTool / SemanticaDecisionTool — LangChain ``BaseTool`` adapters
for LangChain / LangGraph agents.
"""
from __future__ import annotations
import json
from typing import Any, Optional, Type
from pydantic import BaseModel, ConfigDict, Field
from semantica.utils.logging import get_logger
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: LangChain core
# ---------------------------------------------------------------------------
LANGCHAIN_AVAILABLE = False
LANGCHAIN_IMPORT_ERROR: Optional[str] = None
_BaseTool: Any = object
try:
from langchain_core.tools import BaseTool as _BaseTool # type: ignore
LANGCHAIN_AVAILABLE = True
except ImportError: # pragma: no cover
LANGCHAIN_IMPORT_ERROR = (
"langchain-core is not installed. Install with: pip install langchain-core"
)
logger.debug(LANGCHAIN_IMPORT_ERROR)
def _json(payload: Any) -> str:
return json.dumps(payload, default=str, ensure_ascii=False)
class QueryGraphInput(BaseModel):
query: str = Field(..., description="Natural-language or keyword graph query")
limit: int = Field(10, description="Maximum matching nodes to return")
class QueryDecisionsInput(BaseModel):
category: str = Field(
"",
description="Keyword to search recorded decisions; empty returns insights",
)
limit: int = Field(10, description="Maximum results when searching by keyword")
class SemanticaKGTool(_BaseTool): # type: ignore[misc]
"""LangChain tool for querying a Semantica ``ContextGraph``.
Args:
graph: A semantica.context.ContextGraph instance.
Example:
>>> tool = SemanticaKGTool(graph)
>>> agent = create_react_agent(model, tools=[tool])
"""
model_config = ConfigDict(arbitrary_types_allowed=True)
name: str = "semantica_query_graph"
description: str = (
"Query Semantica's shared context graph with a natural-language "
"keyword query. Returns matching entities and relationships."
)
args_schema: Type[BaseModel] = QueryGraphInput
graph: Any = None
def __init__(self, graph: Any = None, **kwargs: Any) -> None:
if LANGCHAIN_AVAILABLE:
super().__init__(graph=graph, **kwargs)
else:
super().__init__()
self.graph = graph
def build(self) -> Any:
"""Return this tool, or None if langchain-core is missing."""
return self if LANGCHAIN_AVAILABLE else None
def _run(self, query: str, limit: int = 10, **kwargs: Any) -> str:
try:
return _json(self.graph.query(query, limit=limit))
except Exception as exc:
return _json({"error": str(exc)})
async def _arun(self, query: str, limit: int = 10, **kwargs: Any) -> str:
return self._run(query, limit=limit)
class SemanticaDecisionTool(_BaseTool): # type: ignore[misc]
"""LangChain tool for searching Semantica's recorded decision log.
Args:
graph: A semantica.context.ContextGraph instance.
"""
model_config = ConfigDict(arbitrary_types_allowed=True)
name: str = "semantica_query_decisions"
description: str = (
"Search Semantica's recorded decision log with a keyword query. "
"Returns decisions, rationale, and context."
)
args_schema: Type[BaseModel] = QueryDecisionsInput
graph: Any = None
def __init__(self, graph: Any = None, **kwargs: Any) -> None:
if LANGCHAIN_AVAILABLE:
super().__init__(graph=graph, **kwargs)
else:
super().__init__()
self.graph = graph
def build(self) -> Any:
"""Return this tool, or None if langchain-core is missing."""
return self if LANGCHAIN_AVAILABLE else None
def _run(self, category: str = "", limit: int = 10, **kwargs: Any) -> str:
try:
if category:
return _json(self.graph.query(category, limit=limit))
return _json(self.graph.get_decision_insights())
except Exception as exc:
return _json({"error": str(exc)})
async def _arun(self, category: str = "", limit: int = 10, **kwargs: Any) -> str:
return self._run(category=category, limit=limit)
integrations/langchain/vectorstore.py
+143
View File
@@ -0,0 +1,143 @@
"""
SemanticaVectorStore — LangChain ``VectorStore`` adapter over Semantica's
hybrid search (``semantica.vector_store.HybridSearch``).
"""
from __future__ import annotations
from typing import Any, Dict, Iterable, List, Optional
from semantica.utils.logging import get_logger
from .retriever import _hit_content, _hit_id, _hit_score, _hit_type, _hit_layers
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: LangChain core
# ---------------------------------------------------------------------------
LANGCHAIN_AVAILABLE = False
LANGCHAIN_IMPORT_ERROR: Optional[str] = None
_VectorStoreBase: Any = object
_Document: Any = None
def _make_document(**kwargs: Any) -> Any:
if _Document is None: # pragma: no cover
raise RuntimeError(LANGCHAIN_IMPORT_ERROR or "langchain-core not installed")
return _Document(**kwargs)
try:
from langchain_core.documents import Document as _Document # type: ignore
from langchain_core.vectorstores import (
VectorStore as _VectorStoreBase, # type: ignore
)
LANGCHAIN_AVAILABLE = True
except ImportError: # pragma: no cover
LANGCHAIN_IMPORT_ERROR = (
"langchain-core is not installed. Install with: pip install langchain-core"
)
logger.debug(LANGCHAIN_IMPORT_ERROR)
def _document_from_hit(hit: Dict[str, Any], include_score: bool = True) -> Any:
metadata, _ = _hit_layers(hit)
node_id = _hit_id(hit)
doc_meta = {
**metadata,
"node_id": node_id,
"node_type": _hit_type(hit),
}
if include_score:
doc_meta["score"] = _hit_score(hit, default=0.0)
return _make_document(
page_content=_hit_content(hit),
metadata=doc_meta,
)
class SemanticaVectorStore(_VectorStoreBase): # type: ignore[misc]
"""Wrap Semantica hybrid search as a LangChain ``VectorStore``.
Args:
hybrid: A semantica.vector_store.HybridSearch instance.
vector_store: Optional Semantica vector store passed through to
``HybridSearch.add_texts``.
"""
hybrid: Any
vector_store: Any = None
def __init__(self, hybrid: Any, vector_store: Any = None, **kwargs: Any) -> None:
if LANGCHAIN_AVAILABLE:
super().__init__(**kwargs)
else:
super().__init__()
self.hybrid = hybrid
self.vector_store = vector_store
# -- required VectorStore API ------------------------------------------
def add_texts(
self,
texts: Iterable[str],
metadatas: Optional[List[Dict[str, Any]]] = None,
**kwargs: Any,
) -> List[str]:
"""Embed and store texts; return the generated IDs.
Delegates to the Semantica ``VectorStore.add_documents`` backing the
HybridSearch instance (or to ``hybrid.vector_store`` if provided).
"""
if self.vector_store is not None:
return self.vector_store.add_documents(
list(texts), metadata=metadatas, **kwargs
)
vs = getattr(self.hybrid, "vector_store", None)
if vs is not None and hasattr(vs, "add_documents"):
return vs.add_documents(list(texts), metadata=metadatas, **kwargs)
raise ValueError(
"SemanticaVectorStore requires a Semantica vector store with "
"add_documents (pass vector_store=... to the HybridSearch or to "
"SemanticaVectorStore)"
)
def similarity_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Any]:
"""Return documents most similar to the query."""
return [_document_from_hit(hit) for hit in self.hybrid.search(query, k=k)]
def similarity_search_with_score(
self, query: str, k: int = 4, **kwargs: Any
) -> List[Any]:
"""Return (document, score) pairs."""
return [
(
_document_from_hit(hit, include_score=False),
_hit_score(hit, default=0.0),
)
for hit in self.hybrid.search(query, k=k)
]
@classmethod
def from_texts(
cls,
texts: List[str],
embedding: Any = None,
metadatas: Optional[List[Dict[str, Any]]] = None,
**kwargs: Any,
) -> "SemanticaVectorStore":
"""Build a store from a list of texts (LangChain convention).
Requires a pre-configured ``hybrid`` instance passed via kwargs.
"""
hybrid = kwargs.pop("hybrid", None)
if hybrid is None:
raise ValueError(
"SemanticaVectorStore.from_texts requires a 'hybrid' "
"HybridSearch instance as a keyword argument"
)
store = cls(hybrid=hybrid, **kwargs)
store.add_texts(texts, metadatas=metadatas)
return store
integrations/openclaw/mcp_tool.py
+35 -1
View File
@@ -116,7 +116,41 @@ class OpenClawKGTool:
)
def __init__(self, base_url: str = "http://localhost:8000", timeout: int = 30) -> None:
self.base_url = base_url.rstrip("/")
# Validate base_url at construction time so callers get an immediate,
# actionable error rather than a cryptic failure on the first request.
# allow_private_ips=True because the documented default (localhost:8000)
# is intentionally a local Semantica server; the scheme check and
# URL-structure check still apply unconditionally.
try:
from semantica.ingest.ssrf import validate_url_for_request
validate_url_for_request(base_url, allow_private_ips=True)
except ImportError:
# semantica.ingest not installed in minimal openclaw-only environments;
# mirror the structural checks that validate_url_for_request performs
# unconditionally (before allow_private_ips is consulted), so the
# guarantee in the comment above — "scheme check and URL-structure check
# still apply unconditionally" — holds in this path too.
from urllib.parse import urlparse as _urlparse
if not isinstance(base_url, str) or not base_url.strip():
raise ValueError("OpenClawKGTool base_url must be a non-empty string.")
_parsed = _urlparse(base_url.strip())
_scheme = (_parsed.scheme or "").lower()
if _scheme not in ("http", "https"):
raise ValueError(
f"OpenClawKGTool base_url scheme '{_parsed.scheme}' is not permitted. "
"Only http and https are allowed."
)
if not _parsed.netloc:
raise ValueError(
f"Invalid OpenClawKGTool base_url '{base_url}': "
"URL must include a netloc (domain or host)."
)
if not _parsed.hostname:
raise ValueError(
f"Invalid OpenClawKGTool base_url '{base_url}': "
"URL must include a hostname."
)
self.base_url = base_url.strip().rstrip("/")
self.timeout = timeout
self._session: Any = None
mcp/__init__.py
+11
View File
@@ -21,6 +21,17 @@ Configure in Claude Desktop, Windsurf, Cline, Continue, VS Code:
}
"""
import os
# MCP stdio framing IS stdout: any progress bar or console renderer that writes
# to stdout would interleave with the JSON-RPC stream and corrupt framing for
# every client. This package is always used as an MCP stdio server, so force
# progress tracking off for the entire process. Set before importing server /
# tools so the Semantica progress-tracker singleton is never created with
# output enabled (the singleton reads this variable at construction time and
# the enabled.setter re-checks it, so later re-enable attempts are also blocked).
os.environ["SEMANTICA_DISABLE_PROGRESS"] = "1"
# `semantica.__version__` is the authoritative package version — see
# semantica/mcp_server/__init__.py for why it is used directly rather than
# importlib.metadata.version("semantica").
mcp/tools/export.py
+6 -1
View File
@@ -80,7 +80,12 @@ def handle_export_graph(args: dict) -> dict:
if rdf_fmt:
try:
from semantica.export import RDFExporter
rdf_str = RDFExporter().export_to_rdf(graph, format=rdf_fmt)
# RDFExporter.export_to_rdf() expects the canonical kg dict
# {"entities": [...], "relationships": [...]}, not a ContextGraph
# object. Convert before handing off; passing the raw graph
# caused AttributeError: 'ContextGraph' object has no attribute
# 'get' on every RDF format.
rdf_str = RDFExporter().export_to_rdf(graph.to_kg_dict(), format=rdf_fmt)
return {"format": rdf_fmt, "data": rdf_str}
except Exception as exc:
return {"error": f"RDF export failed: {exc}"}
plugins/.claude-plugin/README.md
+1 -1
View File
@@ -53,7 +53,7 @@ plugins/
## Prerequisites
```bash
git clone https://github.com/Hawksight-AI/semantica.git
git clone https://github.com/semantica-agi/semantica.git
cd semantica
pip install semantica # Python 3.10+
```
plugins/.claude-plugin/marketplace.json
+2 -2
View File
@@ -1,8 +1,8 @@
{
"name": "semantica-local",
"owner": {
"name": "Hawksight AI",
"url": "https://github.com/Hawksight-AI/semantica"
"name": "Semantica",
"url": "https://github.com/semantica-agi/semantica"
},
"plugins": [
{
plugins/.claude-plugin/plugin.json
+2 -2
View File
@@ -5,8 +5,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.cline-plugin/plugin.json
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.codex-plugin/plugin.json
+2 -2
View File
@@ -5,8 +5,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.continue-plugin/plugin.json
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.cursor-plugin/plugin.json
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.openclaw-plugin/plugin.json
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.vscode-plugin/plugin.json
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
plugins/.windsurf-plugin/plugin.json
+2 -2
View File
@@ -6,8 +6,8 @@
"author": {
"name": "Semantica Contributors"
},
"homepage": "https://github.com/Hawksight-AI/semantica",
"repository": "https://github.com/Hawksight-AI/semantica",
"homepage": "https://github.com/semantica-agi/semantica",
"repository": "https://github.com/semantica-agi/semantica",
"license": "MIT",
"keywords": [
"semantica",
poc_runner.py
-9
View File
@@ -187,15 +187,6 @@ def poc_vuln3():
})
return nodes
# Simulate the CSV parser — mirrors export_import.py lines 131-133
def parse_import_csv_row(row: dict) -> dict:
"""Mirrors export_import.py CSV node ID extraction (no sanitization)."""
node_id = row.get("id") or row.get("node_id") or row.get(":ID") or row.get("_id")
return {
"id": str(node_id), # ← UNSANITIZED
"type": row.get("type", "entity"),
}
# Attack payloads
payloads = [
# Header injection payload (chained with VULN-1)
pyproject.toml
+5 -3
View File
@@ -85,7 +85,8 @@ dependencies = [
"loguru>=0.7.3",
"structlog>=22.1.0",
"gensim>=4.4.0",
"httpx<0.29.0"
"httpx<0.29.0",
"pyarrow>=14.0.0"
]
[project.urls]
@@ -103,7 +104,7 @@ Discord = "https://discord.gg/sV34vps5hH"
llm-openai = ["openai>=1.0.0"]
llm-groq = ["groq>=0.4.0"]
llm-gemini = ["google-genai>=0.1.0"]
llm-anthropic = ["anthropic>=0.18.0"]
llm-anthropic = ["anthropic>=0.122.0"]
llm-ollama = ["ollama>=0.1.0"]
llm-deepseek = ["openai>=1.0.0"]
llm-litellm = ["litellm>=1.83.9"]
@@ -205,6 +206,7 @@ agno = ["agno>=1.0.0"]
# needed (it pulls vulnerable transitive deps like chromadb) and would only
# duplicate the prebuilt tooling users can install separately.
crewai = ["crewai>=0.80.0"]
langchain = ["langchain-core>=0.3.0"]
# ---- File Watching ----
watch = ["watchdog>=6.0.0"]
@@ -252,7 +254,7 @@ explorer-lite = [
# dependency-audit/security gates. Install it explicitly via ``semantica[crewai]``.
all = [
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno,langchain]"
]
# ---------------- ENTRYPOINTS ----------------
requirements-ci.txt
+800 -323
View File
File diff suppressed because it is too large Load Diff
semantica/change_management/change_management_usage.md
+1 -1
View File
@@ -1039,6 +1039,6 @@ manager = TemporalVersionManager(storage_path="large_data.db")
## Support
For questions or issues:
- GitHub Issues: https://github.com/Hawksight-AI/semantica/issues
- GitHub Issues: https://github.com/semantica-agi/semantica/issues
- Documentation: https://semantica.readthedocs.io
- Community: https://discord.gg/sV34vps5hH
semantica/change_management/version_storage.py
+14
View File
@@ -31,6 +31,7 @@ import hashlib
import json
import sqlite3
import threading
import warnings
from abc import ABC, abstractmethod
from datetime import datetime
from pathlib import Path
@@ -62,6 +63,12 @@ def create_graph_snapshot_record(
"""
Creates a standardized snapshot metadata record for a named graph.
.. deprecated::
``create_graph_snapshot_record()`` is deprecated and will be removed in
a future major version. It has no callers inside Semantica; build the
record inline and checksum it with
:func:`semantica.change_management.compute_checksum` instead.
Args:
version_id: Unique identifier for this snapshot
graph_uri: The underlying named graph URI in the triplet store
@@ -69,6 +76,13 @@ def create_graph_snapshot_record(
description: Purpose or context of the snapshot
metadata: Additional tags or pipeline context
"""
warnings.warn(
"create_graph_snapshot_record() is deprecated and will be removed in a "
"future major version. Build the snapshot record inline and use "
"semantica.change_management.compute_checksum() instead.",
DeprecationWarning,
stacklevel=2,
)
record = {
"label": version_id,
semantica/cli.py
+193 -4
View File
@@ -773,10 +773,22 @@ def changelog(cli_ctx: CLIContext, local_json: bool) -> None:
_run_with_error_handling(_action)
class _DeepEmbeddingFailure(Exception):
"""A deep-probe failure from doctor's embedding checks.
Marks failures that happened AFTER the backend imported cleanly — model
load, probe, or runtime problems — so the check's hint can point at the
real remediation instead of `pip install`.
"""
@main.command()
@click.option("--json", "local_json", is_flag=True, default=False)
@click.option("--deep-embeddings", "deep_embeddings", is_flag=True, default=False,
help="Also instantiate the local embedding backends and embed a probe "
"text (catches backends that import cleanly but cannot load).")
@click.pass_obj
def doctor(cli_ctx: CLIContext, local_json: bool) -> None:
def doctor(cli_ctx: CLIContext, local_json: bool, deep_embeddings: bool) -> None:
"""Run a health check on all Semantica components and backends."""
import importlib.metadata
cli_ctx = _require_ctx(cli_ctx)
@@ -787,6 +799,16 @@ def doctor(cli_ctx: CLIContext, local_json: bool) -> None:
try:
note = fn()
return label, "ok", note, None
except _DeepEmbeddingFailure as exc:
# A deep-probe failure means the package IMPORTED fine: the pip
# hint would be the wrong remediation for what is actually a
# runtime/model-load problem (broken torch, failed model
# download, missing shared libs).
return label, "fail", str(exc), (
"runtime/model-load failure — reinstalling the package usually "
"does not help; check the warnings above (torch install, model "
"download, disk space)"
)
except Exception as exc:
return label, "fail", str(exc), hint
@@ -827,6 +849,50 @@ def doctor(cli_ctx: CLIContext, local_json: bool) -> None:
return f"{backend} importable"
checks.append(_check("Vector store", _vector, hint="pip install semantica[vectorstore-…]"))
# Embedding backends (#994): `doctor` used to report all green while
# every local embedding backend was non-functional — import success
# says nothing about model loading. Default checks stay cheap
# (import + version); --deep-embeddings (or
# SEMANTICA_DOCTOR_DEEP_EMBEDDINGS=1) instantiates the backend through
# TextEmbedder and embeds a probe, which is the only level that
# catches a backend that imports cleanly but cannot actually load.
deep = deep_embeddings or os.environ.get("SEMANTICA_DOCTOR_DEEP_EMBEDDINGS", "").strip().lower() in ("1", "true", "yes", "on")
def _embedding_backend(method: str) -> str:
if method == "sentence_transformers":
import sentence_transformers # noqa: F401
note = f"importable ({importlib.metadata.version('sentence-transformers')})"
else:
import fastembed # noqa: F401
note = f"importable ({importlib.metadata.version('fastembed')})"
if not deep:
return note
try:
from .embeddings import TextEmbedder
embedder = TextEmbedder(method=method)
if embedder.model is None and embedder.fastembed_model is None:
raise RuntimeError(
"model failed to load — the hash fallback is active "
"(see warnings above); embedding quality is degraded"
)
probe = embedder.embed_text("semantica doctor embedding probe")
except _DeepEmbeddingFailure:
raise
except Exception as exc:
raise _DeepEmbeddingFailure(str(exc)) from exc
return f"{note}; deep probe ok ({len(probe)}-dim)"
checks.append(_check(
"Embeddings (sentence-transformers)",
lambda: _embedding_backend("sentence_transformers"),
hint="pip install sentence-transformers",
))
checks.append(_check(
"Embeddings (fastembed)",
lambda: _embedding_backend("fastembed"),
hint="pip install fastembed",
))
# LLM provider keys
for provider, var in [("OpenAI", "OPENAI_API_KEY"), ("Anthropic", "ANTHROPIC_API_KEY"),
("Groq", "GROQ_API_KEY")]:
@@ -1666,6 +1732,93 @@ def embed(ctx: click.Context) -> None:
click.echo(ctx.get_help())
def _json_default(obj) -> object:
"""JSON serialiser that converts NumPy scalars/arrays to native Python types.
Falls back to ``str()`` for everything else so the writer never crashes on
unexpected types (e.g. ``datetime``, custom domain objects).
"""
try:
import numpy as np # local import — only needed when result contains numpy
if isinstance(obj, np.ndarray):
return obj.tolist()
if isinstance(obj, np.generic):
return obj.item()
except ImportError:
pass
return str(obj)
def _write_result_output(out_path: Path, result) -> None:
"""Serialize a structured CLI result (dict or list) for ``--output``.
Domain commands like ``deduplicate`` and ``ontology align`` produce dicts
and lists, not numeric matrices — routing them through the embeddings
writer rejected their shapes and extensions (.csv is documented for
deduplicate). JSON-family formats serialize anything; CSV serializes a
list of dicts (or a single dict as one row).
Accepted extensions: .json, .jsonl, .csv (no-extension and .txt are
rejected so the path reported to the caller always matches the file
actually created, consistent with every other --output in the CLI).
"""
import json as _json
suffix = out_path.suffix.lower()
# ── JSON ────────────────────────────────────────────────────────────────
if suffix == ".json":
with open(out_path, "w", encoding="utf-8") as fh:
_json.dump(result, fh, indent=2, default=_json_default)
return
# ── JSON Lines ──────────────────────────────────────────────────────────
# Every record must occupy exactly one line. Wrap a bare dict in a list
# so callers never need to know whether their result is singular or plural.
if suffix == ".jsonl":
items = result if isinstance(result, list) else [result]
with open(out_path, "w", encoding="utf-8") as fh:
for item in items:
fh.write(_json.dumps(item, default=_json_default) + "\n")
return
# ── CSV ─────────────────────────────────────────────────────────────────
if suffix == ".csv":
import pandas as pd
rows = result if isinstance(result, list) else [result]
if not rows:
raise click.ClickException(
"No results to write — output file not created."
)
# Normalise numpy scalars/arrays to Python natives so to_csv() does
# not fall back to repr() strings for array-valued cells.
def _normalise(row):
if not isinstance(row, dict):
return row
out = {}
for k, v in row.items():
try:
import numpy as np
if isinstance(v, np.ndarray):
v = v.tolist()
elif isinstance(v, np.generic):
v = v.item()
except ImportError:
pass
out[k] = v
return out
pd.DataFrame([_normalise(r) for r in rows]).to_csv(out_path, index=False)
return
# ── unsupported ─────────────────────────────────────────────────────────
display = suffix if suffix else "(no extension)"
raise click.ClickException(
f"Unsupported output format '{display}'. Use .json, .jsonl, or .csv"
)
@embed.command("generate")
@click.argument("input_path")
@click.option("--model",
@@ -1708,7 +1861,43 @@ def embed_generate(cli_ctx: CLIContext, input_path: str, model: str,
except ImportError as exc:
raise click.ClickException(f"Embeddings module not available: {exc}") from exc
if output:
Path(output).write_text(json.dumps(result, default=str), encoding="utf-8")
output_path = Path(output)
suffix = output_path.suffix.lower()
try:
import numpy as np
import pandas as pd
arr = np.asarray(result)
if arr.ndim == 1:
arr = arr[np.newaxis, :]
if arr.ndim != 2:
raise click.ClickException(
f"embed generate --output expects a 1-D or 2-D array, "
f"got {arr.ndim}-D (shape {arr.shape})"
)
rows = [list(row) for row in arr]
if suffix == ".parquet":
# Schema: single 'embedding' column (list[float] per row).
# embed index detects vector columns via
# isinstance(df[c].iloc[0], (list, np.ndarray)).
df = pd.DataFrame({"embedding": rows})
df.to_parquet(output_path, index=False)
elif suffix in (".json", ".jsonl"):
df = pd.DataFrame({"embedding": rows})
df.to_json(
output_path,
orient="records",
lines=(suffix == ".jsonl"),
)
else:
raise click.ClickException(
f"Unsupported output format '{suffix}'. "
"Use .parquet, .json, or .jsonl"
)
except ImportError as exc:
raise click.ClickException(
f"Missing dependency for --output: {exc}. "
"Install pyarrow with: pip install pyarrow"
) from exc
_ok(cli_ctx, f"Wrote {output}")
elif _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, dict) else {"status": "ok"})
@@ -1930,7 +2119,7 @@ def deduplicate(
except ImportError as exc:
raise click.ClickException(f"Deduplication module not available: {exc}") from exc
if output:
Path(output).write_text(json.dumps(result, default=str), encoding="utf-8")
_write_result_output(Path(output), result)
_ok(cli_ctx, f"Wrote {output}")
elif _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, (dict, list)) else {"result": str(result)})
@@ -3054,7 +3243,7 @@ def ontology_align(cli_ctx: CLIContext, source: str, target: str, strategy: str,
except ImportError as exc:
raise click.ClickException(f"Ontology module not available: {exc}") from exc
if output:
Path(output).write_text(json.dumps(result, default=str), encoding="utf-8")
_write_result_output(Path(output), result)
_ok(cli_ctx, f"Wrote {output}")
elif _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, dict) else {"alignments": str(result)})
semantica/context/_markdown_filesystem.py
+32
View File
@@ -0,0 +1,32 @@
"""Filesystem safety helpers for human-editable Markdown persistence."""
import os
import stat
from pathlib import Path
from typing import Optional
def is_filesystem_link(path: Path) -> bool:
"""Return whether *path* is a symlink, junction, or Windows reparse point."""
if path.is_symlink():
return True
isjunction = getattr(os.path, "isjunction", None)
if isjunction is not None and isjunction(path):
return True
try:
attributes = getattr(os.lstat(path), "st_file_attributes", 0)
except (FileNotFoundError, NotADirectoryError):
return False
reparse_point = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400)
return bool(attributes & reparse_point)
def find_filesystem_link(path: Path) -> Optional[Path]:
"""Return the first linked component in *path*, including its ancestors."""
for candidate in (path, *path.parents):
if is_filesystem_link(candidate):
return candidate
return None
semantica/context/agent_memory.py
+49 -37
View File
@@ -77,6 +77,7 @@ import yaml
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ..utils.types import EntityDict, RelationshipDict
from ._markdown_filesystem import find_filesystem_link
class _UniqueKeySafeLoader(yaml.SafeLoader):
@@ -1742,9 +1743,10 @@ class AgentMemory:
@staticmethod
def _write_markdown_file(file_path: Path, document: str) -> None:
"""Atomically replace a Markdown file without following output symlinks."""
if file_path.is_symlink():
if find_filesystem_link(file_path) is not None:
raise ValueError(
f"Refusing to overwrite Markdown symbolic link: {file_path}"
"Refusing to overwrite Markdown symbolic link or junction: "
f"{file_path}"
)
temporary_path = None
@@ -1867,7 +1869,8 @@ class AgentMemory:
if "\n" not in data and "\r" not in data:
candidate = Path(data)
try:
candidate_exists = candidate.exists()
candidate_is_link = find_filesystem_link(candidate) is not None
candidate_exists = candidate_is_link or candidate.exists()
except OSError as exc:
error_message = (
"Failed to inspect possible Markdown import "
@@ -1909,62 +1912,71 @@ class AgentMemory:
return memories
def _read_markdown_file_content(self, file_path: Path) -> str:
if file_path.is_symlink():
raise ValueError(f"Symlink Markdown import paths are rejected: {file_path}")
if find_filesystem_link(file_path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links and "
f"junctions are unsafe: {file_path}"
)
flags = os.O_RDONLY
if hasattr(os, "O_NOFOLLOW"):
# On POSIX, O_NOFOLLOW makes os.open() fail with ELOOP if the
# final path component is a symlink, atomically closing the TOCTOU
# window between the is_symlink() check above and the open call.
# On Windows, O_NOFOLLOW is not available; the is_symlink() pre-check
# above is the only symlink defense and remains vulnerable to a narrow
# race. The fstat()/S_ISREG guard below still rejects special files
# (FIFOs, devices) on both platforms.
flags |= os.O_NOFOLLOW
nofollow_flag = getattr(os, "O_NOFOLLOW", 0)
flags |= nofollow_flag
try:
fd = os.open(str(file_path), flags)
except OSError as exc:
if exc.errno == getattr(errno, "ELOOP", None):
if (
(nofollow_flag and exc.errno == errno.ELOOP)
or find_filesystem_link(file_path) is not None
):
raise ValueError(
f"Symlink Markdown import paths are rejected: {file_path}"
"Symlink Markdown import paths are rejected; symbolic links "
f"and junctions are unsafe: {file_path}"
) from exc
raise
try:
stat_res = os.fstat(fd)
if not stat.S_ISREG(stat_res.st_mode):
if find_filesystem_link(file_path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links "
f"and junctions are unsafe: {file_path}"
)
if not stat.S_ISREG(os.fstat(fd).st_mode):
raise ValueError(
f"Markdown import path is not a regular file: {file_path}"
)
with open(fd, "r", encoding="utf-8", closefd=True) as f:
return f.read()
except Exception:
try:
with os.fdopen(fd, mode="r", encoding="utf-8") as source:
fd = -1
return source.read()
finally:
if fd >= 0:
os.close(fd)
except OSError:
pass
raise
def _read_markdown_path(self, path: Path) -> List[Tuple[str, str]]:
if path.is_symlink():
raise ValueError(f"Symlink Markdown import paths are rejected: {path}")
if find_filesystem_link(path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links and "
f"junctions are unsafe: {path}"
)
if not path.exists():
raise FileNotFoundError(f"Markdown import path does not exist: {path}")
if path.is_dir():
file_paths = sorted(
(
file_path
for file_path in path.iterdir()
if file_path.is_file()
and not file_path.is_symlink()
and file_path.suffix.lower() in self._MARKDOWN_EXTENSIONS
),
key=lambda file_path: (file_path.name.casefold(), file_path.name),
)
file_paths = []
for file_path in path.iterdir():
if file_path.suffix.lower() not in self._MARKDOWN_EXTENSIONS:
continue
if find_filesystem_link(file_path) is not None:
continue
if file_path.is_file():
file_paths.append(file_path)
if find_filesystem_link(path) is not None:
raise ValueError(
"Symlink Markdown import paths are rejected; symbolic links "
f"and junctions are unsafe: {path}"
)
file_paths.sort(key=lambda item: (item.name.casefold(), item.name))
elif path.is_file():
file_paths = [path]
else:
semantica/context/context_graph.py
+1248 -42
View File
File diff suppressed because it is too large Load Diff
semantica/context/graph_schema.py
+15
View File
@@ -6,6 +6,7 @@ including node labels, relationship types, and indexes for graph databases.
"""
import json
import warnings
from typing import Dict, Any, List
from ..graph_store import GraphStore
@@ -460,11 +461,25 @@ def drop_decision_schema(graph_store: GraphStore) -> None:
"""
Drop decision tracking schema (for cleanup/testing).
.. deprecated::
``drop_decision_schema()`` is deprecated and will be removed in a future
major version. It has no callers inside Semantica; issue the DROP
CONSTRAINT / DROP INDEX / DETACH DELETE statements directly against your
:class:`~semantica.graph_store.GraphStore` instead.
Args:
graph_store: Graph database instance
"""
logger = get_logger(__name__)
warnings.warn(
"drop_decision_schema() is deprecated and will be removed in a future "
"major version. Issue the DROP CONSTRAINT / DROP INDEX / DETACH DELETE "
"statements directly against your GraphStore instead.",
DeprecationWarning,
stacklevel=2,
)
try:
# Drop constraints
constraints = [
semantica/deduplication/duplicate_detector.py
+36 -8
View File
@@ -278,8 +278,12 @@ class DuplicateDetector:
for i, (entity1, entity2, score) in enumerate(similarities):
candidate = self._create_duplicate_candidate(entity1, entity2, score)
# Filter by confidence threshold
if candidate.confidence >= self.confidence_threshold:
# Filter by confidence threshold; type mismatches are excluded
# structurally so no threshold value can admit them.
if (
candidate.confidence >= self.confidence_threshold
and "type_mismatch" not in candidate.reasons
):
candidates.append(candidate)
remaining = total_similarities - (i + 1)
@@ -624,8 +628,12 @@ class DuplicateDetector:
new_entity, existing_entity, similarity.score
)
# Filter by confidence threshold
if candidate.confidence >= self.confidence_threshold:
# Filter by confidence threshold; type mismatches are
# excluded structurally regardless of the threshold.
if (
candidate.confidence >= self.confidence_threshold
and "type_mismatch" not in candidate.reasons
):
candidates.append(candidate)
processed += 1
@@ -723,7 +731,9 @@ class DuplicateDetector:
if key == "name":
return getattr(entity, "text", default)
if key == "type":
return getattr(entity, "label", default)
# Entity objects store the type on .type; extraction entities
# may expose .label. Missing .label never means "no type".
return getattr(entity, "type", default) or getattr(entity, "label", default)
if key == "properties":
# Check metadata for properties
metadata = getattr(entity, "metadata", {})
@@ -757,6 +767,25 @@ class DuplicateDetector:
reasons = []
confidence = similarity_score
# Check entity type mismatch first: two entities with different
# explicit types are not duplicates, whatever their similarity.
entity_type1 = self._get_entity_value(entity1, "type")
entity_type2 = self._get_entity_value(entity2, "type")
if entity_type1 and entity_type2 and entity_type1 != entity_type2:
return DuplicateCandidate(
entity1=entity1,
entity2=entity2,
similarity_score=similarity_score,
confidence=0.0,
reasons=["type_mismatch"],
metadata={
"name_match": False,
"common_properties": 0,
"type_match": False,
"type_mismatch": True,
},
)
# Check for exact name match (strong indicator)
name1 = str(self._get_entity_value(entity1, "name", "")).lower().strip()
name2 = str(self._get_entity_value(entity2, "name", "")).lower().strip()
@@ -779,9 +808,8 @@ class DuplicateDetector:
# Boost confidence for each matching property
confidence += 0.05 * prop_matches
# Check entity type match
entity_type1 = self._get_entity_value(entity1, "type")
entity_type2 = self._get_entity_value(entity2, "type")
# Check entity type match (only boosts when types are equal; mismatch
# is handled above)
if entity_type1 and entity_type2 and entity_type1 == entity_type2:
reasons.append("same_type")
confidence += 0.05
semantica/embeddings/embeddings_provenance.py
+9
View File
@@ -69,6 +69,15 @@ class EmbeddingGeneratorWithProvenance:
return embeddings
def __getattr__(self, name):
# __getattr__ only runs when normal lookup fails. Accessing
# self._generator by attribute syntax HERE would re-enter
# __getattr__ for ever when _generator itself is missing — the shape
# pickle/copy protocol probes hit when __init__ never completed
# (#994's RecursionError family). Fail fast on private probes.
if name.startswith("_"):
raise AttributeError(
f"{type(self).__name__!r} object has no attribute {name!r}"
)
return getattr(self._generator, name)
semantica/embeddings/methods.py
+33 -32
View File
@@ -80,6 +80,7 @@ import numpy as np
from ..utils.exceptions import ConfigurationError, ProcessingError
from ..utils.logging import get_logger
from ..utils.custom_methods import CUSTOM_METHOD_FELL_BACK, call_custom_method
from .config import embeddings_config
from .embedding_generator import EmbeddingGenerator
from .pooling_strategies import PoolingStrategyFactory
@@ -116,15 +117,15 @@ def generate_embeddings(
>>> emb = generate_embeddings("Hello world", method="default")
>>> embs = generate_embeddings(["text1", "text2"], method="text")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("generation", method)
if custom_method:
try:
return custom_method(data, data_type=data_type, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
if custom_method and custom_method is not generate_embeddings:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, data_type=data_type, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
if method == "default":
@@ -164,15 +165,15 @@ def embed_text(
>>> emb = embed_text("Hello world", method="sentence_transformers")
>>> embs = embed_text(["text1", "text2"], method="sentence_transformers")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("text", method)
if custom_method:
try:
return custom_method(text, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
if custom_method and custom_method is not embed_text:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, text, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -224,15 +225,15 @@ def calculate_similarity(
>>> similarity = calculate_similarity(emb1, emb2, method="cosine")
>>> print(f"Similarity: {similarity:.3f}")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("similarity", method)
if custom_method:
try:
return custom_method(embedding1, embedding2, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
if custom_method and custom_method is not calculate_similarity:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, embedding1, embedding2, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
generator = EmbeddingGenerator(**kwargs)
@@ -271,15 +272,15 @@ def pool_embeddings(
>>> pooled = pool_embeddings(embeddings, method="mean")
>>> attention_pooled = pool_embeddings(embeddings, method="attention")
"""
# Check for custom method in registry
# Check for custom method in registry, skip self-reference
custom_method = method_registry.get("pooling", method)
if custom_method:
try:
return custom_method(embeddings, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
if custom_method and custom_method is not pool_embeddings:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, embeddings, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
strategy = PoolingStrategyFactory.create(method, **kwargs)
semantica/explorer/dependencies.py
+3 -13
View File
@@ -2,8 +2,9 @@
Semantica Explorer : FastAPI Dependencies
Provides ``Depends()``-compatible callables for injecting the
current ``GraphSession`` and ``ConnectionManager`` into route handlers,
and for enforcing API-key authentication on protected routes.
current ``GraphSession`` into route handlers, and for enforcing API-key
authentication on protected routes. WebSocket manager access is handled
directly via ``app.state.ws_manager``.
"""
import hmac
@@ -14,7 +15,6 @@ from fastapi import Request, HTTPException, Security, status
from fastapi.security.api_key import APIKeyHeader
from .session import GraphSession
from .ws import ConnectionManager
_api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)
@@ -80,13 +80,3 @@ def get_session(request: Request) -> GraphSession:
detail="GraphSession not initialized."
)
return request.app.state.session
def get_ws_manager(request: Request) -> ConnectionManager:
"""Retrieve the ConnectionManager stored on ``app.state``."""
if not hasattr(request.app.state, "ws_manager") or request.app.state.ws_manager is None:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="WebSocket manager not initialized.",
)
return request.app.state.ws_manager
semantica/explorer/routes/graph.py
-60
View File
@@ -78,66 +78,6 @@ def _parse_bbox(raw_bbox: Optional[str]) -> Optional[tuple[float, float, float,
return min_x, min_y, max_x, max_y
def _coerce_embedding_vector(value: object) -> Optional[List[float]]:
if isinstance(value, dict):
# Probe keys in priority order: generic first, then framework-specific.
# Must stay aligned with the top-level keys in _extract_node_embeddings.
for key in ("embedding", "embeddings", "vector", "values", "node2vec", "semantic"):
nested = _coerce_embedding_vector(value.get(key))
if nested is not None:
return nested
return None
if not isinstance(value, (list, tuple)):
return None
vector: List[float] = []
for item in value:
try:
vector.append(float(item))
except (TypeError, ValueError):
return None
return vector if vector else None
def _extract_node_embeddings(graph_dict: dict) -> dict[str, List[float]]:
"""Extract embeddings from graph dictionary."""
# Top-level keys to probe on each entity (and its metadata/properties dicts).
# Priority: generic names first, then KG-extras-specific names.
# Must stay aligned with the inner probe list in _coerce_embedding_vector.
embedding_keys = (
"embedding",
"embeddings",
"vector",
"node_embedding",
"node2vec_embedding",
"semantic_embedding",
"reasoning_embedding",
)
embeddings: dict[str, List[float]] = {}
for entity in graph_dict.get("entities") or graph_dict.get("nodes") or []:
if not isinstance(entity, dict):
continue
node_id = entity.get("id") or entity.get("node_id")
if not node_id:
continue
metadata = entity.get("metadata") if isinstance(entity.get("metadata"), dict) else {}
properties = entity.get("properties") if isinstance(entity.get("properties"), dict) else {}
for key in embedding_keys:
vector = _coerce_embedding_vector(
entity.get(key, metadata.get(key, properties.get(key)))
)
if vector is not None:
embeddings[str(node_id)] = vector
break
return embeddings
def _get_cached_embeddings(session: GraphSession) -> dict[str, List[float]]:
"""Get embeddings from session cache for optimal performance."""
return session.get_cached_embeddings()
semantica/explorer/routes/ontology.py
-4
View File
@@ -456,10 +456,6 @@ class DraftResponse(BaseModel):
updated_at: str
class ProposalState(BaseModel):
state: Literal["draft", "proposed", "approved", "published", "rejected"]
class ProposalRequest(BaseModel):
draft_id: str
ontology_uri: str
semantica/explorer/schemas.py
-23
View File
@@ -8,11 +8,6 @@ from typing import Any, Dict, List, Literal, Optional, Tuple
from pydantic import BaseModel, Field, field_validator
class ErrorResponse(BaseModel):
detail: str
status_code: int = 500
class NodeResponse(BaseModel):
id: str
type: str
@@ -187,12 +182,6 @@ class ComplianceResponse(BaseModel):
violations: List[Dict[str, Any]] = Field(default_factory=list)
class TemporalSnapshotResponse(BaseModel):
timestamp: str
active_nodes: List[NodeResponse]
active_node_count: int
class TemporalDiffResponse(BaseModel):
from_time: str
to_time: str
@@ -256,13 +245,6 @@ class ExportRequest(BaseModel):
node_ids: Optional[List[str]] = None
class ExportResponse(BaseModel):
format: str
content_type: str
filename: str
size_bytes: int = 0
class ImportResponse(BaseModel):
status: str = "success"
message: str = "Import successful"
@@ -272,11 +254,6 @@ class ImportResponse(BaseModel):
edges_imported: Optional[int] = None
class StandardMessageResponse(BaseModel):
status: str
message: str
class AnnotationCreate(BaseModel):
node_id: str
content: str
semantica/export/json_exporter.py
+136 -9
View File
@@ -28,12 +28,54 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory, utc_now_iso, write_json_file
from ..utils.helpers import ensure_directory, hash_data, utc_now_iso, write_json_file
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri
def _content_iri(prefix: str, payload: Any) -> str:
"""Mint a document IRI from what was exported, not when.
Minting from ``utc_now_iso()`` gave every export of the same graph a new
identity a few microseconds apart, so re-exporting an unchanged graph was
never idempotent and merging exports duplicated every node (#1147). This
mirrors ``mint_entity_iri`` (#1109): identical content hashes to the same
IRI, and any change to the content changes it too. ``default=str`` keeps
the hash defined for values ``json.dumps`` would otherwise reject, such as
``datetime`` objects a caller may have left in the graph.
Args:
prefix: IRI prefix the digest is appended to
payload: JSON-serializable value whose content determines the digest
Returns:
A stable IRI of the form ``{prefix}{16-hex-char digest}``
"""
canonical = json.dumps(payload, sort_keys=True, default=str)
digest = hash_data(canonical)[:16]
return f"{prefix}{digest}"
def _is_jsonld_document(data: Dict[str, Any]) -> bool:
"""
Report whether a dictionary is already a JSON-LD document.
``export_knowledge_graph`` converts a knowledge graph to JSON-LD and then
hands the finished document to ``export()``, which converted it a second
time. The converted document no longer carries ``entities``/
``relationships`` keys, so the second pass treated it as an opaque value and
buried it inside ``@graph``.
Args:
data: Dictionary to test
Returns:
True when the dictionary declares a JSON-LD context
"""
return "@context" in data
class JSONExporter:
"""
JSON exporter for knowledge graphs and semantic data.
@@ -211,7 +253,10 @@ class JSONExporter:
- statistics: Statistics dictionary (optional)
file_path: Output JSON file path
format: Export format - 'json' or 'json-ld' (default: self.format)
**options: Additional options passed to conversion methods
**options: Additional options passed to conversion methods:
- graph_uri: Caller-supplied IRI for the graph node when
format='json-ld', overriding the default content-derived
IRI (see #1147)
Example:
>>> kg = {
@@ -382,7 +427,9 @@ class JSONExporter:
data: Data to convert (dict, list, or any value)
include_metadata: Whether to include metadata (default: True)
include_provenance: Whether to include provenance (default: True)
**options: Additional options passed to knowledge graph conversion
**options: Additional options passed to knowledge graph conversion:
- document_uri: Caller-supplied IRI for the document node,
overriding the default content-derived IRI (see #1147)
Returns:
Dictionary in JSON-LD format with @context, @graph/@value, and metadata
@@ -397,9 +444,29 @@ class JSONExporter:
# Convert data based on type
if isinstance(data, dict):
# A knowledge graph is converted even when it carries a context of
# its own: the specialized conversion is what mints entity ids and
# relationship endpoints, and skipping it leaves them raw keys.
if "entities" in data or "relationships" in data:
# Knowledge graph structure - use specialized conversion
jsonld.update(self._convert_kg_to_jsonld(data, **options))
elif _is_jsonld_document(data):
# Already JSON-LD: merge it rather than nesting it. Wrapping a
# converted document in @graph re-typed the payload as a named
# graph and doubled the @context, which is what happened when
# export_knowledge_graph handed its own output back to export().
context = data.get("@context")
if isinstance(context, dict):
jsonld["@context"].update(context)
elif context is not None:
# A context may also be a URL or an array of them, which
# cannot be merged key by key. Keeping both as an array
# preserves the caller's term expansion, which wins over
# ours, while still defining the semantica prefix. An
# explicit null is left alone: in an array it would reset
# the active context and take our own terms with it.
jsonld["@context"] = [jsonld["@context"], context]
jsonld.update({k: v for k, v in data.items() if k != "@context"})
else:
# Generic dictionary - wrap in @graph
jsonld["@graph"] = [data]
@@ -412,13 +479,67 @@ class JSONExporter:
# Add metadata and provenance if requested
if include_metadata:
jsonld["@id"] = f"https://semantica.dev/data/{utc_now_iso()}"
if include_provenance:
jsonld["semantica:exportedAt"] = utc_now_iso()
jsonld["semantica:format"] = "json-ld"
self._attach_document_metadata(
jsonld, include_provenance, options.get("document_uri")
)
return jsonld
@staticmethod
def _attach_document_metadata(
jsonld: Dict[str, Any],
include_provenance: bool,
document_uri: Optional[str] = None,
) -> None:
"""
Attach the export's own metadata without naming the graph.
A top-level ``@id`` alongside a top-level ``@graph`` is a *named graph*:
the members of ``@graph`` become quads named by that ``@id`` and leave
the default graph empty. ``rdflib.Graph.parse()`` keeps only the default
graph, so every statement in the export was discarded without an error
(2 of 21 statements survived a two-entity knowledge graph). When the
payload lives in ``@graph``, the document node goes in beside it as one
more node; otherwise it is the document itself.
Args:
jsonld: Document being built, modified in place
include_provenance: Whether to record how and when it was exported
document_uri: Caller-supplied IRI for the document node. Falls back
to a content-derived IRI (#1147) so re-exporting unchanged data
is idempotent instead of minting a new identity every time.
"""
# A caller may hand us a document that is deliberately a named graph.
# That name is theirs to keep, but our own statements must not end up
# inside it, where a default-graph reader would never see them.
payload_is_named_graph = "@id" in jsonld and "@graph" in jsonld
document: Dict[str, Any] = {}
# Do not overwrite an identifier the payload already carries: the
# knowledge-graph conversion names its own document node.
if "@id" not in jsonld or payload_is_named_graph:
content = {key: value for key, value in jsonld.items() if key != "@context"}
document["@id"] = document_uri or _content_iri(
"https://semantica.dev/data/", content
)
if include_provenance:
document["semantica:exportedAt"] = utc_now_iso()
document["semantica:format"] = "json-ld"
if payload_is_named_graph:
named = {key: value for key, value in jsonld.items() if key != "@context"}
for key in [key for key in jsonld if key != "@context"]:
del jsonld[key]
jsonld["@graph"] = [named, document]
elif "@graph" in jsonld:
# @graph may be a single node object as well as an array. list() on
# a dictionary yields its keys, which would discard the node.
members = jsonld["@graph"]
members = list(members) if isinstance(members, list) else [members]
jsonld["@graph"] = members + [document]
else:
jsonld.update(document)
def _convert_kg_to_json(self, kg: Dict[str, Any], **options) -> Dict[str, Any]:
"""
Convert knowledge graph to JSON format.
@@ -470,7 +591,9 @@ class JSONExporter:
- entities: List of entity dictionaries
- relationships: List of relationship dictionaries
- metadata: Metadata dictionary (optional)
**options: Additional options (unused)
**options: Additional options:
- graph_uri: Caller-supplied IRI for the graph node,
overriding the default content-derived IRI (see #1147)
Returns:
Dictionary in JSON-LD format with @context, @id, @type, and graph data
@@ -483,7 +606,11 @@ class JSONExporter:
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
},
"@id": f"https://semantica.dev/graph/{utc_now_iso()}",
# Minted from the graph's own content rather than the wall clock
# (#1147): re-exporting an unchanged graph must produce the same
# subject, or merging repeated exports duplicates every node.
"@id": options.get("graph_uri")
or _content_iri("https://semantica.dev/graph/", kg),
"@type": "semantica:KnowledgeGraph",
}
semantica/export/methods.py
+79 -78
View File
@@ -164,6 +164,7 @@ from typing import Any, Callable, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
from ..utils.custom_methods import CUSTOM_METHOD_FELL_BACK, call_custom_method
from .arango_aql_exporter import ArangoAQLExporter
from .arrow_exporter import ArrowExporter
from .config import export_config
@@ -221,12 +222,12 @@ def export_rdf(
# Check for custom method in registry
custom_method = method_registry.get("rdf", method)
if custom_method and custom_method is not export_rdf:
try:
return custom_method(data, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, format=format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -270,12 +271,12 @@ def export_json(
# Check for custom method in registry
custom_method = method_registry.get("json", method)
if custom_method and custom_method is not export_json:
try:
return custom_method(data, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, format=format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -316,12 +317,12 @@ def export_csv(
# Check for custom method in registry
custom_method = method_registry.get("csv", method)
if custom_method and custom_method is not export_csv:
try:
return custom_method(data, file_path, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -361,12 +362,12 @@ def export_arrow(
# Check for custom method in registry
custom_method = method_registry.get("arrow", method)
if custom_method and custom_method is not export_arrow:
try:
return custom_method(data, file_path, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -421,12 +422,12 @@ def export_parquet(
# Check for custom method in registry
custom_method = method_registry.get("parquet", method)
if custom_method and custom_method is not export_parquet:
try:
return custom_method(data, file_path, compression=compression, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, compression=compression, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -472,12 +473,12 @@ def export_graph(
# Check for custom method in registry
custom_method = method_registry.get("graph", method)
if custom_method and custom_method is not export_graph:
try:
return custom_method(graph_data, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, graph_data, file_path, format=format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -538,12 +539,12 @@ def export_yaml(
# Check for custom method in registry
custom_method = method_registry.get("yaml", method)
if custom_method and custom_method is not export_yaml:
try:
return custom_method(data, file_path, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -595,12 +596,12 @@ def export_owl(
# Check for custom method in registry
custom_method = method_registry.get("owl", method)
if custom_method and custom_method is not export_owl:
try:
return custom_method(ontology, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, ontology, file_path, format=format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -647,12 +648,12 @@ def export_vector(
# Check for custom method in registry
custom_method = method_registry.get("vector", method)
if custom_method and custom_method is not export_vector:
try:
return custom_method(vectors, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, vectors, file_path, format=format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -695,12 +696,12 @@ def export_lpg(
# Check for custom method in registry
custom_method = method_registry.get("lpg", method)
if custom_method and custom_method is not export_lpg:
try:
return custom_method(knowledge_graph, file_path, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, knowledge_graph, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -738,12 +739,12 @@ def export_neo4j_csv(
"""
custom_method = method_registry.get("neo4j_csv", method)
if custom_method and custom_method is not export_neo4j_csv:
try:
return custom_method(knowledge_graph, output_dir, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, knowledge_graph, output_dir, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
config = export_config.get_method_config("neo4j_csv")
@@ -808,12 +809,12 @@ def export_arango(
# Check for custom method in registry
custom_method = method_registry.get("arango", method)
if custom_method and custom_method is not export_arango:
try:
return custom_method(knowledge_graph, file_path, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, knowledge_graph, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -859,12 +860,12 @@ def generate_report(
# Check for custom method in registry
custom_method = method_registry.get("report", method)
if custom_method and custom_method is not generate_report:
try:
return custom_method(data, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, data, file_path, format=format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
semantica/export/owl_exporter.py
+362 -90
View File
@@ -22,8 +22,10 @@ Author: Semantica Contributors
License: MIT
"""
import re
from datetime import datetime
from pathlib import Path
from urllib.parse import quote
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
@@ -37,6 +39,9 @@ from ..utils.progress_tracker import get_progress_tracker
# PROV-exported URIs co-resolve under one shared namespace by default.
from ..provenance.manager import DEFAULT_BASE_URI
#: Module-level logger, for the classmethod helpers that have no instance.
logger = get_logger("owl_exporter")
class OWLExporter:
"""
@@ -225,6 +230,7 @@ class OWLExporter:
Returns:
String containing OWL-XML serialization
"""
esc_xml = self._escape_xml
ontology_uri = ontology.get("uri") or self.ontology_uri
ontology_name = ontology.get("name", "SemanticaOntology")
version = ontology.get("version") or self.version
@@ -237,97 +243,125 @@ class OWLExporter:
lines.append("")
# Ontology declaration
lines.append(f' <owl:Ontology rdf:about="{ontology_uri}">')
lines.append(f" <rdfs:label>{ontology_name}</rdfs:label>")
lines.append(f" <owl:versionInfo>{version}</owl:versionInfo>")
lines.append(f' <owl:Ontology rdf:about="{esc_xml(ontology_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(ontology_name)}</rdfs:label>")
lines.append(f" <owl:versionInfo>{esc_xml(version)}</owl:versionInfo>")
if ontology.get("description"):
lines.append(
f' <rdfs:comment>{ontology.get("description")}</rdfs:comment>'
f' <rdfs:comment>{esc_xml(ontology.get("description"))}</rdfs:comment>'
)
lines.append(" </owl:Ontology>")
lines.append("")
class_index = self._class_iri_index(ontology, ontology_uri)
object_properties, data_properties = self._split_properties(ontology)
def _as_list(value):
if value is None:
return []
return value if isinstance(value, list) else [value]
# Classes
classes = ontology.get("classes", [])
for cls in classes:
class_uri = cls.get("uri") or cls.get("id", "")
for cls in ontology.get("classes", []) or []:
if not isinstance(cls, dict):
continue
class_uri = self._term_iri(cls, ontology_uri)
if not class_uri:
self.logger.warning(
"Skipping a class with no name, uri or id: it would serialise "
"as an empty rdf:about"
)
continue
class_name = cls.get("name") or cls.get("label", "")
lines.append(f' <owl:Class rdf:about="{class_uri}">')
lines.append(f" <rdfs:label>{class_name}</rdfs:label>")
lines.append(f' <owl:Class rdf:about="{esc_xml(class_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(class_name)}</rdfs:label>")
if cls.get("comment"):
lines.append(f' <rdfs:comment>{cls.get("comment")}</rdfs:comment>')
comment = cls.get("comment") or cls.get("description")
if comment:
lines.append(f" <rdfs:comment>{esc_xml(comment)}</rdfs:comment>")
# Subclass relationships
if cls.get("subClassOf"):
parent = cls.get("subClassOf")
lines.append(f' <rdfs:subClassOf rdf:resource="{parent}"/>')
for parent in _as_list(cls.get("subClassOf") or cls.get("parent")):
parent_iri = self._resolve_class_ref(parent, ontology_uri, class_index)
if parent_iri:
lines.append(
f' <rdfs:subClassOf rdf:resource="{esc_xml(parent_iri)}"/>'
)
# Equivalent classes
if cls.get("equivalentClass"):
equiv = cls.get("equivalentClass")
lines.append(f' <owl:equivalentClass rdf:resource="{equiv}"/>')
for equiv in _as_list(cls.get("equivalentClass")):
equiv_iri = self._resolve_class_ref(equiv, ontology_uri, class_index)
if equiv_iri:
lines.append(
f' <owl:equivalentClass rdf:resource="{esc_xml(equiv_iri)}"/>'
)
lines.append(" </owl:Class>")
lines.append("")
# Object properties
object_properties = ontology.get("object_properties", [])
for prop in object_properties:
prop_uri = prop.get("uri") or prop.get("id", "")
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning(
"Skipping an object property with no name, uri or id"
)
continue
prop_name = prop.get("name") or prop.get("label", "")
lines.append(f' <owl:ObjectProperty rdf:about="{prop_uri}">')
lines.append(f" <rdfs:label>{prop_name}</rdfs:label>")
lines.append(f' <owl:ObjectProperty rdf:about="{esc_xml(prop_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(prop_name)}</rdfs:label>")
if prop.get("comment"):
lines.append(f' <rdfs:comment>{prop.get("comment")}</rdfs:comment>')
comment = prop.get("comment") or prop.get("description")
if comment:
lines.append(f" <rdfs:comment>{esc_xml(comment)}</rdfs:comment>")
# Domain
if prop.get("domain"):
domain = prop.get("domain")
if isinstance(domain, list):
for d in domain:
lines.append(f' <rdfs:domain rdf:resource="{d}"/>')
else:
lines.append(f' <rdfs:domain rdf:resource="{domain}"/>')
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
lines.append(
f' <rdfs:domain rdf:resource="{esc_xml(domain_iri)}"/>'
)
# Range
if prop.get("range"):
range_val = prop.get("range")
if isinstance(range_val, list):
for r in range_val:
lines.append(f' <rdfs:range rdf:resource="{r}"/>')
else:
lines.append(f' <rdfs:range rdf:resource="{range_val}"/>')
for range_val in _as_list(prop.get("range")):
range_iri = self._resolve_class_ref(range_val, ontology_uri, class_index)
if range_iri:
lines.append(
f' <rdfs:range rdf:resource="{esc_xml(range_iri)}"/>'
)
lines.append(" </owl:ObjectProperty>")
lines.append("")
# Data properties
data_properties = ontology.get("data_properties", [])
for prop in data_properties:
prop_uri = prop.get("uri") or prop.get("id", "")
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning("Skipping a data property with no name, uri or id")
continue
prop_name = prop.get("name") or prop.get("label", "")
lines.append(f' <owl:DatatypeProperty rdf:about="{prop_uri}">')
lines.append(f" <rdfs:label>{prop_name}</rdfs:label>")
lines.append(f' <owl:DatatypeProperty rdf:about="{esc_xml(prop_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(prop_name)}</rdfs:label>")
if prop.get("comment"):
lines.append(f' <rdfs:comment>{prop.get("comment")}</rdfs:comment>')
comment = prop.get("comment") or prop.get("description")
if comment:
lines.append(f" <rdfs:comment>{esc_xml(comment)}</rdfs:comment>")
# Domain
if prop.get("domain"):
domain = prop.get("domain")
lines.append(f' <rdfs:domain rdf:resource="{domain}"/>')
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
lines.append(
f' <rdfs:domain rdf:resource="{esc_xml(domain_iri)}"/>'
)
# Range
if prop.get("range"):
range_type = prop.get("range", "xsd:string")
lines.append(
f' <rdfs:range rdf:resource="http://www.w3.org/2001/XMLSchema#{range_type}"/>'
)
for range_val in _as_list(prop.get("range")):
range_iri = self._resolve_datatype_iri(range_val)
if range_iri:
lines.append(
f' <rdfs:range rdf:resource="{esc_xml(range_iri)}"/>'
)
lines.append(" </owl:DatatypeProperty>")
lines.append("")
@@ -335,6 +369,223 @@ class OWLExporter:
lines.append("</rdf:RDF>")
return "\n".join(lines)
# ── Ontology-dict normalisation ───────────────────────────────────────────
#
# OntologyGenerator emits a single `properties` list tagged with
# type/@type, while hand-authored ontologies use `object_properties` and
# `data_properties`. Both shapes are accepted; everything below works from
# the normalised view so the two cannot drift apart again (#1103).
_XSD_NS = "http://www.w3.org/2001/XMLSchema#"
#: Prefixes the generator and hand-authored ontologies actually use. A
#: prefixed name is not an absolute IRI: `owl:Thing` matches the generic
#: scheme grammar, so treating it as one produced <owl:Thing> as a domain,
#: which is a different term from http://www.w3.org/2002/07/owl#Thing.
_KNOWN_PREFIXES = {
"owl": "http://www.w3.org/2002/07/owl#",
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"xsd": _XSD_NS,
"skos": "http://www.w3.org/2004/02/skos/core#",
"dc": "http://purl.org/dc/elements/1.1/",
"dcterms": "http://purl.org/dc/terms/",
"foaf": "http://xmlns.com/foaf/0.1/",
"sem": "https://semantica.dev/ns#",
"semantica": "https://semantica.dev/ns#",
}
#: Schemes that really do introduce an absolute IRI without `//`.
_ABSOLUTE_SCHEMES = ("urn:", "doi:", "mailto:", "tag:", "uuid:")
@classmethod
def _is_absolute_iri(cls, value: str) -> bool:
if not isinstance(value, str):
return False
value = value.strip()
if "://" in value:
return bool(re.match(r"^[A-Za-z][A-Za-z0-9+.\-]*://", value))
return value.lower().startswith(cls._ABSOLUTE_SCHEMES)
@classmethod
def _expand_prefixed_name(cls, value: str) -> str:
"""Expand a known prefixed name, or return "" when it cannot be expanded."""
prefix, _, local = value.partition(":")
namespace = cls._KNOWN_PREFIXES.get(prefix)
return f"{namespace}{local}" if namespace and local else ""
@staticmethod
def _iri_safe(local: str) -> str:
"""
Percent-encode a local name so it can sit inside <>.
A name is free text. "Customer Account" pasted onto a base gives an IRI
with a space in it, which rdflib only warns about and Oxigraph rejects
with "Invalid IRI code point".
"""
return quote(local.strip(), safe="~._-!$&'()*+,;=:@/?")
@classmethod
def _join_iri(cls, base: str, local: str) -> str:
"""Append a local name to a base IRI, respecting hash and slash bases."""
if not base:
return ""
local = cls._iri_safe(local)
if not local:
return ""
separator = "" if base.endswith(("#", "/", ":")) else "#"
return f"{base}{separator}{local}"
@classmethod
def _term_iri(cls, term: Dict[str, Any], base: str) -> str:
"""
Resolve the IRI of a class or property.
Returns "" when the term carries nothing usable, so the caller can skip
it. Interpolating an empty string into <> silently resolves against the
parser's base — under rdflib that is the current working directory — and
collapses every such term onto one subject.
"""
for key in ("uri", "iri", "id"):
value = term.get(key)
if isinstance(value, str) and value.strip():
value = value.strip()
return value if cls._is_absolute_iri(value) else cls._join_iri(base, value)
name = term.get("name") or term.get("label")
if isinstance(name, str) and name.strip():
return cls._join_iri(base, name.strip())
return ""
@classmethod
def _class_iri_index(cls, ontology: Dict[str, Any], base: str) -> Dict[str, str]:
"""Map class name and label to the IRI that class is actually exported under."""
index: Dict[str, str] = {}
for class_def in ontology.get("classes", []) or []:
if not isinstance(class_def, dict):
continue
iri = cls._term_iri(class_def, base)
if not iri:
continue
for key in (class_def.get("name"), class_def.get("label")):
if isinstance(key, str) and key.strip():
index.setdefault(key.strip(), iri)
return index
@classmethod
def _resolve_class_ref(cls, value: Any, base: str, index: Dict[str, str]) -> str:
"""
Resolve a domain/range reference to an absolute IRI.
The generator writes bare class names here. Looking the name up in the
class index first means a reference always lands on the IRI that class
was exported under, rather than on a re-derived guess.
"""
if not isinstance(value, str) or not value.strip():
return ""
value = value.strip()
if cls._is_absolute_iri(value):
return value
if value in index:
return index[value]
if ":" in value:
return cls._expand_prefixed_name(value)
return cls._join_iri(base, value)
@classmethod
def _resolve_datatype_iri(cls, value: Any) -> str:
"""
Resolve a data property range to an absolute datatype IRI.
Accepts "string", "xsd:string" and a full IRI alike. The previous
`xsd:{range}` interpolation doubled the prefix whenever the generator
had already written "xsd:string".
"""
if not isinstance(value, str) or not value.strip():
return ""
value = value.strip()
if value.startswith(("xsd:", "XSD:")):
return cls._XSD_NS + value.split(":", 1)[1]
if cls._is_absolute_iri(value):
return value
return cls._XSD_NS + value
@classmethod
def _ttl_datatype_ref(cls, value: Any) -> str:
"""
Render a data property range for Turtle.
XSD datatypes are written with the xsd: prefix the header already
declares; anything else is written as a full IRI. Both are the same
term, this only keeps the compact style the module was written in.
"""
iri = cls._resolve_datatype_iri(value)
if not iri:
return ""
if iri.startswith(cls._XSD_NS):
return f"xsd:{iri[len(cls._XSD_NS):]}"
return f"<{iri}>"
@classmethod
def _split_properties(
cls, ontology: Dict[str, Any]
) -> "tuple[List[Dict[str, Any]], List[Dict[str, Any]]]":
"""
Return (object_properties, data_properties) across both dict shapes.
A property listed under an explicit key keeps that key's kind. A
property from the generator's combined `properties` list is classified
by its own type/@type, defaulting to a data property.
"""
object_props: List[Dict[str, Any]] = []
data_props: List[Dict[str, Any]] = []
for prop in ontology.get("object_properties", []) or []:
if isinstance(prop, dict):
object_props.append(prop)
for prop in ontology.get("data_properties", []) or []:
if isinstance(prop, dict):
data_props.append(prop)
skipped = 0
untyped = []
for prop in ontology.get("properties", []) or []:
if not isinstance(prop, dict):
skipped += 1
continue
kind = str(prop.get("type") or "").strip().lower()
owl_type = str(prop.get("@type") or "").strip().lower()
if kind in ("object", "objectproperty") or owl_type.endswith("objectproperty"):
object_props.append(prop)
else:
if not kind and not owl_type:
untyped.append(prop.get("name") or prop.get("uri") or "<unnamed>")
data_props.append(prop)
if skipped:
logger.warning(
f"Skipped {skipped} entr(y/ies) in 'properties' that are not "
"dictionaries and cannot be exported"
)
if untyped:
logger.warning(
"Exported as data properties because they declare no type or "
f"@type: {', '.join(str(name) for name in untyped)}"
)
return object_props, data_props
@staticmethod
def _escape_xml(value: Any) -> str:
"""Escape a value for safe embedding in XML text or an attribute value."""
return (
str(value)
.replace("&", "&amp;")
.replace("<", "&lt;")
.replace(">", "&gt;")
.replace('"', "&quot;")
)
@staticmethod
def _escape_ttl_str(value: str) -> str:
"""Escape a string value for safe embedding in a Turtle string literal."""
@@ -387,62 +638,83 @@ class OWLExporter:
lines.append(self._ttl_block(ontology_uri, "owl:Ontology", onto_predicates))
lines.append("")
class_index = self._class_iri_index(ontology, ontology_uri)
object_properties, data_properties = self._split_properties(ontology)
def _as_list(value):
if value is None:
return []
return value if isinstance(value, list) else [value]
# Classes
for cls in ontology.get("classes", []):
class_uri = cls.get("uri") or cls.get("id", "")
for cls in ontology.get("classes", []) or []:
if not isinstance(cls, dict):
continue
class_uri = self._term_iri(cls, ontology_uri)
if not class_uri:
self.logger.warning(
"Skipping a class with no name, uri or id: it would serialise as <>"
)
continue
class_name = cls.get("name") or cls.get("label", "")
predicates = [f'rdfs:label "{esc(class_name)}"']
comment = cls.get("comment")
comment = cls.get("comment") or cls.get("description")
if comment:
predicates.append(f'rdfs:comment "{esc(comment)}"')
sub_class = cls.get("subClassOf")
if sub_class:
predicates.append(f"rdfs:subClassOf <{sub_class}>")
equiv = cls.get("equivalentClass")
if equiv:
predicates.append(f"owl:equivalentClass <{equiv}>")
for parent in _as_list(cls.get("subClassOf") or cls.get("parent")):
parent_iri = self._resolve_class_ref(parent, ontology_uri, class_index)
if parent_iri:
predicates.append(f"rdfs:subClassOf <{parent_iri}>")
for equiv in _as_list(cls.get("equivalentClass")):
equiv_iri = self._resolve_class_ref(equiv, ontology_uri, class_index)
if equiv_iri:
predicates.append(f"owl:equivalentClass <{equiv_iri}>")
lines.append(self._ttl_block(class_uri, "owl:Class", predicates))
lines.append("")
# Object properties
for prop in ontology.get("object_properties", []):
prop_uri = prop.get("uri") or prop.get("id", "")
for prop in object_properties:
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning(
"Skipping an object property with no name, uri or id"
)
continue
prop_name = prop.get("name") or prop.get("label", "")
predicates = [f'rdfs:label "{esc(prop_name)}"']
comment = prop.get("comment")
comment = prop.get("comment") or prop.get("description")
if comment:
predicates.append(f'rdfs:comment "{esc(comment)}"')
domain = prop.get("domain")
if domain:
if isinstance(domain, list):
for d in domain:
predicates.append(f"rdfs:domain <{d}>")
else:
predicates.append(f"rdfs:domain <{domain}>")
range_val = prop.get("range")
if range_val:
if isinstance(range_val, list):
for r in range_val:
predicates.append(f"rdfs:range <{r}>")
else:
predicates.append(f"rdfs:range <{range_val}>")
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
predicates.append(f"rdfs:domain <{domain_iri}>")
for range_val in _as_list(prop.get("range")):
range_iri = self._resolve_class_ref(range_val, ontology_uri, class_index)
if range_iri:
predicates.append(f"rdfs:range <{range_iri}>")
lines.append(self._ttl_block(prop_uri, "owl:ObjectProperty", predicates))
lines.append("")
# Data properties
for prop in ontology.get("data_properties", []):
prop_uri = prop.get("uri") or prop.get("id", "")
for prop in data_properties:
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning("Skipping a data property with no name, uri or id")
continue
prop_name = prop.get("name") or prop.get("label", "")
predicates = [f'rdfs:label "{esc(prop_name)}"']
comment = prop.get("comment")
comment = prop.get("comment") or prop.get("description")
if comment:
predicates.append(f'rdfs:comment "{esc(comment)}"')
domain = prop.get("domain")
if domain:
predicates.append(f"rdfs:domain <{domain}>")
range_type = prop.get("range")
if range_type:
predicates.append(f"rdfs:range xsd:{range_type}")
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
predicates.append(f"rdfs:domain <{domain_iri}>")
for range_val in _as_list(prop.get("range")):
range_ref = self._ttl_datatype_ref(range_val)
if range_ref:
predicates.append(f"rdfs:range {range_ref}")
lines.append(self._ttl_block(prop_uri, "owl:DatatypeProperty", predicates))
lines.append("")
semantica/export/rdf_exporter.py
+709 -49
View File
@@ -29,8 +29,12 @@ Author: Semantica Contributors
License: MIT
"""
import re
from pathlib import Path
from decimal import Decimal, InvalidOperation
from html import escape as xml_escape
from typing import Any, Dict, List, Optional, Set, Union
from urllib.parse import quote, urlsplit
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory, hash_data
@@ -49,6 +53,79 @@ DEFAULT_ENTITY_TYPE = f"{SEMANTICA_NS}Entity"
#: Written when a relationship carries no type of its own. Same reasoning.
DEFAULT_RELATION_TYPE = f"{SEMANTICA_NS}related_to"
#: The one datatype every serializer writes confidence in.
#:
#: The four paths used to disagree: Turtle wrote the value bare, which the
#: Turtle grammar reads as xsd:decimal, N-Triples typed it xsd:float, RDF/XML
#: emitted a plain literal with no datatype, and JSON-LD emitted a native
#: number, which becomes xsd:double. Those are four distinct RDF terms for one
#: value (issue #1100).
#:
#: xsd:decimal is the choice because it is what the Turtle path already
#: produced, so the most used output is unchanged, and because it is exact:
#: xsd:float is 32 bit binary, and cannot represent 0.9 or 0.95 at all.
CONFIDENCE_DATATYPE = "http://www.w3.org/2001/XMLSchema#decimal"
#: Largest power of ten a confidence may carry. xsd:decimal has no exponent
#: notation, so a value has to be written out in full, and a compact literal
#: such as "1e100000000" would expand to a hundred million digits.
MAX_CONFIDENCE_EXPONENT = 100
def normalize_confidence(value: Any) -> Optional[str]:
"""
Return the canonical xsd:decimal lexical form of a confidence value.
Returns None when the value cannot be a decimal, so callers omit the triple
rather than writing something the vocabulary contradicts. The Turtle path
used to interpolate the raw value, so a confidence of "high" produced
`semantica:confidence high .` and made the whole document unparseable
(issue #1102).
Booleans are rejected. `bool` is a subclass of `int` in Python, so True
would otherwise silently become a confidence of 1.
"""
if value is None or isinstance(value, bool):
return None
if isinstance(value, str):
value = value.strip()
if not value:
return None
if not isinstance(value, (int, float, str, Decimal)):
return None
try:
decimal_value = Decimal(str(value))
except (InvalidOperation, ValueError, TypeError):
return None
# NaN and the infinities are Decimal values with no xsd:decimal form.
if not decimal_value.is_finite():
return None
# xsd:decimal has no exponent notation, so writing one means expanding it.
# "1e100000000" is eleven characters that expand to a hundred million, and
# the export path continues past validation errors, so a single malformed
# field could exhaust memory. Nothing near this magnitude is a confidence.
if (
not -MAX_CONFIDENCE_EXPONENT
<= decimal_value.adjusted()
<= MAX_CONFIDENCE_EXPONENT
):
return None
# `str(Decimal("0.00001"))` gives "0.00001", but a float that has already
# been through repr can arrive as "1e-05", which xsd:decimal does not allow.
formatted = format(decimal_value, "f")
if "." in formatted:
formatted = formatted.rstrip("0").rstrip(".") or "0"
# Decimal keeps the sign of zero, so 0.0 and -0.0 would serialise as two
# distinct RDF terms and defeat the point of a canonical form.
if formatted.lstrip("-").strip("0.") == "":
formatted = "0"
return formatted
def mint_entity_iri(text: str) -> str:
"""Mint a stable IRI for an entity that arrived without an id.
@@ -69,6 +146,294 @@ def mint_relationship_iri(index: int, source: Any, target: Any) -> str:
return f"{SEMANTICA_NS}rel_{index}_{digest}"
#: The metadata keys Semantica itself produces, and the terms they are written
#: as. GraphBuilder.build_graph writes the first five, create_snapshot writes
#: snapshot_time, and load_from_neo4j writes source / uri / database. These are
#: Semantica's own vocabulary, so they are minted in the declared namespace and
#: declared in semantica-ns.ttl.
#:
#: A key the caller supplied is a different matter. Which namespace an
#: arbitrary metadata key belongs in is issue #1146, and until that is settled
#: the exporter refuses to guess: it warns and skips, and a caller who already
#: knows the answer passes ``metadata_terms``.
#:
#: The map is key -> term rather than key -> namespace because two of the keys
#: cannot keep their own name. ``source`` on a graph loaded from Neo4j is the
#: system it came from, while sem:source is already the ObjectProperty holding
#: the subject of a reified relationship; reusing it would put a string where
#: an entity belongs.
DEFAULT_METADATA_TERMS: Dict[str, str] = {
"num_entities": f"{SEMANTICA_NS}numEntities",
"num_relationships": f"{SEMANTICA_NS}numRelationships",
"temporal_enabled": f"{SEMANTICA_NS}temporalEnabled",
"entity_resolution_applied": f"{SEMANTICA_NS}entityResolutionApplied",
"timestamp": f"{SEMANTICA_NS}builtAt",
"snapshot_time": f"{SEMANTICA_NS}snapshotAt",
"source": f"{SEMANTICA_NS}sourceSystem",
"uri": f"{SEMANTICA_NS}sourceUri",
"database": f"{SEMANTICA_NS}sourceDatabase",
}
#: Terms whose value is a node rather than a string. Everything else stays a
#: literal: a metadata value that merely looks like a URL is not thereby a
#: reference to one.
IRI_VALUED_METADATA_TERMS: Set[str] = {f"{SEMANTICA_NS}sourceUri"}
_XSD_NS = "http://www.w3.org/2001/XMLSchema#"
def _escape_literal(value: str) -> str:
"""Escape a string for a Turtle or N-Triples quoted literal."""
return (
value.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\r", "\\r")
.replace("\t", "\\t")
)
def _escape_temporal_literal(value: Any) -> str:
"""Escape a temporal bound for a Turtle ``dateTimeStamp`` literal.
Bounds are normally strings, but callers may hand us a ``datetime`` or
``None``. ``_escape_literal`` is str-only, so stringify non-str values
first instead of calling ``.replace()`` on them; ``None`` yields an empty
bound rather than crashing. Datetimes must use ISO 8601 so the
``xsd:dateTimeStamp`` ``T`` separator is preserved — ``str()`` yields a
space ("00:00:00+00:00"), which is a lexically invalid timestamp.
"""
if value is None:
return ""
if isinstance(value, str):
return _escape_literal(value)
if hasattr(value, "isoformat"):
return value.isoformat()
return str(value)
#: Turtle/N-Triples IRIREF grammar excludes these unescaped between `<` and
#: `>`: control characters, space, and <>"{}|^`\. An IRI-valued metadata
#: value (currently only sem:sourceUri, from the caller-controlled "uri"
#: metadata key) is written as `<{value}>` with no other quoting, so a value
#: containing one of these characters — a ">" followed by a full triple, for
#: instance — closes the IRIREF early and lets the rest of the string be
#: parsed as further RDF statements. This is the same shape of defect the
#: entity/relationship IRIs were hardened against; that hardening resolves
#: prefixes as well, which a metadata value never needs, so this stays a
#: narrower, dedicated guard rather than reusing _as_turtle_iri.
_IRI_REF_UNSAFE_RE = re.compile(r'[\x00-\x20<>"{}|^`\\]')
def _safe_iri_ref(value: str) -> str:
"""Percent-encode the characters an IRIREF may not contain unescaped."""
return _IRI_REF_UNSAFE_RE.sub(lambda m: quote(m.group(0), safe=""), value)
def _escape_xml(value: str) -> str:
"""Escape a string for either XML element text or an attribute value.
The quotes matter. This helper feeds `rdf:about`, `rdf:resource` and
`xmlns:` attribute values, which are delimited by double quotes, so a value
carrying one would close the attribute early and produce a document that
does not parse. Escaping them in element text as well is harmless and
means one helper cannot be used in the wrong place.
"""
return (
value.replace("&", "&amp;")
.replace("<", "&lt;")
.replace(">", "&gt;")
.replace('"', "&quot;")
.replace("'", "&apos;")
)
def _is_ncname(value: str) -> bool:
"""Whether a string can be an XML NCName, which is what RDF/XML requires.
Checked over the ASCII range rather than the full XML production: the
grammar also admits combining characters and extenders, so this is
deliberately conservative. It refuses names it could have accepted, and it
never accepts one that would produce a document a parser rejects. The
earlier check tested only that the first character was not a digit, which
let through every other way a local name can fail to be a name.
"""
if not value:
return False
if not (value[0].isascii() and (value[0].isalpha() or value[0] == "_")):
return False
return all(c.isascii() and (c.isalnum() or c in "._-") for c in value[1:])
def _split_iri(iri: str) -> Optional[tuple]:
"""Split an IRI into (namespace, local name) for RDF/XML's QName syntax.
Returns None when no split yields a usable local name. RDF/XML is the only
serialization here that cannot write an arbitrary predicate IRI, so this is
the one place a term can be unrepresentable, and the caller reports it
rather than dropping it quietly.
"""
for sep in ("#", "/"):
index = iri.rfind(sep)
if index != -1 and index + 1 < len(iri):
local = iri[index + 1 :]
if _is_ncname(local):
return iri[: index + 1], local
return None
def _metadata_statements(
metadata: Any,
terms: Dict[str, str],
logger: Any,
) -> List[tuple]:
"""Resolve a metadata mapping to a list of (term IRI, value) pairs.
A key with no term is skipped and reported. Silence is the defect this
fixes, so an unmapped key must be louder than a mapped one, not quieter.
"""
if not isinstance(metadata, dict):
return []
statements: List[tuple] = []
for key, value in metadata.items():
term = terms.get(key)
if term is None:
logger.warning(
"Metadata key %r has no term and was not exported. Which "
"namespace a caller-supplied key belongs in is issue #1146; "
"pass metadata_terms={%r: '<iri>'} to export it now.",
key,
key,
)
continue
if value is None:
continue
if isinstance(value, (dict, list, tuple, set)):
logger.warning(
"Metadata key %r holds a %s, which has no modelled RDF shape "
"yet, and was not exported.",
key,
type(value).__name__,
)
continue
statements.append((term, value))
return statements
def _resolve_metadata_terms(overrides: Optional[Dict[str, str]]) -> Dict[str, str]:
if not overrides:
return DEFAULT_METADATA_TERMS
return {**DEFAULT_METADATA_TERMS, **overrides}
def _typed_literal_parts(term: str, value: Any) -> tuple:
"""Return (kind, lexical, datatype) for one metadata value.
kind is "iri" or "literal". The lexical form and datatype are chosen once,
here, so that the four serializers cannot disagree about them the way they
disagreed about confidence in #1100.
"""
if term in IRI_VALUED_METADATA_TERMS and isinstance(value, str):
return "iri", value, None
if isinstance(value, bool):
return "literal", "true" if value else "false", f"{_XSD_NS}boolean"
if isinstance(value, int):
return "literal", str(value), f"{_XSD_NS}integer"
if isinstance(value, float):
# xsd:double, not xsd:decimal. `repr(1e-05)` is "1e-05" and
# `repr(float("nan"))` is "nan", and xsd:decimal admits neither the
# exponent form nor the special values, so typing a float as decimal
# produced lexicals a strict parser rejects. A Python float is an IEEE
# 754 double; xsd:double has legal lexicals for all of them, and it is
# also the honest claim, since nothing that arrived as a float was ever
# exact. `normalize_confidence` keeps xsd:decimal for confidence
# deliberately: that is a bounded score where exactness is meaningful
# and NaN is not a confidence at all.
if value != value:
lexical = "NaN"
elif value == float("inf"):
lexical = "INF"
elif value == float("-inf"):
lexical = "-INF"
else:
lexical = repr(value)
return "literal", lexical, f"{_XSD_NS}double"
return "literal", str(value), None
def _turtle_object(term: str, value: Any) -> str:
kind, lexical, datatype = _typed_literal_parts(term, value)
if kind == "iri":
return f"<{_safe_iri_ref(lexical)}>"
if datatype is None:
return f'"{_escape_literal(lexical)}"'
return f'"{lexical}"^^<{datatype}>'
def _turtle_metadata_clauses(statements: List[tuple]) -> List[str]:
return [f"<{term}> {_turtle_object(term, value)}" for term, value in statements]
def _ntriples_metadata_lines(subject: str, statements: List[tuple]) -> List[str]:
return [
f"<{subject}> <{term}> {_turtle_object(term, value)} ."
for term, value in statements
]
def _rdfxml_metadata_lines(
statements: List[tuple], indent: str, logger: Any = None
) -> List[str]:
"""RDF/XML needs a QName, so an unprefixed term declares its own prefix.
A term with no QName form has no RDF/XML representation at all, and this is
the only serialization with that restriction. Skipping it quietly would
reintroduce, in one format, exactly the silent metadata loss this module
was changed to stop, so it is reported and the other three formats still
carry the statement in full.
"""
lines: List[str] = []
for position, (term, value) in enumerate(statements):
split = _split_iri(term)
if split is None:
if logger is not None:
logger.warning(
"Term %r has no QName form, so it cannot be written in "
"RDF/XML and was omitted from that serialization only. "
"Turtle, N-Triples and JSON-LD carry it in full.",
term,
)
continue
namespace, local = split
kind, lexical, datatype = _typed_literal_parts(term, value)
prefix = f"md{position}"
opening = f'{indent}<{prefix}:{local} xmlns:{prefix}="{_escape_xml(namespace)}"'
if kind == "iri":
lines.append(f'{opening} rdf:resource="{_escape_xml(lexical)}"/>')
continue
if datatype is not None:
opening += f' rdf:datatype="{_escape_xml(datatype)}"'
lines.append(f"{opening}>{_escape_xml(lexical)}</{prefix}:{local}>")
return lines
def _jsonld_metadata_entries(statements: List[tuple]) -> Dict[str, Any]:
"""Absolute IRIs as keys, and explicit @value/@type rather than JSON's own
types: JSON's number is xsd:double, which would make the JSON-LD export
disagree with the other three about the datatype of an integer."""
entries: Dict[str, Any] = {}
for term, value in statements:
kind, lexical, datatype = _typed_literal_parts(term, value)
if kind == "iri":
entries[term] = {"@id": lexical}
elif datatype is None:
entries[term] = lexical
else:
entries[term] = {"@value": lexical, "@type": datatype}
return entries
class NamespaceManager:
"""
RDF namespace management engine.
@@ -325,6 +690,66 @@ class RDFSerializer:
# OWL-Time namespace URI
_OWL_TIME_NS = "http://www.w3.org/2006/time#"
_SEMANTICA_NS = "https://semantica.dev/ns#"
# Matches an already-valid percent-escape so it can be passed through
# unchanged instead of being re-encoded into e.g. %2520.
_PERCENT_ESCAPE_RE = re.compile(r"%[0-9A-Fa-f]{2}")
@classmethod
def _quote_preserving_escapes(cls, value: str, safe: str) -> str:
"""quote() that leaves existing valid %XX escapes untouched.
Blanket-quoting an absolute IRI double-encodes any percent-escape it
already carries (%20 -> %2520), which changes the identity of every
previously-valid IRI containing one. Only the spans between existing
valid escapes are quoted; a bare '%' that isn't part of a valid
escape (e.g. "%zz") still gets encoded to %25, keeping the malformed
case handled.
"""
parts = []
pos = 0
for match in cls._PERCENT_ESCAPE_RE.finditer(value):
parts.append(quote(value[pos : match.start()], safe=safe))
parts.append(match.group(0))
pos = match.end()
parts.append(quote(value[pos:], safe=safe))
return "".join(parts)
def _as_turtle_iri(
self, value: Any, namespaces: Optional[Dict[str, str]] = None
) -> str:
"""Return an absolute, safely encoded IRI for a Turtle resource."""
value = str(value)
try:
parsed = urlsplit(value)
except ValueError:
parsed = urlsplit("")
if parsed.scheme:
prefix, separator, local_name = value.partition(":")
# Built-in namespaces (semantica:, rdf:, rdfs:, owl:, ...) must
# always be resolvable, not only when the caller passes no
# namespaces of its own — otherwise a value like "semantica:Foo"
# resolves fine with no @context but stops resolving the moment
# any @context is present, since callers pass extract_namespaces()
# (context-only) here without merging in the built-ins.
effective_namespaces = {
**self.namespace_manager.namespaces,
**(namespaces or {}),
}
namespace = effective_namespaces.get(prefix)
if namespace and separator:
return self._quote_preserving_escapes(
namespace + local_name, safe=":/?#[]@!$&'()*+,;="
)
# A scheme with at least two characters is an absolute IRI,
# including opaque forms such as mailto:foo and isbn:0451450523.
# Keep one-character schemes as the existing Windows drive-path case.
if len(prefix) >= 2:
return self._quote_preserving_escapes(
value, safe=":/?#[]@!$&'()*+,;="
)
return self._SEMANTICA_NS + quote(value, safe="")
# Design decision — TemporalBound.OPEN in RDF:
# OWL-Time has no standard predicate for "no known end date." We use
@@ -364,6 +789,8 @@ class RDFSerializer:
"""
include_temporal: bool = options.pop("include_temporal", False)
time_axis: str = options.pop("time_axis", "valid")
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
lines = []
@@ -395,11 +822,34 @@ class RDFSerializer:
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
text = entity.get("text") or entity.get("label", "")
confidence = entity.get("confidence", 1.0)
confidence = normalize_confidence(entity.get("confidence", 1.0))
lines.append(f"<{entity_id}> a <{entity_type}> ;")
lines.append(f' semantica:text "{text}" ;')
lines.append(f" semantica:confidence {confidence} .")
clauses = [
f"a <{self._as_turtle_iri(entity_type, merged_namespaces)}>",
f'semantica:text "{_escape_literal(text)}"',
]
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
else:
clauses.append(
f'semantica:confidence "{confidence}"^^<{CONFIDENCE_DATATYPE}>'
)
clauses.extend(
_turtle_metadata_clauses(
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
)
)
)
entity_iri = self._as_turtle_iri(entity_id, merged_namespaces)
lines.append(f"<{entity_iri}> {clauses[0]} ;")
for clause in clauses[1:-1]:
lines.append(f" {clause} ;")
lines.append(f" {clauses[-1]} .")
lines.append("")
# Convert relationships to RDF triplets
@@ -409,17 +859,110 @@ class RDFSerializer:
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", DEFAULT_RELATION_TYPE)
lines.append(f"<{source_id}> <{rel_type}> <{target_id}> .")
lines.append(
f"<{self._as_turtle_iri(source_id, merged_namespaces)}> "
f"<{self._as_turtle_iri(rel_type, merged_namespaces)}> "
f"<{self._as_turtle_iri(target_id, merged_namespaces)}> ."
)
if include_temporal:
owl_lines = self._owl_time_triples_for_rel(rel, idx, time_axis)
owl_lines = self._owl_time_triples_for_rel(
rel, idx, time_axis, merged_namespaces
)
if owl_lines:
# The interval hangs off the relationship's own IRI, and a
# relationship written as a single triple has no such node
# in the graph. Without this the timestamps are well formed
# and unreachable: no query can get from the edge to its
# validity interval (#1106). The shape matches the JSON-LD
# export, and every term is declared in the vocabulary.
lines.extend(
self._reified_relationship_triples(
rel, idx, source_id, target_id, rel_type, merged_namespaces
)
)
lines.extend(owl_lines)
# Graph-level metadata needs a subject, and this serializer has never
# minted a document node. Rather than invent one here, it is written
# only when the caller names the graph; issue #1147 is where the
# default subject comes from once that lands.
graph_clauses = (
_turtle_metadata_clauses(
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
)
)
if graph_uri
else []
)
if graph_clauses:
graph_iri = self._as_turtle_iri(graph_uri, merged_namespaces)
lines.append("")
lines.append(
f"<{graph_iri}> {graph_clauses[0]} "
+ (";" if len(graph_clauses) > 1 else ".")
)
for clause in graph_clauses[1:-1]:
lines.append(f" {clause} ;")
if len(graph_clauses) > 1:
lines.append(f" {graph_clauses[-1]} .")
return "\n".join(lines)
def _reified_relationship_triples(
self,
rel: Dict[str, Any],
idx: int,
source_id: str,
target_id: str,
rel_type: str,
namespaces: Optional[Dict[str, str]] = None,
) -> List[str]:
"""
Emit the reified relationship node that OWL-Time triples hang off.
The direct triple stays. This adds a subject the interval can attach
to, using the same sem:Relationship shape the JSON-LD export already
writes, so the two serializations describe relationships the same way.
"""
rel_id = self._as_turtle_iri(
rel.get("id")
or mint_relationship_iri(idx, source_id or "", target_id or ""),
namespaces,
)
# The full predicate, not its local name. Truncating to the fragment
# made https://a.example/ns#employs and https://b.example/ns#employs the
# same literal, so the temporal node no longer said which predicate it
# described, and it disagreed with the direct triple beside it.
escaped = (
str(rel_type)
.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\r", "\\r")
)
predicates = ["a semantica:Relationship"]
if source_id:
predicates.append(
f"semantica:source <{self._as_turtle_iri(source_id, namespaces)}>"
)
if target_id:
predicates.append(
f"semantica:target <{self._as_turtle_iri(target_id, namespaces)}>"
)
predicates.append(f'semantica:type "{escaped}"')
return ["", f"<{rel_id}> " + " ;\n ".join(predicates) + " ."]
def _owl_time_triples_for_rel(
self, rel: Dict[str, Any], idx: int, time_axis: str
self,
rel: Dict[str, Any],
idx: int,
time_axis: str,
namespaces: Optional[Dict[str, str]] = None,
) -> List[str]:
"""
Emit OWL-Time Turtle triples for a relationship that carries temporal metadata.
@@ -434,7 +977,7 @@ class RDFSerializer:
def _is_open(v: Any) -> bool:
if v is None:
return False
if hasattr(v, "value"): # TemporalBound enum
if hasattr(v, "value"): # TemporalBound enum
return v.value == _OPEN_SENTINEL
return str(v).strip().upper() == _OPEN_SENTINEL
@@ -451,7 +994,10 @@ class RDFSerializer:
# deterministic IRI.
source_id = rel.get("source_id") or rel.get("source") or ""
target_id = rel.get("target_id") or rel.get("target") or ""
rel_base_id = rel.get("id") or mint_relationship_iri(idx, source_id, target_id)
rel_base_id = self._as_turtle_iri(
rel.get("id") or mint_relationship_iri(idx, source_id, target_id),
namespaces,
)
lines = [""] # blank separator
for axis_name, from_val, until_val in axes:
@@ -466,22 +1012,22 @@ class RDFSerializer:
lines.append(f" time:hasBeginning <{begin_id}> ;")
if _is_open(until_val):
lines.append(
' semantica:openEndedInterval "true"^^xsd:boolean .'
)
lines.append(' semantica:openEndedInterval "true"^^xsd:boolean .')
elif until_val is not None:
end_id = f"{rel_base_id}__{axis_name}_end"
lines.append(f" time:hasEnd <{end_id}> .")
lines.append(f"<{end_id}> a time:Instant ;")
lines.append(
f' time:inXSDDateTimeStamp "{until_val}"^^xsd:dateTimeStamp .'
f' time:inXSDDateTimeStamp "{_escape_temporal_literal(until_val)}"^^xsd:dateTimeStamp .'
)
else:
lines[-1] = lines[-1].rstrip(" ;") + " ." # close interval without hasEnd
lines[-1] = (
lines[-1].rstrip(" ;") + " ."
) # close interval without hasEnd
lines.append(f"<{begin_id}> a time:Instant ;")
lines.append(
f' time:inXSDDateTimeStamp "{from_val}"^^xsd:dateTimeStamp .'
f' time:inXSDDateTimeStamp "{_escape_temporal_literal(from_val)}"^^xsd:dateTimeStamp .'
)
lines.append("")
@@ -510,12 +1056,17 @@ class RDFSerializer:
... }
>>> rdfxml = serializer.serialize_to_rdfxml(rdf_data)
"""
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
lines = ['<?xml version="1.0" encoding="UTF-8"?>']
lines.append('<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"')
lines.append(' xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"')
lines.append(' xmlns:semantica="https://semantica.dev/ns#">')
lines.append("")
namespaces = self.namespace_manager.extract_namespaces(rdf_data)
# Convert entities to RDF/XML
entities = rdf_data.get("entities", [])
for entity in entities:
@@ -525,16 +1076,41 @@ class RDFSerializer:
entity_text = entity.get("text", "")
entity_id = mint_entity_iri(entity_text)
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
entity_type = entity.get("type") or DEFAULT_ENTITY_TYPE
text = entity.get("text") or entity.get("label", "")
confidence = entity.get("confidence", 1.0)
confidence = normalize_confidence(entity.get("confidence", 1.0))
# RDF/XML syntax: rdf:Description with rdf:about
lines.append(f' <rdf:Description rdf:about="{entity_id}">')
lines.append(f' <rdf:type rdf:resource="{entity_type}"/>')
# Attribute values are delimited by quotes, and both of these
# are caller input. Element text is left alone deliberately: that
# is #1098, and it is being fixed on its own path.
entity_iri = xml_escape(
self._as_turtle_iri(entity_id, namespaces), quote=True
)
entity_type_iri = xml_escape(
self._as_turtle_iri(entity_type, namespaces), quote=True
)
lines.append(f' <rdf:Description rdf:about="{entity_iri}">')
lines.append(f' <rdf:type rdf:resource="{entity_type_iri}"/>')
lines.append(f" <semantica:text>{text}</semantica:text>")
lines.append(
f" <semantica:confidence>{confidence}</semantica:confidence>"
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
else:
lines.append(
f' <semantica:confidence rdf:datatype="{CONFIDENCE_DATATYPE}">'
f"{confidence}</semantica:confidence>"
)
lines.extend(
_rdfxml_metadata_lines(
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
),
" ",
self.logger,
)
)
lines.append(" </rdf:Description>")
lines.append("")
@@ -544,11 +1120,39 @@ class RDFSerializer:
for rel in relationships:
source_id = rel.get("source_id") or rel.get("source")
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", "semantica:related_to")
# RDF/XML predicates are emitted as QNames, unlike resource
# attributes which use the shared absolute-IRI normalizer.
rel_type = rel.get("type") or "semantica:related_to"
# Relationship as property on source entity
lines.append(f' <rdf:Description rdf:about="{source_id}">')
lines.append(f' <{rel_type} rdf:resource="{target_id}"/>')
source_iri = xml_escape(
self._as_turtle_iri(source_id, namespaces), quote=True
)
target_iri = xml_escape(
self._as_turtle_iri(target_id, namespaces), quote=True
)
lines.append(f' <rdf:Description rdf:about="{source_iri}">')
lines.append(f' <{rel_type} rdf:resource="{target_iri}"/>')
lines.append(" </rdf:Description>")
lines.append("")
graph_lines = (
_rdfxml_metadata_lines(
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
),
" ",
self.logger,
)
if graph_uri
else []
)
if graph_lines:
graph_iri = xml_escape(
self._as_turtle_iri(graph_uri, namespaces), quote=True
)
lines.append(f' <rdf:Description rdf:about="{graph_iri}">')
lines.extend(graph_lines)
lines.append(" </rdf:Description>")
lines.append("")
@@ -582,6 +1186,9 @@ class RDFSerializer:
"""
import json
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
# Initialize JSON-LD structure with context
jsonld = {
"@context": {
@@ -608,14 +1215,32 @@ class RDFSerializer:
# and was dropped in full by a JSON-LD parser, silently.
entity_id = entity.get("id") or mint_entity_iri(entity.get("text", ""))
jsonld["@graph"].append(
{
"@id": entity_id,
"@type": entity.get("type", "semantica:Entity"),
"semantica:text": entity.get("text") or entity.get("label", ""),
"semantica:confidence": entity.get("confidence", 1.0),
node = {
"@id": entity_id,
"@type": entity.get("type", "semantica:Entity"),
"semantica:text": entity.get("text") or entity.get("label", ""),
}
confidence = normalize_confidence(entity.get("confidence", 1.0))
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
else:
# A native JSON number becomes xsd:double once expanded, so the
# value is written as a typed literal instead.
node["semantica:confidence"] = {
"@value": confidence,
"@type": CONFIDENCE_DATATYPE,
}
node.update(
_jsonld_metadata_entries(
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
)
)
)
jsonld["@graph"].append(node)
# Convert relationships to JSON-LD
relationships = rdf_data.get("relationships", [])
@@ -639,6 +1264,18 @@ class RDFSerializer:
}
)
graph_entries = (
_jsonld_metadata_entries(
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
)
)
if graph_uri
else {}
)
if graph_entries:
jsonld["@graph"].append({"@id": graph_uri, **graph_entries})
return json.dumps(jsonld, indent=2, ensure_ascii=False)
def serialize_to_ntriples(self, rdf_data: Dict[str, Any], **options) -> str:
@@ -655,22 +1292,17 @@ class RDFSerializer:
Returns:
String containing N-Triples serialization
"""
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
lines = []
namespaces = self.namespace_manager.extract_namespaces(rdf_data)
def expand_uri(uri: str) -> str:
if not uri:
return ""
if uri.startswith("http"):
return f"<{uri}>"
if uri.startswith("semantica:"):
return f"<https://semantica.dev/ns#{uri.split(':', 1)[1]}>"
if uri.startswith("rdf:"):
return f"<http://www.w3.org/1999/02/22-rdf-syntax-ns#{uri.split(':', 1)[1]}>"
if uri.startswith("rdfs:"):
return f"<http://www.w3.org/2000/01/rdf-schema#{uri.split(':', 1)[1]}>"
if ":" in uri:
return f"<{uri}>"
return f"<https://semantica.dev/ns#{uri}>"
return f"<{self._as_turtle_iri(uri, namespaces)}>"
# Convert entities
entities = rdf_data.get("entities", [])
@@ -684,7 +1316,7 @@ class RDFSerializer:
subject = expand_uri(entity_id)
# Type triple
entity_type = entity.get("type", "semantica:Entity")
entity_type = entity.get("type") or DEFAULT_ENTITY_TYPE
lines.append(
f"{subject} <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> {expand_uri(entity_type)} ."
)
@@ -692,30 +1324,58 @@ class RDFSerializer:
# Text property
text = entity.get("text") or entity.get("label", "")
if text:
safe_text = text.replace('"', '\\"').replace("\n", "\\n")
safe_text = _escape_literal(text)
lines.append(
f'{subject} {expand_uri("semantica:text")} "{safe_text}" .'
)
# Confidence property
confidence = entity.get("confidence")
if confidence is not None:
lines.append(
f'{subject} {expand_uri("semantica:confidence")} "{confidence}"^^<http://www.w3.org/2001/XMLSchema#float> .'
# Confidence property. The default matches the other serializers,
# which have always written one; omitting it here was half of why
# Turtle and N-Triples of one KG were different graphs (#1100).
raw_confidence = entity.get("confidence", 1.0)
confidence = normalize_confidence(raw_confidence)
if confidence is None:
self.logger.warning(
f"Entity {entity.get('id')} has a confidence that is not a "
f"number ({raw_confidence!r}), so no confidence is written"
)
else:
lines.append(
f'{subject} {expand_uri("semantica:confidence")} '
f'"{confidence}"^^<{CONFIDENCE_DATATYPE}> .'
)
lines.extend(
_ntriples_metadata_lines(
subject.strip("<>"),
_metadata_statements(
entity.get("metadata"), metadata_terms, self.logger
),
)
)
# Convert relationships
relationships = rdf_data.get("relationships", [])
for rel in relationships:
source_id = rel.get("source_id") or rel.get("source")
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", "semantica:related_to")
rel_type = rel.get("type") or DEFAULT_RELATION_TYPE
if source_id and target_id:
lines.append(
f"{expand_uri(source_id)} {expand_uri(rel_type)} {expand_uri(target_id)} ."
)
if graph_uri:
lines.extend(
_ntriples_metadata_lines(
graph_uri,
_metadata_statements(
rdf_data.get("metadata"), metadata_terms, self.logger
),
)
)
return "\n".join(lines)
semantica/ingest/methods.py
+79 -78
View File
@@ -180,6 +180,7 @@ from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
from ..utils.exceptions import ConfigurationError, ProcessingError
from ..utils.logging import get_logger
from ..utils.custom_methods import CUSTOM_METHOD_FELL_BACK, call_custom_method
from .config import ingest_config
from .file_ingestor import FileIngestor, FileObject
from .registry import method_registry
@@ -248,12 +249,12 @@ def ingest_file(
# Check for custom method in registry
custom_method = method_registry.get("file", method)
if custom_method and custom_method != ingest_file:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
# Get config
@@ -314,12 +315,12 @@ def ingest_parquet(
"""
custom_method = method_registry.get("parquet", method)
if custom_method and custom_method != ingest_parquet:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
try:
@@ -393,12 +394,12 @@ def ingest_arrow(
"""
custom_method = method_registry.get("arrow", method)
if custom_method and custom_method != ingest_arrow:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
try:
@@ -477,12 +478,12 @@ def ingest_xml(
"""
custom_method = method_registry.get("xml", method)
if custom_method and custom_method != ingest_xml:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .xml_ingestor import XMLIngestor
@@ -541,12 +542,12 @@ def ingest_web(
# Check for custom method in registry
custom_method = method_registry.get("web", method)
if custom_method and custom_method != ingest_web:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
try:
@@ -631,12 +632,12 @@ def ingest_public_api(
"""
custom_method = method_registry.get("public_api", method)
if custom_method and custom_method != ingest_public_api:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .public_api_ingestor import PublicAPIExamples, PublicAPIIngestor
@@ -718,12 +719,12 @@ def ingest_feed(
# Check for custom method in registry
custom_method = method_registry.get("feed", method)
if custom_method and custom_method != ingest_feed:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
try:
@@ -787,12 +788,12 @@ def ingest_stream(
# Check for custom method in registry
custom_method = method_registry.get("stream", method)
if custom_method and custom_method != ingest_stream:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .stream_ingestor import StreamIngestor
@@ -864,12 +865,12 @@ def ingest_repository(
# Check for custom method in registry
custom_method = method_registry.get("repo", method)
if custom_method and custom_method != ingest_repository:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
try:
@@ -936,12 +937,12 @@ def ingest_email(
# Check for custom method in registry
custom_method = method_registry.get("email", method)
if custom_method and custom_method != ingest_email:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
try:
@@ -1015,12 +1016,12 @@ def ingest_ontology(
# Check for custom method in registry
custom_method = method_registry.get("ontology", method)
if custom_method and custom_method != ingest_ontology:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .ontology_ingestor import OntologyIngestor
@@ -1081,12 +1082,12 @@ def ingest_database(
if method:
custom_method = method_registry.get("db", method)
if custom_method and custom_method != ingest_database:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .db_ingestor import DBIngestor
@@ -1188,12 +1189,12 @@ def ingest_mcp(
# Check for custom method in registry
custom_method = method_registry.get("mcp", method)
if custom_method and custom_method != ingest_mcp:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .mcp_ingestor import MCPIngestor
semantica/ingest/ontology_ingestor.py
+17 -6
View File
@@ -40,7 +40,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import rdflib
from rdflib import RDF, RDFS, OWL, Graph
from rdflib import RDF, RDFS, OWL, Dataset, Graph
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -106,15 +106,24 @@ class OntologyIngestor:
raise ValidationError(f"File not found: {file_path}")
self.progress.update_tracking(tracking_id, message="Parsing RDF graph...")
g = Graph()
# `Dataset`, not `Graph`: a JSON-LD document with a top-level `@id` *and*
# `@graph` places its terms in a NAMED graph. `Graph.parse()` loads only the
# default graph and discards the rest without an error, so every class and
# property in such a document was dropped while the load reported success.
# Same migration #757 made for JenaStore; the ingest path was not covered by it.
# `default_union=True` makes the Dataset itself present triples from every
# graph as one merged view (it is an rdflib.Graph subclass, so it satisfies
# _convert_to_dict()'s Graph-typed contract directly) instead of copying every
# quad into a second in-memory Graph.
ds = Dataset(default_union=True)
# Use provided format or let rdflib guess based on extension
parse_kwargs = kwargs.copy()
if format:
parse_kwargs['format'] = format
try:
g.parse(file_path, **parse_kwargs)
ds.parse(file_path, **parse_kwargs)
except Exception as e:
# Fallback: try to guess format from extension if not provided and initial parse failed
if not format:
@@ -130,12 +139,14 @@ class OntologyIngestor:
guessed_fmt = fmt_map.get(ext)
if guessed_fmt:
self.logger.info(f"Retrying with guessed format: {guessed_fmt}")
g.parse(file_path, format=guessed_fmt, **kwargs)
ds.parse(file_path, format=guessed_fmt, **kwargs)
else:
raise e
else:
raise e
g = ds
self.progress.update_tracking(tracking_id, message="Converting to internal format...")
# Determine format for metadata

Some files were not shown because too many files have changed in this diff Show More