Compare commits

..
Author SHA1 Message Date
Zohaib Hassnain 390b82c08a Merge branch 'main' into docs-learning-more-audit 2026-09-07 16:21:01 +05:00
Zohaib Hassnain 3265dbc420 docs(learning-more): address review feedback on snippets and config defaults
- Define sample tasks in concurrency snippet to avoid NameError

- Define document_text in batch extraction snippet

- Define sample entities in deduplication snippet

- Correct GRAPH_STORE_DEFAULT_BACKEND default to neo4j

- Replace ineffective SEMANTICA_PORT with SEMANTICA_API_KEY in config table
2026-09-07 16:20:19 +05:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Mohd Kaif
97840da51b security(deps): bump pinecone from 9.1.0 to 10.0.0 (#1495)
Bumps [pinecone](https://github.com/pinecone-io/python-sdk) from 9.1.0 to 10.0.0.
- [Release notes](https://github.com/pinecone-io/python-sdk/releases)
- [Commits](https://github.com/pinecone-io/python-sdk/compare/v9.1.0...v10.0.0)

---
updated-dependencies:
- dependency-name: pinecone
  dependency-version: 10.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
2026-09-07 16:23:13 +05:30
Zohaib Hassnain 94ccbe1613 Merge branch 'main' into docs-learning-more-audit 2026-09-07 15:17:28 +05:00
Zohaib Hassnain f6110c4b70 docs(learning-more): fix broken pipeline and dedup snippets 2026-09-07 15:15:13 +05:00
Mohd KaifandClaude Sonnet 5 b8011eb44a fix(docker): revert runtime base image to python:3.13-slim (#1509)
#1290 bumped the runtime stage to python:3.14-slim. gensim is a base
(non-extras-gated) dependency and publishes no cp314 wheel on PyPI, so
pip falls back to building it from source, which needs a C compiler
this slim image doesn't carry:

  error: [Errno 2] No such file or directory: 'gcc'

This has broken the Docker build (and Container Security Scan) on
every push to main since #1290 merged. Confirmed via
`pip index versions gensim` / the PyPI files listing that gensim 4.4.0
ships cp313 wheels but no cp314 ones; the earlier lockfile fix in
#1290 only validated that `uv pip compile` could *resolve* dependencies
for 3.14, which reads sdist metadata and doesn't need a compiler --
it doesn't validate that `pip install` can actually *build* them,
which is what fails here.

Reverts to the exact base image digest that was building successfully
before #1290 and regenerates explorer-extra-py313.txt against today's
PyPI state. Once gensim (and anything else pulled in transitively)
ships cp314 wheels, the 3.14 bump can be retried.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-07 15:29:41 +05:30
2 changed files with 52 additions and 30 deletions
+43 -21
View File
@@ -64,7 +64,7 @@ Whether you're running your first pipeline or deploying Semantica in production,
[Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb): `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries.
</Step>
<Step title="Ontology-driven knowledge bases">
[Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb): auto-generation, SHACL validation, Ontology Hub (v0.5.0).
[Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb): auto-generation, SHACL validation, Ontology Hub.
</Step>
<Step title="Advanced visualization">
[Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb): UMAP, t-SNE, community layouts, embedding projections.
@@ -86,10 +86,10 @@ All settings can be overridden with environment variables: no code changes neede
| OpenAI API Key | `OPENAI_API_KEY` | `None` |
| Groq API Key | `GROQ_API_KEY` | `None` |
| Anthropic API Key | `ANTHROPIC_API_KEY` | `None` |
| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `"openai"` |
| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `"networkx"` |
| Log Level | `SEMANTICA_LOG_LEVEL` | `"INFO"` |
| Log Format | `SEMANTICA_LOG_FORMAT` | `"text"` |
| Graph Store Backend | `GRAPH_STORE_DEFAULT_BACKEND` | `"neo4j"` |
| Vector Store Backend | `VECTOR_STORE_DEFAULT_BACKEND` | `"faiss"` |
| Server Host | `SEMANTICA_HOST` | `"127.0.0.1"` |
| Server API Key | `SEMANTICA_API_KEY` | `None` |
## Troubleshooting
@@ -146,10 +146,15 @@ Also reduce batch sizes and enable streaming ingestion for large corpora.
Enable parallel execution and GPU acceleration:
```python
from semantica.pipeline import Pipeline
from semantica.pipeline import ParallelismManager, Task
pipeline = Pipeline(workers=8, batch_size=32)
pipeline.run(sources)
# Run pipeline tasks concurrently across worker threads
manager = ParallelismManager(max_workers=8)
tasks = [
Task("task_1", lambda: "process part 1"),
Task("task_2", lambda: "process part 2"),
]
results = manager.execute_parallel(tasks)
```
```bash
@@ -160,19 +165,19 @@ pip install "semantica[gpu]" # CUDA-backed embeddings
<Accordion title="Windows [all] installation fails" icon="windows">
Fixed in **v0.5.0**. Upgrade:
Upgrade to the latest release:
```bash
pip install --upgrade semantica
```
Or install extras individually: `pip install "semantica[core]"`, then add `[llm-openai]`, `[gpu]`, etc. as needed.
Or install extras individually: `pip install semantica`, then add `[llm-openai]`, `[gpu]`, etc. as needed.
</Accordion>
<Accordion title="cp1252 encoding crash on Windows" icon="windows">
Fixed in **v0.5.0**. For earlier versions, set the encoding environment variable:
Set the encoding environment variable:
```bash
set PYTHONIOENCODING=utf-8
@@ -202,27 +207,44 @@ Use NetworkX for local development and prototyping. Switch to a persistent backe
<Accordion title="Batch processing for large corpora" icon="layer-group">
Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM: a good starting point is 1,000 documents per batch on a 16 GB machine.
Process documents in batches rather than one at a time. Split large texts into chunks and extract entities in batches:
```python
from semantica.pipeline import Pipeline
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor
pipeline = Pipeline(workers=8, batch_size=32)
pipeline.run(sources)
document_text = "Acme Corp announced record revenue in Seattle. CEO Jane Doe presented results."
splitter = TextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split(document_text)
extractor = NERExtractor()
batch_entities = extractor.extract_entities_batch([c.text for c in chunks])
```
</Accordion>
<Accordion title="Deduplication v2: up to 7× faster" icon="bolt">
If deduplication is a bottleneck, switch from v1 strategies to the v2 engine:
If deduplication is a bottleneck, use candidate blocking to reduce O(n²) comparisons before similarity scoring:
```python
resolver = EntityResolver()
merged = resolver.resolve(entities, strategy="semantic_v2") # up to 7x faster
from semantica.deduplication import DuplicateDetector, EntityMerger
entities = [
{"id": "1", "name": "Acme Corp", "type": "Company"},
{"id": "2", "name": "Acme Corporation", "type": "Company"},
{"id": "3", "name": "Globex", "type": "Company"},
]
# Fast candidate blocking for large entity sets
detector = DuplicateDetector(similarity_threshold=0.8)
duplicates = detector.detect_duplicates(entities, candidate_strategy="blocking_v2")
merger = EntityMerger()
merged = merger.merge_duplicates(entities, strategy="keep_most_complete")
```
The `blocking_v2`, `hybrid_v2`, and `semantic_v2` strategies reduce O(n²) comparisons via candidate blocking before similarity scoring.
The `blocking_v2` and `hybrid_v2` candidate strategies filter candidate pairs before calculating fine-grained similarity.
</Accordion>
@@ -233,8 +255,8 @@ The `blocking_v2`, `hybrid_v2`, and `semantic_v2` strategies reduce O(n²) compa
- **API keys**: store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule
- **Sensitive data**: use local embedding models (Ollama, HuggingFace) for PII or classified content; avoid sending sensitive data to external APIs without data handling agreements
- **Graph exports**: encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways
- **XML ingestion**: always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser
- **Graph exports**: encrypt sensitive exports at rest; use SSRF-safe `base_url` validation when configuring custom LLM gateways
- **XML ingestion**: always use `XMLIngestor`, which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser
- [Cookbook](/cookbook): interactive Jupyter notebooks from beginner to advanced.
- [FAQ](/faq): common questions answered.
+9 -9
View File
@@ -4409,15 +4409,15 @@ pillow==12.3.0 \
# python-pptx
# rapidocr
# torchvision
pinecone==9.1.0 \
--hash=sha256:461632bb07919da32b943100b8a047c74be53a6aa15c8b7679bff7a0f834c939 \
--hash=sha256:6c3a6dfa577dc11aed3197e1b221e65522603e9e1f6bd27a1b504a0909b3559f \
--hash=sha256:d3871bd3f39cb430ae8470158dc9c5dcffbac5ae31d144d9a7c3b351ac51755f \
--hash=sha256:d53fe6f4978ab0642eb2d3a0ee3b2576ccfeebaa11e0690b18e67dac4e057047 \
--hash=sha256:e930ba819f5b7e20aac688d04c840a8b6fbc6d12630d71303bb2130881a9d169 \
--hash=sha256:fc71ec431108de2df1a1978d3a24ac16f74ba3d8f3265c3760f969386e8742b8 \
--hash=sha256:fe6aeaf6515e9021984755ebc162f643c79d98056059aab2e765962a7538818c \
--hash=sha256:ffae8fb7cbb4056b920586629f15b08107350be4802a5637d10b31e2ad841f9c
pinecone==10.0.0 \
--hash=sha256:0994270c514b16c72ec94dd6c29ff2708b81d30ff8467e19de192a28a7c86b7e \
--hash=sha256:0e05956a3201b1fbb54a1861277df919318a4941797f2d87fd558ac5ec232151 \
--hash=sha256:2f4e3200ee3562d195802b363487dd7fe6039a8c13630fc25fa3e8726c7a8654 \
--hash=sha256:3ab0c843b4fb04fbac22f1b8455e389063208a50f6a95d7a90627939968198de \
--hash=sha256:6066bbe9a7ae1d667cde08d262deb6fbea6feb35deb9177dd47141b55bbd9833 \
--hash=sha256:94d4c64779f3213a5cc538d3bd10a873da192cb9b0039db56690e556ba00b55c \
--hash=sha256:995c06e905940b10bb2aefe653225340b5a3f56f3efb373fe07f4e57b5043705 \
--hash=sha256:d482ed27a805cbd4aca2660da212008dd6e41d87d255279f405ff13b725970e2
# via semantica (pyproject.toml)
platformdirs==4.11.7 \
--hash=sha256:4f41487eeeeeb07f3a6625e61d9bc0ae6809f92d3386dbd74392fbb76108104d \