Compare commits

...
Author SHA1 Message Date
Zohaib Hassnain f30c3f8ab0 docs(quickstart): qodo finding fixed 2026-09-03 04:17:44 +05:00
Zohaib Hassnain b4b12ae705 docs(quickstart): qodo findings addressed 2026-09-03 04:11:20 +05:00
Harsh Arora 45915e50a3 fix(context): vector_store=False must suppress AgentMemory's internal vector cascade in ErasureCoordinator (#1395)
* fix(erasure): ensure vector_store=False disables internal vector cascade in AgentMemory

* fix(erasure): ensure skip_vector=True does not orphan local vector ID tracking
2026-09-03 04:05:53 +05:00
Zohaib Hassnain b7b60d4a17 docs(quickstart): fix broken code against real APIs (#1401) 2026-09-03 04:05:19 +05:00
Zohaib Hassnain 25d2ea5fe9 docs: update stale latest version claims 2026-09-03 03:50:50 +05:00
6 changed files with 254 additions and 84 deletions
+6 -6
View File
@@ -16,7 +16,7 @@ icon: "circle-question"
| Python version? | 3.8+ (3.11+ recommended) |
| API key required? | Optional: pattern extraction works with no keys |
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
| Production-ready? | Yes: 1,000+ tests, security fixes shipped in every release (see [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md)) |
| Latest version? | **v0.6.7** (August 2026) |
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
@@ -70,9 +70,9 @@ Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities r
<Accordion title="What's the latest version?" icon="star">
**v0.5.0**: released May 2026.
**v0.6.7**: released August 2026.
Highlights: Ontology Hub, Distance Intelligence, Parquet/XML ingestion, 12 security fixes, Graph Explorer redesign, NER gateway fix.
Highlights: first-class LangChain integration, SAP OData ingestor, human-editable Markdown round-trip persistence for `ContextGraph`, a structured Action layer for the reasoning engine, and a public `run_shacl_validation` entry point. The 0.6.x line also added first-class CrewAI support and the Semantica RDF vocabulary with deterministic IRIs. See the [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md) for the full history.
```bash
pip install --upgrade semantica
@@ -173,7 +173,7 @@ This includes PyTorch with CUDA, FAISS GPU, and CuPy.
<Accordion title="How does Semantica handle large datasets?" icon="layer-group">
- **Batching**: process documents in configurable chunks to control memory usage
- **Parallel processing**: `Pipeline(workers=N)` runs extraction steps concurrently
- **Parallel processing**: the `semantica.pipeline` module can run independent, parallel-safe steps in the same dependency layer concurrently (see the [Pipeline guide](guides/pipeline))
- **Delta processing**: update graphs incrementally without full recompute on new data
- **Persistent backends**: swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs
@@ -269,13 +269,13 @@ Groq, OpenAI, Anthropic, Google Gemini, Ollama (fully local), DeepSeek, Novita A
<Accordion title="Is Semantica production-ready?" icon="shield-check">
Yes. v0.5.0 ships with:
Yes. Every release ships with:
- 1,000+ passing tests across Python 3.83.12
- `PipelineValidator` and `FailureHandler` with exponential backoff and configurable retry policies
- W3C PROV-O provenance tracking across all modules
- Change management with SHA-256 checksums and full audit trails
- 12 security vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal, and more
- Ongoing security hardening: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, and path traversal fixes have all landed across recent releases (see the [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md) security sections)
</Accordion>
+1 -1
View File
@@ -404,7 +404,7 @@ Semantica was designed for domains where every decision must be explainable and
- 1,000+ passing tests with full regression coverage
- `PipelineValidator` catches configuration errors at startup
- `FailureHandler` with exponential backoff and dead-letter queues
- 12 security vulnerabilities fixed in v0.5.0
- Ongoing security hardening: fixes shipped in every release ([CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md))
**Modular by Design** — Import only what you need.
- Use `NERExtractor` without a graph store
+85 -62
View File
@@ -5,7 +5,7 @@ icon: "rocket"
---
<Info>
**v0.5.0**Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. <a href="https://github.com/semantica-agi/semantica/releases" style={{color:"#10B981",fontWeight:600,textDecoration:"none"}}>What's new →</a>
**v0.6.7**first-class LangChain integration, SAP OData ingestor, human-editable Markdown persistence for `ContextGraph`, and a structured Action layer for the reasoning engine. <a href="https://github.com/semantica-agi/semantica/releases" style={{color:"#10B981",fontWeight:600,textDecoration:"none"}}>What's new →</a>
</Info>
This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional: pattern-based extraction works out of the box.
@@ -35,7 +35,7 @@ Verify:
```bash
python -c "import semantica; print(semantica.__version__)"
# 0.5.0
# 0.6.7
```
@@ -47,36 +47,24 @@ python -c "import semantica; print(semantica.__version__)"
<Step title="Ingest">
Load a document from a file, directory, URL, or database.
Load a document from a file or directory. The rest of this walkthrough follows
the file path; other sources are shown afterwards.
<CodeGroup>
```python File
```python
from semantica.ingest import FileIngestor
ingestor = FileIngestor()
sources = ingestor.ingest("data/report.pdf")
# Also accepts: .docx, .html, .json, .csv, .xlsx, .pptx, .parquet, .xml
# Also accepts a directory, .docx, .html, .json, .csv, .xlsx, .pptx, .parquet, .xml
```
```python Web
from semantica.ingest import WebIngestor
ingestor = WebIngestor(max_depth=2)
sources = ingestor.ingest("https://example.com/article")
```
```python Parquet / XML (v0.5.0)
from semantica.ingest import ParquetIngestor, XMLIngestor
# Single file or Hive-partitioned directory
sources = ParquetIngestor().ingest("data/events.parquet")
# XML with XSD schema validation
sources = XMLIngestor(validate_xsd="schema.xsd").ingest("data/records/")
```
</CodeGroup>
<Tip>
**Other sources.** `WebIngestor().ingest_url(url)` returns a `WebContent` whose
`.text` you can feed straight into the Extract step (no parsing needed).
`ParquetIngestor().ingest(path)` and `XMLIngestor().ingest(path, schema_path=...)`
return structured records rather than documents; build a graph from those with
`GraphBuilder().build({"entities": [...], "relationships": [...]})` directly.
</Tip>
</Step>
@@ -88,22 +76,24 @@ Extract structured text and layout from raw documents.
from semantica.parse import DocumentParser
parser = DocumentParser()
parsed = parser.parse(sources[0])
parsed = parser.parse(sources[0].path) # parse() takes a path string
print(parsed.text[:200]) # extracted text
print(parsed.metadata) # title, author, date, source
print(parsed["text"][:200]) # extracted text
print(parsed["metadata"]) # file_path, encoding, size, and format-specific keys
```
`parse()` returns a `dict` with `text`, `full_text`, and `metadata` keys.
<Tip>
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser`: it applies advanced layout analysis and returns structured table data alongside text.
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser` (`pip install semantica[parse-docling]`): it applies advanced layout analysis and returns structured table data alongside text.
</Tip>
```python
from semantica.parse import DoclingParser
parser = DoclingParser()
parsed = parser.parse(sources[0])
print(parsed.tables) # structured table objects
parsed = parser.parse(sources[0].path)
print(parsed["tables"]) # structured table data
```
</Step>
@@ -117,26 +107,28 @@ Identify named entities and extract typed relationships between them.
```python Pattern-based (fast, no API key)
from semantica.semantic_extract import NERExtractor, RelationExtractor
ner = NERExtractor(method="pattern")
entities = ner.extract(parsed)
# Returns: [{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98}, ...]
text = parsed["text"]
rel = RelationExtractor(method="rule")
relationships = rel.extract(parsed, entities=entities)
# Returns: [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc."}, ...]
ner = NERExtractor(method="pattern")
entities = ner.extract(text)
# Returns: [Entity(text="Apple Inc.", label="ORG", start_char=0, end_char=10, confidence=0.7), ...]
rel = RelationExtractor(method="pattern")
relationships = rel.extract(text, entities=entities)
# Returns: [Relation(subject=Entity(...), predicate="founded_by", object=Entity(...), confidence=0.7), ...]
```
```python LLM-powered (higher accuracy)
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.llms import Groq
llm = Groq(model="llama-3.3-70b-versatile")
# Reads GROQ_API_KEY from the environment; provider/llm_model select the backend
text = parsed["text"]
ner = NERExtractor(method="llm", llm_provider=llm)
entities = ner.extract(parsed)
ner = NERExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
entities = ner.extract(text)
rel = RelationExtractor(method="llm", llm_provider=llm)
relationships = rel.extract(parsed, entities=entities)
rel = RelationExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
relationships = rel.extract(text, entities=entities)
```
</CodeGroup>
@@ -198,16 +190,17 @@ exporter.export(graph, file_path="graph.nt", format="nt")
from semantica.export import ParquetExporter
exporter = ParquetExporter()
exporter.export(graph, file_path="output/graph.parquet")
# Writes nodes.parquet + edges.parquet: ready for Spark, BigQuery, Databricks
exporter.export(graph, file_path="output/graph")
# Dict input writes one file per key: output/graph_entities.parquet and
# output/graph_relationships.parquet: ready for Spark, BigQuery, Databricks
```
```python ArangoDB
from semantica.export import ArangoAQLExporter
exporter = ArangoAQLExporter()
aql = exporter.export(graph)
# Returns ready-to-run AQL INSERT statements
exporter.export(graph, file_path="graph.aql")
# Writes ready-to-run AQL INSERT statements to graph.aql
```
</CodeGroup>
@@ -272,14 +265,21 @@ relationships = rel.extract(text, entities=entities)
<Accordion title="Multi-source incremental graph build" icon="layer-group">
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
builder = GraphBuilder(merge_entities=True)
all_entities, all_rels = [], []
parser = DocumentParser()
ner = NERExtractor(method="pattern")
rel = RelationExtractor(method="pattern")
builder = GraphBuilder(merge_entities=True)
for doc in parsed_docs:
entities = ner.extract(doc)
rels = rel.extract(doc, entities=entities)
all_entities, all_rels = [], []
for source in FileIngestor().ingest("data/reports/"):
text = parser.parse(source.path)["text"]
entities = ner.extract(text)
rels = rel.extract(text, entities=entities)
all_entities.extend(entities)
all_rels.extend(rels)
@@ -359,7 +359,8 @@ graph = builder.build({"entities": entities, "relationships": relationships})
# Retrieve full lineage for any entity
sources = prov.get_all_sources("Apple Inc.")
print(sources[0])
# {"source": "data/report.pdf", "location": None, "timestamp": "...", "confidence": 0.98}
# {"source": "data/report.pdf", "location": None, "timestamp": "...",
# "confidence": 1.0, "metadata": {"confidence": 0.98}}
```
</Accordion>
@@ -373,32 +374,54 @@ print(sources[0])
<Accordion title="No entities extracted" icon="magnifying-glass">
The document likely contains scanned images rather than machine-readable text. Enable OCR:
The document likely contains scanned images rather than machine-readable text. `DocumentParser` warns when a PDF has no text layer; switch to `DoclingParser` with OCR enabled:
```python
from semantica.parse import DocumentParser
from semantica.parse import DoclingParser # pip install semantica[parse-docling]
parser = DocumentParser(ocr=True) # enables Tesseract OCR
parsed = parser.parse(sources[0])
parser = DoclingParser(enable_ocr=True)
parsed = parser.parse(sources[0].path)
```
</Accordion>
<Accordion title="Slow processing on large corpora" icon="gauge">
Enable parallel processing and GPU acceleration:
Install the GPU extras so embedding and ML inference run on CUDA:
```bash
pip install semantica[gpu]
```
```python
from semantica.pipeline import Pipeline
Scan the directory for paths first (no file contents are read), then handle one
document at a time and write to a persistent graph backend instead of the
in-memory graph:
pipeline = Pipeline(workers=8, batch_size=32)
pipeline.run(sources)
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.graph_store import GraphStore
from semantica.kg import GraphBuilder
ingestor = FileIngestor()
parser = DocumentParser()
ner = NERExtractor(method="pattern")
rel = RelationExtractor(method="pattern")
store = GraphStore(backend="neo4j", uri="bolt://localhost:7687",
user="neo4j", password="password")
builder = GraphBuilder(merge_entities=True, graph_store=store)
for info in ingestor.scan_directory("data/reports/", recursive=True):
text = parser.parse(info["path"])["text"] # one document loaded at a time
entities = ner.extract(text)
rels = rel.extract(text, entities=entities)
builder.build({"entities": entities, "relationships": rels})
```
For multi-step orchestration with configurable parallelism, see the
[Pipeline guide](guides/pipeline).
</Accordion>
<Accordion title="Memory errors on large graphs" icon="memory">
+15 -12
View File
@@ -588,16 +588,15 @@ class AgentMemory:
return False
# Remove from vector store unless a caller is staging an atomic local update.
if not skip_vector:
if self.vector_store:
try:
vector_ids = list(self._vector_ids.get(memory_id, [])) or [
memory_id
]
self._delete_vector_ids(vector_ids)
except Exception as e:
self.logger.warning(f"Failed to delete from vector store: {e}")
self._vector_ids.pop(memory_id, None)
if not skip_vector and self.vector_store:
try:
vector_ids = list(self._vector_ids.get(memory_id, [])) or [memory_id]
self._delete_vector_ids(vector_ids)
except Exception as e:
self.logger.warning(f"Failed to delete from vector store: {e}")
# Bookkeeping runs unconditionally: a skip_vector delete still removes the
# item, so leaving its tracked ids behind would orphan them permanently.
self._vector_ids.pop(memory_id, None)
memory_item = self.memory_items[memory_id]
@@ -1588,12 +1587,16 @@ class AgentMemory:
memory_ids.append(memory_id)
return memory_ids
def batch_delete(self, memory_ids: List[str]) -> int:
def batch_delete(self, memory_ids: List[str], *, skip_vector: bool = False) -> int:
"""
Batch delete.
Args:
memory_ids: List of memory IDs to delete
skip_vector: If True, skip each item's own vector-store cascade
(see ``delete_memory``). A caller that is already erasing these
ids' vectors itself passes this to avoid a redundant,
best-effort delete against the vector store.
Returns:
Number of memories deleted
@@ -1603,7 +1606,7 @@ class AgentMemory:
"""
deleted = 0
for memory_id in memory_ids:
if self.delete_memory(memory_id):
if self.delete_memory(memory_id, skip_vector=skip_vector):
deleted += 1
return deleted
+62 -1
View File
@@ -34,6 +34,7 @@ Example:
'unsupported'
"""
import inspect
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Union
@@ -150,6 +151,24 @@ class ErasureCoordinator:
more than actually occurred. Erasing the graph last means a partial
failure leaves the node present and the receipt incomplete, which is
recoverable and honest.
Note:
An explicit ``vector_store=False`` also suppresses ``AgentMemory``'s
own internal vector cascade, not just the coordinator's leg (#1378).
``AgentMemory.delete_memory()`` deletes an item's vectors best-effort:
it catches a vector-store failure, logs it, and still returns ``True``,
so without this a caller who opted out of the vector leg could still
have ``memory.vector_store`` mutated underneath them while the receipt
read ``vectors: not_configured``. ``vector_store=False`` is taken to
mean "no vector activity at all", so the coordinator passes
``skip_vector=True`` through to ``memory.batch_delete()`` in that case,
and ``receipt.stores["vectors"]["status"]`` stays ``"not_configured"``
honestly -- the caller opted the vector store out entirely, rather than
the coordinator having erased it. This only applies when
``vector_store=False`` was passed explicitly; when no vector store
exists anywhere (no ``memory`` was supplied, or ``memory`` has no
``vector_store`` attribute), there is nothing to suppress and
``memory.batch_delete()`` is called as before.
"""
def __init__(
@@ -170,6 +189,12 @@ class ErasureCoordinator:
self.graph = graph
self.memory = memory
# Distinct from `self.vector_store is None`: that's also true when no
# vector store exists anywhere (no memory, or memory with no
# vector_store attribute), where there is nothing to suppress and
# forcing skip_vector onto a duck-typed memory would break callers
# whose batch_delete() doesn't accept that kwarg.
self._vector_leg_disabled = vector_store is False
if vector_store is False:
self.vector_store: Optional[Any] = None
elif vector_store is not None:
@@ -424,6 +449,20 @@ class ErasureCoordinator:
return {"status": STATUS_NOT_CONFIGURED}
deleted = 0
skip_vector = self._vector_leg_disabled and _accepts_skip_vector(
self.memory.batch_delete
)
if self._vector_leg_disabled and not skip_vector:
# The class docstring only requires find_by_entity/batch_delete; a
# duck-typed adapter is not required to support skip_vector. Falling
# back to the plain call keeps the memory leg working -- the
# adapter's own cascade (if it has one) just can't be suppressed.
self.logger.warning(
"Memory adapter %r has no skip_vector support; its own vector "
"cascade (if any) could not be suppressed for %r",
type(self.memory).__name__,
entity_id,
)
try:
# Sweep in pages until dry rather than passing one large limit:
# ``find_by_entity`` has historically defaulted to ``limit=10`` and
@@ -454,7 +493,10 @@ class ErasureCoordinator:
"detail": "memory items carry no 'memory_id'",
}
removed = self.memory.batch_delete(memory_ids)
if skip_vector:
removed = self.memory.batch_delete(memory_ids, skip_vector=True)
else:
removed = self.memory.batch_delete(memory_ids)
deleted += removed
if removed == 0:
# No progress: another page would return the same items.
@@ -564,6 +606,25 @@ def _memory_item_id(item: Any) -> Optional[str]:
return str(memory_id) if memory_id else None
def _accepts_skip_vector(batch_delete: Any) -> bool:
"""True when ``batch_delete`` takes a ``skip_vector`` keyword.
``skip_vector`` is an ``AgentMemory``-specific extension, not part of the
duck-typed contract the class docstring promises (``find_by_entity`` and
``batch_delete`` only). Passing it to an adapter that doesn't accept it
would raise ``TypeError`` and fail the whole memory leg, so this is
checked before ever passing the kwarg.
"""
try:
signature = inspect.signature(batch_delete)
except (TypeError, ValueError):
return False
for parameter in signature.parameters.values():
if parameter.name == "skip_vector" or parameter.kind == inspect.Parameter.VAR_KEYWORD:
return True
return False
#: Dict keys a backend uses to report whether a delete succeeded, and the
#: values that mean it did not. Qdrant returns ``{"status": <UpdateStatus>}``
#: and Pinecone ``{"deleted": True}``; neither is a bool, so a bare
+85 -2
View File
@@ -678,7 +678,7 @@ class TestSeparateVectorStoreHandling(unittest.TestCase):
"""
def test_vector_store_false_disables_vector_leg_entirely(self):
"""vector_store=False must disable the vector leg, not try memory.vector_store."""
"""vector_store=False must disable the vector leg AND memory's own cascade (#1378)."""
memory_store = _SelectiveDeleteStore()
memory = _memory_with_embedding("customer-4471", memory_store)
@@ -689,8 +689,91 @@ class TestSeparateVectorStoreHandling(unittest.TestCase):
# Vector leg should report not_configured, not attempt deletion
self.assertEqual(receipt.stores["vectors"]["status"], STATUS_NOT_CONFIGURED)
# Memory's own cascade still runs, but coordinator doesn't track it
self.assertTrue(receipt.complete)
# Memory's own internal vector cascade must be suppressed too, not just
# unreported: the embedding memory owns is left untouched, and the
# backend's delete method is never even called.
self.assertEqual(memory_store.attempts, [])
self.assertTrue(memory_store.live)
def test_vector_store_false_regression_refusing_backend_never_called(self):
"""Regression for #1378: a refusing backend must not be called at all.
Reproduces the exact bug report -- a vector store whose delete_vectors()
always returns False (refuses) bound as memory.vector_store, with the
coordinator's own vector leg disabled via vector_store=False. Before the
fix, delete_memory()'s internal cascade would still call the refusing
store, catch the failure, log a warning, and return True regardless --
so receipt.complete read True while the embedding stayed live and the
backend had in fact been asked to delete it. Pinned here so the delete
method call count can't silently regress back to nonzero.
"""
refusing_store = _SelectiveDeleteStore(refuse={"vec-0"})
memory = _memory_with_embedding("customer-4471", refusing_store)
receipt = ErasureCoordinator(
memory=memory, vector_store=False
).erase_entity("customer-4471")
self.assertTrue(receipt.complete)
self.assertEqual(receipt.stores["vectors"]["status"], STATUS_NOT_CONFIGURED)
self.assertEqual(len(refusing_store.attempts), 0) # delete_calls == 0
def test_skip_vector_deletion_does_not_orphan_local_vector_id_tracking(self):
"""skip_vector=True must still pop the item's own _vector_ids entry.
Regression: delete_memory(skip_vector=True) used to leave the item's
entry in AgentMemory._vector_ids behind since the pop() lived inside
the `if not skip_vector` block alongside the actual vector-store
delete. That orphaned entry never got cleaned up and leaked into
to_dict()/from_dict() snapshots.
"""
memory = _memory_with_embedding("customer-4471", _SelectiveDeleteStore())
memory_id = next(iter(memory.memory_items))
self.assertIn(memory_id, memory._vector_ids)
ErasureCoordinator(memory=memory, vector_store=False).erase_entity(
"customer-4471"
)
self.assertNotIn(memory_id, memory.memory_items)
self.assertNotIn(memory_id, memory._vector_ids)
def test_memory_adapter_without_skip_vector_support_is_not_broken(self):
"""A duck-typed memory whose batch_delete() lacks skip_vector must still work.
The class docstring only requires find_by_entity and batch_delete; an
adapter is not obligated to support skip_vector. The coordinator must
detect that and fall back to the plain call rather than raising
TypeError and failing the whole memory leg.
"""
class _PlainAdapter:
def __init__(self):
self.items = {"m1": {"memory_id": "m1", "entities": [{"id": "customer-4471"}]}}
def find_by_entity(self, entity_id, limit=None):
return [
item
for item in self.items.values()
if any(e.get("id") == entity_id for e in item.get("entities", []))
]
def batch_delete(self, memory_ids):
removed = 0
for memory_id in memory_ids:
if self.items.pop(memory_id, None) is not None:
removed += 1
return removed
adapter = _PlainAdapter()
receipt = ErasureCoordinator(
memory=adapter, vector_store=False
).erase_entity("customer-4471")
self.assertEqual(receipt.stores["memory"]["status"], STATUS_ERASED)
self.assertEqual(adapter.items, {})
def test_separate_vector_store_only_handles_coordinator_store(self):
"""When coordinator has a different vector_store, it only handles that one.