mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-03 04:00:18 +00:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
96c62b2fcc | ||
|
|
088ed55cea |
+6
-6
@@ -16,7 +16,7 @@ icon: "circle-question"
|
||||
| Python version? | 3.8+ (3.11+ recommended) |
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, security fixes shipped in every release (see [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md)) |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Latest version? | **v0.6.7** (August 2026) |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
@@ -70,9 +70,9 @@ Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities r
|
||||
|
||||
<Accordion title="What's the latest version?" icon="star">
|
||||
|
||||
**v0.6.7**: released August 2026.
|
||||
**v0.5.0**: released May 2026.
|
||||
|
||||
Highlights: first-class LangChain integration, SAP OData ingestor, human-editable Markdown round-trip persistence for `ContextGraph`, a structured Action layer for the reasoning engine, and a public `run_shacl_validation` entry point. The 0.6.x line also added first-class CrewAI support and the Semantica RDF vocabulary with deterministic IRIs. See the [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md) for the full history.
|
||||
Highlights: Ontology Hub, Distance Intelligence, Parquet/XML ingestion, 12 security fixes, Graph Explorer redesign, NER gateway fix.
|
||||
|
||||
```bash
|
||||
pip install --upgrade semantica
|
||||
@@ -173,7 +173,7 @@ This includes PyTorch with CUDA, FAISS GPU, and CuPy.
|
||||
<Accordion title="How does Semantica handle large datasets?" icon="layer-group">
|
||||
|
||||
- **Batching**: process documents in configurable chunks to control memory usage
|
||||
- **Parallel processing**: the `semantica.pipeline` module can run independent, parallel-safe steps in the same dependency layer concurrently (see the [Pipeline guide](guides/pipeline))
|
||||
- **Parallel processing**: `Pipeline(workers=N)` runs extraction steps concurrently
|
||||
- **Delta processing**: update graphs incrementally without full recompute on new data
|
||||
- **Persistent backends**: swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs
|
||||
|
||||
@@ -269,13 +269,13 @@ Groq, OpenAI, Anthropic, Google Gemini, Ollama (fully local), DeepSeek, Novita A
|
||||
|
||||
<Accordion title="Is Semantica production-ready?" icon="shield-check">
|
||||
|
||||
Yes. Every release ships with:
|
||||
Yes. v0.5.0 ships with:
|
||||
|
||||
- 1,000+ passing tests across Python 3.8–3.12
|
||||
- `PipelineValidator` and `FailureHandler` with exponential backoff and configurable retry policies
|
||||
- W3C PROV-O provenance tracking across all modules
|
||||
- Change management with SHA-256 checksums and full audit trails
|
||||
- Ongoing security hardening: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, and path traversal fixes have all landed across recent releases (see the [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md) security sections)
|
||||
- 12 security vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal, and more
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
+1
-1
@@ -404,7 +404,7 @@ Semantica was designed for domains where every decision must be explainable and
|
||||
- 1,000+ passing tests with full regression coverage
|
||||
- `PipelineValidator` catches configuration errors at startup
|
||||
- `FailureHandler` with exponential backoff and dead-letter queues
|
||||
- Ongoing security hardening: fixes shipped in every release ([CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md))
|
||||
- 12 security vulnerabilities fixed in v0.5.0
|
||||
|
||||
**Modular by Design** — Import only what you need.
|
||||
- Use `NERExtractor` without a graph store
|
||||
|
||||
@@ -46,7 +46,7 @@ Whether you're running your first pipeline or deploying Semantica in production,
|
||||
[Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb): multi-source, deduplication, conflict resolution.
|
||||
</Step>
|
||||
<Step title="Add semantic search">
|
||||
[Embedding Generation notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/12_Embedding_Generation.ipynb): generating embeddings, provider and model switching, dimensions. Then [Vector Store notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb): storing and searching vectors for retrieval.
|
||||
[Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb): providers, pooling strategies, vector stores.
|
||||
</Step>
|
||||
<Step title="Multi-source integration">
|
||||
[Multi-Source Data Integration notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb) for multi-source patterns.
|
||||
|
||||
+60
-83
@@ -5,7 +5,7 @@ icon: "rocket"
|
||||
---
|
||||
|
||||
<Info>
|
||||
**v0.6.7** — first-class LangChain integration, SAP OData ingestor, human-editable Markdown persistence for `ContextGraph`, and a structured Action layer for the reasoning engine. <a href="https://github.com/semantica-agi/semantica/releases" style={{color:"#10B981",fontWeight:600,textDecoration:"none"}}>What's new →</a>
|
||||
**v0.5.0** — Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. <a href="https://github.com/semantica-agi/semantica/releases" style={{color:"#10B981",fontWeight:600,textDecoration:"none"}}>What's new →</a>
|
||||
</Info>
|
||||
|
||||
This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional: pattern-based extraction works out of the box.
|
||||
@@ -35,7 +35,7 @@ Verify:
|
||||
|
||||
```bash
|
||||
python -c "import semantica; print(semantica.__version__)"
|
||||
# 0.6.7
|
||||
# 0.5.0
|
||||
```
|
||||
|
||||
|
||||
@@ -47,24 +47,36 @@ python -c "import semantica; print(semantica.__version__)"
|
||||
|
||||
<Step title="Ingest">
|
||||
|
||||
Load a document from a file or directory. The rest of this walkthrough follows
|
||||
the file path; other sources are shown afterwards.
|
||||
Load a document from a file, directory, URL, or database.
|
||||
|
||||
```python
|
||||
<CodeGroup>
|
||||
|
||||
```python File
|
||||
from semantica.ingest import FileIngestor
|
||||
|
||||
ingestor = FileIngestor()
|
||||
sources = ingestor.ingest("data/report.pdf")
|
||||
# Also accepts a directory, .docx, .html, .json, .csv, .xlsx, .pptx, .parquet, .xml
|
||||
# Also accepts: .docx, .html, .json, .csv, .xlsx, .pptx, .parquet, .xml
|
||||
```
|
||||
|
||||
<Tip>
|
||||
**Other sources.** `WebIngestor().ingest_url(url)` returns a `WebContent` whose
|
||||
`.text` you can feed straight into the Extract step (no parsing needed).
|
||||
`ParquetIngestor().ingest(path)` and `XMLIngestor().ingest(path, schema_path=...)`
|
||||
return structured records rather than documents; build a graph from those with
|
||||
`GraphBuilder().build({"entities": [...], "relationships": [...]})` directly.
|
||||
</Tip>
|
||||
```python Web
|
||||
from semantica.ingest import WebIngestor
|
||||
|
||||
ingestor = WebIngestor(max_depth=2)
|
||||
sources = ingestor.ingest("https://example.com/article")
|
||||
```
|
||||
|
||||
```python Parquet / XML (v0.5.0)
|
||||
from semantica.ingest import ParquetIngestor, XMLIngestor
|
||||
|
||||
# Single file or Hive-partitioned directory
|
||||
sources = ParquetIngestor().ingest("data/events.parquet")
|
||||
|
||||
# XML with XSD schema validation
|
||||
sources = XMLIngestor(validate_xsd="schema.xsd").ingest("data/records/")
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
|
||||
</Step>
|
||||
|
||||
@@ -76,24 +88,22 @@ Extract structured text and layout from raw documents.
|
||||
from semantica.parse import DocumentParser
|
||||
|
||||
parser = DocumentParser()
|
||||
parsed = parser.parse(sources[0].path) # parse() takes a path string
|
||||
parsed = parser.parse(sources[0])
|
||||
|
||||
print(parsed["text"][:200]) # extracted text
|
||||
print(parsed["metadata"]) # file_path, encoding, size, and format-specific keys
|
||||
print(parsed.text[:200]) # extracted text
|
||||
print(parsed.metadata) # title, author, date, source
|
||||
```
|
||||
|
||||
`parse()` returns a `dict` with `text`, `full_text`, and `metadata` keys.
|
||||
|
||||
<Tip>
|
||||
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser` (`pip install semantica[parse-docling]`): it applies advanced layout analysis and returns structured table data alongside text.
|
||||
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser`: it applies advanced layout analysis and returns structured table data alongside text.
|
||||
</Tip>
|
||||
|
||||
```python
|
||||
from semantica.parse import DoclingParser
|
||||
|
||||
parser = DoclingParser()
|
||||
parsed = parser.parse(sources[0].path)
|
||||
print(parsed["tables"]) # structured table data
|
||||
parsed = parser.parse(sources[0])
|
||||
print(parsed.tables) # structured table objects
|
||||
```
|
||||
|
||||
</Step>
|
||||
@@ -107,28 +117,26 @@ Identify named entities and extract typed relationships between them.
|
||||
```python Pattern-based (fast, no API key)
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
|
||||
text = parsed["text"]
|
||||
|
||||
ner = NERExtractor(method="pattern")
|
||||
entities = ner.extract(text)
|
||||
# Returns: [Entity(text="Apple Inc.", label="ORG", start_char=0, end_char=10, confidence=0.7), ...]
|
||||
entities = ner.extract(parsed)
|
||||
# Returns: [{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98}, ...]
|
||||
|
||||
rel = RelationExtractor(method="pattern")
|
||||
relationships = rel.extract(text, entities=entities)
|
||||
# Returns: [Relation(subject=Entity(...), predicate="founded_by", object=Entity(...), confidence=0.7), ...]
|
||||
rel = RelationExtractor(method="rule")
|
||||
relationships = rel.extract(parsed, entities=entities)
|
||||
# Returns: [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc."}, ...]
|
||||
```
|
||||
|
||||
```python LLM-powered (higher accuracy)
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.llms import Groq
|
||||
|
||||
# Reads GROQ_API_KEY from the environment; provider/llm_model select the backend
|
||||
text = parsed["text"]
|
||||
llm = Groq(model="llama-3.3-70b-versatile")
|
||||
|
||||
ner = NERExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
|
||||
entities = ner.extract(text)
|
||||
ner = NERExtractor(method="llm", llm_provider=llm)
|
||||
entities = ner.extract(parsed)
|
||||
|
||||
rel = RelationExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
|
||||
relationships = rel.extract(text, entities=entities)
|
||||
rel = RelationExtractor(method="llm", llm_provider=llm)
|
||||
relationships = rel.extract(parsed, entities=entities)
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
@@ -190,17 +198,16 @@ exporter.export(graph, file_path="graph.nt", format="nt")
|
||||
from semantica.export import ParquetExporter
|
||||
|
||||
exporter = ParquetExporter()
|
||||
exporter.export(graph, file_path="output/graph")
|
||||
# Dict input writes one file per key: output/graph_entities.parquet and
|
||||
# output/graph_relationships.parquet: ready for Spark, BigQuery, Databricks
|
||||
exporter.export(graph, file_path="output/graph.parquet")
|
||||
# Writes nodes.parquet + edges.parquet: ready for Spark, BigQuery, Databricks
|
||||
```
|
||||
|
||||
```python ArangoDB
|
||||
from semantica.export import ArangoAQLExporter
|
||||
|
||||
exporter = ArangoAQLExporter()
|
||||
exporter.export(graph, file_path="graph.aql")
|
||||
# Writes ready-to-run AQL INSERT statements to graph.aql
|
||||
aql = exporter.export(graph)
|
||||
# Returns ready-to-run AQL INSERT statements
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
@@ -265,21 +272,14 @@ relationships = rel.extract(text, entities=entities)
|
||||
<Accordion title="Multi-source incremental graph build" icon="layer-group">
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
parser = DocumentParser()
|
||||
ner = NERExtractor(method="pattern")
|
||||
rel = RelationExtractor(method="pattern")
|
||||
builder = GraphBuilder(merge_entities=True)
|
||||
|
||||
builder = GraphBuilder(merge_entities=True)
|
||||
all_entities, all_rels = [], []
|
||||
for source in FileIngestor().ingest("data/reports/"):
|
||||
text = parser.parse(source.path)["text"]
|
||||
entities = ner.extract(text)
|
||||
rels = rel.extract(text, entities=entities)
|
||||
|
||||
for doc in parsed_docs:
|
||||
entities = ner.extract(doc)
|
||||
rels = rel.extract(doc, entities=entities)
|
||||
all_entities.extend(entities)
|
||||
all_rels.extend(rels)
|
||||
|
||||
@@ -359,8 +359,7 @@ graph = builder.build({"entities": entities, "relationships": relationships})
|
||||
# Retrieve full lineage for any entity
|
||||
sources = prov.get_all_sources("Apple Inc.")
|
||||
print(sources[0])
|
||||
# {"source": "data/report.pdf", "location": None, "timestamp": "...",
|
||||
# "confidence": 1.0, "metadata": {"confidence": 0.98}}
|
||||
# {"source": "data/report.pdf", "location": None, "timestamp": "...", "confidence": 0.98}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -374,54 +373,32 @@ print(sources[0])
|
||||
|
||||
<Accordion title="No entities extracted" icon="magnifying-glass">
|
||||
|
||||
The document likely contains scanned images rather than machine-readable text. `DocumentParser` warns when a PDF has no text layer; switch to `DoclingParser` with OCR enabled:
|
||||
The document likely contains scanned images rather than machine-readable text. Enable OCR:
|
||||
|
||||
```python
|
||||
from semantica.parse import DoclingParser # pip install semantica[parse-docling]
|
||||
from semantica.parse import DocumentParser
|
||||
|
||||
parser = DoclingParser(enable_ocr=True)
|
||||
parsed = parser.parse(sources[0].path)
|
||||
parser = DocumentParser(ocr=True) # enables Tesseract OCR
|
||||
parsed = parser.parse(sources[0])
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Slow processing on large corpora" icon="gauge">
|
||||
|
||||
Install the GPU extras so embedding and ML inference run on CUDA:
|
||||
Enable parallel processing and GPU acceleration:
|
||||
|
||||
```bash
|
||||
pip install semantica[gpu]
|
||||
```
|
||||
|
||||
Scan the directory for paths first (no file contents are read), then handle one
|
||||
document at a time and write to a persistent graph backend instead of the
|
||||
in-memory graph:
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.graph_store import GraphStore
|
||||
from semantica.kg import GraphBuilder
|
||||
from semantica.pipeline import Pipeline
|
||||
|
||||
ingestor = FileIngestor()
|
||||
parser = DocumentParser()
|
||||
ner = NERExtractor(method="pattern")
|
||||
rel = RelationExtractor(method="pattern")
|
||||
store = GraphStore(backend="neo4j", uri="bolt://localhost:7687",
|
||||
user="neo4j", password="password")
|
||||
builder = GraphBuilder(merge_entities=True, graph_store=store)
|
||||
|
||||
for info in ingestor.scan_directory("data/reports/", recursive=True):
|
||||
text = parser.parse(info["path"])["text"] # one document loaded at a time
|
||||
entities = ner.extract(text)
|
||||
rels = rel.extract(text, entities=entities)
|
||||
builder.build({"entities": entities, "relationships": rels})
|
||||
pipeline = Pipeline(workers=8, batch_size=32)
|
||||
pipeline.run(sources)
|
||||
```
|
||||
|
||||
For multi-step orchestration with configurable parallelism, see the
|
||||
[Pipeline guide](guides/pipeline).
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Memory errors on large graphs" icon="memory">
|
||||
|
||||
@@ -611,3 +611,5 @@ The Knowledge Explorer embeds Distance Intelligence directly in the browser dash
|
||||
- [Knowledge Graph Module](kg) — `NodeEmbedder`, `SimilarityCalculator`, and graph analytics.
|
||||
- [Visualization](visualization) — Programmatic distance heatmaps and ego-mode graph renders.
|
||||
- [Explorer](explorer) — Knowledge Explorer with built-in Distance Intelligence dashboard.
|
||||
|
||||
- [Distance Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/12_Distance_Intelligence.ipynb) — Semantic neighborhoods and distance matrices · Advanced
|
||||
|
||||
@@ -588,15 +588,16 @@ class AgentMemory:
|
||||
return False
|
||||
|
||||
# Remove from vector store unless a caller is staging an atomic local update.
|
||||
if not skip_vector and self.vector_store:
|
||||
try:
|
||||
vector_ids = list(self._vector_ids.get(memory_id, [])) or [memory_id]
|
||||
self._delete_vector_ids(vector_ids)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to delete from vector store: {e}")
|
||||
# Bookkeeping runs unconditionally: a skip_vector delete still removes the
|
||||
# item, so leaving its tracked ids behind would orphan them permanently.
|
||||
self._vector_ids.pop(memory_id, None)
|
||||
if not skip_vector:
|
||||
if self.vector_store:
|
||||
try:
|
||||
vector_ids = list(self._vector_ids.get(memory_id, [])) or [
|
||||
memory_id
|
||||
]
|
||||
self._delete_vector_ids(vector_ids)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to delete from vector store: {e}")
|
||||
self._vector_ids.pop(memory_id, None)
|
||||
|
||||
memory_item = self.memory_items[memory_id]
|
||||
|
||||
@@ -1587,16 +1588,12 @@ class AgentMemory:
|
||||
memory_ids.append(memory_id)
|
||||
return memory_ids
|
||||
|
||||
def batch_delete(self, memory_ids: List[str], *, skip_vector: bool = False) -> int:
|
||||
def batch_delete(self, memory_ids: List[str]) -> int:
|
||||
"""
|
||||
Batch delete.
|
||||
|
||||
Args:
|
||||
memory_ids: List of memory IDs to delete
|
||||
skip_vector: If True, skip each item's own vector-store cascade
|
||||
(see ``delete_memory``). A caller that is already erasing these
|
||||
ids' vectors itself passes this to avoid a redundant,
|
||||
best-effort delete against the vector store.
|
||||
|
||||
Returns:
|
||||
Number of memories deleted
|
||||
@@ -1606,7 +1603,7 @@ class AgentMemory:
|
||||
"""
|
||||
deleted = 0
|
||||
for memory_id in memory_ids:
|
||||
if self.delete_memory(memory_id, skip_vector=skip_vector):
|
||||
if self.delete_memory(memory_id):
|
||||
deleted += 1
|
||||
return deleted
|
||||
|
||||
|
||||
@@ -34,7 +34,6 @@ Example:
|
||||
'unsupported'
|
||||
"""
|
||||
|
||||
import inspect
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Union
|
||||
@@ -151,24 +150,6 @@ class ErasureCoordinator:
|
||||
more than actually occurred. Erasing the graph last means a partial
|
||||
failure leaves the node present and the receipt incomplete, which is
|
||||
recoverable and honest.
|
||||
|
||||
Note:
|
||||
An explicit ``vector_store=False`` also suppresses ``AgentMemory``'s
|
||||
own internal vector cascade, not just the coordinator's leg (#1378).
|
||||
``AgentMemory.delete_memory()`` deletes an item's vectors best-effort:
|
||||
it catches a vector-store failure, logs it, and still returns ``True``,
|
||||
so without this a caller who opted out of the vector leg could still
|
||||
have ``memory.vector_store`` mutated underneath them while the receipt
|
||||
read ``vectors: not_configured``. ``vector_store=False`` is taken to
|
||||
mean "no vector activity at all", so the coordinator passes
|
||||
``skip_vector=True`` through to ``memory.batch_delete()`` in that case,
|
||||
and ``receipt.stores["vectors"]["status"]`` stays ``"not_configured"``
|
||||
honestly -- the caller opted the vector store out entirely, rather than
|
||||
the coordinator having erased it. This only applies when
|
||||
``vector_store=False`` was passed explicitly; when no vector store
|
||||
exists anywhere (no ``memory`` was supplied, or ``memory`` has no
|
||||
``vector_store`` attribute), there is nothing to suppress and
|
||||
``memory.batch_delete()`` is called as before.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -189,12 +170,6 @@ class ErasureCoordinator:
|
||||
|
||||
self.graph = graph
|
||||
self.memory = memory
|
||||
# Distinct from `self.vector_store is None`: that's also true when no
|
||||
# vector store exists anywhere (no memory, or memory with no
|
||||
# vector_store attribute), where there is nothing to suppress and
|
||||
# forcing skip_vector onto a duck-typed memory would break callers
|
||||
# whose batch_delete() doesn't accept that kwarg.
|
||||
self._vector_leg_disabled = vector_store is False
|
||||
if vector_store is False:
|
||||
self.vector_store: Optional[Any] = None
|
||||
elif vector_store is not None:
|
||||
@@ -449,20 +424,6 @@ class ErasureCoordinator:
|
||||
return {"status": STATUS_NOT_CONFIGURED}
|
||||
|
||||
deleted = 0
|
||||
skip_vector = self._vector_leg_disabled and _accepts_skip_vector(
|
||||
self.memory.batch_delete
|
||||
)
|
||||
if self._vector_leg_disabled and not skip_vector:
|
||||
# The class docstring only requires find_by_entity/batch_delete; a
|
||||
# duck-typed adapter is not required to support skip_vector. Falling
|
||||
# back to the plain call keeps the memory leg working -- the
|
||||
# adapter's own cascade (if it has one) just can't be suppressed.
|
||||
self.logger.warning(
|
||||
"Memory adapter %r has no skip_vector support; its own vector "
|
||||
"cascade (if any) could not be suppressed for %r",
|
||||
type(self.memory).__name__,
|
||||
entity_id,
|
||||
)
|
||||
try:
|
||||
# Sweep in pages until dry rather than passing one large limit:
|
||||
# ``find_by_entity`` has historically defaulted to ``limit=10`` and
|
||||
@@ -493,10 +454,7 @@ class ErasureCoordinator:
|
||||
"detail": "memory items carry no 'memory_id'",
|
||||
}
|
||||
|
||||
if skip_vector:
|
||||
removed = self.memory.batch_delete(memory_ids, skip_vector=True)
|
||||
else:
|
||||
removed = self.memory.batch_delete(memory_ids)
|
||||
removed = self.memory.batch_delete(memory_ids)
|
||||
deleted += removed
|
||||
if removed == 0:
|
||||
# No progress: another page would return the same items.
|
||||
@@ -606,25 +564,6 @@ def _memory_item_id(item: Any) -> Optional[str]:
|
||||
return str(memory_id) if memory_id else None
|
||||
|
||||
|
||||
def _accepts_skip_vector(batch_delete: Any) -> bool:
|
||||
"""True when ``batch_delete`` takes a ``skip_vector`` keyword.
|
||||
|
||||
``skip_vector`` is an ``AgentMemory``-specific extension, not part of the
|
||||
duck-typed contract the class docstring promises (``find_by_entity`` and
|
||||
``batch_delete`` only). Passing it to an adapter that doesn't accept it
|
||||
would raise ``TypeError`` and fail the whole memory leg, so this is
|
||||
checked before ever passing the kwarg.
|
||||
"""
|
||||
try:
|
||||
signature = inspect.signature(batch_delete)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
for parameter in signature.parameters.values():
|
||||
if parameter.name == "skip_vector" or parameter.kind == inspect.Parameter.VAR_KEYWORD:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
#: Dict keys a backend uses to report whether a delete succeeded, and the
|
||||
#: values that mean it did not. Qdrant returns ``{"status": <UpdateStatus>}``
|
||||
#: and Pinecone ``{"deleted": True}``; neither is a bool, so a bare
|
||||
|
||||
@@ -678,7 +678,7 @@ class TestSeparateVectorStoreHandling(unittest.TestCase):
|
||||
"""
|
||||
|
||||
def test_vector_store_false_disables_vector_leg_entirely(self):
|
||||
"""vector_store=False must disable the vector leg AND memory's own cascade (#1378)."""
|
||||
"""vector_store=False must disable the vector leg, not try memory.vector_store."""
|
||||
memory_store = _SelectiveDeleteStore()
|
||||
memory = _memory_with_embedding("customer-4471", memory_store)
|
||||
|
||||
@@ -689,91 +689,8 @@ class TestSeparateVectorStoreHandling(unittest.TestCase):
|
||||
|
||||
# Vector leg should report not_configured, not attempt deletion
|
||||
self.assertEqual(receipt.stores["vectors"]["status"], STATUS_NOT_CONFIGURED)
|
||||
# Memory's own cascade still runs, but coordinator doesn't track it
|
||||
self.assertTrue(receipt.complete)
|
||||
# Memory's own internal vector cascade must be suppressed too, not just
|
||||
# unreported: the embedding memory owns is left untouched, and the
|
||||
# backend's delete method is never even called.
|
||||
self.assertEqual(memory_store.attempts, [])
|
||||
self.assertTrue(memory_store.live)
|
||||
|
||||
def test_vector_store_false_regression_refusing_backend_never_called(self):
|
||||
"""Regression for #1378: a refusing backend must not be called at all.
|
||||
|
||||
Reproduces the exact bug report -- a vector store whose delete_vectors()
|
||||
always returns False (refuses) bound as memory.vector_store, with the
|
||||
coordinator's own vector leg disabled via vector_store=False. Before the
|
||||
fix, delete_memory()'s internal cascade would still call the refusing
|
||||
store, catch the failure, log a warning, and return True regardless --
|
||||
so receipt.complete read True while the embedding stayed live and the
|
||||
backend had in fact been asked to delete it. Pinned here so the delete
|
||||
method call count can't silently regress back to nonzero.
|
||||
"""
|
||||
refusing_store = _SelectiveDeleteStore(refuse={"vec-0"})
|
||||
memory = _memory_with_embedding("customer-4471", refusing_store)
|
||||
|
||||
receipt = ErasureCoordinator(
|
||||
memory=memory, vector_store=False
|
||||
).erase_entity("customer-4471")
|
||||
|
||||
self.assertTrue(receipt.complete)
|
||||
self.assertEqual(receipt.stores["vectors"]["status"], STATUS_NOT_CONFIGURED)
|
||||
self.assertEqual(len(refusing_store.attempts), 0) # delete_calls == 0
|
||||
|
||||
def test_skip_vector_deletion_does_not_orphan_local_vector_id_tracking(self):
|
||||
"""skip_vector=True must still pop the item's own _vector_ids entry.
|
||||
|
||||
Regression: delete_memory(skip_vector=True) used to leave the item's
|
||||
entry in AgentMemory._vector_ids behind since the pop() lived inside
|
||||
the `if not skip_vector` block alongside the actual vector-store
|
||||
delete. That orphaned entry never got cleaned up and leaked into
|
||||
to_dict()/from_dict() snapshots.
|
||||
"""
|
||||
memory = _memory_with_embedding("customer-4471", _SelectiveDeleteStore())
|
||||
memory_id = next(iter(memory.memory_items))
|
||||
self.assertIn(memory_id, memory._vector_ids)
|
||||
|
||||
ErasureCoordinator(memory=memory, vector_store=False).erase_entity(
|
||||
"customer-4471"
|
||||
)
|
||||
|
||||
self.assertNotIn(memory_id, memory.memory_items)
|
||||
self.assertNotIn(memory_id, memory._vector_ids)
|
||||
|
||||
def test_memory_adapter_without_skip_vector_support_is_not_broken(self):
|
||||
"""A duck-typed memory whose batch_delete() lacks skip_vector must still work.
|
||||
|
||||
The class docstring only requires find_by_entity and batch_delete; an
|
||||
adapter is not obligated to support skip_vector. The coordinator must
|
||||
detect that and fall back to the plain call rather than raising
|
||||
TypeError and failing the whole memory leg.
|
||||
"""
|
||||
|
||||
class _PlainAdapter:
|
||||
def __init__(self):
|
||||
self.items = {"m1": {"memory_id": "m1", "entities": [{"id": "customer-4471"}]}}
|
||||
|
||||
def find_by_entity(self, entity_id, limit=None):
|
||||
return [
|
||||
item
|
||||
for item in self.items.values()
|
||||
if any(e.get("id") == entity_id for e in item.get("entities", []))
|
||||
]
|
||||
|
||||
def batch_delete(self, memory_ids):
|
||||
removed = 0
|
||||
for memory_id in memory_ids:
|
||||
if self.items.pop(memory_id, None) is not None:
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
adapter = _PlainAdapter()
|
||||
|
||||
receipt = ErasureCoordinator(
|
||||
memory=adapter, vector_store=False
|
||||
).erase_entity("customer-4471")
|
||||
|
||||
self.assertEqual(receipt.stores["memory"]["status"], STATUS_ERASED)
|
||||
self.assertEqual(adapter.items, {})
|
||||
|
||||
def test_separate_vector_store_only_handles_coordinator_store(self):
|
||||
"""When coordinator has a different vector_store, it only handles that one.
|
||||
|
||||
Reference in New Issue
Block a user