* docs(ontology): document quality gate threshold semantics
The Ontology Quality Gate section (#1397) showed a thresholds={...}
example but never explained what min_coverage, max_errors,
max_warnings, or fail_on_warnings actually mean, or that
fail_on_warnings is a separate parameter rather than a thresholds
key. Add a concise defaults/semantics table, verified against
OntologyQualityGate.DEFAULT_THRESHOLDS and __init__ in quality_gate.py.
* docs(ontology): explain thresholds as prose instead of a table
A four-row table was heavier than this needed; each threshold's
meaning reads faster as two connected sentences.
Flagship pass establishing the crisp-prose style for the rest of
docs/: remove em dashes from explanatory prose (leave them in
simulated document/alert string literals, which are data, not our
voice), replace colon-as-dramatic-pause constructions, and fix two
broken relative links in reference/context.md ([Reasoning](reasoning)
and [Provenance](provenance) were missing their leading slash and
would 404 on the live site, the same class of bug fixed sitewide in
PR #1407). concepts.md's intro also picks up the new context/semantic
layer tagline. No code examples, tables, or technical content
changed.
* feat(context): add ErasureCoordinator for cross-store entity erasure
purge_node() is graph-scope by design (#957), so an entity removed from the
graph can survive verbatim as an AgentMemory item and as an embedding. The
changelog names GDPR Article 17 as purge's motivation, and an Article 17
erasure the vector store can still answer queries from is not an erasure --
it is worse than none, because purge_node() returns True and writes a
tombstone attesting the content is gone.
ErasureCoordinator composes the existing public APIs to drive the cascade and
returns an ErasureReceipt recording what each store reported. Nothing in
context_graph.py or agent_memory.py changes behaviorally; ContextGraph keeps
its documented graph-scope contract instead of acquiring references that would
invert the dependency.
Honest partial reporting is the point. Stores report erased / not_found /
not_configured / unsupported / failed, and complete is False when any store
reports unsupported or failed. FAISS, Milvus and Weaviate expose no delete at
all, so erasure genuinely cannot be completed on them today -- the receipt
says so rather than reporting a success it did not achieve.
Erasure runs outward-in (vectors, memory, graph). The tombstone is the durable
attestation, so writing it first would let a crash mid-cascade leave a record
claiming more than happened; erasing the graph last leaves a partial failure
recoverable and honest.
The memory sweep pages until dry and re-queries afterwards rather than
trusting one find_by_entity() call, whose limit=10 default silently truncates
the very check a caller uses to decide the erasure is done. Unsupported vector
backends are detected by probing the wrapped backend, since the VectorStore
facade declares delete_vectors() for every backend and only raises
NotImplementedError once called.
27 tests against real ContextGraph/AgentMemory instances, including the
25-items-on-one-entity regression that fails against a naive single-call
sweep. Full tests/context/ suite: 596 passed.
Closes#1018
* docs(context): document ErasureCoordinator in the context API reference
* test(context): exercise ErasureCoordinator against a real VectorStore
The vector-leg tests asserted the three backend shapes the coordinator
expects -- delete_vectors / delete / neither -- against fakes, which is worth
exactly as much as the assumption that a real store looks like one of them.
VectorStore(backend="inmemory") runs without external services, so it can
hold that assumption to account.
Adds the end-to-end case the receipt actually attests to: a real ContextGraph,
AgentMemory and VectorStore, where the embedding is written by
AgentMemory.store() and has to be gone afterwards. That exercises the memory
leg's own delete_memory() vector cascade rather than the coordinator's model
of it.
The real backend also pins a limit worth knowing before trusting the receipt:
it pops the ids and returns True whether or not they were there, and no
backend offers a portable existence check, so `erased` on the vectors leg
means the store accepted the delete for the ids given -- not that embeddings
were really removed. The memory leg re-queries to confirm and so is the
stronger claim. Documented on STATUS_ERASED, _erase_vectors(), and both status
tables.
tests/context/: 599 passed.
* fix(context): address review findings on ErasureCoordinator
Timestamp drift (high). erase_entity() resolved erased_at up front but passed
the caller's original `at` down to purge_node(), so on the default at=None
path the coordinator and the graph each took their own now() and the receipt
attested to a different instant than the tombstone it points at -- breaking
the invariant this module states most loudly. The resolved value is now what
the graph receives. The existing test passed only because it supplied an
explicit `at`, which hides the drift; the regression test covers at=None,
which is what callers actually use.
Backend delete results. The vectors leg treated anything other than the
literal False as success, but no in-repo backend returns a bool -- Qdrant
returns {"status": <UpdateStatus>} and Pinecone {"deleted": True}, so every
dict read as success and the backend's own account of the delete was thrown
away. Results are now interpreted by shape and the payload is kept in the
receipt as backend_result, stringified so it stays JSON-serializable as an
audit record. Bool markers match by identity so a 0 count isn't read as
False; string markers match as substrings so an enum rendering as
"UpdateStatus.FAILED" isn't read as success.
Falsey vector store. The "at least one store" guard used `not vector_store`,
rejecting a valid store whose __bool__/__len__ makes an empty instance falsey
and then reporting vector_store=None when an object had been passed. It now
separates None (absent) from False (deliberately disabled) from provided, and
echoes what it received.
`at` annotations. Widened to int/float, matching the ContextGraph normalizer
they delegate to, so the coordinator stops advertising less than the API it
wraps.
tests/context/: 608 passed.
* fix(context): report memory-owned vectors that survive erasure (#1018)
A receipt could read complete while an embedding was still in the vector
store. The vector leg deleted `vector_ids` or `[entity_id]`, and the memory
leg relied on `AgentMemory.delete_memory()` to cascade to the vectors each
item owns. That cascade is best-effort: `_delete_vector_ids()` raises when a
backend returns False, `delete_memory()` catches it, logs a warning, and
still returns True. So `batch_delete` counted the item, the residual re-query
found no items, the memory leg reported `erased`, and nothing in the receipt
recorded that the embedding was refused.
Reproduced with a store that deletes the entity-keyed id and refuses the
memory-owned one: `receipt.complete` was True with the embedding still live.
That is the failure mode this module exists to prevent -- a receipt is a
compliance artifact, and one that overstates is worse than none.
Fix by deleting memory-owned vector ids through the coordinator's own vector
leg, which reports honestly, instead of trusting the memory leg's cascade.
The ids are collected before anything is deleted, while the items still exist
to be enumerated, and are unioned with any caller-supplied ids rather than
replacing them.
This needs one addition to AgentMemory: `vector_ids_for(memory_id)`, a
read-only accessor mirroring the fallback in `delete_memory` (an item stored
without tracked ids is keyed by its own memory id). Reaching into
`_vector_ids` from the coordinator would have been the internals-access
pattern this repo keeps getting bitten by. No existing AgentMemory behaviour
changes -- `delete_memory()` still cascades best-effort, so other callers are
unaffected; the coordinator simply no longer depends on that being reliable.
It does mean the vectors are attempted twice, which is a no-op on a working
store and only ever costs a log line.
Note this deviates from the PR's stated "nothing in agent_memory.py changes"
constraint. The constraint could not hold: with `_vector_ids` private and no
portable way to ask a vector store what it still holds, the coordinator had
no way to make the claim truthful without it.
Four tests: the refused-vector case (receipt must be incomplete), that
memory-owned ids reach the store, that explicit `vector_ids` do not displace
them, and the accessor's fallback. The first three were confirmed to fail
against the previous coordinator, on the `receipt.complete` assertion rather
than incidentally. 655 tests pass across tests/context and the agno
integration.
* fix(context): make erasure receipt vector failures honest
* fix(context): optimize erasure pagination handling
* fix(context): use one timestamp for batch erasure
* style: strip trailing whitespace from erasure.py and test file
* docs(changelog): correct test counts to 48 / 738 after review rounds
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Adds `SAPODataConnector`, `SAPODataEntity`, and `SAPIngestor` for ingesting master and transactional data from SAP OData services, mainly things like Business Partners and Sales Orders.
Tested around S/4HANA Cloud, SuccessFactors, and on-prem NetWeaver Gateway style OData endpoints.
Main pieces included:
* OAuth2 client credentials and Basic auth support. Both go through the existing `ssrf.py` checks, including the OAuth token request.
* Small EDMX parser used by `discover_service()` so we don't need to pull in `pyodata`.
* Server-side pagination support for both OData versions:
* v2: `__next`, including plain string and `__deferred` formats
* v4: `@odata.nextLink`
* Keeps the service path in the base URL correctly whether the URL has a trailing slash or not. This is normalized in `SAPIngestor.__init__`.
* Adds an `ingest-sap` extra with just `requests`, so there is no SAP/proprietary SDK dependency.
This is meant to be a fairly small first version of the connector without adding a lot of SAP-specific dependencies.
Closes#1228
* fix(provenance): use timezone-aware UTC and assert stored records (#946)
Replace datetime.utcnow() in ProvenanceManager, ProvenanceEntry,
BridgeAxiom, and GraphBuilderWithProvenance with
datetime.now(timezone.utc), matching PipelineWithProvenance.
KG workflow and integration tests now read provenance back through
get_provenance() and assert algorithm metadata instead of generated
IDs, and call tracker methods that actually persist records.
* fix(provenance): compare provenance timestamps as instants, not strings
query_recorded_between() and audit_log() filtered and sorted on raw ISO
strings. With the timezone-aware change, a store can hold both pre-existing
naive stamps and offset-bearing ones, and the two are not string-comparable:
"...500000+00:00" sorts above "...500000", so a record at the identical
instant as a naive bound falls outside the range that should contain it.
Both now parse through _parse_timestamp() before comparing, reading naive
values as UTC. This mirrors ProvenanceTracker._parse_dt() in kg/, the class
ProvenanceManager replaces, so both sides of the migration answer a range
query the same way. Unparseable stored timestamps are skipped and logged
rather than silently dropped; unparseable bounds raise ValueError.
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
* feat(reasoning): rule-driven actions with provenance
Add a structured Action layer so matched rules can trigger side effects
instead of only deriving new facts, turning the reasoner into a
production-rule system.
L1 - Action type system:
- Action base class with execute(bindings, reasoner) + ?var substitution
- AssertAction (optional write-back to KnowledgeGraph), RetractAction,
CallAction (structured replacement for the unused Rule.handler),
EmitEventAction (delivers to a registered event sink)
- Rule.actions field; wired into Reasoner.forward_chain() and
ReteEngine.execute_matches() (via optional bind_reasoner)
L2 - Provenance-aware actions:
- Reasoner records fired actions (rule, bindings, confidence) to
action_log when provenance is enabled
- Fix dangling import in reasoning_provenance.py (ReasoningEngine ->
Reasoner, infer -> infer_facts)
Backward compatible: rules using the legacy handler still fire (wrapped
as a CallAction); rules without actions behave exactly as before.
Adds tests/reasoning/test_rule_actions.py (9 tests).
Closes#1095
* fix(reasoning): address qodo review findings on rule actions
- Token-aware variable substitution to avoid ?x/?xy prefix collision
- KnowledgeGraph write-back protocol (explicit API -> canonical translation -> ValueError)
- Structured action_log entries with timestamp
- Decouple action firing from conclusion dedup via per-activation tracking
(fires known conclusions once; retract-self no longer loops to max_iterations)
- Add Reasoner.infer_with_results preserving confidence; infer_facts delegates
- Forward provenance flag in ReasoningProvenance; drop **kwargs; propagate confidence
- Populate Rete Match.bindings from rule conditions
- Add regression tests for each fix
* fix(reasoning): persist fired action activations
* fix(reasoning): deduplicate Rete action execution
* fix(reasoning): canonicalize action activation identity
* docs(reasoning): explain action replay controls
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
ProgressTracker attached ConsoleProgressDisplay unconditionally, so any
script or CI job that piped or redirected stdout had one progress bar per
stage written into its output, escape sequences included. A plain
`python demo.py > out.txt` captured 173 bytes of progress-bar noise around
10 bytes of the program's own output.
Console progress is now attached only when stdout is an interactive
terminal, when running under Jupyter, or when SEMANTICA_FORCE_PROGRESS is
set. FileProgressDisplay is untouched, so progress logging still works in
pipelines, and SEMANTICA_DISABLE_PROGRESS keeps its existing meaning and
still takes precedence.
Both progress environment variables are now documented in the README and
the utils reference; SEMANTICA_DISABLE_PROGRESS previously existed only in
the reference page.
Deviations from the issue: the issue suggested disabling the tracker on
non-TTY stdout. This gates the display instead, because disabling the
tracker would short-circuit before FileProgressDisplay and take file
progress logging down with it, and the ~20 modules that set
`progress_tracker.enabled = True` in __init__ would need the property
setter taught about TTY state to avoid undoing it. Gating the display
leaves both alone.
Design note: the claim comment on the issue proposed an
`enabled: Optional[bool] = None` constructor opt-in; during implementation
the opt-in became SEMANTICA_FORCE_PROGRESS, which needs no signature change
and follows the NO_COLOR/FORCE_COLOR convention. Known limitation: TTY
detection runs once at tracker construction (the tracker is a process-wide
singleton), so a process that redirects stdout after first use needs the
env vars to change behaviour.
Fixes#1185
* refactor(export): centralize graph-payload key normalization
Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.
Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.
Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.
Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.
CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.
* fix(export): reject non-mapping input to the YAML exporters
export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.
Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.
Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.
* fix(export): address YAML exporter review findings
- semantica/export/yaml_exporter.py — import Sequence from typing
instead of collections.abc. `Sequence[str]` in _require_mapping's
annotation is evaluated at function-definition time; collections.abc.Sequence
only became subscriptable in Python 3.9, so on the 3.8 this project
declares support for, importing this module raised TypeError.
typing.Sequence has supported subscripting since 3.5.3. Mapping stays
imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
and read exported YAML through a context manager instead of an
unclosed yaml.safe_load(open(...)).
* fix(export): reject YAML export payloads with no recognized key
Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.
- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
collections through normalize_graph_payload(), which raises rather than
returning empty collections for an unrecognized mapping. Adopting the
shared resolver rather than repeating the check locally also brings the
'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
from this library's own graph type to YAML, used in
examples/capability_gap_context_graphs_example.py — exports its records
instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
defaulted lookups and had the same defect; it goes through the resolver
too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
its own key set. Schemas are a separate vocabulary with no aliasing, so
_require_recognized_keys lives in this module rather than in the shared
graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
An export_json envelope carries one, so accepting it would readmit the
case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
output directory, so a rejected export leaves nothing behind.
The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.
Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.
Closes#953.
* fix(export): reject payloads whose records resolve to nothing
Addresses the Qodo findings on #958.
Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.
- utils/helpers.py — split the check in two. _require_recognized_keys keeps
the presence rule; _require_nothing_dropped runs after resolution and
refuses a payload that resolved to nothing while an unread key still holds
records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
always carries a populated 'statistics' dict, and an empty graph must stay
exportable, so 'metadata', 'statistics' and 'count' are named as context
rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
both checks through the shared helpers rather than its own copy, so the
two vocabularies cannot drift apart in what counts as a silent-empty
export.
Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.
- export/yaml_exporter.py — the serialization span now says it serialized,
not that it exported, and export() opens its own span around the
filesystem work that stops as failed on error. Nothing reports a completed
export until the bytes are on disk.
Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.
Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.
* fix(export): validate collection values and make Neo4j mappings strict
Two gaps at the boundary the shared normalizer is supposed to own.
_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.
Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.
Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.
* fix(export): close YAML schema and record validation gaps
Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived. Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.
Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects. Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary. Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.
Tests: 101 tests pass across
tests/utils/test_normalize_graph_payload.py
tests/export/test_yaml_exporter_key_recognition.py
tests/export/test_yaml_exporter_input_validation.py
tests/export/test_neo4j_csv_exporter.py
* fix(export): close exception-type and record-shape gaps in normalize_graph_payload
LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).
Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).
normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.
Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.
* docs(changelog): add entry for #958 YAML export input hardening
Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes#956, #952, #953.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Two review findings on #807/#812:
- retrieve() and trace_lineage() were routed through transaction()'s
BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
serialized behind every other read/write, defeating the WAL
concurrency this PR was meant to add. They now use a dedicated
_read_connection() (configured, no explicit BEGIN).
- track_entity()/track_chunk() swallowed all internal storage
exceptions unconditionally, so a single item's failure inside
track_entities_batch()/track_chunks_batch()'s shared transaction
never reached the batch loop's per-item except, inflating
tracked_count for entries that were never persisted. Both now
re-raise when called with a shared _conn (batch context) while
still degrading gracefully on standalone calls.
Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
Makes enterprise lakehouse/warehouse ingestion (Databricks Unity Catalog +
Delta Lake, Snowflake) a first-class, prominently documented capability
across the README and guides, and adds matching runnable examples to
docs/guides/ingest.md. Also fixes several pre-existing inaccuracies caught
while auditing the ingest module docs against the actual source:
WebIngestor has no ingest_urls() (only singular ingest_url()), XMLIngestor's
XSD option is schema_path (not validate_xsd) and belongs on ingest() not the
constructor, and the "Available ingestors" list was missing DatabricksIngestor
while listing several classes not actually exported from semantica.ingest.
sparql.py handles direct SPARQL query execution against the live graph with no test coverage anywhere in the repo. Adds coverage for the read-only allowlist (the actual security boundary here), row/timeout limits, error handling, and RDF projection fidelity.
Implements #322: ConstructTemplate/ParameterDescriptor/ConstructTemplateRegistry
with injection-safe {{param}} rendering, Blazegraph CONSTRUCT-aware execute_sparql
extension, execute_construct_template (render->execute->parse->persist), and a
construct_template pipeline step. RDF4J/Jena support deferred to a follow-up issue.
Closes#322
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
* docs: improve agent memory guide onboarding and usage guidance
* docs: align Agent Memory guide with persistence implementation
* docs: fix misleading index_path persistence claim across guides
VectorStore's index_path kwarg is silently absorbed into FAISSStore's
**config and never read anywhere in faiss_store.py, so it does not make
the FAISS index persist across restarts as several docs implied. Real
persistence requires an explicit VectorStore.save()/.load() call, or
AgentContext.save()/.load() which cascades to it.
- docs/reference/context.md: rewrite the "Persist your vector store"
tip to explain the actual save()/load() mechanism instead of the
dead index_path kwarg.
- docs/guides/graphrag.md, decision-intelligence.md, ingest.md,
semantic-extraction.md: drop the dead index_path=... kwarg from
VectorStore(backend="faiss", ...) constructor calls.
Follow-up to #691, which fixed the same false claim in
docs/guides/agent-memory.md but missed these other files.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- GCP: remove --allow-unauthenticated, restrict ingress to
internal-and-cloud-load-balancing, replace wildcard ALLOWED_ORIGINS=*
with a substitution variable (_ALLOWED_ORIGINS) so operators supply a
real URL at deploy time; same fix in cloudrun-service.yaml
- Fly.io: replace hardcoded FALKORDB_HOST=localhost with the correct
.internal private-network hostname pattern; update README accordingly
- docker-compose.dev.yml: add missing top-level networks: block so the
frontend service can join the semantica network without --file layering
- K8s/Helm: add readOnlyRootFilesystem: true + runAsUser: 1000 to
container securityContext; mount an emptyDir /tmp so uvicorn can write
temp files
- app.py: fix _read_explorer_settings() or-chain, use in os.environ
checks so an explicit ALLOWED_ORIGINS="" produces an empty allow-list
instead of silently falling through to localhost defaults; remove dead
app.state.falkordb_host/port attributes
- docs: update four locations that still documented {"status":"healthy"}
to reflect the new {"status":"ok"} health response
- tests: update test assertion to read falkordb settings from
app.state.explorer_settings instead of removed top-level attributes
Mohd Kaif