* test(ner): fix NER configuration tests for the typed LLM extraction API
Two of the three failing tests tracked in #1059 were still red after
#1070 was closed because the mocks targeted the pre-typed provider API:
- test_ner_llm_config mocked generate_structured, but the LLM path now
goes through generate_typed with a Pydantic schema. Mock the typed
response (namespace items with .text/.label/.start/.end/.confidence)
and expect extraction_method 'llm_typed'.
- test_ner_pattern_config asserted 'Apple Inc' without the trailing
dot, but the ORG pattern captures it via (?:\.|\b). Assert 'Apple
Inc.' to match current production behavior.
Verified locally: 8/8 pass in test_ner_configurations.py; the
performance-test failures in tests/semantic_extract/ reproduce on a
clean main checkout and are unrelated.
Fixes#1059
Signed-off-by: Yunare Maia <yunare@gmail.com>
* refactor(ner): remove dead _extract_with_spacy method and unused self.nlp
_extract_with_spacy() had no callers: the ML dispatch path goes through
get_entity_method('ml') -> extract_entities_ml(), which loads the spaCy
model lazily via the process-level cache in methods.py. The instance
attribute self.nlp was only read by that dead method, so __init__ now
just validates the runtime (keeping the _ml_runtime_usable gate) instead
of eagerly loading a model that was never used.
Fixes#1058
Signed-off-by: Yunare Maia <yunare@gmail.com>
* test(split): rewrite NERExtractor cache tests to not rely on removed .nlp attribute
NERExtractor.nlp was removed in this PR as part of dead-code cleanup
(the attribute was only used by the equally-dead _extract_with_spacy()).
The three affected tests in TestNERExtractorSpacyModelCache previously
verified cache behavior through .nlp identity comparisons; rewrite them
to use load-call counts and direct se_methods.load_spacy_model() cache
queries instead:
- test_ner_extractor_reuses_cached_model_across_instances: drop the
e1.nlp is e2.nlp is e3.nlp assertion; len(calls)==1 already proves
reuse; add a cache query to confirm the cached object is non-None.
- test_ner_extractor_distinct_model_names_load_separately: store each
mock nlp in a dict keyed by name, then query the cache to assert
sm_cached is loaded['en_core_web_sm'] and sm_cached is not lg_cached.
- test_ner_extractor_failed_load_not_cached_and_retried: replace
extractor.nlp is None/not None with is-not-None construction checks
and a final cache query that verifies the recovered model is the
exact object returned by working_load.
All three tests still exercise the original behavioral contract (no
crash on missing model, failures not cached / retried, successful load
shared across instances); they just no longer rely on a private
instance attribute that no longer exists.
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
fix(config): honor boolean env overrides in Config.get() (#1038)
Config.get() checked int before bool. Since bool subclasses int, boolean
environment values could be ignored or returned as integers.
Check bool first and strip whitespace before parsing boolean environment
values. This applies to the config modules for conflicts, deduplication,
split, embeddings, export, ingest, kg, normalize, ontology, and parse.
Also make _load_env_vars() use the same whitespace handling for mapped and
generic environment variables.
Fixes#1035
* fix(parse): import email.message and repair pdfplumber test mock
- email_parser.py uses email.message.Message at class-definition time but
only did 'import email', so 'import semantica.parse' fails in a fresh
Python process unless something else imported email.message first
- test_pdf_parser patched semantica.parse.pdf_parser.pdfplumber, which
never exists as a module attribute (pdfplumber is imported inside
PDFParser.parse); inject a fake module via sys.modules instead
* fix(parse): warn when PDF parse yields no text layer (scanned PDFs)
Scanned (image-only) PDFs parsed via the default pdfplumber route
returned an empty full_text with progress status 'completed' - no error,
no warning - so the failure only surfaced far downstream. Warn in
PDFParser.parse() when every parsed page yields no text (and extract_text
is enabled), pointing users to method='docling' with enable_ocr=True.
* fix(parse): improve scanned PDF detection
---------
Co-authored-by: shanyu910 <208111055+shanyu910@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(provenance): use timezone-aware UTC and assert stored records (#946)
Replace datetime.utcnow() in ProvenanceManager, ProvenanceEntry,
BridgeAxiom, and GraphBuilderWithProvenance with
datetime.now(timezone.utc), matching PipelineWithProvenance.
KG workflow and integration tests now read provenance back through
get_provenance() and assert algorithm metadata instead of generated
IDs, and call tracker methods that actually persist records.
* fix(provenance): compare provenance timestamps as instants, not strings
query_recorded_between() and audit_log() filtered and sorted on raw ISO
strings. With the timezone-aware change, a store can hold both pre-existing
naive stamps and offset-bearing ones, and the two are not string-comparable:
"...500000+00:00" sorts above "...500000", so a record at the identical
instant as a naive bound falls outside the range that should contain it.
Both now parse through _parse_timestamp() before comparing, reading naive
values as UTC. This mirrors ProvenanceTracker._parse_dt() in kg/, the class
ProvenanceManager replaces, so both sides of the migration answer a range
query the same way. Unparseable stored timestamps are skipped and logged
rather than silently dropped; unparseable bounds raise ValueError.
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Index the four module notebooks merged via #989-#992 (Provenance
Tracking, Reasoning, Change Management, Seed Data) in the cookbook
landing page, as committed in tracking issue #1032.
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
* feat(reasoning): rule-driven actions with provenance
Add a structured Action layer so matched rules can trigger side effects
instead of only deriving new facts, turning the reasoner into a
production-rule system.
L1 - Action type system:
- Action base class with execute(bindings, reasoner) + ?var substitution
- AssertAction (optional write-back to KnowledgeGraph), RetractAction,
CallAction (structured replacement for the unused Rule.handler),
EmitEventAction (delivers to a registered event sink)
- Rule.actions field; wired into Reasoner.forward_chain() and
ReteEngine.execute_matches() (via optional bind_reasoner)
L2 - Provenance-aware actions:
- Reasoner records fired actions (rule, bindings, confidence) to
action_log when provenance is enabled
- Fix dangling import in reasoning_provenance.py (ReasoningEngine ->
Reasoner, infer -> infer_facts)
Backward compatible: rules using the legacy handler still fire (wrapped
as a CallAction); rules without actions behave exactly as before.
Adds tests/reasoning/test_rule_actions.py (9 tests).
Closes#1095
* fix(reasoning): address qodo review findings on rule actions
- Token-aware variable substitution to avoid ?x/?xy prefix collision
- KnowledgeGraph write-back protocol (explicit API -> canonical translation -> ValueError)
- Structured action_log entries with timestamp
- Decouple action firing from conclusion dedup via per-activation tracking
(fires known conclusions once; retract-self no longer loops to max_iterations)
- Add Reasoner.infer_with_results preserving confidence; infer_facts delegates
- Forward provenance flag in ReasoningProvenance; drop **kwargs; propagate confidence
- Populate Rete Match.bindings from rule conditions
- Add regression tests for each fix
* fix(reasoning): persist fired action activations
* fix(reasoning): deduplicate Rete action execution
* fix(reasoning): canonicalize action activation identity
* docs(reasoning): explain action replay controls
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
* docs(cookbook): add Seed Data module notebook
Add cookbook/introduction/25_Seed_Data.ipynb covering the seed module
with verified, executable examples:
- SeedDataManager.register_source with a CSV source
- load_source record enrichment (entity_type/source provenance)
- create_foundation_graph entity/relationship/metadata structure
- validate_quality gating
The seed module ships seed_usage.md but has no cookbook coverage. All
API calls and outputs were executed against
semantica/seed/seed_manager.py.
Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
* docs(cookbook): isolate seed CSV in a temp dir and execute notebook in Jupyter
- Write companies.csv into a session-scoped tempfile.mkdtemp() directory
instead of the working directory, so a user's existing companies.csv
can never be silently clobbered (review finding)
- Run the notebook through a fresh Jupyter kernel (restart + run all +
save): real execution counts, print() cells saved as stream outputs
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
---------
Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
* fix(ingest): import sqlalchemy text where DBIngestor and DataExporter use it
sqlalchemy.text was imported function-locally in DatabaseConnector.connect
and test_connection, but called in DataExporter.export_table_data and
DBIngestor.execute_query, which never imported it. Both raised NameError,
re-wrapped by their except handlers into a ProcessingError reading
'Failed to execute query: name text is not defined' -- a message that
looks like a database fault rather than a missing import.
No test exercised either method, so this also repairs a pre-existing
failure in tests/ingest/test_notebook_02.py::test_08_database_ingestion.
Add SQLite-backed coverage for all three call sites, including the
SELECT COUNT(*) branch that only runs when no limit is passed and would
otherwise stay untested.
Closes#1015
* test(ingest): register setUp cleanups with addCleanup
TemporaryDirectory and the SQLAlchemy engine were released only in tearDown, which unittest skips when setUp raises partway through. Register each cleanup as soon as its resource exists so a failed setUp still disposes the engine and removes the temp directory. LIFO ordering keeps dispose before cleanup, as tearDown had it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
* fix(ontology): preserve #-terminated namespaces in SHACLGenerator base_uri (#1082)
SHACLGenerator.__init__ normalized base_uri with rstrip('/') + '/', turning a #-terminated RDF namespace (e.g. http://example.org/manufacturing#) into ...#/. Every generated URI then landed in a different namespace than the instance data, so SHACL validation silently passed because the shapes targeted nothing.
__init__ now preserves a base_uri already ending in '/' or '#', matching the #-aware normalization generate() already applies. shapes_uri inherits the fix.
Adds test_hash_namespace_base_uri_is_not_mangled (fails on the old normalization), plus a CHANGELOG entry. Full ontology suite green.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ontology): collapse slash runs, only preserve #-terminated base_uri
Qodo review caught that preserving any endswith('/') base left redundant
trailing slashes (e.g. .../ns////) intact, leaking a different namespace
into emitted IRIs. Now only '#'-terminated bases are kept verbatim; slash
runs are collapsed to a single '/', matching generate() normalization.
Adds test_slash_run_normalization_regression.
---------
Co-authored-by: changshenhan <217217832+changshenhan@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
track_relationship() has no dedicated subject/object fields, so the
Step 2 example only stored relationship_id + type, leaving readers
unable to reconstruct which two entities the relationship connects.
Encode subject_entity_id/object_entity_id in metadata by convention,
and note the lack of dedicated fields in the prose.
* fix(cli): write embed generate output in the format embed index reads
* Address review: structured results get their own --output writer
deduplicate --output and ontology align --output were routed through
_write_embeddings_output, a helper for numeric matrices: it rejects the
dict/list shapes these commands produce and the .csv extension deduplicate
documents. New _write_result_output serializes structured results — JSON,
JSON-lines for lists, CSV for rows — and both commands use it. embed
generate keeps the embeddings writer, whose strictness is what #994 fixed.
On the pyarrow gap: the parquet writer already fails with an actionable
message (install pyarrow or use .json). Silently writing JSON bytes to a
.parquet path would recreate #994's magic-bytes failure, so the error stays
an error and the default suggestion stays .json.
* fix(cli): improve structured output serialization
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(export): consolidate duplicate Turtle/N-Triples literal escapers
_escape_literal (module-level) and RDFSerializer._escape_turtle_literal did
identical work in the same order (backslash, double-quote, newline, CR, tab).
Drop the newer static helper added in #1148 and route all call sites through
_escape_literal instead. Behaviour no-op.
Closes#1218.
* fix(export): handle datetime/None temporal bounds safely in OWL-Time
_escape_literal is str-only, so routing datetime or None temporal bounds
through it raised AttributeError during Turtle export. Stringify non-str
bounds (plain f-string semantics) before escaping, and render None as an
empty bound. Add regression tests for datetime bounds and end-only
intervals. Addresses Qodo high-priority finding #2 on #1221.
* fix(export): use isoformat for datetime temporal bounds
str() on a datetime drops the ISO-8601 T separator, producing a lexically invalid xsd:dateTimeStamp. Use isoformat() when available; strengthen the test to assert the exact T-separated form.
---------
_calculate_decision_content_similarity's character-bigram fallback was
unconditional, so ordinary multi-word English queries could pick up
incidental bigram overlap with unrelated decisions via max(word_sim,
bigram_sim). Gate it to only activate for CJK-like scripts or queries
with at most one whitespace token, matching its documented purpose.
Separately, _add_decision_to_graph never persisted recorded_at as a
node property, so _rebuild_decision_indexes/_sync_decision_from_node
(which already read it back) always recovered "" after any reload.
* fix(explorer): coerce decision timestamp to str to prevent 422 on /api/decisions
ContextGraph stores decision timestamps as POSIX floats (e.g. 1786513069.69),
but DecisionResponse.timestamp is typed Optional[str]. Pydantic strict
validation rejects the float and the whole /api/decisions endpoint returns
HTTP 422 "Invalid input", which breaks the Decisions workspace in the
Knowledge Explorer entirely (no decision can be listed).
Coerce the value to str (preserving None) in _node_to_decision so the
response validates. Verified: /api/decisions now returns 200 and the 3
sample decisions render in the Decisions workspace.
* test(explorer): cover decision timestamp coercion in _node_to_decision
Regression tests for the 422 fix in _node_to_decision. Covers the cases
that produced HTTP 422 (float / int timestamps from ContextGraph) and
the ones that must keep working (None, already-string, missing key).
Verified the suite catches the regression: with the fix reverted, the
float / int / nan / inf cases fail with the same ValidationError that
caused the 422; with the fix applied all 6 pass.
* fix(explorer): preserve decision timestamp normalization
The route-level str() cast introduced in the initial fix bypasses
DecisionResponse._normalize_timestamp, the field validator on main that
converts POSIX float epochs to ISO-8601 strings via
datetime.fromtimestamp(value, tz=UTC).isoformat().
With the cast in place the API emits raw numeric strings such as
'1786513069.69' instead of '2026-08-12T05:37:49+00:00', breaking
datetime.fromisoformat() for every caller and failing
TestRecordedDecisions::test_list_decisions_serializes_float_timestamp.
It also silently accepts nan/inf/out-of-range epochs that the validator
is designed to reject.
Restore _node_to_decision() to pass the raw stored value through
unchanged so DecisionResponse._normalize_timestamp remains the single
normalization boundary for all three affected endpoints:
GET /api/decisions
GET /api/decisions/{id}
GET /api/decisions/{id}/precedents
Rewrite test_decision_route_timestamp.py so every assertion uses
datetime.fromisoformat() to verify ISO-8601 output and explicitly
asserts ValidationError for nan, inf, -inf and out-of-range epochs.
Add three TestClient integration tests covering the full production
path: record_decision() -> float stored in graph -> HTTP GET -> JSON.
---------
Co-authored-by: administrator <administrator@administratordeMac-mini.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
RelationExtractor.extract_relations(text, entities, ...) requires
entities, but the tool called it with only text, raising TypeError
on every invocation. Run NER first and pass the resulting entities
through, matching how the rest of the pipeline extracts relations.
* Add tests for max_tokens propagation in LLM methods
This test verifies that the max_tokens parameter is correctly propagated to the generate_typed method for different extraction functions.
* fix(tests): make issue-176 regression tests discoverable by pytest
The contributor's PR added tests/optimize reproduce_issue_176.py — a file
with a space in its name that never matched pytest's test_*.py discovery
pattern, so the regression would have been silently skipped in CI/local runs.
The repository already contained a richer canonical regression file at
tests/reproduce_issue_176.py (11 tests across three classes) which had
the same naming problem: it was also never auto-discovered.
The contributor's file added only TestMaxTokensPropagation (3 tests), which
is a strict subset of what the canonical file already covers. No unique
coverage is lost by removing it.
Changes:
- Rename tests/reproduce_issue_176.py -> tests/test_reproduce_issue_176.py
so all 11 regression tests are collected by 'pytest tests/'
- Remove tests/optimize reproduce_issue_176.py (redundant strict subset)
No production code changes. All 11 regression tests pass.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(export): escape Turtle/N-Triples string literals (fixes#1098)
Add RDFSerializer._escape_turtle_literal and apply it to the semantica:text
literal in serialize_to_turtle and the N-Triples text triple. Backslash,
double quote, newline, CR, and tab are escaped per the RDF 1.1 Turtle
STRING_LITERAL_QUOTE grammar, so entity text containing quotes or control
characters no longer emits invalid Turtle/N-Triples.
N-Triples previously escaped only quotes and newlines; now it also handles
backslashes and tabs via the shared escaper.
* fix(export): escape OWL-Time timestamp literals in Turtle output
Addresses Qodo finding on #1148: the OWL-Time branch of
serialize_to_turtle interpolated from_val/until_val directly into quoted
literals. Apply _escape_turtle_literal there too so timestamps containing
quotes, backslashes, or control characters cannot produce invalid Turtle.
* chore: remove stray local files (AGENTS.md, evals superpowers docs) from PR branch
---------
Profiling the viewer in headless Chromium (real DOM, production React)
separated remark parse time, React commit time and DOM node count across
large-prose, large-code-block, deep-nested-list and GFM-table fixtures.
Two findings, one of which is fixed here.
1. Every re-render re-parsed the whole document and remounted the whole
subtree. remarkPlugins and the ~20-entry components map were inline
literals, so each render allocated fresh arrow components; React saw a new
element type per mapped tag and replaced the DOM rather than updating it. A
DOM-identity probe confirmed the remount on every fixture. Because
react-markdown runs the remark pipeline inside its own render, an unrelated
state change -- clicking Copy, toggling Preview/Source -- re-paid the full
parse. Measured 364ms for a 1000-row GFM table and 1121ms for 2000 rows.
Hoisting both props to module scope and memoising the rendered element on
rawContent drops re-render cost to ~0.1ms across every fixture and removes
the remount (DOM identity now survives). Initial mount and node switching
are unchanged, since those are genuine parses.
2. Initial parse of large GFM tables is quadratic and lives upstream in
remark-gfm: the same table text parses in 12.5ms without the plugin and
1156ms with it at 2000 rows. Not addressed here -- any mitigation is a
product decision and is tracked on the issue.
Note that document size is the wrong threshold for this: 562KB of prose parses
in 85ms while a 27KB GFM table takes 102ms. Row count, not bytes, predicts cost.
Rendered output is unchanged; the components map is moved verbatim. All 66
Explorer graph-workspace tests pass.
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
MarkdownContentViewer.tsx exported the isSafeUrl helper alongside the
component so it could be unit tested, which tripped
react-refresh/only-export-components.
Move the helper into a sibling pure module, markdownUrlSafety.ts,
following the existing GraphWorkspace convention for testable non-component
logic (graphAnalytics.ts, pluginRegistryPredicates.ts,
temporalLifecyclePredicates.ts). The function body is moved verbatim — the
scheme allowlist, protocol-relative rejection, whitespace-only guard and
malformed-URL handling are unchanged — so the existing URL-safety tests pass
untouched apart from the import path.
The component module now exports only its component and prop type, clearing
the lint error without any change to the lint configuration.
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Dataset(default_union=True) presents triples from every named graph as a
single merged view and is itself an rdflib.Graph subclass, so it satisfies
_convert_to_dict()'s Graph-typed contract directly. Drops the O(n) manual
quad-copy loop while keeping the same named-graph fix and behavior.
_tool_export_graph fell through to json.dumps(kg) for any format outside
the RDF set, including values never declared in the tool's own inputSchema
enum. Nothing in this server validates tool-call args against inputSchema
before dispatch, so a typo'd or unsupported format (e.g. "yaml") silently
returned JSON data labeled with the wrong format and no error.
Validate against the declared format list up front and reuse the same
constant for the inputSchema enum so the two can't drift apart again.
* fix(triplet_store): OxigraphStore silently ignores storage_path and skips flush
Two persistence bugs in OxigraphStore:
1. `storage_path=...` was silently swallowed by **config. The __init__
parameter is named `path`, so passing the project-conventional
`storage_path` (used by ProvenanceManager and other stores) left
self.path = None and the store silently degraded to in-memory —
no error, no warning, data gone on exit. Accept `storage_path` as
an alias for `path`.
2. add_triplets never called flush(). pyoxigraph auto-flushes via
background threads but, per its docs, "might lag a little bit" —
that lag is a race where reopening or crashing immediately after a
write observes fewer triples. Call flush() explicitly for on-disk
stores to close the window.
Both verified: with the fix, `OxigraphStore(storage_path=...)` persists
across reopen; without it, data is lost.
* fix(triplet_store): improve oxigraph persistence
* test(triplet_store): clarify oxigraph persistence test
---------
Co-authored-by: administrator <administrator@administratordeMac-mini.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix: preserve generation kwargs in relation extraction
* fix: include generation params in extraction cache keys
* fix: cover provider-specific generation params in extraction cache key
_GENERATION_CACHE_KEYS only covered the common OpenAI-shaped generation
params, so calls that differed only in Anthropic's system/stop_sequences,
Gemini's candidate_count, or Ollama's repeat_penalty/num_ctx/context_window
could still return a stale cached result generated under different settings.
Add these provider-specific keys to the cache key and add regression tests
covering system prompt, stop_sequences, and repeat_penalty.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
``requests.exceptions.RequestException`` subclasses ``OSError``, so the
``except (ImportError, OSError)`` handler in ``load_from_api`` swallowed
genuine network failures (connection errors, timeouts, HTTP errors) and
reported them as "requests library not available", hiding the real cause.
Remove the obsolete handler so those failures fall through to the generic
handler, which reports "Failed to load from API: ..." and chains the real
exception as ``__cause__``. Update the docstring's ``Raises`` section to
match the actual behavior.
Fixes#949
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
* Remove unreachable dead code
Delete symbols with no callers anywhere in the codebase, tests, or docs,
confirmed by a repo-wide search. These are internal/private or app-layer
(explorer) symbols, not part of the importable library's public API
(no __all__ / package re-export), so there is no user-facing change.
Removed:
- poc_runner.py: parse_import_csv_row (unused nested helper)
- change_management/version_storage.py: create_graph_snapshot_record
- context/graph_schema.py: drop_decision_schema
- explorer/dependencies.py: get_ws_manager (+ now-unused ConnectionManager import)
- explorer/routes/graph.py: _extract_node_embeddings (+ stale cross-ref comment)
- explorer/routes/ontology.py: ProposalState
- explorer/schemas.py: ErrorResponse, TemporalSnapshotResponse, ExportResponse,
StandardMessageResponse
- semantic_extract/methods.py: _parse_entity_result, _parse_triplet_result
- triplet_store/methods.py: _get_query_engine (+ now-unused _global_query_engine)
Co-Authored-By: Vinv-AI <309466812+Vinv-AI@users.noreply.github.com>
* Address review: drop now-orphaned helper and fix stale docstring
- Remove _coerce_embedding_vector from explorer/routes/graph.py: its only
non-recursive caller was _extract_node_embeddings (removed in this PR), so
it is now dead. The live coercion logic lives in
GraphSession._coerce_embedding_vector.
- Update explorer/dependencies.py module docstring: it no longer injects
ConnectionManager (get_ws_manager was removed); note that websocket manager
access is via app.state.ws_manager.
Co-Authored-By: Vinv-AI <309466812+Vinv-AI@users.noreply.github.com>
* Keep public helpers with a DeprecationWarning instead of removing them
create_graph_snapshot_record() and drop_decision_schema() are not
underscore-prefixed, so downstream users can import them directly from
their modules even though they are not re-exported from the package
__init__.py. A repo search only proves there are no in-tree callers.
Restore both unchanged and emit a DeprecationWarning on call, with a
matching ".. deprecated::" note in each docstring pointing at the
replacement. This keeps the PR non-breaking; the actual removal can
happen in a future major version.
The underscore-prefixed helper removals are unaffected.
---------
Co-authored-by: noQbot <noQbot@users.noreply.github.com>
Co-authored-by: Vinv-AI <309466812+Vinv-AI@users.noreply.github.com>
Co-authored-by: noQbot <anshul@vinv.ai>
* fix: guard integration HTTP requests against SSRF
* fix(openclaw): complete fallback validation and base URL handling
Address the remaining review findings in the OpenClaw integration.
- Strengthen fallback base_url validation to require a non-empty string, valid HTTP(S) scheme, netloc, and hostname.
- Strip leading and trailing whitespace from base_url before storing it.
- Replace the flaky endpoint-construction test that made a real network connection with mocked session assertions.
- Add coverage for _get and _post endpoint construction and timeout forwarding.
- Add regression tests for whitespace-padded base URLs and the fallback validation path.
These changes complete the Qodo review fixes and harden OpenClaw URL handling without changing the intended localhost/private deployment behavior.
The previous commit's fix to 03_Document_Parsing.ipynb collapsed the
cell's source array into a single string and dropped the trailing
newline. Restore the original array-of-lines formatting so the diff
is limited to the corrected badge URL.
Seven introduction notebooks linked to a different notebook's filename
in their Colab badge (off-by-one numbering), sending readers to the
wrong notebook or a 404. Point each badge back at its own file.
Add a Cite Us section to the README with BibTeX citation info, and
align it with docs/citation.md (author/organization: Semantica, 2026).
Update LICENSE and docs/project-license.md copyright holder to
Semantica, and replace the stale Hawksight-AI GitHub org slug with
semantica-agi across READMEs, plugin manifests, cookbook notebooks,
and GitHub templates.
_turtle_object() wrote an IRI-valued metadata value (currently only
sem:sourceUri, from the "uri" metadata key) straight into `<{value}>`
with no escaping. Turtle/N-Triples IRIREFs exclude control
characters, space, and <>"{}|^`\ unescaped, so a value shaped like
`<goodIRI> . <injected> <p> <o>` closed the reference early and let
the rest of the string be parsed as an attacker-chosen extra triple:
metadata={"uri": "https://x> . <https://injected> <https://p> <https://o"}
produced a well-formed Turtle/N-Triples document containing a triple
the caller never asked for.
RDF/XML was already safe (_rdfxml_metadata_lines runs the value
through _escape_xml before putting it in an rdf:resource attribute),
and JSON-LD is safe by construction (json.dumps makes structural
injection impossible) — only the Turtle/N-Triples "iri" literal path
in _turtle_object was unguarded.
Adds _safe_iri_ref(), a narrow percent-encoder for exactly the
characters an IRIREF may not contain unescaped. It's deliberately not
_as_turtle_iri: that also resolves registered prefixes, which a
metadata value never needs, so a dedicated guard stays simpler than
threading namespaces into a module-level helper that has no `self`.
Two regression tests, parametrised over turtle/ntriples: the `>`
delimiter-breaking payload from the report, and a control-character
(newline/tab) variant covering the other half of the excluded set.
Resolves the conflict in semantica/export/rdf_exporter.py between this
branch's metadata clauses (entity/graph metadata statements) and
main's IRI-normalization and XML-escaping hardening
(_as_turtle_iri / xml_escape, landed after this branch's last sync).
Kept both: entity/relationship/graph subjects and objects now go
through _as_turtle_iri (Turtle) or _as_turtle_iri + xml_escape
(RDF/XML), same as every other identifier in these serializers,
while the metadata-clause list building and graph_uri handling from
this branch are preserved unchanged. graph_uri is now normalized the
same way for consistency with the rest of the file.
Verified: tests/export + tests/ontology (411 tests) and the existing
Turtle-IRI regression suite (test_rdf_exporter_turtle_iris.py, 9
tests) all pass against the merged code.
Matches the README's existing convention for query params inside
HTML attribute URLs (e.g. the Trendshift badge), per review feedback
from Zohaib Hassan and Qodo on this PR.
Co-authored-by: OctoBored <212877535+OctoBored@users.noreply.github.com>
ProgressTracker attached ConsoleProgressDisplay unconditionally, so any
script or CI job that piped or redirected stdout had one progress bar per
stage written into its output, escape sequences included. A plain
`python demo.py > out.txt` captured 173 bytes of progress-bar noise around
10 bytes of the program's own output.
Console progress is now attached only when stdout is an interactive
terminal, when running under Jupyter, or when SEMANTICA_FORCE_PROGRESS is
set. FileProgressDisplay is untouched, so progress logging still works in
pipelines, and SEMANTICA_DISABLE_PROGRESS keeps its existing meaning and
still takes precedence.
Both progress environment variables are now documented in the README and
the utils reference; SEMANTICA_DISABLE_PROGRESS previously existed only in
the reference page.
Deviations from the issue: the issue suggested disabling the tracker on
non-TTY stdout. This gates the display instead, because disabling the
tracker would short-circuit before FileProgressDisplay and take file
progress logging down with it, and the ~20 modules that set
`progress_tracker.enabled = True` in __init__ would need the property
setter taught about TTY state to avoid undoing it. Gating the display
leaves both alone.
Design note: the claim comment on the issue proposed an
`enabled: Optional[bool] = None` constructor opt-in; during implementation
the opt-in became SEMANTICA_FORCE_PROGRESS, which needs no signature change
and follows the NO_COLOR/FORCE_COLOR convention. Known limitation: TTY
detection runs once at tracker construction (the tracker is a process-wide
singleton), so a process that redirects stdout after first use needs the
env vars to change behaviour.
Fixes#1185
_as_turtle_iri() re-encoded absolute IRIs wholesale, turning already-valid
percent-escapes like %20 into %2520. Only spans outside existing valid
%XX escapes are quoted now, so malformed escapes (%zz) still get repaired
while valid ones pass through unchanged.
serialize_to_ntriples()/serialize_to_rdfxml() also passed only the
@context-derived namespaces into _as_turtle_iri(), which shadowed the
built-in semantica:/rdf:/rdfs:/owl: prefixes entirely whenever any
@context was present. _as_turtle_iri() now always merges the built-ins
with whatever namespaces the caller passes.
The adapter inventory and connection examples referenced classes that
don't exist in semantica.graph_store (Neo4jGraphStore, NeptuneGraphStore,
AgeGraphStore) and used constructor kwargs that don't match the actual
adapters (username vs user, host vs endpoint, url vs endpoint, etc.),
verified against each adapter's real __init__ signature and by
constructing every example against the live classes.
- Correct class names: Neo4jStore, AmazonNeptuneStore, ApacheAgeStore
- Fix kwargs for all seven examples to match actual constructors
- Fix ApacheAgeStore's connection_string to libpq keyword=value format
instead of a postgresql:// DSN, which the adapter doesn't accept
- Reclassify Anzo from interface/BYO to built-in — AnzoStore is a real,
exported, tested adapter
- Add the two adapters missing from the inventory: FalkorDBStore and
OxigraphStore
- Replace the literal password='password' example with an env var
- Note a real RDF4JStore bug found while verifying the RDF4J example:
repository_id is a named constructor parameter but the implementation
reads it from **config instead, so it's silently ignored and the
store always connects to the "default" repository
vector_store_config.get_all() always includes a "dimension" key, so
forwarding it via **config into VectorIndexer(dimension=dimension, **config)
raised "got multiple values for keyword argument 'dimension'" any time the
default index-creation path ran with the default config — including
`semantica embed index`, which is exactly the second half of the #994
quick-start pipeline this PR fixes.
Review feedback: analyze_decision_influence(), trace_decision_causality(),
and find_precedents() had the same vocabulary split as get_causal_chain().
The first two read edge_type_index, which is keyed by the RAW edge_type
string, so they now filter index keys by normalized type; find_precedents()
accepts the analyzer's 'precedes' spelling alongside PRECEDENT_FOR.
Adds regression tests for all three call sites.
* docs(shacl): warn that rdfs:range makes sh:class unfalsifiable under entailment (#1130)
* docs(shacl): self-contained pitfall example, sh:node coverage, and wrapper clarifications (#1130)
Review feedback: normalization must not turn invalid inputs into
AttributeError. Non-string relationship types now raise ValueError before
normalization, matching the pre-change behavior; strings are stripped
before alias lookup.
get_causal_chain() matched only the canonical uppercase spellings
(CAUSED, INFLUENCED, PRECEDENT_FOR), while CausalChainAnalyzer's
vocabulary includes the present-tense forms (causes, influences,
leads_to, supports) — and the two differ in word form, not just case,
so case-insensitive matching alone would still miss them. An edge
recorded as "causes" produced an empty audit chain.
Storage normalizes both vocabularies onto the canonical types via
_CAUSAL_EDGE_ALIASES; traversal accepts the union (_CAUSAL_TRAVERSAL_TYPES).
add_causal_relationship() now accepts either spelling and stores the
canonical form.
CodeQL (py/incomplete-url-substring-sanitization) flagged the "https://schema.org/"
in flattened check because it pattern-matches on URL-ish strings tested with `in`.
flattened is always a list here, so the check was already exact membership, not a
substring test on untrusted input, but the ambiguous idiom tripped the scanner.
Rewrite as an explicit equality comparison so the intent is unambiguous.
analyze_evolution() previously appended a constant placeholder (durations.append(1)) for every bounded relationship, so the stability metric was always 1.0 when any bounded relationship existed and 0 otherwise, never reflecting actual valid-time durations.
Stability now computes the mean valid-time duration in seconds ((valid_until - valid_from).total_seconds()) across relationships with both bounds set; unbounded/half-open intervals are skipped and non-positive intervals clamped to 0. Adds unit tests and a CHANGELOG entry.
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
* fix(dedup): never merge entities with different explicit types (fixes#1137)
The duplicate candidate confidence scoring only rewarded same-type pairs
but never penalized different-type pairs, so a Person 'Alice' and an
Organization 'Acme' (different id, type, and name) passed the confidence
threshold and were merged, silently dropping one entity. Add a type guard:
when both entities carry a non-empty type and they differ, the pair is
never a duplicate candidate (confidence 0, reason 'type_mismatch').
Untyped entities and genuinely duplicate same-type pairs keep their
previous behavior. Regression tests cover all three cases.
* fix(dedup): honor Entity.type and exclude mismatch structurally (review fixes)
Two gaps from code review (#1149):
1. _get_entity_value mapped object 'type' exclusively to .label, which
Entity objects never have — their type lives on .type. The mismatch
guard therefore never saw the type of Entity objects, and differently
typed objects could still merge. Read .type first, fall back to .label.
2. The mismatch branch returned a normal candidate with confidence 0.0,
but detection filters with >= confidence_threshold, and 0.0 is a
documented valid threshold, so mismatches slipped through. Exclude
type_mismatch candidates structurally at both filter sites regardless
of threshold.
Adds tests for Entity objects with different types and for
confidence_threshold=0.0. 94 dedup tests pass.
---------
* fix(reasoning): refuse SPARQL query execution instead of returning empty results (#1083)
SPARQLReasoner.execute_query() never executed the query: both branches
returned an empty SPARQLQueryResult, with or without a triplet store, so
callers that trust an empty result as "no matches" silently drew wrong
conclusions. Until a real triplet-store execution path lands, the method
raises NotImplementedError with an explanation, per the issue's
suggestion. The dead cache/inference scaffolding after the execution
point is removed along with it.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(reasoning): align execute_query() docs with the NotImplementedError contract (#1087)
Review feedback: the docstring still carried a "Returns" section and the
reasoning guide showed execute_query() returning bindings, both of which
now mislead. The docstring documents Raises only, the guide demonstrates
expand_query() and points to rdflib for execution until the triplet-store
path lands, and query_cache/clear_cache() are marked as reserved for that
future execution path.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(utils): bound caller-controlled keys in validation error messages (#1001)
_require_recognized_keys() and _require_nothing_dropped() interpolated
supplied keys directly into ValidationError messages, so a megabyte-long
key produced a megabyte-long exception and, through the export wrappers
that log the full exception, an equally large log entry. Keys are now
rendered through _truncate_key(), which bounds the display at 64
characters with an ellipsis; the supplied payload is never modified.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(utils): bound the count of keys shown in validation error messages (#1001)
Review feedback: per-key truncation did not bound the number of keys
shown, so a payload carrying many short unknown keys could still size the
message (and the log entry that records it). _truncate_key_list() caps
the display at 8 keys and appends "and N more", keeping the message
actionable without letting the payload size it.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Convert_units() was validating categories on raw input like "kg" or "ft"
instead of the normalized unit name, so aliases got checked against a
category list that only has canonical names in it. Any alias-based
conversion that should've worked just raised ValidationError instead.
Fixed by normalizing both units before the category check runs.
Also added foot/yard/mile/gallon to the alias map - they already had
conversion factors but weren't mapped to their canonical names, so they'd
still have failed even after the above fix.
Turned out there was a second bug hiding behind the first one: the category
check defaults both sides to None, and None == None is True, so two aliases
from different categories that neither resolved to a real category would
silently pass instead of raising. kg -> ft would just return a number
instead of erroring. Normalizing first fixes this too, since aliases now
resolve to their actual categories and the mismatch gets caught.
Added a regression test locking that second one down - kg->ft and gal->lb
now raise ValidationError instead of silently converting.
Fixes#931.
Docker build was broken on python:3.14-slim because gensim doesn't ship a
3.14 wheel yet (typical of bleeding edge Python), so pip
tries to compile it from source and there's no gcc in the slim image.
gensim's a core dependency so every build hit this.
Went back to 3.13 instead of installing a compiler : simpler, and 3.14 was
just a jump from an automated bump PR anyway.
Fixes#1025.
Per the Qodo review: only string application ids are recorded in (and
resolved through) _app_node_id_map. Internal ids are commonly integers,
so an integer application id could collide with — and silently remap —
a caller-supplied internal id of the same value. Also pass a labels list
to create_node in the regression test, matching the API signature.
GraphStore.add_edges reads application-level string ids from
source_id/target_id and passed them straight to the backend, while
Neo4jStore.create_relationship matches on internal integer ids (id(n)).
Nothing resolved one to the other, so persisting a graph created every
node and zero relationships — each edge failed with 'nodes not found'
as a logger.warning and the call appeared to succeed (#1136).
add_nodes already receives the application-id/internal-id pair from
create_nodes (the app id is preserved in properties['id']) and discarded
it one statement before add_edges needed it. Keep the map on the store,
populate it from both add_nodes and create_node, and resolve known
application ids in create_relationship. Unknown ids pass through
unchanged, so direct internal-id callers and backends whose ids are the
application ids keep their existing behavior.
#1123 through #1127 landed while this was open, and #1125 rewrote the same
four entity loops this branch extends. Confidence is now normalised through
normalize_confidence, which returns None for a value that has no xsd:decimal
form, so the clause can be absent.
Resolved by folding that into the clause list this branch already builds:
the Turtle path assembles its predicate-object clauses and then terminates
the last one, which is what makes a variable-length list work at all, and
an omitted confidence is simply one clause fewer. RDF/XML and JSON-LD take
the upstream conditional as written, with the metadata call after it.
convert_kg_to_rdf copies metadata into the RDF-ready dictionary at
rdf_exporter.py:302 and no serializer has ever read it back out. Turtle,
N-Triples, RDF/XML and RDFExporter's JSON-LD each write an entity's id,
type, text and confidence and nothing else, so an entity keeps its
confidence score and loses what produced it. JSONExporter's json-ld path
keeps the same fields, which is how one knowledge graph exported two ways
carried the user's data through one exporter and none through the other.
Measured on e3405ebc with an entity carrying four metadata keys: 3 triples
per format, 0 of them metadata. With this change: 7 triples per format,
4 of them metadata, and the same four in all four formats.
The keys Semantica itself writes are mapped to declared terms in
DEFAULT_METADATA_TERMS and declared in semantica-ns.ttl. A key the caller
supplied is not: which namespace an arbitrary key belongs in is #1146, and
that issue is open on the maintainer's modelling call, so the exporter
warns and skips rather than inventing an IRI. Callers who already know the
answer pass metadata_terms={key: iri}.
Two keys cannot keep their own name. sem:source is already the
ObjectProperty holding the subject of a reified relationship, so the Neo4j
loader's "source" is written as sem:sourceSystem and its "uri" as
sem:sourceUri, the one term whose value is a node rather than a literal.
sem:builtAt and sem:snapshotAt have range xsd:string, not xsd:dateTime.
GraphBuilder stamps with a timezone-naive datetime.now(), and #1114 is the
demonstration of what typing such a value as xsd:dateTime costs: a
timezone-qualified SPARQL filter over it silently drops the row. #1121
swept export and provenance and deliberately left kg/ alone.
Graph-level metadata is written only when the caller names the graph with
graph_uri, because this serializer has never minted a document node and
#1147 is where that default belongs once it lands.
The lexical form and datatype of a value are chosen once, in
_typed_literal_parts, so the four serializers cannot come to disagree
about them the way they disagreed about confidence in #1100. The JSON-LD
path writes explicit @value/@type rather than JSON's native numbers,
which would have made an integer xsd:double there and xsd:integer
everywhere else.
21 tests, asserting on the parsed graph in all four formats. Output is
unchanged when no metadata is present. Full-suite failure set is identical
to the parent commit: 512 = 512.
A JSON-LD document with a top-level `@id` *and* `@graph` places its terms in a named
graph. `rdflib.Graph.parse()` loads only the default graph and discards the rest
without raising, so every class and property in such a document was dropped while
the load reported success.
`OntologyIngestor.ingest_ontology` now parses into a `Dataset` and flattens the
quads into the working `Graph`, keeping both the default and the named graphs. This
is the same `Graph` -> `Dataset` migration #757 made for `JenaStore` (#756); the
ingest path was not covered by it.
Measured on the 12-line reproduction from the issue:
before classes=0 properties=0
after classes=2 properties=0
On a real ontology the gap is larger: 25 triples / 1 subject against 719 / 88 for
the document that surfaced this.
Tests: `tests/ingest/test_ontology_named_graph.py` covers the named-graph document,
keeps a canary on the default-graph document so the fix cannot trade one blind spot
for another, and asserts that the reported result matches the terms returned.
Reverting `Dataset()` to `Graph()` turns all four red.
`tests/ontology`, `tests/export` and `tests/ingest` pass apart from six failures in
web/feed/database/API ingestion, unrelated to this change and failing the same way
on an unmodified checkout.
Not included, and happy to add here or as a follow-up: making a load that yields
zero classes stop returning `status: "success"`. That value is what made this take
an afternoon to find, but it is a behaviour change on a different layer and seemed
worth reviewing on its own.
* test(visualization): isolate optional dependency mocks
* test(visualization): stop requiring Plotly in unit tests
Removing the global sys.modules stubs left the tests that patch
`...go.Bar`, or call a visualizer, with nothing standing in for the
module level `px` and `go` aliases. Those are None when Plotly is
missing, so patch resolution and _check_dependencies() both failed.
Add a helper that substitutes a double only for the aliases that are
None, leaving the real module in place when Plotly is installed.
---------
* test(ingest): track relationship provenance via ProvenanceManager
kg.ProvenanceTracker has no track_relationship and never did, so
patch.object raised AttributeError before the test body ran.
Closes#1055
* test(ingest): disambiguate relationship keys and pin provenance storage
Addresses review feedback on #1071.
---------
The MCP server's export_graph tool was broken on all formats in 0.6.5/0.6.6:
- json: JSONExporter().export(graph) was called without the required
file_path argument -> TypeError surfaced as {"error": ...}.
- RDF branches: RDFExporter().export_to_rdf(graph, ...) received the
ContextGraph object instead of the canonical kg dict -> AttributeError
(ContextGraph has no 'get').
- All branches: the RDF path printed a rich progress bar to stdout,
corrupting the stdio JSON-RPC framing and hanging the client (observed:
300s timeout over MCP while the same call returns in <1s directly).
Fix: convert via ContextGraph.to_kg_dict() before exporting, serialize the
json branch to a string, and force SEMANTICA_DISABLE_PROGRESS=1 for the
server process — stdout is the protocol channel, not a console.
Tests: tests/test_mcp_server_export_graph.py covers every format, the json
payload shape (entities/relationships), and the progress-disable env var.
All four are in the branch that recognises an already-converted document,
which has to survive every shape JSON-LD allows rather than the one shape
Semantica happens to produce.
A knowledge graph carrying a context of its own took the already-JSON-LD
branch and skipped its own conversion, leaving entity ids, relationship
endpoints, types and confidences as raw keys. The entities/relationships
test now runs first, and a converted document never has those keys, so the
double-conversion guard is unaffected.
A context that is a URL or an array cannot be merged key by key, and was
being dropped in favour of Semantica's defaults, silently changing how every
term expands. Both are kept as an array now, the caller's winning, which is
the same precedence the dictionary branch already used. An explicit null is
left alone on purpose: in an array it resets the active context and would
take the semantica prefix with it.
@graph may be a single node object as well as an array. list() on a
dictionary yields its keys, so an object-valued graph was replaced by a list
of strings.
A caller may hand us a document that is deliberately a named graph. That name
is theirs to keep, so it is no longer flattened; it is nested one level and
the export's own provenance goes beside it, in the default graph, where a
plain reader can see it.
Four tests, one per case, all failing before this commit.
A JSON-LD document with a top-level @id and a top-level @graph is a named
graph. Its members become quads named by that @id, and the default graph is
left empty. rdflib.Graph.parse() keeps the default graph and discards the
rest without reporting anything, so every consumer that loads an export the
ordinary way saw the document header and none of the data.
_convert_to_jsonld wrote the payload into @graph and then stamped a document
@id beside it, which named every list export and every generic-dict export.
export_knowledge_graph made it worse: it converted the graph to JSON-LD and
handed the finished document back to export(), which converted it a second
time. The converted document no longer carries entities/relationships keys,
so the second pass treated it as opaque and buried the whole knowledge graph
inside @graph, under a name that is a wall-clock timestamp.
A two-entity, one-relationship graph exported to JSON-LD parsed as 2 triples
with Graph() and 21 quads with Dataset(). The 19 missing triples were the
entire knowledge graph.
The document node now goes inside @graph when the payload lives there, and is
the document itself otherwise, so no export names its own graph by accident.
An already-converted document is merged rather than nested, which also stops
the export carrying two document nodes and two @context blocks.
Semantica's reader has the mirror of this bug (#1129), so these exports could
not be read back by Semantica either.
Bump version, cut CHANGELOG's Unreleased section into 0.6.6, backfill
changelog entries for merged PRs missing from it, and refresh
version-dependent references in README/docs.
Consolidates the remaining #994 fixes into this PR so it can fully close
the issue, per maintainer request.
From #1005 (yzxcj797):
- EmbeddingGeneratorWithProvenance.__getattr__ self-recursion guard:
accessing self._generator via attribute syntax re-entered __getattr__
forever when _generator was absent (failed __init__, pickle/copy probes
like __deepcopy__). Private-name lookups now raise AttributeError.
- 4 regression tests in TestMethodDispatchRecursion: default dispatch no
longer self-recurses for generation/text, a user-registered custom
method still takes precedence, and a bare provenance wrapper raises
AttributeError instead of RecursionError.
(The methods.py identity guards from #1005 are already present here.)
From #1006 (yzxcj797):
- doctor gains two embedding backend checks, "Embeddings
(sentence-transformers)" and "Embeddings (fastembed)". Default is a
cheap import+version check (uninstalled backend now reports fail with a
pip hint instead of invisible). --deep-embeddings (or
SEMANTICA_DOCTOR_DEEP_EMBEDDINGS=1) instantiates via TextEmbedder and
embeds a probe, catching backends that import cleanly but cannot load
(the #994 failure mode) via the hash-fallback-active signal.
_DeepEmbeddingFailure marks post-import runtime/model-load failures so
they get a remediation hint instead of a misleading pip-install hint.
- 7 tests in TestDoctorEmbeddings and TestDoctorEmbeddingHintsAndEnv.
Validation:
- tests/test_cli_commands.py: 237 passed (7 new)
- tests/test_embedding_providers.py: 9 passed (4 new)
- AST parse + import of all four modules OK
Address Qodo review on #1113:
- Escape entity text for Turtle, RDF/XML and N-Triples so names containing
quotes, XML markup, backslashes or control chars cannot break out of the
literal or inject RDF/XML (High/Security).
- Replace colon-only id split with URI-aware local-name extraction so an id
like https://example.org/acme yields 'acme', not '//example.org/acme'
(Medium/Correctness).
- Add regression tests: escaping (quotes/XML/backslash/CR/LF), parseability
via rdflib, and exact id local-name assertions.
convert_kg_to_rdf() maps an entity's 'name' to 'label'/'text' but was
never invoked from export_to_rdf(), so graphs produced by GraphBuilder
(which emit 'name') exported with an empty semantica:text on every RDF
format (turtle, ntriples, rdfxml, jsonld). Call convert_kg_to_rdf() at
the export boundary before validation/serialization so all formats
benefit from a single normalization step.
Add regression tests asserting a name-only entity exports a non-empty
label across all four serializers and the file-writing entry point,
plus that an explicit 'text' is not clobbered and an id tail is used
as a fallback label.
Closes#1097
Review finding, reproduced. `call_custom_method(..., **kwargs)` builds a
fresh dict from the unpacking, so popping `fallback_on_custom_error`
inside the helper left the caller's own kwargs untouched. On the fallback
path the flag was then forwarded straight into the default
implementation, which is exactly the case the flag exists for.
Instrumenting the default exporter shows it arriving:
config handed to the default exporter: {'fallback_on_custom_error': True}
Most defaults take **kwargs and ignore it, which is why nothing failed
loudly, but any default with a fixed signature raises TypeError on it.
The helper's docstring promised the flag was never forwarded, so the
promise was false rather than merely untidy.
All 58 sites now pop the flag from their own bag and pass it explicitly.
One site in normalize/methods.py names its bag `**context` rather than
`**kwargs`, and is handled too.
3 further tests: the flag reaches neither the default implementation nor
a successful custom method, and a per-module guard that every call site
has a matching pop, since a site that forgets one reintroduces the leak
silently.
Failure set across the six affected modules is unchanged against
upstream/main: 37 pre-existing, none new.
Review finding, reproduced. The reified node reduced the relationship
type to its last fragment or path component, so
https://a.example/ns#employs and https://b.example/ns#employs both became
semantica:type "employs". The temporal node no longer said which
predicate it described, and it disagreed with the direct triple written
beside it, which carries the full IRI.
The full predicate is written instead. I had flagged the local-name form
as a deliberate simplification in the PR description; the collision case
shows it was the wrong call.
2 further tests.
1. An absurd magnitude expanded instead of being rejected. xsd:decimal
has no exponent notation, so the value has to be written out in full,
and "1e100000000" is eleven characters that expand to a hundred
million digits. "1e100000" already produced a 100,001 character string
here. The export path continues past validation errors, so one
malformed field could exhaust memory. Values beyond
MAX_CONFIDENCE_EXPONENT are now omitted like any other unusable value.
1e-9 still round-trips.
2. Decimal keeps the sign of zero, so 0.0 and -0.0 serialised as "0" and
"-0", which are two distinct RDF terms. That is exactly the duplicate
this PR exists to remove, so zero is normalised.
4 further tests.
Four findings from the automated review, all reproduced first.
1. The fix only reached Turtle. `_uri` was the single place I corrected,
and JSON-LD and N-Triples build sh:targetClass, sh:path and sh:class
straight from graph.base_uri, so two of the three formats went on
emitting shapes that match nothing. That is the defect this PR claims
to close, still live wherever the output is not Turtle. All three
serializers now resolve through one `_term_iri`, and the pySHACL
violation test runs against each of them.
2. Classes and properties shared one name-keyed index built with
setdefault, so a property named after a class was permanently mapped
to the class IRI and its sh:path validated the wrong predicate. The
index is now split into class_iris and property_iris, and each call
site says which it wants.
3. OntologyEngine.to_shacl forwarded target_namespace and
attach_domainless_properties through generate(**options), which never
reads them, so both were silently dropped on the public path. They are
now named parameters passed to the constructor, and documented.
4. The opt-in attachment logged at debug. It broadens constraint
generation, so it warns.
7 further tests, including the target-namespace and real-violation checks
parametrised across Turtle, N-Triples and JSON-LD.
Four findings from the automated review, all reproduced first.
1. The name fallback minted invalid IRIs. `_term_iri` pasted a raw name
onto the ontology base, so a class named "Customer Account" produced
<https://example.org/onto/Customer Account>. rdflib only warns about
the space, Oxigraph rejects it with "Invalid IRI code point". That is
the same class of defect this PR set out to fix, introduced by the fix
itself. Local names are now percent-encoded.
2. `improve_coherence` raised AttributeError. It lives on
OntologyOptimizer, which holds no namespace manager, so the URI
fallback I added there crashed on any ontology carrying a class
without a URI. It now mints from the ontology's own base through a
shared module-level helper.
3. `owl:Thing` was treated as an absolute IRI. It matches the generic
scheme grammar, so `_is_absolute_iri` accepted it and domains and
ranges came out as the term <owl:Thing> rather than
<http://www.w3.org/2002/07/owl#Thing>. This is the live path: stage 4
of the generator assigns ["owl:Thing"] to object properties with no
inferred endpoints. Absoluteness is now decided on a real scheme, and
the well-known prefixes expand.
4. Unusable property entries were dropped in silence. Non-dictionary
entries and definitions carrying no type are now named in a warning.
6 further tests, including a strict-parser check through Oxigraph, which
is what catches the space that rdflib waves through.
Every module supporting custom methods wrapped the registered callable in
a bare `except Exception`, logged a warning, and carried on into the
built-in implementation:
try:
return custom_method(data, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(f"Custom method {method} failed: {e}, falling back to default")
That makes a registered method advisory. It can add behaviour, but it
cannot decline. For a gate, a validator or a policy check, declining is
the entire purpose: raising is how such a method says "do not produce
this output". Catching the exception and running the default produces
exactly the output the method was registered to prevent, and the only
trace is a warning.
Demonstrated with a verifier that rejects invalid RDF and deletes the
file. The fallback wrote it straight back.
`call_custom_method` in utils/custom_methods.py now holds the policy in
one place: an exception from a registered method propagates. Callers who
relied on the old behaviour can pass `fallback_on_custom_error=True`,
which restores warn-and-continue for that call and is consumed by the
policy rather than forwarded to the method.
The swallow was in six modules, not only the one the issue was filed
against, so all 58 sites are converted: export 13, ingest 13, normalize
13, parse 12, embeddings 4, kg 3. The rewrite is mechanical and uniform.
Sentinel comparison is by identity, so a custom method returning None, 0,
"" or an empty list is not mistaken for a failure.
13 tests in tests/utils/test_custom_method_can_refuse.py, including the
issue's own demonstration and a guard that no module still carries the
swallow. Across the six affected modules the failure set is identical to
upstream/main: 37 pre-existing failures before and after, none new, with
869 passing against 856 on the baseline.
include_temporal=True emitted a well formed OWL-Time interval hanging off
a relationship IRI that appears nowhere else in the graph. A relationship
is written as a single triple, <e1> <employs> <e2>, so there is no node
for the time to attach to:
<...#rel_0_0940a860> time:hasTime <...#rel_0_0940a860__valid_interval> .
Counting inbound arcs to that subject gives zero. The timestamps parse,
they validate, and no query can reach them from the relationship they
describe, which is the only thing they are for.
The JSON-LD path already reifies relationships as sem:Relationship with
sem:source, sem:target and sem:type, and the vocabulary declares all four
terms. Turtle now emits the same shape when it has temporal data to
attach, so the two serializations describe relationships the same way and
the interval has a reachable subject.
The direct triple is unchanged, and nothing is reified when a
relationship carries no temporal data, so default output is untouched.
7 tests in tests/export/test_owl_time_reachability.py, including a SPARQL
walk from the edge to its validity interval, which is what the dangling
node made impossible, and a check that every emitted term is declared in
the shipped vocabulary. Export and ontology suites pass at 228 tests.
#1100 — the four serializers rendered the same confidence four different
ways. Turtle wrote it bare, which the Turtle grammar reads as
xsd:decimal. N-Triples typed it xsd:float. RDF/XML wrote a plain literal
with no datatype. JSON-LD wrote a native JSON number, which expands to
xsd:double. For confidence 0.9 that is four distinct RDF terms, so a
FILTER matches at most one of them, and merging two exports of one graph
gives an entity two different confidence values.
N-Triples also omitted the triple entirely when confidence was absent,
while the other three wrote the 1.0 default, so the two serializations
differed in the number of triples as well as in their datatype.
`normalize_confidence` now produces one canonical lexical form and every
path writes it with CONFIDENCE_DATATYPE. xsd:decimal is the choice
because it is what the Turtle path already produced, so the most used
output is unchanged, and because it is exact: xsd:float is 32 bit binary
and cannot represent 0.9 at all. Values that arrive in exponent notation
are reformatted, since 1e-05 is not a valid xsd:decimal.
#1102 — the Turtle path interpolated the value with no type check, so a
confidence of "high" produced `semantica:confidence high .` and made the
entire document unparseable. One bad field cost the whole export. A value
that cannot be a decimal is now omitted with a warning naming the entity,
rather than written as something the vocabulary contradicts. Numeric
strings are still accepted. Booleans are not, since bool subclasses int
and True would otherwise become a confidence of 1.
sem:confidence in the shipped vocabulary declared no rdfs:range,
deliberately, because declaring one would have contradicted three of the
four exporters. It now declares xsd:decimal, and a drift guard asserts
the vocabulary and the serializers agree.
20 tests in tests/export/test_confidence_literal_typing.py, comparing the
parsed graphs of all four formats rather than their text. Export and
ontology suites pass at 240 tests.
#1104 — SHACLGenerator used one namespace for two jobs. `base_uri` says
where the shape resources live, and it was also used to expand every
sh:targetClass and sh:path. With the default "https://semantica.dev/shapes/"
that made shapes target <https://semantica.dev/shapes/Person>, while data
carries the ontology's own class IRI or the semantica:ns# vocabulary. The
shapes matched nothing.
That failure is silent. A shape with no focus nodes is vacuously
satisfied, so pySHACL reports conforms=True on data that plainly breaks
the stated constraints. The shipped validator agrees the file is fine.
The two namespaces are now separate. `target_namespace` resolves in this
order: an explicit argument, the ontology's declared namespace, the
namespace of any absolute IRI a term already carries, the ontology URI,
and finally the vocabulary namespace the package ships rather than the
shapes namespace. Every class and property name is indexed to the IRI it
expands to, and `_uri` resolves through that index, so shapes always name
the terms the data uses.
#1105 — a property with no declared domain was attached to every node
shape. That states a constraint the ontology does not, and with minCount 1
it makes every instance of every class invalid. Such a property is now
left unattached, with a warning naming it. Passing
attach_domainless_properties=True restores the old behaviour.
tests/ontology/test_shacl_target_namespace.py adds 17 tests that validate
real data through pySHACL rather than reading the shapes text, so a shape
that targets nothing cannot pass by being ignored. They cover a generated
ontology, one that declares only a namespace, and one that carries only
class URIs.
tests/ontology/test_ontology_advanced.py::test_no_domain_property_attaches_to_all_shapes
asserted the #1105 behaviour, so it pinned the defect in place. It is now
two tests: the old expectation against the explicit opt-in, and the new
default.
Export and ontology suites pass at 239 tests.
OWLExporter read `object_properties` and `data_properties`, while
OntologyGenerator emits one combined `properties` list tagged with
type/@type. Every generated property was therefore dropped, and a
generated ontology exported as classes alone.
Class IRIs were worse. ClassInferrer writes `"uri": None` when it is
given no namespace manager, so the stage 3 guard `if "uri" not in cls`
never fired: the key is present, only its value is missing. The exporter
then interpolated the empty string into `<>`, which is a relative IRI
that resolves against the parser's base. Under rdflib that base is the
current working directory, so a two-class ontology parsed as one subject
carrying two rdfs:label values, and the identity of that subject changed
with the directory the export ran from. Oxigraph rejects the same file
outright with "No scheme found in an absolute IRI".
Changes:
- Accept both dict shapes. `_split_properties` classifies the combined
`properties` list by type/@type and merges it with any explicit
`object_properties` and `data_properties`.
- Resolve class and property IRIs through `_term_iri`, falling back from
uri to iri to id to a name joined onto the ontology base. A term with
none of those is skipped with a warning rather than emitted as `<>`.
- Resolve domain and range references through the class index, so a bare
name such as "Person" lands on the IRI that class was exported under
instead of staying relative.
- Resolve data property ranges properly. "string", "xsd:string" and a
full IRI now all give one well formed datatype. The previous
`rdfs:range xsd:{range}` produced `xsd:xsd:string` for generator output,
which no parser accepts. Turtle keeps the compact xsd: form the module
already used.
- Fix the two `not in` guards in the generator so a present-but-None uri
is minted, and mint an absolute IRI rather than assigning a bare name.
- Escape XML text and attribute values, which were interpolated raw, so a
label containing & or < no longer breaks the document.
Turtle and RDF/XML now serialise the same 25 triples for the same
ontology, and both are accepted by rdflib and by Oxigraph.
10 regression tests in tests/export/test_owl_exporter_generator_schema.py,
driven by a real OntologyGenerator run and asserting on the parsed graph
rather than on serialised text. All 10 fail on the parent commit. The
export and ontology suites pass at 231 tests.
Review finding on #1121, and correct: with new entries carrying +00:00
and entries written earlier carrying nothing, query_recorded_between()
and audit_log() compared ISO strings directly, which orders by how a
timestamp is spelled rather than when it happened.
Two consequences, both introduced by the offset this PR adds:
- An inclusive naive bound naming a stored offset-bearing timestamp
sorts below it, because the stored value is the longer string, so the
record it names is excluded from its own range.
- A bound in another offset lands wherever its digits fall.
"2026-08-19T19:45:00+05:30" is 14:15Z, before an entry at 14:19Z, but
string comparison puts it after.
Both paths now compare instants, through a new to_utc_datetime() helper
that reads a missing offset as UTC. That is what the naive values
actually were: provenance stamped with datetime.utcnow(), so reading
them as UTC keeps a stored naive value and the same instant written with
an offset comparing equal instead of ordering by representation. It is
also the read side the remaining 147 call sites will need whenever the
rest of the package is converted.
A bound that cannot be read as a timestamp keeps the historical string
comparison rather than raising on a call that used to work.
Five new tests cover the inclusive naive bound, the other-offset bound,
legacy and offset-bearing entries ordered together, audit_log's since
filter, and the unreadable-bound fallback. The first two fail with
manager.py reverted; the rest are guards.
569 provenance, export and ontology tests pass, and the full-suite
failure set is unchanged at 329, all from optional dependencies missing
locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
semantica/export/ stamped every value with datetime.now().isoformat(),
which reads the machine's local clock. semantica/provenance/ stamped its
own with datetime.utcnow().isoformat(), which reads UTC. Both return a
naive datetime and both serialize identically, so once the value is out
of the process nothing distinguishes them: the same string means two
different instants depending on which module wrote it.
In RDF the consequence is silent rather than loud. Under XSD 1.1 a value
with no timezone compared against one with a timezone is indeterminate
whenever the two fall inside the 14-hour window, SPARQL turns an
indeterminate comparison into an error, and FILTER discards errors as
non-matches. Loading a Semantica-stamped export into Oxigraph next to two
correctly stamped ones and asking which were written before a given
instant returns the other two and drops ours, with no error anywhere.
prov:generatedAtTime, prov:startedAtTime, prov:endedAtTime and
prov:atTime all carry values written this way, so an audit trail cannot
be ordered against timestamps from any other system.
Adds utc_now()/utc_now_iso() to semantica/utils/helpers.py, exported from
semantica.utils, and uses them at all 29 call sites in export/
(json_exporter, yaml_exporter, report_generator, export_provenance) and
provenance/ (manager, schemas, bridge_axiom). Values now read
2026-08-19T14:19:04.229937+00:00: one unambiguous instant, comparable
against any correctly stamped value, and valid xsd:dateTimeStamp.
sem:exportedAt's range in the vocabulary that landed with #1109 is
tightened from xsd:dateTime to xsd:dateTimeStamp accordingly. Its comment
had to explain why the weaker range was necessary; that reason is gone.
datetime.utcnow() is also deprecated as of Python 3.12 and scheduled for
removal. Constructing a ProvenanceEntry under -W error::DeprecationWarning
on 3.13 raised; it no longer does.
Two new test modules cover offset presence on every export and provenance
path, PROV-O literals valid as xsd:dateTimeStamp, comparison against a
timezone-aware instant without TypeError, the Oxigraph filter that
dropped the naive value, the declared range matching what the exporter
writes, and the document @id remaining a valid IRI with +00:00 in it. The
filter test picks a bound inside the indeterminate window on purpose: a
bound years away is determinate even for a naive value, and the test
would pass without the fix. 13 of the 14 fail with this commit's
semantica/export, semantica/provenance and vocabulary reverted.
The remaining 147 naive call sites, in context/, vector_store/, seed/ and
elsewhere, are deliberately untouched: those timestamps are compared
against values parsed back from previously stored naive strings, so
converting the write side alone would raise TypeError on existing data.
That sweep needs a read-side migration and belongs in its own change.
No new failures across the suite: 329 pre-existing failures before and
after, all from optional dependencies missing in the local environment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The #1101 fix covered serialize_to_turtle, serialize_to_ntriples and
serialize_to_rdfxml. Both JSON-LD writers were left interpolating the
entity's own text into f"semantica:entity/{text}" and the endpoints into
f"semantica:rel/{source}_{target}". Three consequences, all reproducible
on 0.6.5 through the public API:
- An entity whose text contains a space, which is most organisation and
person names an extractor produces, mints an invalid IRI. A JSON-LD
parser drops that node in full and says nothing, so the entity is
simply missing from the export: rdflib reads 6 triples for
{"text": "AcmeCorp"} and 2 for {"text": "Acme Corp"}.
- serialize_to_jsonld resolved endpoints from source_id/target_id only,
while the rest of the module accepts source/target too. Every
relationship carrying the second form minted the identical
"semantica:rel/_", so all of them collapsed onto one node and their
types and endpoints merged into a graph nobody wrote.
- The JSON-LD @id and the Turtle IRI for one entity disagreed
(ns#entity/Acme Corp vs ns#entity_a73cb4563ee2e72c), so the two
serializations of one knowledge graph were two different graphs.
Both writers now use mint_entity_iri/mint_relationship_iri, resolving
endpoints both ways and passing the list index the RDF paths pass, so
one knowledge graph carries one node identity whichever serializer
wrote it.
JSONExporter.export_entities and export_relationships also declare the
semantica prefix their @context was already writing "semantica:entities"
against. Without the declaration a processor reads that as an IRI in the
scheme semantica rather than the namespace expansion, which is the
original #1101 defect on a third path: rdflib returns the predicate
literally as semantica:entities.
tests/export/test_jsonld_iri_minting.py parses each export with a real
JSON-LD processor rather than asserting on the JSON text, and covers all
seven claims above. Each test fails on the parent commit.
236 export and ontology tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
serialize_to_rdfxml still defaulted entity_type to the bare string
"semantica:Entity" written into an rdf:resource attribute, which isn't
namespace-expanded the way a Turtle angle-bracket or XML element name is -
the same #1101 failure mode, just on the path the original tests didn't
cover. Now uses the full-IRI DEFAULT_ENTITY_TYPE like the Turtle path.
json_exporter.py emits semantica:format and @type: "semantica:KnowledgeGraph",
neither of which was declared in the vocabulary or included in
EMITTED_TERMS, so the "undeclared terms fail the build" guarantee didn't
actually cover them. Both are now declared with rdfs:label/comment and
added to the guard set.
MANIFEST.in didn't mirror the pyproject.toml package-data addition, so a
source-distribution install could ship without the vocabulary file.
The cross-process minting-stability test replaced the subprocess's entire
environment with a POSIX-only PATH, breaking it on Windows and any host
needing other inherited env vars; now overrides only PYTHONHASHSEED on top
of the inherited environment.
Also folds mint_entity_iri/mint_relationship_iri's hand-rolled
hashlib.sha256(...).hexdigest() into the existing hash_data() helper this
file already imports alongside.
229 export and ontology tests pass, including a new regression test for
the RDF/XML default-type fix.
Co-Authored-By: fabio-rovai <fabio@thetesseractacademy.com>
Both from review on #1109.
The temporal fallback minted from source_id only, while the main serializer
accepts source_id or source. Relationships using the second form therefore
hashed two empty strings, and once the IRI became deterministic that turned a
latent problem into an active one: unrelated relationships at the same list
index collided on the same IRI across exports, so their temporal data aliased
when loaded together. Endpoints are now resolved the way serialize_to_turtle
resolves them, before minting.
The vocabulary declared sem:confidence with range xsd:decimal, which the
N-Triples serializer contradicts by typing the same value xsd:float. Neither is
safe to declare while the two serializers disagree, since the Turtle path writes
the value bare and the Turtle grammar reads that as xsd:decimal. The range is
dropped with the reasoning recorded on the term and a pointer to #1100, which
tracks the disagreement itself.
Extends the drift guard rather than only fixing the instance: a new test asserts
that any range this vocabulary declares matches the datatype the serializers
actually emit, so the class of contradiction that review caught fails the build
next time.
228 export and ontology tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes#1107, closes#1101.
Every RDF export mints terms in https://semantica.dev/ns#, and nothing declared
what those terms meant. The namespace returns 404 and no vocabulary shipped with
the package, so a consumer receiving an export could not tell semantica:text
from a typo of it: in the open world an undeclared IRI is unknown rather than
wrong, and every RDF tool treats the two alike. Closed-world checking is what
separates them, and it needs a document to check against.
semantica/ontology/vocabulary/semantica-ns.ttl declares the fourteen terms the
exporters actually emit, drawn from the emitting call sites rather than from
what a vocabulary ought to contain. It ships inside the package so it loads
without a network round trip, and is the same document intended to be served at
the namespace IRI once hosting and content negotiation are sorted.
tests/ontology/test_vocabulary.py ties the document to the code: every term the
serializers can write must be declared, so adding a term to an exporter without
declaring it fails the build rather than shipping an undeclared IRI.
The vocabulary alone would not have made those IRIs resolve, because the
fallback path minted them from Python's builtin hash(). That is randomised per
process, so the same entity received a different IRI on every run and exports
could not be diffed, deduplicated against an earlier load, or joined to a
provenance record written by an earlier process. Minting now uses SHA-256 and
writes a full IRI in the declared namespace rather than semantica:entity_N,
which inside angle brackets is an IRI in the scheme "semantica" rather than the
prefix expansion, and so never joined with anything written through the prefix.
The same applies to the default entity and relationship types in the Turtle
path.
134 export tests and 91 ontology tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
explain_violations previously rendered hardcoded placeholders (min_count=1,
max_count=1) and misused the violation message as the datatype/class value,
so plain-English explanations were inaccurate. The root cause is that
_run_pyshacl never read the real constraint parameters from sh:sourceShape
when building each SHACLViolation.
Changes:
- SHACLViolation: add min_count/max_count/datatype/class_ fields and include
them in to_dict()
- _run_pyshacl: back-reference sh:sourceShape to extract the real
sh:minCount/sh:maxCount/sh:datatype/sh:class values
- explain_violations: render the real values, falling back to "?" or
descriptive text when unknown
Note: sh:qualifiedMinCount/qualifiedMaxCount are not handled and fall back to
the "?" placeholder.
Adds regression tests covering both the formatting path and the sh:sourceShape
back-reference (skips when pyshacl/rdflib are absent).
* feat(context): add to_kg_dict() adapter for canonical KG shape
Convert ContextGraph internal nodes/edges/source representation into the canonical entities/relationships/source_id shape consumed by RDFExporter and TemporalGraphQuery. Add entities_only filtering that drops dangling relationships, plus README examples and unit tests.
* fix(context): harden to_kg_dict against null props and non-str node ids
- Guard properties/metadata with 'or {}' so nodes loaded from JSON null
no longer raise TypeError when copied (Qodo bug 1)
- Coerce entity id to str(n.node_id) so it matches ContextEdge's
str-coerced endpoints, preventing valid relationships from being
dropped during entities_only filtering (Qodo bug 3)
* fix(kg): accept source_id/target_id endpoints in validator and temporal query
to_kg_dict() emits canonical source_id/target_id keys, but GraphValidator
and TemporalGraphQuery only read the legacy source/target keys, so its
output failed validation and lost relationships (Qodo bug 2).
- GraphValidator: resolve endpoints from either key variant and treat a
resolvable source/target (plus type) as satisfying required fields
- TemporalGraphQuery.analyze_evolution/find_paths: read either variant
- tests: add regression coverage for null props/metadata (bug 1),
non-string node ids (bug 3), and KG-utility consumability (bug 2)
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
- export_table_data() re-raises ValidationError instead of masking it
as ProcessingError via the blanket except Exception.
- _apply_connection_pin() restores the session's original Host header
state on an unpinned hop instead of unconditionally clearing it,
which was dropping a caller-supplied session's own Host override.
- SQL fragment blocklist now masks quoted string/identifier literal
contents before matching, so legitimate data containing a blocked
keyword (e.g. status = 'union') no longer false-positives; a
malformed/unterminated quote stays unmasked and still scrutinized.
Documents the tarball path traversal, latent SQLi, DNS-rebinding TOCTOU,
stored XSS, and SPARQL injection fixes, plus the follow-up hardening
found in review, under [Unreleased] > Security.
Fixes a set of runtime trust-boundary issues from a private security
disclosure (checkout 7c3372c0): tarball restore path traversal, latent
SQL injection in the DB exporter, a DNS-rebinding TOCTOU gap in the
shared SSRF guard, unescaped HTML in report generation, and unvalidated
SPARQL object IRIs in AnzoStore, plus several lower-severity hardening
items found in the same review.
* docs(context): fix unrunnable ContextGraph docstring example
The module docstring's Example Usage block called add_node/add_edge with
keyword arguments they do not accept. add_node(node_id, node_type, ...) takes
node_type positionally and has no properties parameter, so the documented call
raised TypeError; add_edge's parameter is edge_type, so type= fell through to
**properties and polluted edge metadata while appearing to work.
Two of the three broken forms failed silently rather than raising, storing a
nested properties dict or a stray type key instead of erroring.
Add regression tests that execute the documented calls and assert the docstring
itself does not reintroduce the invalid kwargs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(context): close two blind spots in the docstring regression guards
The guards added in the previous commit could pass while checking nothing.
_example_block() terminated the capture at the first "\n\n". The Example
Usage block already contains ">>> " spacer lines, so any reformatting that
turned one into a bare blank line would truncate the capture -- potentially
to empty -- and the guards would then scan a block that no longer held the
add_node/add_edge calls they exist to police.
Both guards also iterated over re.findall() without asserting a match. Zero
matches meant zero assertions and a green test, so the two failure modes
compounded: a truncated block produced no matches, and no matches produced
a pass.
Terminate the block at the next top-level section header (^\S) or end of
docstring instead, so blank lines inside the example are harmless, and
assert the captured block, the parsed statement list, and each guard's
match list are all non-empty.
Extract statements with doctest.DocTestParser rather than a line regex.
This also catches a call reformatted across "..." continuation lines, which
the ">>> graph.add_node(.*" pattern silently skipped, and lets
test_documented_calls_execute exec the docstring's own statements instead
of a retyped copy that could drift from it. Full doctest.testmod isn't
usable here: add_node/add_edge return True and the docs carry no
expected-output lines, so it reports 4 spurious failures.
Narrow the kwarg check to (?<![\w])type\s*= so a legitimate node_type=
or edge_type= in the docs no longer trips a guard aimed at bare type=.
Verified by mutating the module docstring and re-running the guards: extra
blank lines with a valid example still pass; regressed add_node/add_edge,
a type= on a continuation line, deleted calls, and a deleted section all
fail; a legitimate node_type= passes. 6 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(context): correct precedent lookup in docstring example
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Address review feedback:
- State that only protected routes require the API key and note that
/api/health and /api/info are intentionally unauthenticated.
- Note the CLI warning on non-loopback binds only fires in anonymous
mode or when SEMANTICA_API_KEY is unset.
- Add SEMANTICA_API_KEY and SEMANTICA_ALLOW_ANONYMOUS to the Environment
variables table.
* test(export): guard Parquet tests on pyarrow itself, not the exporter import
Closes#1054
* test(export): guard on PARQUET_AVAILABLE so the skip matches the runtime check
find_spec only proves pyarrow is discoverable, not importable. Addresses
review feedback on #1056.
---------
Two findings from the Qodo review:
- Sigma's edge label renderer draws data.label, but the graph stores the
relationship type in edgeType — enabling renderEdgeLabels alone left
edges blank. The edgeReducer now maps edgeType onto label (suppressed for
hidden edges).
- renderEdgeLabels was hardcoded on with no way to disable it. It now
follows a new edgeLabelsEnabled entry in the Effects panel (default on),
wired through the existing GraphEffectToggle/GraphEffectsState plumbing,
so dense graphs get their label-free edges back.
* fix(security): prevent auth header leakage across redirects
* fix(security): harden redirect credential handling
Address Copilot and Qodo review findings for #947.
- Remove unused variables, imports, and unnecessary pass statements from tests.
- Harden cross-origin redirect handling for per-request auth credentials.
- Strip session-level auth handlers before cross-origin redirect hops.
- Prevent session.auth from regenerating Authorization headers.
- Disable trust_env during cross-origin hops to prevent .netrc credential injection.
- Restore session auth and trust_env state reliably with try/finally.
- Add regression coverage for auth=, session.auth, trust_env, and multi-hop redirects.
- Preserve existing security behavior and same-origin authentication semantics.
Validated with 189/189 security and affected tests passing.
* fix(security): scope allow_private_ips to same-host redirects, fix error handling gaps
Follow-up to review findings on #1067:
- MCPClient hardcoded allow_private_ips=True for every redirect hop, not
just its operator-configured host, so a compromised/malicious MCP server
could 302 into private address space (e.g. cloud metadata) unchecked.
request_with_ssrf_guard() gains allow_private_ips_on_redirect: a redirect
target inherits the original host's private-IP trust only when it matches
that host; MCPClient now pins it to False.
- detect_public_api() only caught requests.exceptions.RequestException, but
the SSRF guard raises ValidationError for blocked hosts/redirects, unlike
its sibling ingest_public_api(). Now catches and re-raises it the same way.
- detect_public_api()/ingest_public_api() forwarded session/allow_private_ips
through **options into request_with_ssrf_guard(), which already passes
both explicitly -- a caller supplying either would hit a duplicate-kwarg
TypeError. Both are now popped from request_options first.
New regression coverage for all three in tests/ingest/, plus a CHANGELOG
entry under Unreleased/Security.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
The star history chart was broken due to GitHub stargazer API restrictions, so it could no longer be rendered. Update the README to point to a working alternative that uses a different data source requiring no API token.
SemanticChunker.__init__ only caught OSError around load_spacy_model(),
while NERExtractor's identical call (fixed earlier in this PR) also
catches generic Exception for a model that is installed but fails at
runtime. Bring SemanticChunker in line so a broken spaCy config
degrades to fallback chunking instead of crashing __init__.
Adds a regression test mirroring the existing NERExtractor case, and a
CHANGELOG entry for #998/#1042.
The Explorer API has required SEMANTICA_API_KEY (X-API-Key header) since
v0.6.5, failing closed with 503 when unconfigured. Both the explorer
README security note and docs/explorer-setup.md still claimed there was
no built-in authentication.
Update both to describe the actual behavior: API-key enforcement,
the 503 fail-closed mode, and the explicit SEMANTICA_ALLOW_ANONYMOUS=true
opt-in for local development.
Fixes#1028
Moves a concise version of the system-level vs. foundation-model
explainability clarification up next to the opening pitch, so it's
visible before readers scroll to the high-stakes-domains section.
Adds a consistent scope note to README and docs (concepts, FAQ, index)
stating Semantica does not expose or reconstruct an LLM's internal
reasoning/chain-of-thought. It explains and audits the AI system
around the model: context, provenance, policies, decisions, and
execution history.
The non-Parquet branch still used json.dumps(result, default=str), which
stringifies numpy arrays to their repr() — the same corrupt-output bug
#994 reports, just for .json/.jsonl extensions instead of .parquet.
embed index reads .json/.jsonl via pd.read_json(lines=...) and detects a
vector column by isinstance(val, (list, np.ndarray)); a repr() string
fails that check, so generate→index still breaks for JSON outputs.
- .json/.jsonl now use pandas to_json(orient='records') with real lists
- Unsupported extensions (.txt, .csv, etc.) now raise ClickException
instead of silently writing JSON text, matching embed index behavior
- Error message corrected: pyarrow is now a core dep, not an extra
* docs(contributing): formalize issue assignment and duplicate-PR triage workflow
Comments are no longer required before an issue can be assigned - maintainers
may assign directly based on recent activity. Also documents the duplicate-PR
priority order for triage (contributor PR, claimed issue, activity tiebreak,
late duplicates, overlapping scope).
* docs(contributing): clarify assignment precedence and define activity tiebreak
Addresses Qodo review feedback on PR #1030: the duplicate-PR priority list
now states these rules apply on top of the assignment workflow (opening a PR
pre-assignment doesn't grant priority), and the "most active" tiebreak now
specifies a concrete 60-day window and signals instead of being subjective.
ExcelParser.__init__ called get_progress_tracker() without importing it,
so every instantiation raised NameError and the class was unusable. The
existing test imported ExcelParser but never constructed it, so nothing
caught it. Same defect as #530 in SimilarityCalculator, which was fixed
without sweeping the rest of the codebase.
Add construction coverage for every parser exported from semantica.parse,
driven off __all__ so later additions are covered automatically. These
live outside test_parse_comprehensive.py, whose setUp patches
get_progress_tracker into each parse module and would mock away the
interaction under test.
Closes#1014
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
* feat(crewai): add first-class CrewAI integration (#962)
Add native CrewAI support so Crew agents can share a ContextGraph and
AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching
the existing agno integration pattern.
- SemanticaKGTool: 5 KG actions (extract_entities, extract_relations,
add_to_graph, query_graph, find_related) with sync run()/async arun()
- SemanticaDecisionTool: 5 decision-intelligence actions
(record_decision, find_precedents, trace_causal_chain,
analyze_impact, check_policy) over AgentContext
- SemanticaKnowledgeSource: serializes a ContextGraph into crew
knowledge storage; bridges legacy load_content() and current
validate_content()/aadd() contracts for crewai>=0.80.0
- All classes degrade gracefully when crewai is absent
- New pip extra crewai=... included in the all bundle
- 70 new tests (stub-based present-case + subprocess degradation path)
- Docs: integrations/crewai.md, docs.json nav, README matrix updates
* fix(crewai): harden tools against real Semantica dataclass shapes (#962)
Bugs found during live testing with crewai 1.15.16:
- SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses
('str' object has no attribute 'end_char'): string names were passed to
extract_relations(entities=...), which requires Entity objects, and the
tool read .name/.source/.target instead of Entity's .text/.label and
Relation's .subject/.object. Add shape-agnostic field helpers.
- SemanticaDecisionTool() created an AgentContext without a knowledge_graph,
so _decision_backend was never set and record_decision raised 'Decision
tracking is not enabled'. Wire in a ContextGraph.
- record_decision hard-failed when the agent omitted optional fields; fall
back to category='general', reasoning='agent decision',
outcome='recorded'.
Add tests covering real Entity/Relation dataclass shapes and the live
auto-created AgentContext path (now 77 crewai tests, 212 total).
* fix(crewai): make find_related traverse edges undirected (#962)
ContextGraph.get_neighbors only follows outgoing edges, so a node whose
only edge is incoming (A -> B) reported no related concepts. Rebuild a
bidirectional adjacency from find_edges() in SemanticaKGTool._find_related
so 'related' honors both directions.
* fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962)
- Exclude live graph/context/extractor state from JSON serialization
(model_dump(mode="json")) so CrewAI checkpointing no longer raises
PydanticSerializationError; model_post_init self-heals defaults on restore
- query_graph now searches node content via graph.query() plus id/type
- trace_causal_chain returns an explicit error when causal tracing is
unavailable instead of substituting similarity precedents; call
trace_decision_causality(..., max_depth=...) with the correct kwarg name
- find_precedents propagates max_precedents/limit to the backend instead of
being silently capped at 10
- Serialize add_to_graph batches under a module lock to prevent concurrent
double-counting; skip nameless entities instead of creating repr()-junk nodes
- aadd() runs CPU-bound serialization in a thread executor
- Mirror crewai args_schema serialize/restore in the conftest stub and add
serialization regression tests (crewai: 92 tests)
* fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962)
- _eval_rule now coerces rule values type-aware: bool("false") was truthy, so
'enabled == false' reported a violation for enabled=false, and string datums
like "0.90" were compared lexicographically instead of numerically
- _trace_causal_chain no longer raises AttributeError (which escaped _run) when
the decision context lacks knowledge_graph; returns honest error JSON
- SemanticaKnowledgeSource storage failures log an actionable ERROR; without a
configured crew embedder agents previously retrieved nothing silently
- add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a
process-global one: independent graphs no longer serialize each other and
re-entrant extractor callbacks cannot deadlock
- entity/relation confidence=None normalizes to 1.0 instead of failing the
whole extraction with float(None)
- add subprocess integration test against real crewai covering Crew-level
serialization round-trip and checkpoint restore (stub tests cannot see it)
- docs: embedder requirement for SemanticaKnowledgeSource; resume contract note
* fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962)
Re-verification against real crewai showed the embedder-missing failure raises
ValueError even though storage IS wired, so the old except-ValueError branch
mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure.
Distinguish by storage presence instead of exception type: storage is None ->
DEBUG keep-in-memory (legitimate standalone use); storage wired but save()
raises -> actionable ERROR. Add regression test mirroring real crewai's
ValueError-on-missing-embedder behavior.
* fix(crewai): expose run()/arun() entry points in degraded mode (#962)
The public crewai contract is run()/arun(); without crewai installed they were
missing (only the private _run existed), so the documented 'usable without
crewai' path raised AttributeError at the entry point. Define them in degraded
mode only, leaving crewai's BaseTool implementations untouched when present.
Extend the degradation subprocess test to exercise run() and arun().
* fix(crewai): standardize query shape, field-name rules, and restore-state flag
- _query_graph: id/type matches now return the same schema as content
matches (id/type/label/content/score) instead of a bare list
- _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys
(e.g. "risk-score >= 0.9") are addressable in policy rules
- add had_live_state/reconstructed_state so checkpoint-restored tools
and knowledge sources signal that their live graph/context was lost
and an empty one reconstructed; knowledge source no longer hides the
loss by eagerly rebuilding its graph inside __init__ (pydantic calls
__init__ during model_validate)
* fix(crewai): address Qodo review — confidence errors, string trim, holistic availability
- record_decision: stop calling float() in _run, so malformed confidence
values surface as JSON errors (via _record_decision's handling) instead
of crashing the tool
- _coerce_value: return the stripped string for non-numeric literals so
whitespace-padded decision_data fields match policy rules
- centralize crewai availability in _availability.py so the exported
CREWAI_AVAILABLE flag is holistic across tools and knowledge source
(previously each module probed crewai independently and the package
flag came from decision_tool only)
* ci: regenerate requirements-ci.txt for the crewai extra
The crewai extra in pyproject.toml brings in crewai, crewai-tools and
transitive deps (chromadb, lancedb, ...). Recompile with
uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes.
* ci: keep crewai out of the locked CI dependency set
crewai (all versions) hard-requires chromadb~=1.1.0, which carries a
pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c)
with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in
the 'all' extra failed pip-audit and the safety check on requirements-ci.txt.
- drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is
unchanged and still installs crewai)
- stop listing crewai-tools in the extra: the integration only uses crewai core
(BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps
- regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0
vulnerabilities, staleness check matches
* docs(crewai): document crewai extra scope and chromadb CVE-2026-45829
- CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not
part of the 'all' bundle, with the chromadb CVE-2026-45829 reason
- integrations/crewai/README.md: add a security warning that installing
the extra pulls chromadb~=1.1.0, which is affected by the unpatched
pre-auth code-injection CVE-2026-45829
---------
* refactor(export): centralize graph-payload key normalization
Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.
Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.
Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.
Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.
CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.
* fix(export): reject non-mapping input to the YAML exporters
export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.
Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.
Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.
* fix(export): address YAML exporter review findings
- semantica/export/yaml_exporter.py — import Sequence from typing
instead of collections.abc. `Sequence[str]` in _require_mapping's
annotation is evaluated at function-definition time; collections.abc.Sequence
only became subscriptable in Python 3.9, so on the 3.8 this project
declares support for, importing this module raised TypeError.
typing.Sequence has supported subscripting since 3.5.3. Mapping stays
imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
and read exported YAML through a context manager instead of an
unclosed yaml.safe_load(open(...)).
* fix(export): reject YAML export payloads with no recognized key
Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.
- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
collections through normalize_graph_payload(), which raises rather than
returning empty collections for an unrecognized mapping. Adopting the
shared resolver rather than repeating the check locally also brings the
'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
from this library's own graph type to YAML, used in
examples/capability_gap_context_graphs_example.py — exports its records
instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
defaulted lookups and had the same defect; it goes through the resolver
too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
its own key set. Schemas are a separate vocabulary with no aliasing, so
_require_recognized_keys lives in this module rather than in the shared
graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
An export_json envelope carries one, so accepting it would readmit the
case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
output directory, so a rejected export leaves nothing behind.
The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.
Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.
Closes#953.
* fix(export): reject payloads whose records resolve to nothing
Addresses the Qodo findings on #958.
Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.
- utils/helpers.py — split the check in two. _require_recognized_keys keeps
the presence rule; _require_nothing_dropped runs after resolution and
refuses a payload that resolved to nothing while an unread key still holds
records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
always carries a populated 'statistics' dict, and an empty graph must stay
exportable, so 'metadata', 'statistics' and 'count' are named as context
rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
both checks through the shared helpers rather than its own copy, so the
two vocabularies cannot drift apart in what counts as a silent-empty
export.
Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.
- export/yaml_exporter.py — the serialization span now says it serialized,
not that it exported, and export() opens its own span around the
filesystem work that stops as failed on error. Nothing reports a completed
export until the bytes are on disk.
Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.
Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.
* fix(export): validate collection values and make Neo4j mappings strict
Two gaps at the boundary the shared normalizer is supposed to own.
_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.
Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.
Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.
* fix(export): close YAML schema and record validation gaps
Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived. Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.
Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects. Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary. Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.
Tests: 101 tests pass across
tests/utils/test_normalize_graph_payload.py
tests/export/test_yaml_exporter_key_recognition.py
tests/export/test_yaml_exporter_input_validation.py
tests/export/test_neo4j_csv_exporter.py
* fix(export): close exception-type and record-shape gaps in normalize_graph_payload
LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).
Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).
normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.
Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.
* docs(changelog): add entry for #958 YAML export input hardening
Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes#956, #952, #953.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Explorer was firing temporal requests before the graph even loaded.
When the backend is down, /api/graph/nodes fails but the temporal
bounds and snapshot effects didn't care , they fired anyway, off in
their own corner, ignoring whether the graph actually came up. Every
page load with no backend meant three failed requests instead of one,
and a scrubber that had nothing to scrub.
Added two small predicate functions and gated the temporal effects on
them. Basically: don't ask for time-based data until you know the
graph itself loaded. An empty graph still counts as loaded, so that
case isn't broken.
Confirmed with the backend down, before and after: three failing
requests down to one.
Fixes#982.
Fix DBIngestor calls in load_from_database , it was never actually reaching the db.
execute_query/export_table need the connection string as their first arg,
but we were only passing it to the constructor's config dict, which
those methods don't read. Every call blew up with a TypeError before
connecting.
Also split the ImportError/OSError handling , they were caught together
so a real connection failure got reported as "module not available",
which sent people looking in the wrong place. OSError now surfaces as
an actual failure with the original exception chained via `from e`.
Fixes#973.
* feat(context): add retraction and purge to ContextGraph
ContextGraph had 56 public methods and none that removed anything: the only
option was clear(), which discards the whole graph. Removing one entity meant
exporting to a dict, filtering by hand and rebuilding, losing provenance.
Add two operations with deliberately different contracts.
retract_node/retract_edge close the entity's validity window. The entity stops
being active going forward, but state_at() before the retraction still returns
it, so decisions recorded against it remain explainable. This reuses the
valid_from/valid_until machinery already present rather than adding a new
subsystem.
purge_node/purge_edge remove the entity outright, from history as well as from
the active view, leaving a tombstone that records that a purge happened and why
but never the purged content. Scope is this graph only; copies in AgentMemory
or a bound vector store are not reached, so it is one step of an erasure
workflow rather than the whole of it.
Both record themselves through the existing mutation_callback path.
MutationRecord already documented REMOVE_NODE/REMOVE_EDGE in its operation
vocabulary, so retraction emits UPDATE_NODE and purge emits REMOVE_NODE with no
changes required to change_management.
Incident-edge lookup scans self.edges rather than _adjacency, which is keyed by
source only and would otherwise leave inbound edges pointing at a removed node.
Purge updates edges, edge_type_index and _adjacency together so the indexes
cannot drift, and clear() now resets the retraction and tombstone records.
* fix(context): address review findings on retraction and purge
* fix(context): close every duplicate when retracting/purging by edge_id
edge_id is content-derived and not yet guaranteed unique (#922, fix
pending in #926): two identical add_edge() calls produce two edge
objects sharing one id. retract_edge()/purge_edge() resolved "the
edge" via the first matching object only, so a duplicate was silently
left untouched (still live, still active) while the call returned
True and recorded a tombstone/retraction claiming it was fully
handled. Repeat purge_edge() calls also silently overwrote the
tombstone's reason/purged_at on each partial attempt instead of
no-op'ing once nothing remained to purge.
retract_node()'s cascade had the same root cause from the other
direction: it checked the live _retractions dict mid-loop, so the
first duplicate's just-written record made the second look already
handled and it was skipped outright, left permanently active.
retract_edge()/purge_edge() now act on every edge matching the id
under a single record; the cascade's dedup check is snapshotted
before the loop starts so within-call duplicates are still closed
rather than skipped.
Adds TestDuplicateEdgeId (5 tests) reproducing all three paths.
* docs(changelog): document retraction/purge feature
Adds an Unreleased/Added entry for #955/#957 covering retract_node,
retract_edge, purge_node, purge_edge and the get/list accessors, plus
the duplicate-edge_id fix caught and applied during review.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
CONSTRUCT_QUERY_RE skipped comments with a bare \#[^\n]*, whose trailing *
backtracks. For '# CONSTRUCT ...\nSELECT ...' the engine gave back everything
after the '#', so the CONSTRUCT inside the comment satisfied the query-form
keyword and a SELECT/ASK was reported as a CONSTRUCT.
All four SPARQL backends delegate to this regex, so such a query took the
CONSTRUCT branch of execute_sparql, which sends Accept: text/turtle and parses
the body as Turtle — failing with a misleading 'Failed to parse CONSTRUCT
response as Turtle'.
Require a comment to reach a line terminator. Both LF and CR are accepted
because the SPARQL grammar ends a comment at either; matching only LF would
regress CR-terminated comments into false negatives.
Add regression tests covering both directions across all four backends.>
* fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#922)
* docs(changelog): document add_edge dedupe fix
Adds an Unreleased/Fixed entry for #922/#926 so the ContextGraph
edge-dedupe bug and its fix are recorded per Keep a Changelog format.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Qodo finding: embed generate wrote scalar dim_* columns, but embed index only
detects embeddings when a column's values are list/np.ndarray. This broke the
generate→index pipeline with 'No vector column found'.
Fix: write a single 'embedding' column where each value is a list[float],
matching what embed index's isinstance(df[c].iloc[0], (list, np.ndarray))
check expects. Row indices serve as ids (embed index will pass ids=None
to create_index, which is acceptable — vectors index correctly regardless).
Also addressed from Qodo review:
- .parquet suffix check is now case-insensitive (.lower())
- pandas already a core dependency (bot was wrong)
- pyarrow dependency remains added
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes
Two Explorer API endpoints fail on every install.
/api/enrich/extract imported extract_entities and extract_relations from
semantic_extract.methods, where neither name is defined — that module ships
only the per-strategy variants (extract_entities_ml, extract_relations_regex,
...), and nothing re-exports a plain facade. The resulting ImportError was
caught and reported as "semantic_extract module not available. Ensure spacy
and transformers are installed.", so a wiring bug looked like a missing
dependency. The route now calls NamedEntityRecognizer and RelationExtractor
directly, the classes the README documents, and feeds the extracted entities
into relation extraction rather than re-deriving them. The 503 branch stays
for a genuinely absent module.
Every /api/decisions* route returned 500 once the graph held a decision:
record_decision() stores timestamp as datetime.now().timestamp(), a float,
while DecisionResponse types the field as str, so pydantic rejected the value
the library itself wrote. A before-mode field validator on DecisionResponse
normalizes float, int and datetime inputs to ISO-8601, covering every route
that builds the model instead of only the list endpoint.
The existing tests missed both: test_extract accepted 503 as a pass, and the
decision fixtures are hand-built nodes carrying no timestamp at all. Both are
tightened, and a TestRecordedDecisions class exercises the routes against
decisions created through record_decision().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(semantic_extract): cache spaCy models instead of loading one per call
extract_entities_ml(), extract_relations_similarity() and
extract_relations_dependency() called spacy.load() on every invocation, so the
model was re-read from disk and re-initialized per call. On a short sentence
that is ~120 ms of loading around ~2 ms of work, and successive calls never got
cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the
pipeline ner_extract step and POST /api/enrich/extract, and process_batch()
multiplies it by the number of documents.
The module already had a cached loader for one code path — get_nlp_model() and
its _nlp_cache global — but the extraction functions bypassed it.
Adds load_spacy_model(), a process-level cache keyed by model name behind a
lock so concurrent callers do not each start a load, and routes the five call
sites through it. Errors are left uncached and propagate unchanged, so the
existing OSError fallbacks to pattern extraction still fire. get_nlp_model()
keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for
similarity work, so its model is not interchangeable with the NER one.
Cache entries record the spacy module object they came from. Several tests
patch methods.spacy with a mock and assert on load calls; without that guard a
name-keyed cache would hand a previous test's mock to a later one.
Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm:
extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(explorer): harden extraction and timestamp handling
* fix(explorer): catch OverflowError/OSError in decision timestamp validator
DecisionResponse._normalize_timestamp only guarded against NaN/inf via
math.isfinite(), but datetime.fromtimestamp() raises OverflowError or
OSError for finite epoch values outside the platform's representable
range (e.g. milliseconds stored where seconds were expected). Those
exceptions escaped the pydantic validator unhandled, reintroducing an
unhandled 500 on /api/decisions* for exactly the bug class this PR
closes. Also exclude bool from the numeric branch, since bool is an
int subclass and was being silently coerced to epoch 0/1.
* docs: add changelog entry for PR #886 (explorer extract/decisions fixes)
Documents the extraction 503, decisions timestamp 500, and folded-in
spaCy caching fixes, plus the review-round hardening from Sameer6305
and the timestamp overflow/bool fix from this follow-up commit.
---------
Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Fixes#994:
1. Prevent self-recursion in methods.py: generate_embeddings, embed_text,
calculate_similarity, and pool_embeddings all registered themselves as
custom methods, causing infinite self-calls when dispatch invoked them
without explicitly passing method parameter.
Fix: check custom_method is not the function itself before recursing.
2. Fix embed generate --output corrupt output: the CLI wrote
json.dumps(result, default=str) which produced plaintext repr of numpy
arrays (e.g. '[1.49e-01 4.85e-02 ...]') instead of proper Parquet.
Fix: detect .parquet extension (case-insensitive), convert numpy array
to pandas DataFrame with dim_* columns and id index, use to_parquet().
Non-parquet extensions fall back to JSON with clear ImportError message.
3. Add pyarrow>=14.0.0 to core dependencies (previously only in
ingest-parquet/ingest-arrow optional extras). The documented quick-start
flow of embed generate --output ... requires pyarrow out of the box.
(Note: pandas>=1.3.0 is already a core dependency; pyarrow is the
missing piece.)
Note: .github/workflows/* files are excluded from this PR as they require
a token with workflow scope. Upstream workflows are unchanged.
The pin was 5595ccaf..., but upstream has since moved the v4 tag to
ff2f1c62.... The Verify Action Pins workflow flags this drift on every
PR that touches any workflow file, regardless of whether that PR
changed codeql.yml or defender-for-devops.yml.
Verified the new SHA against the GitHub API directly (not just the CI
error text) and confirmed .github/scripts/verify-action-pins.sh passes
clean locally (40/40 action references OK, exit 0).
* ci: pin Python dependencies in requirements-ci.txt for reproducible CI
Adds a committed lockfile pinning all transitive dependencies at exact
versions (uv pip compile, Python 3.11, all extras — 1581 lines), the
Python equivalent of explorer/package-lock.json + npm ci.
- CI installs from requirements-ci.txt before building the wheel
- CI verifies the lockfile is byte-identical to a fresh compile (fails
on staleness after pyproject.toml changes)
- CONTRIBUTING documents the regeneration command
Closes#938
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: address Qodo review — security scans use pinned deps, exclude gpu extras
- security-scan.yml installs from requirements-ci.txt instead of
"./[llm-litellm]" so Safety scans the exact CI/release dependency tree
- security.yml runs pip-audit -r requirements-ci.txt for the same parity
- lockfile regenerated with --extra all (the cross-platform set) instead
of --all-extras, which pulled faiss-gpu/cupy from the Linux-only gpu
extra and co-installed faiss-cpu + faiss-gpu in CI
- uv pinned to 0.12.1 (the version that generated the lockfile) in CI and
CONTRIBUTING so regeneration is deterministic
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: make lockfile staleness check immune to upstream releases
The previous check re-resolved pyproject.toml without constraints, so any
upstream package release (e.g. boto3 1.43.69 -> 1.43.70) failed CI even
when nothing in the repo changed — exactly the time-dependent drift Qodo
flagged. The check now re-resolves with requirements-ci.txt as a
constraint and compares only version lines, so it detects intentional
pyproject.toml changes but ignores upstream releases. CONTRIBUTING
updated to match.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: fix security workflows — install pip-audit; order tooling after pinned deps
Security workflow: the pip-audit install step was lost in the rebase
conflict merge — pip-audit was invoked but never installed (exit 127).
Security-scan workflow: installing safety first let the pinned
requirements-ci.txt overwrite its transitive deps (rich), breaking the
safety CLI at runtime (RuntimeError: Type not yet supported). Tooling is
now installed AFTER the pinned set.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): address review — hashes, build isolation, release builds, docs (4/4)
ZohaibHassan16's review flagged 4 supply-chain gaps; all addressed:
1. **Release builds now use the lockfile**: release.yml installs
requirements-ci.txt and runs `python -m build --no-isolation` so the
sdist/wheel is built against the exact tested dependency set.
2. **Build isolation pinned**: [build-system].requires is now
setuptools==84.0.0 + wheel==0.48.0 (exact pins, no ranges).
3. **Hashes**: requirements-ci.txt regenerated with --generate-hashes
(5,708 sha256 hashes, verified against PyPI). Staleness check updated
to strip the `\` line continuations hashes introduce.
4. **CONTRIBUTING.md documents the separate environment**: hashes,
never-install-into-dev note, build-system pins, --no-isolation release
builds.
Validated: stale-check diff clean, hash spot-check matches PyPI.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): apply --no-isolation to CI build + align benchmark to Python 3.11
Follow-up to ZohaibHassan16's second review round:
1. ci.yml was still running `python -m build` with build isolation
(unpinned setuptools/wheel from PyPI) — now `python -m build
--no-isolation` against the pinned deps, matching release.yml.
2. benchmark.yml was on Python 3.12 while the lockfile is compiled for
3.11 — aligned to 3.11 so every workflow runs the same environment.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): install pinned wheel before --no-isolation build
python -m build --no-isolation failed with 'Missing dependencies:
wheel==0.48.0' because wheel is build-time only — uv's lockfile
excludes it, so installing requirements-ci.txt alone left the build
env without it. Both ci.yml and release.yml now install wheel==0.48.0
(the same pin [build-system] declares) before building. Validated
locally: wheel builds clean with --no-isolation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(context): honor explicit causal edges in decision tracing
trace_decision_causality() inferred causes purely from shared NER entities
plus timestamp ordering, so relationships recorded through
add_causal_relationship() never affected the trace. When entity extraction
returned nothing, trace_decision_chain() came back empty even though an
explicit CAUSED edge was stored in the graph.
Traverse the explicit CAUSED/INFLUENCED/PRECEDENT_FOR edges first, since
they are the ground truth the caller recorded, and keep the entity and
timestamp inference as an additive fallback for pairs with no explicit
link. Edges whose source has no decision record (for example a graph
restored via from_dict) are skipped so a stale edge cannot abort the trace.
analyze_decision_influence() now reports explicitly linked decisions as
direct influence rather than surfacing them only as indirect, and no
longer lists the same decision under both direct and indirect.
Closes#975
* fix(context): address review feedback on causal edge tracing
Follow-up to the explicit causal edge fix, covering the issues raised in
review.
A stored edge weight of 0.0 was coerced to the 1.0 default by a truthiness
check, inflating confidence_decay in the causal chain report. add_edge() is
public and can create causal edges with any weight, so use an explicit None
check instead.
Explicit causes were collected into a dict keyed by source_id, so multiple
causal edges between the same pair of decisions overwrote each other and
only the last was traced. Collect every edge instead, keeping a separate set
of source ids for the entity fallback exclusion.
Cycle detection used a single traversal-wide visited set, so a decision
reached through one branch became unreachable through another and branching
graphs silently lost valid chains. Detect cycles per path instead; max_depth
still bounds the traversal.
Build a reverse index of causal edges once per call rather than scanning the
edge list at every visited node, and use edge_type_index in the influence
analysis. The three causal edge types are now a shared constant.
Adds regression tests for zero weights, parallel edges, branching graphs and
cycle termination.
* fix(context): bound causal trace and report truncation
Per-path cycle detection keeps branching graphs correct but makes the
traversal combinatorial in max_depth: on a densely connected graph the
number of distinct causal paths grows by roughly the branching factor per
level, so a raised max_depth could return hundreds of thousands of chain
reports and take seconds of CPU.
Add a max_chains bound, defaulting to 10000. Rather than dropping chains
silently, which is the exact failure this fix set out to eliminate, the
traversal stops at the bound and appends a {"truncated": True, ...} marker
so callers can always tell the trace is incomplete. A warning is logged with
the same detail. Pass max_chains=None for the previous unbounded behaviour.
Graphs that fit within the bound are unaffected.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): lock the repo host DNS resolve cache against concurrent mutation
_REPO_HOST_RESOLVE_CACHE is a module level OrderedDict shared by every
RepoIngestor instance and thread. _resolve_repo_host_ips and
_prune_repo_host_resolve_cache read, wrote, and iterated it with no lock,
so concurrent ingest_repository() calls (e.g. from a thread pool) could
mutate the dict while another thread was iterating it during pruning.
This reliably raised RuntimeError: OrderedDict mutated during iteration
under ordinary concurrent usage, not just adversarial input.
Reproduced with 32 threads hammering _resolve_repo_host_ips with a low
TTL and small cache cap so pruning and eviction happen on nearly every
call; the crash showed up within the first few hundred iterations on
every run before the fix and did not reproduce at all after it.
Fix adds a threading.Lock guarding every read, write, and prune of the
cache. The blocking socket.getaddrinfo call stays outside the lock so a
slow DNS lookup for one host cannot stall cache access for other hosts.
Added a regression test, TestRepoHostResolveCacheThreadSafety, that
drives 32 threads through _resolve_repo_host_ips with a short TTL and
small cache cap and asserts no exception is raised.
Full test suite: 4088 passed, 332 failed, 140 errors both before and
after this change (same counts on main), all from missing optional
dependencies in this local environment (snowflake, sqlite-vec, spaCy
models, faiss/torch version mismatches), not from this fix. The ingest
and SSRF focused test files pass cleanly: 106 passed, 0 failed.
* test(ingest): fail fast on the first hung thread in the resolve-cache race test
join(timeout=30) alone doesn't fail the test if a worker hangs -- it
just returns after the timeout with the thread still running, and the
test falls through to the errors check, which trivially passes since
a hung thread never got far enough to append one. A future deadlock
could slip past this test looking green.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s instead. Worker threads are daemon=True so a genuine hang can't
also block the test process from exiting.
Verified the assertion is load-bearing, not cosmetic: temporarily
injected an artificial 9999s sleep into the first worker in a
throwaway copy of the test and confirmed the test now fails in ~31s
with a clear message, instead of the ~16 minutes a mass hang would
otherwise cost. That copy was never committed.
Addresses the review comment on #979 from ZohaibHassan16 and Qodo's
automated review.
* test(ingest): fail fast on the first hung thread, for real this time
The previous commit (f94e3b38) claimed to check is_alive() right after
each individual join, but a git staging mistake meant it actually
committed the old batched version instead (checking all 32 threads
only after the whole join loop finished) -- ZohaibHassan16 caught this
by timing it directly, 5 hanging threads took ~5x longer than 1
hanging thread, which the per-thread version would not do.
This commit was built by resetting to the current branch tip, verifying
byte-for-byte against a separately saved copy of the intended fix, and
confirming the actual committed git object (not just `git diff`) has
the inline check before pushing anything.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s regardless of how many threads hang.
---------
* fix(explorer): show a retryable error when the graph fails to load
The dependency-pre-bundle overlay had no failure path: on a fetch
error it kept rendering the last progress frame forever with no
retry. Route isError/error out of the load query, surface a real
error card with the underlying message, and let retry re-fetch
without a full page reload.
* fix(explorer): reflect real backend connectivity on the landing page
The status dot and 'System Online' text were static, so a dead
backend still looked healthy. Track checking/online/offline explicitly
and drive both off the same state so they can't disagree.
* feat(explorer): let search results be dismissed, round relevance scores
The results strip had no close affordance and stayed pinned until the
next search. Add a header row with a dismiss button, and round scores
to whole numbers instead of showing three decimals of a raw relevance
value nobody can act on.
* feat(explorer): add typeahead suggestions to graph search
Typing in the search box now debounces a query against the existing
search endpoint and shows a combobox dropdown, with arrow-key
navigation, Enter/click to jump straight to a node, and Escape to
dismiss. Previously nothing happened until the full form was
submitted.
* fix(explorer): abort stale typeahead requests and clear suggestions on error
Clearing the search box while a suggestion fetch was in flight never
aborted it, so a late response could reopen the dropdown with results
for a query that was no longer typed. A non-OK response also left
whatever suggestions were already on screen untouched instead of
clearing them. Abort on every effect cleanup (not just unmount) and
clear suggestions on any non-abort failure.
* docs(changelog): add entry for Explorer backend failure states fix
Documents the (#980, closes#977) fix in the Unreleased/Fixed section.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* test(semantic_extract): skip openai-dependent tests when the SDK is absent, assert logs not stdout
* test(semantic_extract): pass logger name to assertLogs to match suite convention
All 11 existing assertLogs call sites in the suite pass a logger name
string rather than a Logger instance; tests/reasoning/test_reasoner.py
uses this exact .logger.name form. Behaviour is unchanged.
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
* feat(seed): add allow_private_ips opt-in for trusted internal API sources (Closes#943)
SeedDataManager.load_from_api now delegates to the shared SSRF guard
(semantica/ingest/ssrf.py, added in #906) instead of raw requests.get,
gaining redirect validation and bounded DNS resolution for free.
New config option allow_private_ips (parsed via the shared parse_bool
helper) lets trusted internal deployments load from private APIs while
the secure default (block private/loopback/link-local) is unchanged.
Tests updated to mock request_with_ssrf_guard; new tests cover the
block-by-default behavior and the opt-in flag reaching the guard.
19/19 green in test_seed_manager.py, 25/25 across both seed suites.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ssrf): strip sensitive headers on cross-host redirects (Qodo finding)
request_with_ssrf_guard reused the caller's headers on every redirect hop,
so an Authorization bearer token from load_from_api could leak to a
different redirect target host. Now strips Authorization and
Proxy-Authorization when the redirect origin (netloc) changes, while
keeping them for same-host hops (matching requests semantics).
2 new tests: cross-host redirect drops the credential; same-host keeps it.
37/37 green in test_ssrf_protection.py. load_from_api docstring now also
documents cloud-metadata blocking and per-hop redirect validation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ssrf): strip credentials on https->http downgrade redirects (review feedback)
_should_strip_auth now mirrors requests' should_strip_auth semantics:
strip on hostname change, port change, or scheme downgrade; keep the
credential only for the safe http->https upgrade on default ports.
Previously only netloc was compared, so an https->http redirect on the
same host replayed the Authorization header in cleartext.
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
* security: apply SSRF guard to feed ingestion requests
FeedIngestor and FeedMonitor fetched feed and website URLs with plain
requests.get/head calls, bypassing the SSRF validation already used by
web_ingestor.py and api_ingestor.py. This allowed feed URLs pointing at
loopback, link-local, or other private network addresses to be fetched
directly.
Route all outbound requests in feed_ingestor.py through
request_with_ssrf_guard, gated by the same allow_private_ips config
option the other ingestors expose.
* test: mock the correct request boundary in test_discover_feeds_empty
The test still patched requests.get after discover_feeds() moved to
request_with_ssrf_guard(), which calls requests.request and performs
real DNS resolution. That left the test hitting live network/DNS.
* docs(changelog): document FeedIngestor SSRF guard fix (#928, closes#927)
Records the SSRF guard applied to all 5 feed-ingestion request sites,
the Qodo-flagged test-mock fix, independent PoC verification, and the
carried-over exception-swallowing behavior in discover_feeds().
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(vector_store): make VectorManager methods work on persistent backends (#855)
maintain_store() and collect_statistics() reached into VectorStore
internals (.vectors/.metadata), which only exist for the inmemory
backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus,
...) crashed with AttributeError.
Add a public backend-agnostic VectorStore.count() accessor following
the get_vector()/get_metadata() precedent (#843) and the
NotImplementedError-on-unsupported-capability precedent of
_filter_by_metadata() (#848): inmemory counts its dict, persistent
backends delegate to count() when available, and raise
NotImplementedError otherwise. VectorManager methods now go through
count(); maintain_store() keeps the exact inmemory semantics (separate
vector/metadata dict counts) and reports a 1:1 count for persistent
backends, where metadata is stored alongside each vector.
Tests: 10 hermetic unit tests covering inmemory, delegation and the
NotImplementedError path. Core vector_store suite: 40 passed.
* fix(vector_store): raise NotImplementedError when count() unavailable
Address Qodo review findings on #914:
- Persistent backend with no wrapped store no longer silently returns 0
(which masked a missing initialization as an empty, healthy store);
it now raises NotImplementedError like get_vector()/get_metadata().
- A mis-shaped adapter exposing a non-callable 'count' attribute now
surfaces a clean NotImplementedError instead of a TypeError, via a
getattr + callable() capability check.
Adds regression tests for both cases.
* fix(vector_store): implement count() on FAISS/SQLite/PgVector backends (#914)
- FAISSStore.count(): returns len(index.vector_ids); 0 when no index exists yet
- SQLiteVecStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- PgVectorStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- VectorStore.count(): fix misleading NotImplementedError message; now describes
how to add count() support to a backend adapter rather than claiming only the
inmemory backend can ever support counting
- VectorManager.maintain_store(): split inmemory and persistent paths:
* inmemory: independently reads len(vectors) and len(metadata) and compares
them as an integrity check (original semantics preserved)
* persistent: calls store.count(); returns metadata_count=None because
metadata is co-located with vectors in the backend and cannot be counted
independently; never manufactures metadata_count=vector_count as a vacuous
tautology (#914 Qodo review)
- Tests: rewrite test_vector_manager_persistent.py with 31 tests covering
dispatch logic, inmemory divergence detection, persistent metadata_count=None
invariant, FAISSStore/PgVectorStore via mocks, and SQLiteVecStore via real
in-memory SQLite (skipped when sqlite-vec absent)
* docs(changelog): document VectorManager persistent-backend count fix (#914, closes#855)
Records the VectorStore.count() accessor, the FAISS/SQLite/PgVector
implementations added during review, and the maintain_store()
metadata_count fix (no longer fabricates equality for persistent backends).
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
* fix(ingest): harden RepoIngestor against GitPython URL and option injection
Bump GitPython to >=3.1.58, allowlist clone kwargs, and validate repo URLs
before clone_from to close env-var exfiltration and option-injection paths.
* fix(ingest): accept scp-like SSH remotes in RepoIngestor URL validation
* fix(ingest): resolve repo hostnames to block SSRF via private IPs
* fix(ingest): map malformed repo URL parse errors to ValidationError
* fix(ingest): bound and prune repo host resolve cache
Cap the repository host DNS cache, prune expired entries on access, and evict the oldest entries so long-running processes cannot accumulate unbounded host lookups from user-supplied repo URLs.
* fix(ingest): cap host resolve cache and tighten env-var token checks
Bound the repo host DNS cache with pruning and oldest-entry eviction, and narrow URL env-var blocking to actual $VAR/${VAR} tokens so literal dollar signs are not rejected.
* fix(ingest): preserve repo path compatibility and NAT64 support
* docs(changelog): document RepoIngestor GitPython hardening (#905, closes#868)
Records the clone-surface hardening (GitPython floor, clone-option
allowlist, URL/SSRF validation), the two fixes made during review
(NAT64 false-positive, local-path regression), and a known residual
gap: the SSRF host check doesn't classify RFC 6598 CGNAT space
(100.64.0.0/10) as blocked since ipaddress.is_private doesn't cover it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@wifi-10-43-175-99.wifi.berkeley.edu>
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Bug fixes:
- _get_graph: call load_from_file (graph.load does not exist; SEMANTICA_KG_PATH was silently ignored and the graph started empty).
- query_decisions: read category from metadata.category (top-level category was always empty, so category filtering returned nothing).
- find_precedents / query: lower default similarity threshold to 0.05 so short CJK queries can match.
- extract_entities/extract_relations: return the entity text field (previously returned the spaCy type label as 'label' and dropped the actual text); expose model/language/method params so non-English (e.g. zh_core_web_sm) NER works.
New tools:
- query_graph: node detail / bidirectional neighbours (up to 5 hops, in-edges included) / keyword search.
- update_node: update node properties (e.g. action status todo/doing/done) and persist to SEMANTICA_KG_PATH.
- delete_node: soft-archive a node (status=archived) and persist.
Co-Authored-By: Claude <noreply@anthropic.com>
DecisionResponse.timestamp is typed str, but decision nodes store a float epoch. Coerce non-str timestamps so GET /api/decisions stops returning 422 Unprocessable Content.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a character-bigram overlap-coefficient fallback to _calculate_decision_content_similarity so CJK scenarios (no whitespace tokenization) can match recorded decisions; the previous whitespace Jaccard was always 0 for CJK.
Rebuild _decisions/_decision_index/_entity_index/_temporal_index from persisted decision nodes at the end of load_from_file, otherwise find_precedents_by_scenario and decision_count break after a reload since save_to_file does not serialize the internal decision indexes.
Co-Authored-By: Claude <noreply@anthropic.com>
gensim (core dependency) has no prebuilt cp314 wheel, and the slim base image lacks gcc to build from source, so 'pip install .[explorer]' fails on python:3.14-slim. Pin to python:3.13-slim (still satisfies requires-python>=3.8) until gensim ships a cp314 wheel.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(context): clarify get_node_property not-found contract (#877)
Add default= param to get_node_property and get_node_attributes so
callers can distinguish node-missing from property-missing using a
sentinel. Fix add_node_attribute calling mutation_callback outside
the lock. Tests added for all cases.
* fix(context): address Qodo review findings (#877)
* fix(context): wrap add_node_attribute mutation_callback in try/except (#877)
The PR claimed to move the callback back inside `with self._lock`, but
the diff only dropped a stray blank line -- the call stayed outside the
lock, unchanged. That's actually correct: self._lock is an RLock, and
_add_internal_node/_add_internal_edge deliberately release the lock
before invoking the callback too, so a slow/misbehaving callback never
holds up other threads. The real gap was that, unlike those two
siblings, this call site didn't catch exceptions from the callback.
Wrapped it the same way, with a regression test.
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(export): log DistanceExporter metric computation failures instead of swallowing them
The four private metric helpers in DistanceExporter (_betweenness,
_hop_distance, _weighted_distance, _semantic_similarity) each catch a bare
Exception and return None/{} with no signal. That makes an exported None
indistinguishable from a legitimate "no path exists" result, corrupting
downstream CSV/JSONL/DataFrame exports with no way to tell a real gap from a
swallowed error.
Log each caught exception at warning level with the offending source/target
before returning the existing sentinel. The exported row shape and values are
unchanged; only the observability of the failure changes.
Fixes#874
* fix(export): route DistanceExporter warnings through the semantica logger tree
get_logger(__name__) doubled the semantica. prefix (__name__ is already
semantica.export.distance_exporter), so the warnings this PR adds landed on
semantica.semantica.export.distance_exporter, a branch setup_logging() never
configures and does not reach the app's log handler. Also reworded the three
except-Exception log messages: they said "recording as no path", which
overclaims what a generic exception means.
Addresses review feedback from @KaifAhmad1 on #879.
* docs(changelog): add DistanceExporter logging fix entry
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract
_extract_from_text() defaulted ner_method, relation_method and
triplet_method to "llm" and ran relation extraction unconditionally,
contradicting the build() docstring ("ml"/"pattern"/False) and the
standalone extractor defaults. Any raw-text build() therefore required a
provider, an API key, and network access without saying so.
Defaults are now ml/pattern/pattern with extract_relations=False. LLM
extraction is unchanged and now opt-in via explicit kwargs.
Also documents relation_method and extract_triplets, which the docstring
never listed, and drops the stale "Default to LLM methods as per
requirement" comment.
Closes#930
* perf(kg): reuse extractors across texts instead of rebuilding per source
Addresses review feedback on #941. NERExtractor.__init__ loads its spaCy
model eagerly when the method includes "ml", so switching the default
from "llm" to "ml" made _extract_from_text() reload the model once per
source in a multi-document build.
Extractors are now cached per (kind, method) on the builder. Adds tests
asserting single construction across repeated texts, that distinct
methods still get distinct extractors, and that the default path runs
end to end without any provider call.
* fix(kg): keep fallback method lists working with the extractor cache
The extractor cache keyed directly on `method`, but all three extractors
accept a list for fallback ordering (e.g. ner_method=["pattern", "ml"]),
so a list argument raised TypeError: unhashable type: 'list' before
extraction started. Lists are now converted to tuples for the cache key
only; the extractor still receives the original value.
Also seeds _extraction_stats in __init__. It was previously created only
in build(), so calling _extract_from_text() directly — as the report's
repro does — raised an AttributeError that the broad except swallowed and
logged as "Entity extraction failed".
Adds coverage for list methods on all three extractors, cache reuse for
equal lists, and distinct entries for different orderings.
* fix(kg): forward extracted relations into triplet extraction
_extract_from_text() passed only entities= to extract_triplets(), so
TripletExtractor re-derived relations itself whenever relations is None,
using a method taken from triplet_method rather than relation_method.
That duplicated work and could yield triplets inconsistent with the
relations already extracted.
relations is now initialized to None, holds the extracted list when
extract_relations=True succeeds, and is forwarded to extract_triplets().
When extraction is disabled or fails, None is passed and
TripletExtractor's existing self-derivation is unchanged.
Folded in at maintainer request rather than tracked as #944.
* docs(changelog): note that #878 documented the LLM defaults before this landed
#878 merged while this was in review and resolved the same code/docstring
mismatch in the opposite direction. Records that #930's decision makes
the code the side that changes, and that #878's docstring formatting is
retained.
* docs(kg): document GraphBuilder public methods
* test(kg): skip module-level doctest to fix suite run
* docs(kg): restore GraphBuilder option documentation
* docs(kg): document default values for build() extraction options
extract_relations, extract_triplets, ner_method, relation_method, and
triplet_method all have concrete defaults in _extract_from_text(), but
the build() docstring only stated a default for extract, inconsistent
with CONTRIBUTING.md's docstring convention of noting parameter
defaults.
* docs: add changelog entry for GraphBuilder docstrings (#878, #876)
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(security): sanitize node_id in Content-Disposition to prevent header injection (CWE-113)
* fix(security): cap link prediction at 10k nodes with semaphore to prevent OOM DoS (CWE-770)
* fix(security): sanitize imported node IDs to prevent stored header injection chain (CWE-20)
* test(security): add self-contained PoC runner with real measured output
* test(security): add regression tests for header injection, DoS cap, import sanitization
* fix(security): comprehensive fix for header injection, DoS, and import ID sanitization
* fix: move semaphore to wrap entire data-load+scoring region, use node-specific edge queries (Qodo #2, #3)
* fix: sanitize edge source/target IDs to match sanitized node IDs (Qodo #4)
* fix: scope 999_999 check to predict_links function via AST (Qodo #1)
* fix: add explicit None guard to _sanitize_import_node_id
* fix(security): close import-sanitizer bypass, enforce link-prediction cap before the expensive scan
Follow-up to the fixes in this PR, found in review:
- export_import.py's "properties" in raw_node fast path stored the id
verbatim, completely skipping _sanitize_import_node_id() -- a node
payload of {"id": "<crlf>", "properties": {}} (the shape this app's
own /api/export produces) bypassed the VULN-3 fix entirely. That
branch now sanitizes id before storing.
- The link-prediction 10k-node cap checked `total` only after calling
session.get_nodes()/get_edges(), which normalize the graph's entire
matching set before applying `limit` -- so the DoS guard ran after
the expensive work it exists to prevent had already happened, on
every request regardless of graph size. Added
GraphSession.get_raw_counts(), an O(1) check against the raw
len(graph.nodes)/len(graph.edges), and moved the size check ahead of
the normalizing calls (also added an edge-count cap).
- 5 of the existing regression tests asserted that literal words like
"Set-Cookie"/"Content-Type" disappear from the sanitized value -- the
sanitizer strips \r\n\x00"\ , not letters, so those assertions failed
against this PR's own fix as submitted. Corrected to assert on the
actual security property (no \r/\n survives), and added end-to-end
tests that exercise the real /api/import -> /api/provenance/report
route chain so the properties-key bypass has regression coverage.
Full explorer suite: 241 passed. tests/test_security_regression_pr2.py: 30 passed.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(context): take the lock in ContextGraph.to_dict()
to_dict() iterated self.nodes.values() and self.edges without holding
self._lock, so a concurrent writer raised "RuntimeError: dictionary changed
size during iteration". It was the only reader on the class that did not take
the lock -- stats(), density(), find_nodes(), find_edges(), get_neighbors(),
get_nodes_by_label(), state_at() and save_to_file() all hold it.
Commit 1d1ae398 introduced the RLock and added 26 "with self._lock:" blocks;
to_dict already existed and was not among them. save_to_file is safe only
incidentally -- it holds the lock and builds its payload inline rather than
delegating to to_dict, so it never reaches the unguarded loops.
Beyond the RuntimeError, the unguarded body could also return a torn snapshot:
the statistics block reads len(self.nodes)/len(self.edges) after building the
node and edge lists, so a write landing in between yields counts that
contradict the payload they describe.
self._lock is an RLock, so this composes with the callers that already hold it
(build_from_conversation and build_from_documents both return self.to_dict()
from inside a locked block). Neither external caller -- agent_context's
_capture_checkpoint_state nor triplet_store's knowledge-graph conversion --
defines a lock of its own, so there is no ordering inversion.
Add tests/context/test_context_graph_thread_safety.py: a deterministic check
that to_dict() blocks while another thread holds _lock (no race window
needed), a reentrancy check, and three checks under concurrent writes covering
the RuntimeError, statistics/payload agreement, and duplicate node ids. Four
of the five fail against the unfixed method.
Closes#923
* test(context): make to_dict lock tests deterministic and hang-proof
Wait for the worker thread to actually start before asserting to_dict()
blocks on _lock, and run the reentrancy check in a joined worker so a
non-reentrant lock fails the test instead of hanging CI.
* test(context): assert worker threads actually stopped after timed joins
A join(timeout=...) on a daemon thread returns even if the thread is
still running, so a deadlock would leak a live thread into subsequent
tests instead of failing. Assert not is_alive() after each timed join.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* security(deps): bump fastapi floor to >=0.109.1 (PYSEC-2024-38)
The [explorer] extra declared fastapi>=0.100.0, which allows the
vulnerable 0.109.0 (PYSEC-2024-38, HTTP response splitting). Raise the
floor to 0.109.1, the patched release. One-line change, no functional
impact -- the 0.109.x API is stable and backward-compatible.
Fixes#869
* fix(deps): bump fastapi to >=0.109.2 and python-multipart to >=0.0.7 for PYSEC-2024-38
PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p) is a ReDoS in
python-multipart < 0.0.7: an attacker sends a crafted Content-Type header
that causes catastrophic backtracking in the multipart regex, stalling the
event loop and causing a DoS on any endpoint that parses form data.
The original PR bumped fastapi to >=0.109.1, but that version pins
starlette<0.36.0,>=0.35.0 and cannot install starlette 0.36.2+ (which
contains the fix via python-multipart>=0.0.7). FastAPI 0.109.2 is the
first version that pins starlette>=0.36.3 (verified against PyPI metadata).
Two changes are necessary:
1. fastapi>=0.109.1 -> fastapi>=0.109.2: ensures starlette>=0.36.3 is
installed as a transitive dependency, which in turn pulls the fixed
python-multipart>=0.0.7.
2. python-multipart>=0.0.6 -> python-multipart>=0.0.7: closes the direct
dependency path. python-multipart is listed explicitly in the explorer
extra, so without this floor a resolver could still install 0.0.6 and
leave the vulnerability present even with the fastapi bump.
The fix targets only the 'explorer' optional dependency group, which is
the only code surface where FastAPI and form-data parsing are used.
No functional API changes between 0.109.1 and 0.109.2; 239 Explorer tests
pass without modification.
* ci(security): gate pip-audit on explorer-extra dependency PRs, add changelog entry for PYSEC-2024-38
The Security workflow's pip-audit job ran weekly against a bare Python
env with none of Semantica's optional extras installed, and always
continue-on-error'd -- it would never have flagged the vulnerable
fastapi/python-multipart floors this PR fixes, or the first attempt at
the fix that left python-multipart>=0.0.6 in place. security-scan.yml's
Safety check has the same blind spot (only installs [llm-litellm]).
pip-audit now also runs on pull_request when pyproject.toml changes,
installs semantica[all] so it can actually see extras like [explorer],
and fails the build on findings for that trigger. Scheduled/dispatch
runs stay non-blocking pending a full pass over the [all] tree.
Also documents the fix (#871, closes#869) in CHANGELOG.md, including
the correction made during review after the original fastapi-only bump
turned out not to close the vulnerability.
* fix(deps): raise setuptools floor to >=83.0.0 (CVE-2026-59890), harden audit env
The new pull_request pip-audit gate (previous commit) caught this on its
first run: pip install -e ".[all]" resolved setuptools==79.0.1, vulnerable
to CVE-2026-59890 / GHSA-h35f-9h28-mq5c / PYSEC-2026-3447 (Unicode
normalization lets a MANIFEST.in exclude/prune pattern be bypassed on
macOS APFS/HFS+, leaking excluded files into a built sdist). Fixed in
setuptools 83.0.0.
[build-system] requires had the same too-permissive floor this whole PR
is about (setuptools>=61.0). Raised to >=83.0.0. Also upgrade pip/
setuptools explicitly in the Security workflow before running pip-audit,
since [build-system] requires only governs isolated build environments,
not the ambient one actions/setup-python provisions and pip-audit scans.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
semantica/mcp_server/__init__.py was fixed to stop hardcoding 0.4.0,
but the separate top-level mcp/ package (run via `python -m
mcp.server`, documented in mcp/__init__.py as a supported way to
configure Claude Desktop/Windsurf/etc. from a source checkout) still
hardcoded 0.4.0 in three places: mcp/__init__.py, mcp/server.py, and
mcp/resources/registry.py.
Reuses semantica.__version__ directly, matching the pattern just
adopted in semantica/mcp_server/__init__.py, so both implementations
stay in sync with the package version going forward.
The previous implementation used importlib.metadata.version('semantica') as
the primary version source with a PackageNotFoundError fallback to
semantica.__version__. This caused two of the three new regression tests to
fail in editable/development installs, where dist-info (egg-info) is written
at install time and is not automatically updated on subsequent version bumps.
In this repo, pyproject.toml declares version as a static field (not dynamic),
and semantica/__init__.py maintains __version__ in sync with it by convention.
semantica.__version__ is therefore the authoritative source of truth and is
always present whenever semantica.mcp_server is importable -- the importlib
.metadata indirection adds no value and can return a stale value.
Changes:
- semantica/mcp_server/__init__.py: replace the importlib.metadata try/except
block with a direct 'from semantica import __version__ as _SEMANTICA_VERSION'
- tests/test_mcp_server_version.py: rewrite tests to assert both MCP version
surfaces (SERVER_INFO['version'] and semantica://schema/info) against
semantica.__version__ as the single ground truth; add 0.4.0 regression
canaries and a cross-surface consistency assertion; remove the mirrored
importlib.metadata resolution that masked the staleness problem
The root-level mcp/ directory (a separate unpublished companion implementation
not included in the built package) is intentionally left unchanged -- it is
outside the scope of issue #863 which targets the semantica-mcp entry point.
- pinecone_store: call self.index.describe_index_stats() instead of the
nonexistent self.describe_index_stats(), and use a unit query vector
instead of an all-zero vector so filter_by_metadata() works on
cosine-metric indexes (the library's own default)
- pgvector_store: apply the existing lowercase true/false bool handling
to the list-filter branch too, and use the jsonb ?| operator so
list-valued metadata fields match on intersection instead of being
compared as a single JSON-text blob
- sqlite_vec_store: use json_each() with a json_type guard so list-valued
metadata fields match on intersection, mirroring the in-memory
backend's set-intersection semantics
- faiss_store: filter_by_metadata(limit=0) now returns [] instead of one
result
- milvus_store: reject NaN/Infinity filter values up front with a clear
ValidationError instead of building an invalid expression that gets
silently swallowed
- update the #848 FAISS NotImplementedError test to reflect that FAISS
now implements real filter_by_metadata() (this PR's whole point)
- add regression tests for each fix; sqlite tests run against the real
sqlite-vec extension
* fix(pipeline): resolve broken import and missing run() in PipelineWithProvenance
Fix two bugs in pipeline_provenance.py:
1. Wrong import path: `from .pipeline import Pipeline` fails because
`semantica/pipeline/pipeline.py` does not exist. Pipeline lives in
`pipeline_builder.py`. Fixed to `from .pipeline_builder import Pipeline`.
2. Pipeline dataclass has no run() method. PipelineWithProvenance.run()
now delegates to ExecutionEngine.execute_pipeline(), which is the
intended execution path for built pipelines.
Additional changes:
- Constructor now accepts a built Pipeline instance (breaking the previous
unusable API that tried to instantiate a dataclass with **config).
- Replace deprecated datetime.utcnow() with datetime.now(timezone.utc).
- Add test suite covering import, instantiation, execution, attribute
delegation, and provenance graceful degradation.
Fixes#858
* test: address Qodo review findings
- Remove redundant test_import_succeeds (module-level import already
guards against import regression at collection time).
- Fix test_provenance_disabled_when_import_fails to deterministically
simulate ImportError via sys.modules patch and assert provenance is
actually toggled off (runner.provenance is False).
* fix(pipeline): update provenance callers for Pipeline API
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Russell Jurney <russell.jurney@gmail.com>
CORSMiddleware doesn't cover WebSocket handshakes at all (Starlette's CORS
support only wraps HTTP), so under SEMANTICA_ALLOW_ANONYMOUS=true --
the mode docker-compose.dev.yml ships -- is_valid_api_key's anonymous
bypass accepted a /ws/graph-updates connection from any origin. Loopback
binding isn't a boundary against a browser: any page the operator has
open can still reach ws://localhost:8000/ws/graph-updates directly, and
ConnectionManager.broadcast sends every graph_mutation to every
connected socket with no per-connection scoping. Combined with
/api/import accepting multipart/form-data (a CORS-safelisted content
type that skips preflight), a hostile page could write to the graph
over REST and read the result back over the unauthenticated WebSocket
-- demonstrated end-to-end in the report with a real client.
Not affected: any deployment with SEMANTICA_API_KEY configured -- the
handshake already rejects without a valid key in that mode. This is an
anonymous-mode-only, development-configuration exposure.
Fix: check the handshake's Origin header against
app.state.explorer_settings['allowed_origins'], the same list
CORSMiddleware already enforces for HTTP, before the key check. A
missing Origin (native/CLI clients, which never set the header --
only browsers do) is still allowed through, since the browser is the
only threat this closes.
4 new tests in test_explorer_auth.py: hostile Origin rejected under
anonymous mode; hostile Origin rejected even with a correct key
(Origin is checked before the key, so a leaked key alone can't
hijack the socket); an allowlisted Origin still connects under
anonymous mode; a missing Origin still connects under anonymous mode
(native clients keep working). Full explorer suite: 226 passed.
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Qodo's re-review confirmed the multi-IP fallback fix but kept the proxy
finding open: logging-and-falling-back when a proxy applies still let
the DNS-pinning protection be silently skipped under proxy
configuration, rather than enforcing a clear policy either way.
Implemented Qodo's preferred option: proxies are now disabled outright
for this SSRF-sensitive fetcher via session.trust_env = False, so
HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars are never consulted in the
first place (a configured proxy would perform its own DNS resolution
of the target host outside this process's control, reopening the
DNS check-then-use race pinning exists to close). The adapter also
keeps a fail-closed backstop: if a proxy is somehow still configured
despite trust_env=False (e.g. set explicitly by future code), it now
raises a clear 502 instead of silently connecting through the proxy
unpinned.
_validate_fetch_url's destination classification (blocking private/
internal targets) is unaffected either way — it runs before any of
this and doesn't depend on proxy configuration.
4 new tests: trust_env is disabled on every pinned session; an
HTTP_PROXY env var pointed at an address that would fail if contacted
is confirmed genuinely unused (real local-server fetch still succeeds
directly); and the fail-closed backstop actually raises when a proxy
is forced onto the session. Full explorer + triplet_store suite: 572
passed.
Four findings from PR #916's automated review, all addressed:
- CodeQL (HIGH): the test HTTPS server's SSLContext allowed TLSv1/TLSv1.1
by not setting a minimum version. Added
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2.
- github-code-quality: unused `cryptography` local in
_make_self_signed_cert — importorskip's return value was never used.
- Qodo (reliability): _validate_fetch_url() only returned the first
validated IP, and _make_pinned_session() pinned to just that one
address, so a fetch would fail outright if the first-returned A/AAAA
record happened to be unreachable even though a later one would work.
_validate_fetch_url() now returns every validated IP (deduplicated,
in resolution order); _make_pinned_session() takes the full list and
falls back through each one via a custom Connection._new_conn
override, matching the fallback behavior a normal DNS-resolving
connection would already get for free. Verified with a real test:
pin to an unreachable loopback address followed by a real one, confirm
the fetch still succeeds by falling back; and a real test confirming
it still raises (rather than silently re-resolving the hostname) when
every pinned address is unreachable.
- Qodo (security): when an HTTP(S) proxy applies, the adapter falls back
to the unpinned path rather than pinning. This is a real, but
architecturally unavoidable, limitation from the client side: for a
forward proxy, the *proxy* performs its own DNS resolution of the
target host on the application's behalf, a resolution this process
has no visibility into or control over — there's no client-side pin
that closes that race. _validate_fetch_url's destination
classification still fully applies either way; only the secondary
DNS-pinning hardening doesn't extend through a proxy. Added an info
log when this fallback path is taken so it's observable rather than
silent, and expanded the code comment to make the reasoning explicit
for the next reader/reviewer rather than looking like an oversight.
Tests: 3 new tests in test_ontology_dns_pinning.py (multi-IP fallback
success, all-unreachable failure, deduplicated multi-record resolution).
Full explorer + triplet_store suite: 569 passed.
Two follow-up hardening items flagged as secondary/deferred during
GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236's fixes:
1. DNS check-then-use (TOCTOU) window in the ontology URL fetcher.
_validate_fetch_url() resolved and validated a hostname once, but
_fetch_url_sync() then let requests resolve the same hostname again
independently at connect time — a low-TTL or rebinding DNS answer
could differ between the two lookups, reopening the SSRF window the
validation exists to close.
_validate_fetch_url() now returns the validated IP, and a new
_make_pinned_session() builds a per-hop requests.Session whose
connection pool is pinned directly to that IP (bypassing DNS
resolution for the connection entirely), while explicitly restoring
the real hostname as the outgoing HTTP Host header and, for HTTPS,
the TLS SNI server_hostname/assert_hostname — so the connection
reaches the validated IP but still presents (and is verified
against) the real hostname's identity, keeping virtual hosting and
certificate validation correct.
Note: an earlier version of this fix set `_dns_host` post-construction
assuming it was decoupled from `host`, matching some other urllib3
releases; in the installed version (2.7.0), `host` is a property
that reads/writes `_dns_host` directly, so that approach silently
changed the Host header too. Verified with a real (non-mocked) local
HTTP server, a real local HTTPS server with a self-signed cert
(proving SNI/cert-hostname verification checks the real hostname,
not the pinned IP), and a negative control confirming a hostname/cert
mismatch is still correctly rejected — not silently bypassed.
2. Pre-wrapped object IRIs skipped full validation in
_format_object_for_sparql/_format_object_for_ntriples (Blazegraph,
RDF4J). A triplet object already wrapped in `<...>` only had its
inner content checked for a literal space or `>`, not run through
sparql_escaping.validate_uri() like the unwrapped-object branch —
flagged by automated review during GHSA-8vgg-8mr4-r236's fix. Both
branches now validate identically.
Tests: tests/explorer/test_ontology_dns_pinning.py (6 tests, including
2 real local-server end-to-end checks and 2 real-TLS checks with a
generated self-signed cert, gracefully skipped if `cryptography` isn't
installed); updated tests/explorer/test_ontology_ssrf.py for the new
per-hop session construction; 4 new tests in
tests/triplet_store/test_sparql_injection.py for the object-IRI fix.
Full explorer + triplet_store suite: 566 passed.
Two follow-up fixes to the initial ReDoS patch (CodeQL py/polynomial-redos
#1897), raised during code review:
--- Fix 1: _PREFIX_DECL regression — inline prologues and CRLF (#review-1) ---
The first ReDoS fix replaced the ambiguous trailing \s* with [ \t]*(?:\n|$),
but that introduced a behavioral regression:
* Inline prologues — PREFIX ex: <...> SELECT ... on a single line were no
longer stripped because the mandatory (?:\n|$) anchor never matched when
non-whitespace content followed the IRI on the same line.
* CRLF line endings — PREFIX ex: <...>\r\n failed because \r is not in
[ \t]* and the anchor expected a bare \n.
Root cause: the end-of-line anchor was unnecessary; the only thing needed
to eliminate backtracking ambiguity is ensuring the IRI body character class
and the trailing whitespace quantifier are disjoint.
Fix: change the IRI body from <[^>]*> to <[^>\r\n]*>, which:
- excludes CR and LF from the IRI match (semantically correct — SPARQL
IRIs cannot span line boundaries)
- makes [^>\r\n]* and the trailing [ \t]* have zero character overlap,
eliminating all backtracking ambiguity without any end-of-line anchor
No anchor is used, so both inline prologues and CRLF/LF endings work
naturally. ReDoS payloads (base< + !< x 10,000) still complete in <1 ms.
--- Fix 2: oversized-query length guard obscured error (#review-2) ---
The initial patch placed the _SPARQL_MAX_QUERY_LEN guard inside
_is_read_only_query(), which caused execute_sparql() to return the same
generic 'Only SELECT' error for both genuinely disallowed query types and
oversized inputs. Clients could not distinguish the two rejection reasons.
Fix: move the length check out of _is_read_only_query() and into
execute_sparql() as an explicit early gate, alongside the other resource
limits (_SPARQL_MAX_ROWS, _SPARQL_MAX_GRAPH_NODES). Oversized queries now
return a specific message naming the limit, the received length, and the
remediation step. _is_read_only_query() is documented to be length-agnostic.
_SPARQL_MAX_QUERY_LEN is relocated to the resource-limits block with the
other constants.
--- Tests added ---
tests/test_security_regression.py:
- test_inline_prefix_before_select_allowed (Fix 1 regression)
- test_crlf_line_endings_with_prefix (Fix 1 regression)
- test_crlf_multiple_prefixes_then_select (Fix 1 regression)
- test_inline_prefix_before_insert_still_blocked (Fix 1 security check)
- test_long_valid_query_not_rejected_by_is_read_only (Fix 2 separation)
tests/explorer/test_sparql_route.py:
- test_oversized_query_returns_distinct_length_error (Fix 2 error message)
- test_oversized_query_never_touches_the_graph (Fix 2 short-circuit)
- test_query_exactly_at_length_limit_is_accepted (Fix 2 boundary)
All 82 tests pass.
The _PREFIX_DECL pattern used \s* as a trailing quantifier after
<[^>]*>. On inputs that start with ase< but contain no closing >
(e.g. ase<!<<!<<!<...), the regex engine explores exponentially many
ways to split the match between [^>]* and \s*, causing polynomial
backtracking against user-controlled SPARQL query input.
Fix:
- Replace ^\s* / \s+ / \s* with ^[ \t]* / [ \t]+ / [ \t]*
so the leading/internal whitespace quantifiers only match horizontal
whitespace (no overlap with the <[^>]*> IRI part).
- Replace the ambiguous trailing \s* with [ \t]*(?:\n|$), which
matches only horizontal whitespace followed by a hard line boundary.
[^>]* and [ \t]* have disjoint character sets, eliminating the
backtracking ambiguity entirely.
- Add _SPARQL_MAX_QUERY_LEN = 10_000 guard at the top of
_is_read_only_query as defence-in-depth: rejects oversized input
before any regex work, bounding worst-case cost even if a future
pattern change reintroduces ambiguity.
Verified: ReDoS payload ase< + !< x 5000 completes in <1 ms.
Normal PREFIX/BASE stripping and read-only query detection unchanged.
Fixes: CodeQL py/polynomial-redos alert #1897
CWE: CWE-1333, CWE-730, CWE-400
* security: sanitize Cypher labels/relationship types/property keys (GHSA-482h-hw99-h62p)
Node labels and property keys passed to create_node/create_relationship
were interpolated directly into Cypher strings in the Neptune, Neo4j, and
FalkorDB graph stores. Property values are parameterized, but labels and
keys can't be bound as parameters, and nothing validated them, so a
document-derived entity type or property name could close the current
Cypher token early and append arbitrary statements (e.g. DETACH DELETE),
running with the application's database credentials.
- New shared semantica/graph_store/query_sanitize.py: sanitize_identifier()
generalizes age_store.py's existing _sanitize_label/_sanitize_rel_type
(the only backend that already validated this) into a helper the other
backends can import without an import cycle with graph_store.py/methods.py.
- Applied at every label/relationship-type/property-key interpolation site
in amazon_neptune.py, neo4j_store.py, falkordb_store.py, graph_store.py
(degree_centrality's own query builder), and methods.py
(update_relationship's own query builder) — create_node, create_nodes,
create_relationship, get_nodes, get_relationships, get_neighbors,
shortest_path, update_node, create_index, and all relationship-type
filters.
- depth/max_depth path-length parameters are also cast to int before
interpolation as defense-in-depth (they're already typed int, but
Python doesn't enforce that at runtime).
Added tests/graph_store/test_cypher_injection.py (12 tests covering the
sanitizer directly and reproducing the advisory's injection payload
against Neptune/Neo4j/FalkorDB create_node/create_relationship — asserts
the malicious query is never built or sent), plus regression tests for
graph_store.py's degree_centrality and methods.py's update_relationship.
Full graph_store test suite (224 tests) passes with no regressions.
* fix(graph-store): prevent depth-based Cypher injection
* test(graph-store): tighten injection regression assertions
* docs(changelog): add PR #910 (GHSA-482h Cypher injection) entry
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Two issues in the last round of commits:
1. explorer/auth.py added a new, opt-in APIKeyAuthMiddleware
(EXPLORER_API_KEY) and wired it into create_app(), but in doing so
removed the Depends(require_auth) dependency from every router and
deleted the /ws/graph-updates handshake check entirely. The new
middleware also fails OPEN (allows all requests) when its key is
unset, the opposite of require_auth's fail-closed design. Since
GHSA-j4mq-hprp-987v (the unauthenticated-Explorer-API advisory) is
already merged into main via require_auth, this would have reverted
a merged Critical fix the moment this branch merges. Removed
explorer/auth.py, restored the per-router dependencies and the
WebSocket auth check. Kept auth.py's one genuine improvement (adding
X-API-Key to the CORS allow_headers list) by folding it into the
existing CORS middleware config.
2. sparql.py's new _is_read_only_query() hardening (comment/PREFIX
stripping + forbidden-keyword scan) used `#[^\n]*` to strip SPARQL
comments, but a bare '#' also appears inside standard RDF namespace
IRIs (e.g. ".../1999/02/22-rdf-syntax-ns#") — the regex struck
everything after that '#' as a "comment", corrupting the query and
rejecting any legitimate SELECT using rdf:/rdfs:-style PREFIX
declarations. Confirmed by the fact the new hardening's own inlined
test copy failed against two of its own cases. Fixed by only
treating '#' as a comment-start at line-start or after whitespace,
which distinguishes ".../ns#" (preceded by a word character) from an
actual comment (preceded by whitespace/newline in every realistic
case, including the attacker's own comment-hiding PoC). Also fixed
the companion PREFIX/BASE regex, which required a prefix-name token
between the keyword and the IRI even for bare `BASE <...>`
declarations (which have none).
tests/test_security_regression.py's SPARQL section now imports the real
_is_read_only_query instead of maintaining a parallel inlined copy that
had silently drifted from — and shared the same bug as — the real
implementation; removed its TestAPIKeyAuth class (tested the now-deleted
auth.py) since equivalent, more thorough coverage already exists in
tests/explorer/test_explorer_auth.py. Updated tests/explorer/test_sparql_route.py's
multi-statement-injection test to reflect that the keyword scan now
catches "SELECT ... ; DROP ALL" itself rather than relying on rdflib's
parser, and added a new test confirming the parser still catches
multi-statement syntax that doesn't contain any forbidden keyword.
Full explorer/vector_store/security-regression/age_store suite: 543
passed (the only failures are 6 pre-existing, unrelated Pinecone-client
mocking issues).
The previous rework of the redirect loop closed the response on each
redirect hop but dropped the try/finally around the success path, so the
terminal response (the one actually read and returned) was left
unclosed, leaking the connection back to the pool unclosed under load.
Triplet.subject and Triplet.predicate (and, in some builders, .object)
were interpolated directly into SPARQL update/query strings in the
Blazegraph and RDF4J stores, and into a SELECT filter in the Jena store.
A subject containing '>' closes the '<...>' IRI token early, so the rest
of the value is parsed as more SPARQL. Entity names are document text in
the normal ingest pipeline, so anyone whose content gets processed could
append operations like CLEAR ALL, running with the application's store
credentials.
Applied the existing sparql_escaping.validate_uri (already used by
anzo_store.py, the one backend that was already hardened) at every
subject/predicate/object interpolation site:
- blazegraph_store.py: _build_insert_data, _triplets_to_rdf (unreachable
dead code today but same fix applied for consistency/future-proofing),
bulk_load's graph option, get_triplets's filter, delete_triplet.
- rdf4j_store.py: _triplets_to_ntriples, get_triplets's filter,
delete_triplet. (add_triplets's graph option was already validated.)
- jena_store.py: get_triplets's filter — the only vulnerable site;
add_triplets/delete_triplet already use rdflib's native Python API
(Graph.add/.remove with URIRef) rather than building query strings, so
they were never exploitable this way.
Added tests/triplet_store/test_sparql_injection.py (12 tests) reproducing
the advisory's own injection payload against all three backends' write
and read paths, asserting the malicious query is never built or sent.
Full triplet_store suite (330 tests) passes with no regressions.
Note: while adding read-path test coverage, found that jena_store.py's
get_triplets() WHERE-clause filter syntax is malformed SPARQL (missing a
FILTER()/separator before the equality conditions) — a pre-existing
correctness bug unrelated to this fix, worth a separate follow-up.
* security: require API-key auth on all Explorer API routes (GHSA-j4mq-hprp-987v)
Every Explorer route (bulk import/export, delete, LLM-backed ontology
generation, SPARQL, etc.) was mounted with no authentication, and both
server entrypoints bind 0.0.0.0 by default. Anyone reaching the port got
full read/write/delete on the graph.
- Add require_auth dependency (explorer/dependencies.py): checks
X-API-Key against SEMANTICA_API_KEY, fails closed with 503 if
unconfigured (not silently anonymous), 401 on wrong/missing key.
SEMANTICA_ALLOW_ANONYMOUS=true opts out explicitly for local dev.
- Wire dependencies=[Depends(require_auth)] into all 11 API routers in
both explorer/app.py and server.py. /health, /api/info, static assets,
and the SPA catch-all stay public.
- /ws/graph-updates handshake now checks the same key via header or
?api_key= query param (browsers can't set custom WS headers) before
accepting the connection.
- Default bind changed from 0.0.0.0 to 127.0.0.1 in server.py's main()
and cli.py's `server start`; the CLI warns if a non-loopback host is
passed explicitly without a key configured.
- Startup logging reports the resolved auth mode in both app factories.
- Document/generate SEMANTICA_API_KEY in the deploy recipes that expose
a public endpoint by default: docker-compose, Railway, Fly, Render.
Added tests/explorer/test_explorer_auth.py covering fail-closed default,
wrong/missing/correct key, anonymous opt-in, public-route exemptions, and
the WS handshake. Added tests/explorer/conftest.py defaulting the
pre-existing ~200 explorer tests to SEMANTICA_ALLOW_ANONYMOUS=true so
they keep exercising route logic without needing a key.
* fix CORS
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): add SSRF protection for WebIngestor and RESTIngestor
Block non-http(s) schemes and private/loopback/link-local targets before
outbound requests, with allow_private_ips opt-in for trusted deployments.
* fix(ingest): fail closed on SSRF DNS resolution errors
* fix(ingest): validate SSRF targets on every HTTP redirect hop
* fix(ingest): avoid blocking on SSRF DNS executor shutdown
* fix(ingest): parse allow_private_ips without truthy-string pitfalls
* docs(ingest): clarify robots.txt SSRF/validation comment
---------
Co-authored-by: Pravit Ampapathini
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
before re-validating (previously any relative redirect was rejected
outright), and close every response instead of leaking the connection
across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
error handling so the graph-size cap returns a clean SparqlResponse
error instead of an unhandled 500.
- add regression tests for all three.
* test(deduplication): cover ClusterBuilder, MergeStrategyManager, and batch paths
Add focused coverage for union-find clustering, property merge rules,
merge_duplicates, embedding similarity, and incremental detection (#866).
* test(deduplication): assert unrelated clusters without conditional skip
Make cluster-separation coverage fail closed by using mocked pairs and
unconditional assertions for distinct Apple vs Microsoft cluster IDs.
* test(deduplication): tighten update_clusters attachment assertions
Require the incremental path to place the new near-duplicate in the
same rebuilt cluster instead of accepting a vacuous cluster-count check.
* test(deduplication): strengthen incremental detect_duplicates wrapper checks
Assert real DuplicateCandidate matches, score threshold, and new×existing
routing instead of only checking that the wrapper returns a list.
* test(deduplication): verify metadata provenance behavior
Assert that preserve_provenance writes metadata.provenance fields and add a disabled-path test so regressions do not pass through merge_entities metadata alone.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@users.noreply.github.com>
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.
Also adds regression tests for scores >= 1 and a CHANGELOG entry.
Adds similarity_unavailable marker and warning logs to build_decision_context and explain_decision when a persistent backend (like FAISS) fails to reconstruct a vector. Updates docstrings to explicitly state this degraded-path behavior and guarantees schema stability. Adds regression tests to test vector retrieval failure behavior via caplog and context assertions.
- build_decision_context() and explain_decision(include_paths=True) both
accessed self.vectors directly, which is only initialized for the
inmemory backend, crashing with AttributeError on any persistent
backend (FAISS, Qdrant, Pinecone, etc.). Replaced with self.get_vector()
(#843's backend-agnostic accessor) + an is-not-None check — a verified
1:1 behavioral equivalent for the old 'decision_id in self.vectors'
guard on the inmemory path.
- Found a third, undocumented instance of the same bug during
verification: _filter_by_metadata() also accessed self.metadata/
self.vectors directly. Initial fix silently returned [] for persistent
backends, which was itself a new silent-failure bug (indistinguishable
from a genuine zero-match result). Reconciled to raise
NotImplementedError instead, matching the established precedent from
get_vector()/get_metadata() (#843) for 'backend exists but doesn't
support this operation' — confirmed via full grep of all 7 backend
wrapper classes that none currently implement filter_by_metadata,
so this path was previously dead-code-masked-as-working.
Tests: 14 new tests across two rounds — inmemory behavioral equivalence,
real (non-mocked) FAISS backend regression tests for all three methods,
and explicit coverage proving the NotImplementedError fires with a clear
message rather than the old silent-[] behavior. Full suite: 53 passed,
0 failed, 0 regressions across the 39 pre-existing tests.
self.indexer is only set for backend="inmemory", so save()/load() still
raised AttributeError for persistent backends (faiss, qdrant, etc.) even
after this PR's getattr() guards on self.vectors/self.metadata, since the
unguarded `self.indexer` access happened first. Guard it the same way and
delegate to the backend store's native save_index/load_index (currently
only FAISSStore implements these) so persistent-backend saves actually
persist instead of silently no-oping.
- Removed total=False from SearchResult TypedDict so all fields are strictly required
- Ensured distance: None is returned from backends that don't natively expose distance (Qdrant, Pinecone, SQLite, pgvector, in-memory)
- Standardized search result score to a consistent 0.0 - 1.0 similarity metric scale across all backend adapters
- Relaxed SearchResult id type to Union[str, int] to accommodate native integer IDs from Milvus and Qdrant without casting
- Updated schema verification tests
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
- Added insert_vectors alias to add_vectors for backward compatibility.
- Sanitized vector_id in get_vector and get_metadata to prevent query injection.
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
- Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
- Added metadata storage using JSON.
- Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
- Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
- Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
- VectorStore.get_vector() and get_metadata() were hardcoded to access
self.vectors and self.metadata dicts, which are only initialized for
the inmemory backend, causing AttributeError on all persistent backends
(FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).
Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
self.backend == 'inmemory' (zero behavior change) and delegate to
self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
- FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
NotImplementedError (FAISS has no metadata storage natively).
- QdrantStore: uses client.retrieve() with with_vectors/with_payload.
- PineconeStore: wraps existing fetch_vectors() call.
- MilvusStore: raises NotImplementedError (auto_id=True schema discards
string IDs at insert time, making by-ID lookup impossible in this
wrapper's current schema).
- WeaviateStore: uses collection.query.fetch_object_by_id().
- PgVectorStore: wraps existing get_vectors() SQL method.
- SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
FAISS backends with real (non-mocked) assertions.
All 28 tests pass.
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
Adds an Unreleased/Fixed entry for #841 (closes#840) — QdrantStore
search results were keyed "payload" instead of "metadata", breaking
HybridSearch.filter_by_metadata() for Qdrant results.
Adds an Unreleased/Added entry for #838 (closes#834), including the
follow-up fix that preserves ImportError for a missing pyoxigraph
install instead of masking it as a generic ProcessingError.
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
- Wire the Explorer frontend's node --test suites (test:graph-store,
test:graph-workspace, and the new test:plugin-registry regression
test) into CI. Previously only `npm run build` ran, so none of the
frontend tests -- including this fix's own regression coverage --
executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
transition doesn't leave the dev diagnostics panel stale.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
* fix(vector_store): stop dropping metadata for add_vectors-only backends
VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.
Fixes#832
* fix(vector_store): guard signature introspection in store_vectors
inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.
* docs(changelog): document VectorStore metadata-drop fix (#832, #835)
* test(vector_store): add regression coverage for metadata forwarding
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
- pluginRegistryPredicates.ts: consolidate 8-line JSDoc to 5 lines,
removing redundant detail that restated implementation mechanics
already obvious from the code.
- GraphWorkspace.tsx: shorten the lastScrubberMsRef comment from 5 lines
to 2; trim the handleDiagnosticsChange block comment by removing the
'rather than bailing out' implementation-alternative sentence; tighten
the distanceVisual inline comment.
- pluginRegistry.temporal.test.mjs: replace 17-line file-level JSDoc
with 9 lines focused on the invariant rather than the root-cause
narrative (already covered in pluginRegistryPredicates.ts); remove
two tsx loader implementation-detail comments; tighten two test-level
inline comments.
No logic, types, or test assertions changed. All 42 tests pass.
- Legacy top_k kwarg was read but not removed from options, so it got
forwarded via **options into VectorStore.search_vectors(), colliding
with backends that call search(..., top_k=k, **options) (e.g. sqlite,
pgvector) and raising "got multiple values for keyword argument
'top_k'". Now popped instead of just read.
- VectorStore.search_vectors()'s dispatch only recognized backend
methods named search/search_similar, so HybridSearch's delegation
still hit NotImplementedError for qdrant/milvus/pinecone, which name
their method search_vectors() with a differently-named count
parameter (limit vs k). Added a third dispatch branch that binds the
count positionally so it works regardless of the backend's parameter
name.
- Backend-delegated results defaulted a missing "distance" to the raw
score, silently reusing the local path's cosine-similarity convention
(distance = 1 - score) even for backends using unrelated metrics
(L2, inner product). A missing distance is now left as None instead
of a fabricated, metric-inconsistent value.
Two issues addressed:
1. Plugin-loading useEffect unnecessarily depended on temporalState.
After the #830 fix, no shouldLoad predicate reads temporalState, but
the effect's dep array still included it, causing extra re-runs on
every scrubber update. Removed temporalState from the dep array and
the shouldLoad call site. Made temporalState optional in the
LazyPluginRegistryEntry shouldLoad context type to match.
2. Regression test imported a local copy of shouldLoad instead of the
production predicate. Extracted all three shouldLoad predicates into
pluginRegistryPredicates.ts (pure module, no React/DOM dependencies),
wired GraphWorkspace.tsx to use the imported functions, and updated
the test to import and exercise the real production code via tsx.
Verified: introducing the old broken condition causes the test to fail;
the correct implementation passes all 7 assertions.
HybridSearch.search() directly accessed self.vector_store.vectors, a dict
that VectorStore only creates for backend="inmemory". Every other backend
(faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised
AttributeError. It now delegates to VectorStore.search_vectors() for
non-inmemory backends, applying metadata_filter as a post-filter and
normalizing results to a consistent {id, score, distance, metadata} shape.
Also fixes two related bugs surfaced while testing the backend-delegated
path end to end:
- vector_ids could remain None when explicit vectors/metadata were passed
without vector_ids, crashing downstream indexing.
- query_vector passed as a plain list crashed backend stores (e.g.
FAISSStore.search_similar) that call .ndim on it; now normalized to a
numpy array up front.
And in vector_store.py: VectorStore.store_vectors() silently dropped
metadata for FAISS (and any add_vectors-only backend) because it called
add_vectors(vectors, **options) without forwarding metadata, even though
FAISSStore.add_vectors() accepts it. This blocked HybridSearch's metadata
filtering from working at all against FAISS.
Fixes#833
temporalDiffState.ts belongs to feat/793-temporal-diff-ui and should not
appear in the #830 diff. Remove it from this branch's tracked files.
Add the pluginRegistry.temporal.test.mjs regression test that covers the
shouldLoad fix committed in the main #830 commit (it was never committed).
Add test:plugin-registry script to package.json so the regression test
can be run via npm run test:plugin-registry.
Two independent render loops were causing the Temporal panel to remain
stuck on 'Loading temporal...' in npm run dev:
Loop 1 — diagnostics state churn (GraphWorkspace.tsx):
handleDiagnosticsChange unconditionally called setGraphDiagnosticsState
with a new object on every invocation. buildEffectAvailability (called
inside GraphCanvas's diagnostics useEffect) always returns a new object,
so setGraphDiagnosticsState was called on every effect run, creating a
cycle: setGraphDiagnosticsState graphDiagnosticsState new
diagnosticsSnapshot new pluginContext new handleInteractionStateChange
new GraphCanvas re-renders diagnostics effect fires again.
Fix: before calling setGraphDiagnosticsState, compare the incoming
diagnostics field-by-field against the last accepted snapshot via a ref
(lastDiagnosticsRef). All effectAvailability entries, edgeClasses.updatedAt,
structureLayer.cacheKey/lastDrawAt/enabled, and distanceVisual identity
must differ for a state update to proceed. The ref approach avoids
scheduling a re-render at all, rather than bailing out inside a functional
updater after the render has already been committed.
Loop 2 — scrubberTime churn (GraphWorkspace.tsx + GraphWorkspaceShell.tsx):
TimelinePanel.tsx calls onTimeChange(defaultTime) whenever its useEffect
re-runs. React 18 concurrent mode re-runs effects with structurally-new
Date objects for the same timestamp when speculative renders discard
useMemo caches, causing setScrubberTime to be called repeatedly with a
new Date that has the same millisecond value — triggering temporalState
churn, the diagnostics effect, and eventually the same loop.
Fix: wrap setScrubberTime in an onTimeChange useCallback that compares the
incoming time's millisecond value against the last sent value (via
lastScrubberMsRef). Redundant calls with the same timestamp are dropped
before reaching setScrubberTime. Stable useCallback identity also prevents
TimelinePanel's useEffect from re-firing solely due to prop identity churn.
Both fixes applied to GraphWorkspace.tsx and identically to
GraphWorkspaceShell.tsx which has the same pattern.
Verified:
- npm run dev: 0 'Maximum update depth exceeded' errors
- Temporal panel renders with real data in dev mode
- Effects and Neighbors panels unaffected
- npm run build + preview: identical behavior, 0 errors
- All 42 frontend tests pass (34 graph-workspace, 1 graph-store, 7 plugin-registry)
fixed qodo review
applyDiffHighlight/clearDiffHighlight were writing baseColor only to
graphStore.graph (the store singleton), but Sigma is constructed with
displayGraphRef.current and the nodeReducer reads attributes from that
instance. When the display graph is a derived copy (aggregated,
focused, or grouped view), the store write has no effect on the
currently-rendered frame -- sigma.scheduleRefresh() flushes the
reducer over the display graph, which did not receive the mutation.
Fix: introduce writeBaseColor(context, nodeId, color) which writes to
BOTH the store graph (so the color propagates into the next display
graph rebuild via aggregateDisplayGraph's shallow attribute copy) AND
context.displayGraph (the live Graph instance currently bound to
Sigma, so the change is visible in the current frame immediately).
The dg !== graph guard skips the display-graph write when they happen
to be the same object (non-aggregated full view), avoiding a redundant
double-write in that case.
Original baseColor is still captured from the store graph (the
authoritative source, since aggregateDisplayGraph copies from there),
so restore remains correct across all view modes.
Adds a Compare section to the existing Temporal Context panel
(temporalOverlayPlugin.tsx) that lets a user pick two ISO timestamps
and diff the graph's node set between them via the existing, previously
UI-less GET /api/temporal/diff backend route.
- New temporalDiffState.ts: typed fetch wrapper (fetchTemporalDiff)
matching the route's added_nodes/removed_nodes response shape.
- Diff results recolor affected nodes via baseColor (not
ringColor/haloColor -- traced and confirmed those are only read by
the sigma reducer for hovered/selected/path-state nodes and are
silently discarded for default-state nodes).
- Validates both timestamps are present, parseable, and from < to
before firing a request.
- Distinct idle/loading/error/empty/success states -- an empty diff
(no changes) is rendered as its own state, not as an error.
- Cancels any in-flight request via AbortController on re-submission
and on unmount; restores each highlighted node's original baseColor
(captured before overwrite, not cleared to a fallback default) on
both paths.
- Reuses existing theme tokens (GRAPH_THEME.palette.semantic[2],
ui.control.dangerText) and existing button/input/loading/error
visual patterns already established in this same plugins directory
and in GraphInspectorPanel.tsx, rather than introducing new styling.
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
older database with the new code would break on the first insert/select
since the row width and _row_to_entry's fixed indices grew past the old
schema. Added test_migrates_pre_existing_old_schema_database.
- verify_chain() now also checks that sequence_id is exactly the
predecessor's plus one (no gap, no duplicate), in addition to the existing
previous_checksum comparison. Hardens against the narrow case where
compute_checksum()'s deliberate exclusion of entity_id could let two
distinct rows coincidentally share a checksum, which alone would let a
checksum-only comparison miss a gap. Added
test_verify_chain_detects_tampered_sequence_gap.
- Explorer provenance route: edge ids now include direction
(f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
already included it. The same (src, target) pair can legitimately appear
in both the upstream and downstream chains (cycles/overlap), and without
this the two edges collided on the same id. Added
test_add_chain_edges_ids_distinguish_direction.
- Removed an unused `Any` import in parse_provenance.py.
Part A - high-stakes trust blockers:
- Invalidation tombstones via ProvenanceManager.invalidate() (archive-then-append,
never mutates or deletes) instead of hard delete
- Hash-chained integrity: sequence_id/previous_checksum chain every entry to its
predecessor; new verify_chain() detects wholesale row deletion that a lone
per-row checksum cannot
- Typed Agent (AgentRecord: agent_type/is_automated) and Activity (ActivityRecord:
start/end timing), wired through all 18 *_provenance.py wrappers
- Split parent_entity_id into previous_version_id (correction) vs derived_from_id
(cross-source derivation), additive alongside the legacy combined field
- Downstream/descendant lineage traversal (get_descendants/trace_descendants,
reverse BFS) closing the dead direction="downstream" code path in the
Explorer's provenance route
- Qualified Association+hadRole and Invalidation in export_prov()
- New CLI: provenance invalidate|verify-chain|descendants
Part B - general PROV-O spec completeness:
- Qualified Generation/Usage/Derivation in export_prov()
- wasAssociatedWith, actedOnBehalfOf, wasInformedBy relations
- Bitemporal fields (valid_from/valid_until/revision_type/supersedes) plus
revision_history()/query_recorded_between(), closing the deprecated
kg.ProvenanceTracker's "no direct equivalent yet" migration gaps
- prov:Bundle/hadMember membership via bundle_id
- Configurable base_uri (--base-uri CLI flag), shared by RDFExporter's
NamespaceManager and OWLExporter's default ontology_uri so KG/OWL/PROV
exports co-resolve under one namespace instead of three hardcoded ones
Bugs fixed along the way:
- agent_id was a dead field: no track_* method read it from kwargs
- track_entities_batch silently absorbed typed kwargs into the metadata blob
- compute_checksum() had to exclude entity_id itself: hashing it made
track_entity's versioning-archive relabel permanently orphan any entry
already chained from the pre-relabel checksum, a false-positive "broken
chain" for a legitimate rename
- InMemoryStorage.get_chain_head() ignored the committed head whenever the
current transaction had staged entries, corrupting the next chain link
- several new ProvenanceEntry fields were wired into the dataclass and
export_prov() but not into SQLiteStorage's DDL/INSERT/row-mapping;
InMemoryStorage masked the gap. Added a permanent round-trip regression
test to catch this class of bug for future field additions
Flagged, not fixed (separate pre-existing issues, out of scope for #825):
- pipeline/pipeline_provenance.py imports a nonexistent module and wraps a
Pipeline dataclass with no run() method
- most *_provenance.py wrappers' backing classes are themselves missing or
incomplete (context_manager, deduplicator, normalizer, etc.)
- kg_provenance.py passes entity_type inside metadata={} instead of as a
top-level track_entity() kwarg across most of its call sites
* security: SHA-pin all Actions, harden release pipeline, add pin verification
Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.
- Pin every third-party GitHub Action across all workflows to a full commit
SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
check that confirms via the GitHub API that each pin still matches its
tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
contents: read); add a concurrency group so simultaneous tag pushes can't
race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
pull-requests: write and silently failing; add bounded artifact retention
for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
adopters, including what's enforced and what a fork needs to reconfigure
for itself (environment/branch protection, Trusted Publishing trust).
Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).
* fix: harden verify-action-pins per PR #824 bot review
Addresses real findings from the automated review on #824:
- The script previously only matched uses: lines that already contained a
40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
would never be scanned at all and the check would pass silently. It now
matches every uses: line and hard-fails on any ref that isn't a full
commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
previously only logged a warning and continued; that's now a hard
failure too, since an unverifiable pin is exactly the failure mode this
check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
so an edit to the verifier script itself wouldn't run the check that
verifies it. Added the script path to both trigger filters.
The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.
Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).
* fix: repair broken Safety scan and PR comment formatting
The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:
- Every line in the JS comment builder used \n (escaped backslash-n)
inside template literals, which JS renders as the literal two-character
string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
check_id - hence "undefined: <path>" for every entry.
Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).
While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:
- `safety check --json --output safety-report.json` is invalid in
Safety 3.x: --output now selects a console format (json/text/screen),
not a file path. The command errored on every run (swallowed by
`|| true`), so safety-report.json was never created and the PR comment
always fell back to a generic "scan completed" message. Switched to
`--save-json`, which is the correct flag for writing a JSON report to
disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
package_name.
- The job never installed Semantica's own dependencies before scanning,
so `safety check` (which defaults to scanning the environment) was
auditing the scanner tools' own dependencies, not Semantica's. Added
`pip install -e ".[llm-litellm]"` so the project's actual dependency
tree - including the LiteLLM extra this whole hardening effort is
about - is what gets scanned.
Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.
* fix: remove unused pypdf2 dependency (CVE-2023-36464)
Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.
PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.
* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use
Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").
Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").
Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.
Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).
* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening
Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).
* fix: close two remaining gaps missed by upstream bot-review fixes
verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
by the existing regex, so a SHA-pinned action written with quotes would
silently skip verification. Updated the main ERE to accept an optional
leading/trailing single or double quote around the owner/action@ref
value, and excluded quote chars from the inner character classes so the
ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
guard so the command doesn't fail when no *.yaml files exist.
security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
five scanner-failure modes (file missing, empty file, malformed JSON,
valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
genuine zero-vuln and real-vuln cases still behave correctly.
* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI
The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:
syntax error in conditional expression: unexpected token ')'
at line 27 on every CI run.
Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.
Verified in real bash 5.2.21 (Git for Windows):
No syntax error on the real 40-pin workflow tree (Checked 40)
Unquoted SHA pin: MATCH, correct repo+ref extracted
Double-quoted SHA pin: MATCH, correct repo+ref extracted
Single-quoted SHA pin: MATCH, correct repo+ref extracted
.yaml extension file: MATCH, correct repo+ref extracted
./local-action: NO MATCH (correct)
docker://: NO MATCH (correct)
* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md
The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:
- GitHub secret scanning and push protection (repo settings, not copied
on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
whether the upload-sarif step in codeql.yml does anything)
Added as items 5, 6, 7 matching the existing numbered bullet style.
* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)
verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:
old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
new: d1ba80a13dd99fba24a470575428917156a28b43
Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* fix(agno): surface unevaluable policy rules (#778)
* fix(agno): fixed qodo reviews
check_policy previously let unevaluable policy rules silently return
compliant=True with no signal (issue #778): a rule referencing a field
missing from decision_data, or a rule string not matching the expected
<field> <op> <value> format, both fell through _eval_rule's `return
True` and were treated as passed.
Both now raise ValueError, which routes through check_policy's existing
exception handler and surfaces as a `warnings` entry instead. compliant/
violations semantics are unchanged for every case that previously worked
correctly; an unevaluable rule is not counted as a violation since it's
genuinely unknown whether it would have passed.
Follow-up fixes from code review:
- policy_rules decoded via json.loads without checking it was a list;
a JSON-encoded bare string decoded to a Python str, so iterating it
evaluated one "rule" per character, amplifying a single input-shape
mistake into a wall of per-character warnings. A decoded string is
now treated as a single rule; any other non-list shape or non-string
list element produces exactly one warning instead.
- _eval_rule used `data.get(field) is None` to detect a missing field,
which can't distinguish an absent key from a key present with JSON
null - both produced the same "undefined field" warning. Field
presence is now checked with `field not in data` first, and a
present-but-null value gets its own distinct message.
Added regression tests for all of the above in
tests/integrations/agno/test_decision_kit.py (38 tests in the file,
128 passing across tests/integrations/agno/).
* fix(agno): reject non-object decision_data in check_policy
check_policy only validated that decision_data was well-formed JSON,
not that it decoded to an object. When it decoded to a list, `field
not in data` in _eval_rule silently became list-membership testing
of values instead of a dict key check - e.g. "confidence" not in
["confidence", 0.95] evaluates to False - so a matching rule fell
through to data["confidence"], raising a raw internal TypeError
("list indices must be integers or slices, not str") instead of any
meaningful diagnostic. Numbers, strings, and bools produced similarly
opaque TypeErrors deep inside _eval_rule.
check_policy now checks isinstance(data, dict) right after decoding
and rejects any other shape with a single clear violations entry,
the same way it already rejects malformed JSON.
Added 5 regression tests in tests/integrations/agno/test_decision_kit.py
covering list/number/string/bool/null decision_data shapes (43 tests
in the file, 133 passing across tests/integrations/agno/).
Addresses Copilot PR review comment on the #778 fix branch.
* docs(changelog): reference PR #822 in the check_policy changelog entry
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(provenance): log tracking failures and return None on storage error (closes#783)
* docs(provenance): document Optional return types and failure behavior (#783)
* fixed qodo reviews
- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)
- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True
- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable
- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()
- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Removed unused `validate_skos_hierarchy` import from
test_ontology_subissue3.py (flake8 F401); the test uses a `wraps=` spy
on the real add_nodes_and_edges instead of calling the helper directly.
- refresh_ontology tests now percent-encode the ontology URI with
urllib.parse.quote before interpolating it into the {ontology_uri:path}
request path, matching the already-encoded unknown-uri refresh test in
the same file instead of embedding a raw http://... URI with slashes.
- Reworded the cyclic-SKOS refresh test's comment and section header:
GraphSession.add_nodes_and_edges() documents pre-write validation and
lock-based mutual exclusion, not transactional rollback, so "atomic"
was replaced with "single combined add_nodes_and_edges() call" to avoid
implying rollback guarantees that don't exist.
Verified: tests/explorer/test_ontology_subissue3.py (34 passed) and
tests/explorer/ (204 passed), no regressions.
- validate_skos_hierarchy() re-walked every existing hierarchy edge in
the graph on each write, so one pre-existing cycle anywhere would
block all unrelated future SKOS writes. It now only traverses
concepts touched by the edges actually being written, while still
checking those against existing edges for cross-boundary cycles.
- In /api/ontology/load, `except HTTPException: raise` sat after a
broader `except Exception` clause that already matched HTTPException,
so a 422 raised after a successful OntologyIngestor parse was
silently swallowed and retried via the fallback RDF parser instead of
reaching the caller. Reordered the except clauses.
Co-authored-by: mikemikimike <13286568797@163.com>
- Added exc_info=True to both store failed and record_decision failed warning logs in _AgentScopedStore.upsert_memory() to preserve full traceback context for debugging
- Updated CHANGELOG.md entry to document traceback preservation
- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)
- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True
- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable
- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()
- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False
Signature introspection and the resulting call were sharing one
try/except, so a genuine bug inside a backend's get_causal_chain
(raising an unrelated TypeError) was misread as a signature mismatch,
causing an identical retry call before the real error surfaced.
Split introspection from the call so a successfully-introspected call
happens exactly once; the trial-and-error cascade now only runs when
inspect.signature itself fails. Also adds the CHANGELOG entry for
#781/#817, which was missing.
- Add safe input validation and bounds clamping on max_depth (1..100) to prevent DoS/memory exhaustion
- Use inspect.signature for accurate keyword dispatch with precise TypeError fallback
- Prevent masking of genuine internal TypeError exceptions inside graph backends
- Add security and input hardening regression tests
- Support legacy (depth kwarg) and positional-only get_causal_chain backend signatures in fallback path
- Add regression tests for signature compatibility
- Return explicit error dictionary when graph lacks get_causal_chain instead of silent empty list
- Forward direction and max_depth in fallback graph.get_causal_chain call
- Add regression tests for error signaling and parameter forwarding
* feat(triplet_store): add Altair Anzo triplet store backend
Adds AnzoStore as a fourth peer to BlazegraphStore/RDF4JStore/JenaStore,
speaking plain SPARQL 1.1 over HTTP (no new dependency needed). The one
structural difference from the existing backends is that Anzo addresses
data by a dataset/graphmart URI rather than a short namespace/repository
name, so the endpoint path percent-encodes it. Reuses the shared
sparql_escaping.py helpers and wires "anzo" into TripletStore's backend
dispatch and config env vars.
Closes#813
* fix(triplet_store): correct AnzoStore SPARQL syntax and validate IRIs
Addresses review findings from Qodo and Codex on PR #814:
- get_triplets(): constraints are now expressed via FILTER(...) instead of
bare equality expressions appended inside the WHERE group graph pattern
(e.g. "?s ?p ?o ?s = <...>"), which is not valid SPARQL and was rejected
by standards-compliant endpoints.
- bulk_load(): named-graph inserts now nest the GRAPH block inside the
INSERT DATA braces (INSERT DATA { GRAPH <g> { ... } }) per the SPARQL 1.1
Update grammar, instead of "INSERT DATA GRAPH <g> { ... }".
- bulk_load()/_build_insert_data()/delete_triplet()/get_triplets() now
validate subject/predicate/graph URIs via sparql_escaping.validate_uri
before interpolating them into SPARQL Update/Query strings, closing an
injection path where a value containing ">" or "}" could break out of
the intended <...> token.
- Corrected the store_type docstring/usage example: Anzo's linked-data-set
store type is "lds", not "dataset".
Extended tests/triplet_store/test_anzo_store.py with coverage for the
corrected query shapes and the new validation/injection-rejection paths
(38 tests total, up from 32). Full tests/triplet_store/ suite: 299/299
passing.
* test(triplet_store): expand AnzoStore regression coverage
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(provenance): centralize duplicated store-and-swallow logic into _save_entry() (closes#784)
- Added ProvenanceManager._save_entry(entry) as the single shared
checksum-compute + storage.store() + graceful-failure-swallow
pipeline, previously duplicated identically across track_entity,
track_relationship, track_chunk, and track_property_source.
- track_entities_batch/track_chunks_batch already delegate to
track_entity/track_chunk in a loop, so they inherit the fix for
free — left untouched, confirmed no direct duplication there.
- Byte-for-byte preserves today's swallow-and-continue behavior and
comment text; this is an architecture-only refactor. The silent-
failure behavior itself is unchanged and out of scope here — a fix
to it now only needs to happen in one place instead of four.
- Added 4 new regression tests (previously 0 of the 4 single-item
methods had failure-path coverage) proving storage.store() raising
is still caught and each method still returns its ProvenanceEntry.
Tests: tests/provenance/ 228 passed (+4 new), tests/explorer/test_provenance_manager_wiring.py 8 passed. 236/236, 0 failed.
* fix(provenance): drop out-of-transaction store attempt in track_entity fallback
The _save_entry refactor changed track_entity's pre-build exception
fallback (entry is None branch) to call _save_entry(), which makes a
real self.storage.store(entry) call. The original code only computed
a checksum here and never attempted storage again, since this branch
fires when something already failed before the entry was built inside
the atomic transaction. Storing outside that transaction bypasses the
BEGIN IMMEDIATE serialization #807 added, risking the same race it
fixed. Restored checksum-only behavior and added a regression test
asserting storage.store is not called on this path.
Also removed an untested hasattr(_store_with_conn) defensive branch
added during the refactor that wasn't in the original code, and added
a changelog entry.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Two review findings on #807/#812:
- retrieve() and trace_lineage() were routed through transaction()'s
BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
serialized behind every other read/write, defeating the WAL
concurrency this PR was meant to add. They now use a dedicated
_read_connection() (configured, no explicit BEGIN).
- track_entity()/track_chunk() swallowed all internal storage
exceptions unconditionally, so a single item's failure inside
track_entities_batch()/track_chunks_batch()'s shared transaction
never reached the batch loop's per-item except, inflating
tracked_count for entries that were never persisted. Both now
re-raise when called with a shared _conn (batch context) while
still degrading gracefully on standalone calls.
Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
- Reuse lineage['integrity_verified'] in _build_provenance in O(1) time when available, eliminating redundant SHA-256 verification loops across lineage chains.
- Improve compute_checksum dictionary handling in semantica/provenance/integrity.py so None values fall back cleanly to ProvenanceEntry defaults.
- Move json and verify_checksum imports to module top-level in semantica/provenance/manager.py to avoid function-local import overhead during get_lineage calls.
- Extend ProvenanceNode in semantica/explorer/schemas.py with audit evidence fields: source_document, source_location, source_quote, confidence, and checksum.
- Update _transform_audit_lineage in semantica/explorer/routes/provenance.py to populate these evidence fields for each lineage node from ProvenanceEntry records, while keeping default None values for orphan nodes.
- Include source_document, confidence, and checksum in markdown report rendering (_render_markdown) so exported markdown reports surface attribution and integrity evidence.
- Add unit test test_provenance_audit_evidence_fields_preserved in test_provenance_manager_wiring.py verifying that evidence fields are present across /api/provenance JSON responses and exported JSON/markdown reports.
- Update verify_checksum and compute_checksum in semantica/provenance/integrity.py to support both ProvenanceEntry objects and serialized dictionary entries.
- Add integrity_verified flag computed via verify_checksum to the dictionary returned by ProvenanceManager.get_lineage().
- Update _build_provenance in semantica/explorer/routes/provenance.py to verify every returned lineage entry before labeling the result as source=audit. If verification fails due to missing checksums or corrupted records, log a warning and fall back cleanly to graph traversal.
- Add unit test test_provenance_manager_wiring_checksum_failure_falls_back in test_provenance_manager_wiring.py verifying that tampered lineage entries trigger fallback to source=graph_traversal.
- Fix _transform_audit_lineage to classify all non-downstream ancestor derivation edges as 'upstream' instead of 'lateral', correcting multi-hop lineage direction in JSON and markdown reports.
- Add GraphSession.set_provenance_storage_path() to explicitly reject conflicting preconfigured storage paths or path mutations after provenance_manager initialization.
- Update create_app() to call active_session.set_provenance_storage_path(prov_path), preventing silent retention of conflicting paths or un-redirectable cached managers.
- Remove unused logging import in app.py.
- Add comprehensive unit tests in test_provenance_manager_wiring.py for upstream edge classification, markdown report grouping, conflicting path rejection, and manager initialization lockouts.
- Explorer's /api/provenance now queries the audit-grade ProvenanceManager
(SQLite-backed, checksummed) first, falling back to the naive 2-hop
graph traversal when no audit records exist for a node.
- Fixed a process-global mutable-state risk in the initial approach:
provenance storage path is threaded per-session via GraphSession,
not via ProvenanceManager's global set_default_storage_path classmethod.
- Added source: 'audit' | 'graph_traversal' to the response so callers
can distinguish which path served the data.
- Documented a known limitation: ProvenanceManager currently only
traces upstream/ancestor lineage, not descendants — the naive
fallback remains the only source for downstream relationships until
ProvenanceManager gains a reverse lookup (tracked separately).
- Warns (rather than silently no-ops) if a provided session's
provenance_manager was already constructed before create_app()
applied a provenance_storage_path.
- Never lets a provenance-manager failure crash the route; degrades
to the naive path with a logged warning instead.
Tests: 5 new tests in test_provenance_manager_wiring.py covering the
audit path, empty-record fallback, storage-failure degradation, app
startup wiring, and cross-session storage isolation. Full
tests/explorer/ + tests/provenance/ suite passing, order-invariant.
- README: get_table_lineage() takes table_name first, then catalog/schema
keyword args — the example had them in the wrong order, which would have
queried lineage for the wrong fully-qualified table when copy-pasted.
- modules.md: the ingest example used DatabricksIngestor without importing
it, causing a NameError if copy-pasted as-is.
- guides/ingest.md: corrected the claim that Databricks/Snowflake ingestors
return "the same shape as DBIngestor" — DBIngestor.execute_query() returns
a raw List[Dict] with no wrapper, unlike DatabricksData/SnowflakeData.
Makes enterprise lakehouse/warehouse ingestion (Databricks Unity Catalog +
Delta Lake, Snowflake) a first-class, prominently documented capability
across the README and guides, and adds matching runnable examples to
docs/guides/ingest.md. Also fixes several pre-existing inaccuracies caught
while auditing the ingest module docs against the actual source:
WebIngestor has no ingest_urls() (only singular ingest_url()), XMLIngestor's
XSD option is schema_path (not validate_xsd) and belongs on ingest() not the
constructor, and the "Available ingestors" list was missing DatabricksIngestor
while listing several classes not actually exported from semantica.ingest.
Extracts the row-cap-and-truncate loop (duplicated between the
CONSTRUCT/DESCRIBE and SELECT branches) into a shared _cap_rows()
helper, and adds a test for the previously-uncovered CONSTRUCT/DESCRIBE
truncation path. Addresses review nits on PR #805.
- Revert create_ontology silently falling back to a near-empty ontology on
generation failure; restores the HTTPException(500) behavior from #770/#787
that this PR had accidentally undone (and re-enables TestOntologyCreateFailures)
- Fold sh:Warning/sh:Info severity pySHACL results into the /shacl/validate
response's violations array instead of silently dropping them, so a
non-conforming report is never returned with an empty violations list
- Share a single nodes/edges fetch between _generated_shacl_for_uri and
_data_graph_turtle_for_uri via new _fetch_analysis_graph(), so /health
no longer re-queries and re-truncation-checks the same ontology twice
* fix(security): restrict Neptune cookbook SG, add VPC flow logs, harden IaC scan suppressions
Addresses open GHAS code scanning alerts:
- Neptune cookbook stack (neptune-setup.yaml) no longer opens the Bolt/OpenCypher
port to 0.0.0.0/0; a required ClientCidr parameter must be supplied instead.
Updated 21_Amazon_Neptune_Store.ipynb deploy instructions to match.
- Added VPC Flow Logs (CloudWatch Logs + IAM role) to the same stack.
- Documented why an account-wide IAM password policy resource does not belong
in a disposable per-learner CFN stack, with a justified ts:skip.
- Added inline `checkov:skip` / `ts:skip` comments to the knowledge-explorer
Helm templates (deployment/service/configmap) as a second suppression path
for the CKV_K8S_21/AC_K8S_0086/AC_K8S_0080 false positives, since the prior
annotation-only suppression was not being honored by the scanner.
* docs(changelog): document the Neptune and Helm chart security scan fixes
* fix(security): correct flow-log IAM scope and ClientCidr regex from review
- FlowLogRole granted logs:CreateLogStream/PutLogEvents on the bare log
group ARN, but those actions apply to log streams, not the group itself;
scoped them to "${FlowLogGroup.Arn}:log-stream:*" instead and moved the
Describe* actions (which don't support group/stream-level resource
restriction) to Resource: "*", matching AWS's documented flow-log IAM
policy shape. Without this, flow log delivery could silently fail.
- ClientCidr's AllowedPattern only checked digit count (1-3 digits per
octet), so malformed values like 999.999.999.999/32 passed parameter
validation and would only fail later when CloudFormation tried to
create the security group rule. Tightened the regex to enforce valid
IPv4 octet ranges (0-255) and prefix lengths (0-32).
* fix(security): harden IAM policy in neptune-setup and standardize Helm chart scan suppressions
- neptune-setup.yaml: split FlowLogRole policy into account-level statement (CreateLogGroup, DescribeLogGroups, DescribeLogStreams with Resource: '*') and log-group-scoped statement (CreateLogStream, PutLogEvents with !GetAtt FlowLogGroup.Arn) per AWS VPC Flow Logs least-privilege documentation.
- deployment.yaml: remove unreliable file-header skip comments (# checkov:skip / # ts:skip) and replace with resource-level metadata.annotations (checkov.io/skip and runterrascan.io/skip). Update seccomp rule ID from CKV_K8S_28 to checkov's actual seccomp rule CKV_K8S_31 on both Deployment and pod-template metadata.
- configmap.yaml / service.yaml: remove stale # ts:skip=AC_K8S_0086 file-header comments and add runterrascan.io/skip resource-level metadata annotations for consistency across all chart templates.
- .checkov.yaml: update documentation to explain resource-level metadata.annotations and reference CKV_K8S_31.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
sparql.py handles direct SPARQL query execution against the live graph with no test coverage anywhere in the repo. Adds coverage for the read-only allowlist (the actual security boundary here), row/timeout limits, error handling, and RDF projection fidelity.
- track_entity() no longer aliases a caller-supplied used_entities list
(it stored the reference directly and later mutated it via .append())
- Remove dead fallback branches in orchestrator.py/manager.py that
duplicated what Config.get()'s dotted-path resolution already does
- Add local --dry-run to `provenance audit` for parity with
`provenance export`
- `provenance check --strict` now warns instead of printing a success
checkmark before raising on a failed check
Add an Unreleased/Added entry for #786 covering the new export/import
Markdown format, idempotency and rollback guarantees, and the
Explorer/ContextGraph scoping decision from #765.
- Add entries alias in get_lineage() return dictionary so CLI and programmatic callers can access lineage entries via either key
- Update lineage() wrapper method to fallback to lineage_chain when entries is missing
- Add assertions in test_cli_lineage confirming lineage and entries lists are non-empty
- Add _default_storage_path, set_default_storage_path(), and test-isolation context manager default_storage_path() in ProvenanceManager
- Accept config kwarg in ProvenanceManager.__init__ to fix CLI initialization bug
- Implement audit_log(), lineage(), export_prov(), and check() on ProvenanceManager matching cli.py expectations
- Wire provenance.storage_path in Semantica.__init__ before pipeline stages execute
- Add comprehensive unit tests in tests/provenance/test_manager.py for CLI methods and test isolation
The set-state-in-effect refactor inlined each initial-fetch effect as a
standalone `fetchInitial`, duplicating the logic of the existing
reload/fetchOverview/fetchRegistry/loadVersions callbacks instead of
reusing them (required, since eslint-plugin-react-hooks v7 flags calling
an outside setState-touching function directly from an effect body, even
through an async gap - verified via a local lint probe). The duplicates
dropped the setError/flashMsg calls the originals had, so a failed
initial page load in AlignmentsTab, KGOverviewTab, OntologyManager, and
VersionsTab now failed silently instead of showing an error - a
regression of the exact bug #767/#790 fixed for these same files.
Also fixes LineageDiagram only clearing nodes/edges when the new
activeId was falsy, leaving the previous lineage view's stale diagram
on screen while switching directly between two ids.
The single-line checkov:skip=CKV_K8S_21 comment added in ed44260 was 286
characters, exceeding the repo's yamllint line-length rule (max 120,
.pre-commit-config.yaml). Split into three short comment lines: the skip
directive itself, then the rationale, in service.yaml, deployment.yaml,
and configmap.yaml.
Checkov's helm framework renders the chart without a namespace override,
so metadata.namespace (set to .Release.Namespace, bound only at install
time) always resolves to "default" and trips CKV_K8S_21 on service.yaml,
deployment.yaml, and configmap.yaml even though the chart is
namespace-agnostic by design.
Suppressed via per-file checkov:skip comments, following the same
convention already used for the Cloud Run false positives in
deploy/gcp/cloudrun-service.yaml.
* fix(#768): add ErrorBoundary to workspace Suspense blocks
* fix(#768): ensure ErrorBoundary retryCount only resets on recovery transition
* fix(#768): remove componentDidUpdate auto-reset to avoid premature reset on Suspense fallback
* fix(#768): reset ErrorBoundary retryCount only after a retry settles
Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.
KGOverviewTab dropped the nodes-fetch 207 warning whenever stats also
returned 207; HealthTab and AlignmentsTab still had the exact
silent-swallow pattern this PR set out to fix elsewhere in the same
folder. Also documents all of #790's fixes in the changelog.
* Fix#788: pin httpx<0.28.0 globally to fix TestClient breakage
Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.
Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.
Closes#788
* Add CHANGELOG entry for #788 httpx pin fix
Documents the httpx<0.28.0 global pin from this PR under Unreleased/Fixed.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Ensures network errors, API failures, and 207 Partial Success statuses are surfaced correctly to the user instead of failing silently in the frontend UI.
207 alone is indistinguishable from 200 to callers that only check
response.ok, so /api/analytics now raises 500 when every requested
metric fails and reserves 207 for genuine partial failure. Adds
regression tests for the temporal, analytics, and ontology-create
failure paths introduced in this PR, and logs the fix in the
changelog's Unreleased section.
Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.
Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.
Closes#788
- routes/temporal.py: temporal_patterns raises HTTPException(500) instead of
silently returning an empty-but-valid TemporalPatternResponse on exception
- routes/analytics.py: preserves existing partial-success body shape
(frontend already parses this), but sets response.status_code = 207 when
any individual metric computation fails, so callers get a real signal
instead of an indistinguishable 200
- routes/ontology.py: POST /create now raises HTTPException(500) on
generation failure instead of silently falling back to a partial/minimal
ontology and returning 200 with a misleading nodes_added count
Verified via git stash comparison that pre-existing test suite failures
(58 errors, Starlette TestClient/httpx version mismatch) are unrelated to
this change - identical failure count on modified and unmodified code.
Closes#770
Promotes the Unreleased changelog section (Databricks connector, SQLite
vector store, SPARQL CONSTRUCT templates, JenaStore named-graph support)
to 0.6.0 and syncs version references across pyproject.toml, __init__.py,
and docs.
- TemporalGraphQuery.query_time_range() and RDFExporter.export() both
expect {entities/relationships} (or {relationships} with source_id/
target_id keys), not ContextGraph.to_dict()'s {nodes, edges} shape.
Map the output before passing it in, and add an actual temporally-
bounded edge to the Temporal Intelligence example so the query has
something to find.
- add_causal_relationship() only accepts relationship_type values of
CAUSED, INFLUENCED, or PRECEDENT_FOR; replace the invented "triggers"/
"enables" values used in Decision Intelligence and the audit-trail
recipe, which would otherwise raise ValueError immediately.
PipelineBuilder.add_step() returns the created PipelineStep, not the
builder, so chaining .add_step().add_step() raised AttributeError.
Only connect_steps() and set_parallelism() return the builder and can
be chained.
Consolidates the platform reference into a single, premium README with
collapsible module/recipe sections so the docs and the deep-dive reference
no longer live in two places. Every code example was checked against the
actual semantica/ source and corrected where the API had drifted:
resolve_conflicts, register_source, ValidationResult.valid, clean_data,
execute_pipeline, ParquetExporter/LPGExporter/ReportGenerator calls,
graph.to_dict(), TemporalGraphQuery/TemporalNormalizer usage, the
Reasoner/ExplanationGenerator API, and the REST endpoint paths. Also
removed duplicated titles, snippets, and repeated example scenarios that
had crept in during the merge.
The CodeQL Analyze Python job failed on the #757 merge commit with
ECONNRESET while streaming the CodeQL bundle download in
codeql-action/init's "Setup CodeQL tools" step. This is unrelated to
the merged code — it's a known, currently-unaddressed gap in
codeql-action: the download error is retryable but the action doesn't
retry it internally (confirmed via codeql-action's issue tracker and
changelog).
Since a `uses:` step can't be wrapped by a shell-level retry action,
Initialize CodeQL now runs up to 3 times, cascading to the next
attempt only if the previous one failed, so the common case (success
on attempt 1) costs nothing extra.
Dataset.remove() on a bare 3-tuple resolves context=None internally,
which the underlying store treats as a wildcard and deletes the
matching triple from every graph, not just the default graph the
docstring promises. Pass self.graph.default_graph explicitly as the
context so delete_triplet stays scoped to the default graph, matching
the isolation guarantee default_union=False is meant to provide.
Also corrects a misleading comment: SPARQLStore is graph_aware=True
too, so graph-awareness isn't what requires SPARQLUpdateStore here —
it's SPARQLStore being read-only (.add()/.remove() raise TypeError).
Adds regression tests and a CHANGELOG entry for PR #757.
Updates the tagline, adds Ontology Management/SKOS to the feature pills, swaps
the yellow-highlight subhead for a cleaner italic style, and surfaces a
regulated-domains teaser linking to the existing "Built for High-Stakes
Domains" section.
Trims the README from a full module/API dump into a scannable pitch (hero,
why-Semantica, quick start, architecture, decision intelligence, one flagship
audit-trail recipe) and moves the exhaustive per-module reference, extra
recipes, and full integrations matrix into a new PLATFORM_REFERENCE.md.
1. Endpoint derivation regression: Detect if self.endpoint already contains
a Fuseki service suffix (/query, /update, /sparql) to prevent double-appending
(e.g., /ds/query/query). If it does, derive the base and construct both
paths properly.
2. Misleading serialize warning: Limit the named-graph data loss warning
to single-graph serializer formats (turtle, xml, n3, etc.). Multi-graph
formats (trig, nquads, nt) will correctly serialize all graphs without warning.
3. Zero-added error misdiagnosis: Track malformed triples accurately in
add_triplets(). If every triplet fails the local validation (ValueError/
AttributeError), raise a formatting-oriented ProcessingError instead of
assuming a store connectivity issue.
Includes comprehensive regression tests for all three cases.
Migrates JenaStore from rdflib.Graph to rdflib.Dataset with default_union=False
explicitly set, per maintainer-confirmed architecture for issue #756.
Changes:
- _initialize_graph: construct self.graph as Dataset(default_union=False) for
the in-memory path, and Dataset(store=SPARQLUpdateStore(...), default_union=False)
for the remote path. SPARQLUpdateStore.graph_aware=True satisfies Dataset's
hard requirement. Both paths verified against rdflib source.
- add_triplets: accept and honor graph= option. When supplied, Dataset.graph(uri)
creates/retrieves the named-graph context and the triple is written via a
4-tuple (which SPARQLUpdateStore maps to INSERT DATA { GRAPH <uri> { ... } }).
When graph= is omitted, the 3-tuple path routes to Dataset's default graph,
preserving pre-migration semantics exactly.
- serialize: add WARNING log when named-graph content would be silently dropped
by a single-graph serializer (turtle/xml/n3). Log includes triple count and
recommends trig/nquads formats. No warning when only the default graph is used.
- create_model: document that triplet_count now counts triples across all graphs
(default + named) as a consequence of this migration. Semantics shift made
visible, not silent.
- delete_triplet: document that graph= parity is a known gap, deferred to a
future follow-up per maintainer's stated scope (add_triplets only).
Decisions applied:
1. triplet_count semantics shift: documented in create_model docstring
2. delete_triplet graph= parity: explicitly out of scope, noted in docstring
3. Existing store.graph=Graph() tests: left unchanged; new tests added
to cover the real _initialize_graph path
Tests added (TestJenaStoreDatasetMigration):
- test_initialize_graph_produces_dataset_not_graph
- test_initialize_graph_dataset_has_default_union_false
- test_add_triplets_with_graph_option_writes_to_named_graph
- test_add_triplets_without_graph_option_writes_to_default_graph
- test_add_triplets_named_graph_isolated_from_default_query
- test_serialize_logs_warning_when_named_graph_content_present
- test_serialize_no_warning_when_only_default_graph_used
Also updated test_add_triplets_remote_endpoint_fires_insert_data_via_update_store
to patch Dataset instead of Graph (the remote path now creates Dataset(store=...)).
Full suite: 269 passed, 0 failed (tests/triplet_store/ + tests/pipeline/)
The exception-propagation comment and Raises docstring in
execute_construct_template stated that add_triplets signals failure
exclusively via a returned dict. This became stale after the JenaStore fix
(previous commit) which introduced ProcessingError propagation for complete
batch failures.
Updated to document both paths:
- dict-based failure: success=False in returned dict (BlazegraphStore, RDF4J, etc.)
- raised ProcessingError: JenaStore full-batch failure now raises directly
No logic changed. 262 tests pass.
The remote-endpoint path in _initialize_graph was instantiating the read-only
rdflib SPARQLStore, causing every add_triplets() call against a remote Fuseki
endpoint to silently fail: SPARQLStore.add() raises TypeError which was swallowed
by the broad except Exception per-triplet handler and returned as success=True/added=0.
Changes:
- Import SPARQLUpdateStore alongside SPARQLStore
- _initialize_graph: use SPARQLUpdateStore(query_endpoint=<base>/query,
update_endpoint=<base>/update) per standard Fuseki REST API conventions
- Fix constructor: self.endpoint=config.get('endpoint') always returned None
because the named positional 'endpoint' param captures the kwarg before **config;
now uses endpoint or config.get('endpoint')
- Narrow per-triplet except to (ValueError, AttributeError); add ProcessingError
when entire batch fails to prevent misleading success=True/added=0 return
Tests added (TestJenaStoreRemoteEndpointUsesUpdateStore): 4 new test cases
Full suite: 262 passed (tests/triplet_store/ + tests/pipeline/)
_format_object_for_ntriples decided IRI vs. literal purely from the
presence of datatype/lang metadata, defaulting anything without it to
<obj>. Any plain literal object (e.g. typical NER/extraction output
like "Alice", or an untyped Turtle literal round-tripped through the
new CONSTRUCT path) was wrapped as an invalid IRI instead of a quoted
literal, diverging from BlazegraphStore's _is_uri_value-first check.
Port _is_uri_value from BlazegraphStore so RDF4JStore checks whether
the object is actually URI-shaped before falling back to literal
handling, with a plain-quoted-literal fallback instead of <obj>.
Extends CONSTRUCT support to JenaStore, which uses rdflib.Graph natively rather
than an HTTP protocol - CONSTRUCT results come as native 3-tuples with no
Accept-header/parsing dance needed, unlike Blazegraph/RDF4J.
- CONSTRUCT-aware execute_sparql: reuses shared sparql_escaping.CONSTRUCT_QUERY_RE,
extracts datatype/language from rdflib Literal objects into the same 4-tuple
metadata contract used by Blazegraph/RDF4J
- Non-CONSTRUCT path (SELECT/ASK) confirmed byte-for-byte unchanged (Property 9)
- execute_construct_template confirmed backend-agnostic against JenaStore, zero
changes needed
- Named-graph support explicitly out of scope - JenaStore wraps a single
rdflib.Graph with no named-graph concept; add_triplets continues to silently
ignore graph= exactly as before. Tracked separately as a follow-up issue
requiring a Graph -> ConjunctiveGraph/Dataset migration.
Extends the Blazegraph-only CONSTRUCT support from #322 (commit 4f2c6c82's
approved pattern) to RDF4JStore:
- CONSTRUCT-aware execute_sparql: Accept: text/turtle, rdflib Turtle parsing,
4-tuple (s, p, o, metadata) contract with datatype/language preservation
- Named-graph writes via RDF4J's REST context parameter, N-Triples-encoded
(angle-bracket-wrapped IRI), confirmed against RDF4J's Protocol.java source
- graph=None preserves existing behavior exactly (no context param sent,
not context=null - verified as a distinct, deliberate choice)
- _CONSTRUCT_QUERY_RE moved to sparql_escaping.py as a shared, backend-agnostic
constant; Blazegraph now delegates to it, zero behavioral change confirmed
- execute_construct_template (construct_templates.py) required zero changes -
confirmed backend-agnostic via end-to-end integration tests against RDF4JStore
29 new tests, full suite 245/245 passing. Jena support remains out of scope
for this PR - tracked separately in #754's remaining scope.
Replace remaining Hawksight-AI GitHub org links and the old
semantica-dev noreply email with the current semantica-agi org
and kaif@getsemantica.ai contact, so security/support contacts
match pyproject.toml.
Implements #322: ConstructTemplate/ParameterDescriptor/ConstructTemplateRegistry
with injection-safe {{param}} rendering, Blazegraph CONSTRUCT-aware execute_sparql
extension, execute_construct_template (render->execute->parse->persist), and a
construct_template pipeline step. RDF4J/Jena support deferred to a follow-up issue.
Closes#322
Documents the 9 pre-existing test failures caused by never-implemented
kg.ProvenanceTracker compatibility methods, the deprecation fix, and
the follow-up migration guide addition in this PR.
Every deprecation warning added in this PR (and the class docstring)
points to docs/migration/kg-provenance-tracker.md, but the file was
never added, so the reference was dead. Adds the guide with a
method-mapping table to semantica.provenance.ProvenanceManager.
Add graph-apache-age extra (psycopg2-binary) which was previously
undeclared despite age_store.py depending on it, wire it into
graph-all, and document install commands for FalkorDB/AGE/Neptune
alongside Neo4j. Note that RDF triple stores need no extra since they
talk SPARQL over HTTP via the core `requests` dependency. Also align
README's "Triplet Stores" table label to "Triple Stores (RDF)" to
match the standard term used elsewhere in the docs, while keeping the
TripletStore interface name in backticks.
Semantica already ships both an RDF triplet-store stack (Blazegraph,
Apache Jena, Eclipse RDF4J via a unified TripletStore/SPARQL interface)
and an LPG graph-store stack (Neo4j, FalkorDB, Apache AGE, AWS Neptune
via Cypher), but the README only surfaced the LPG side. Add a hero
highlight line, a "What Semantica gives you" bullet, and split the
Features-at-a-Glance table row so both formats and all backends are
named explicitly.
- get_table_lineage() gains include_column_lineage=True, resolving
per-column upstream/downstream references via Unity Catalog's
column-lineage API (one request per column, opt-in)
- DatabricksConnector.connect() now reuses an already-open connection
instead of opening a second one; ingest_table()/ingest_query() only
close the connection they opened themselves, so using the ingestor
as a context manager no longer leaks the connection opened by
__enter__
- get_table_schema()/get_table_lineage()/list_tables() now validate
both catalog and schema are resolved before calling Unity Catalog,
matching list_tables()'s existing catalog check
- 8 new regression tests (35 total)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
Review follow-up: only append archived_history_id to used_entities when
explicit_parent_supplied is True. Previously it was appended unconditionally,
so the no-explicit-parent re-track path ended up with the same history id in
both parent_entity_id and used_entities, duplicating the reference in
get_lineage() output.
- get_lineage() aggregated metadata by iterating trace_lineage()'s BFS
order and calling dict.update() on each entry, so ancestor metadata
(now reachable via derived_from chains) could overwrite the queried
entity's own metadata on conflicting keys. Reverse the iteration so
the queried entity (always lineage_entries[0]) is applied last and
wins, matching the documented "most recent entry's metadata takes
precedence" intent.
- track_entity()'s derived_from guard only accepted a concrete dict,
silently ignoring other collections.abc.Mapping implementations
(e.g. types.MappingProxyType). Switch the isinstance check to
Mapping so any mapping-like metadata is honored.
Addresses Qodo review findings on PR #741.
track_entity() only auto-linked a parent by looking up `source` as an
existing entity_id, so two entities sharing a real source URL (e.g. a
document and a decision derived from it) never got connected, and
metadata["derived_from"] was stored but never consulted by any linking
or traversal code.
track_entity() now treats metadata["derived_from"] as an explicit
parent link (unless parent_entity_id was already passed directly), so
the existing BFS in trace_lineage() picks it up for free.
Closes#735
Rule is a mutable dataclass, so an already-registered rule's priority
could change after being added; the dedup early-return skipped the
priority re-sort, so re-adding a rule after mutating its priority
left self.rules stale relative to that change. The duplicate branch
now re-sorts before returning, matching the append path.
- add_rule()'s duplicate-skip path now logs at warning level instead
of debug, so a skipped duplicate is visible by default rather than
silent in typical logging configs
- The duplicate-rule log message now stringifies conditions via
map(str, ...) before joining, since Rule.conditions is List[Any]
and non-string entries would otherwise raise TypeError
add_rule() unconditionally appended to self.rules, so re-running the
same setup code on an existing Reasoner instance (e.g. re-executing a
Jupyter cell) duplicated every rule; forward_chain() would then match
the duplicated rules but silently return no new results since the
conclusions were already in self.facts, with no error or warning.
add_rule() now compares an incoming rule's rule_type, conditions, and
conclusion against existing rules and returns the existing Rule
instead of appending a duplicate, keeping repeated add_rule() calls
with the same definition idempotent.
* Add shacl extra to pyproject.toml (fixes#736)
* docs(changelog): add entry for shacl extra fix (#736)
by @Sameer6305
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
'By default, initializing ... with storage_path=...' read as if passing
storage_path were the default, contradicting the very next sentence
about the no-argument in-memory default. Rephrased so the in-memory
default isn't undercut by the first sentence.
* docs: improve SHACL validation guide onboarding and workflow guidance
* docs: fix SHACL validation implementation mismatches
* docs: fix stale violation URIs and drop unused imports in SHACL guide
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
Condense the CLI section to the essential install/usage snippet and
command groups, pointing to docs.getsemantica.ai for the full
reference instead of maintaining static terminal mockups in the README.
Consolidate the hero around a single category claim (Context and
Accountability Layer for AI agents), drop the named-competitor
comparison table (LangChain/LlamaIndex/Mem0/Zep/Palantir Foundry),
remove GitHub alert-box tips/notes, cut redundant module/changelog
sections, strip vanity feature counts, and trim em dashes for a
cleaner, more premium read.
- Add SQLITE_VEC_AVAILABLE flag via importlib.util.find_spec so the test
suite's skipif actually reflects whether sqlite-vec is installed; it was
previously undefined, causing all sqlite vector store tests to be
silently skipped regardless of installation state.
- Actually apply PRAGMA synchronous=NORMAL alongside journal_mode=WAL when
use_wal=True, matching the documented behavior; document use_wal as an
opt-in kwarg in the docstring and usage guide.
- Correct _is_safe_identifier error messages (regex never allowed hyphens).
- Batch get() and update() with IN(...)/executemany instead of per-id
round trips, consistent with add()/delete().
- Fix flaky test_update_vectors assertion that relied on list.index()
over dicts containing numpy arrays.
- Reorder sqlite_vec_store import alphabetically in vector_store/__init__.py.
Co-Authored-By: Luffy2208 <209925020+Luffy2208@users.noreply.github.com>
Revert badge rows to flat-square (for-the-badge rendered as
mismatched oversized blocks), convert the subtitle to a native
blockquote for GitHub's built-in muted-grey text styling, and
trim "The" from the tagline.
Update the hero tagline, subtitle, and comparison table to frame
Semantica as Palantir-grade knowledge/decision intelligence that is
open source, self-hostable, and priced for startups through
Fortune 500, not just enterprise budgets.
* docs: improve conflict resolution guide onboarding and workflow guidance
* docs: fix conflict resolution implementation mismatches
* docs: correct credibility-weighted example output values
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
The merging example told readers to use merge_entity_group() for
already-confirmed duplicate groups, but the code right below it still
called merge_duplicates() on group.entities, which re-runs duplicate
detection redundantly. Update the call to match the stated guidance.
- Correct the unsupported-rule-key pitfall: absent keys fail compliance,
present keys (any value) pass — the previous wording had this backwards.
- Remove required_ltv/pd/lgd/dsti/credit_score: True from the mortgage
example. required_* checks equality against the given value, so True
against a real numeric field silently marks compliant decisions as
non-compliant (verified: a fully passing decision still returned False).
The min_/max_ rules already enforce presence of ltv/dsti/credit_score.
* docs: improve export guide onboarding and workflow guidance
* docs: fix export guide implementation mismatches
* docs: revert .content to .text in export guide examples
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
* docs: improve agent memory guide onboarding and usage guidance
* docs: align Agent Memory guide with persistence implementation
* docs: fix misleading index_path persistence claim across guides
VectorStore's index_path kwarg is silently absorbed into FAISSStore's
**config and never read anywhere in faiss_store.py, so it does not make
the FAISS index persist across restarts as several docs implied. Real
persistence requires an explicit VectorStore.save()/.load() call, or
AgentContext.save()/.load() which cascades to it.
- docs/reference/context.md: rewrite the "Persist your vector store"
tip to explain the actual save()/load() mechanism instead of the
dead index_path kwarg.
- docs/guides/graphrag.md, decision-intelligence.md, ingest.md,
semantic-extraction.md: drop the dead index_path=... kwarg from
VectorStore(backend="faiss", ...) constructor calls.
Follow-up to #691, which fixed the same false claim in
docs/guides/agent-memory.md but missed these other files.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
@@ -69,5 +69,5 @@ If you have ideas on how this could be implemented, please share.
---
**Note**: For feature requests that are ready to be implemented, consider creating a [Feature Request issue](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md) instead.
**Note**: For feature requests that are ready to be implemented, consider creating a [Feature Request issue](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md) instead.
@@ -46,8 +46,8 @@ If applicable, paste any error messages or describe unexpected behavior:
## Checklist
- [ ] I have searched existing [discussions](https://github.com/Hawksight-AI/semantica/discussions) and [issues](https://github.com/Hawksight-AI/semantica/issues)
- [ ] I have checked the [documentation](https://github.com/Hawksight-AI/semantica/tree/main/docs) and [FAQ](https://github.com/Hawksight-AI/semantica/blob/main/docs/faq.md)
- [ ] I have searched existing [discussions](https://github.com/semantica-agi/semantica/discussions) and [issues](https://github.com/semantica-agi/semantica/issues)
- [ ] I have checked the [documentation](https://github.com/semantica-agi/semantica/tree/main/docs) and [FAQ](https://github.com/semantica-agi/semantica/blob/main/docs/faq.md)
- [ ] I have provided a minimal code example (if applicable)
- [ ] I have included error messages (if applicable)
Check the [docs folder](https://github.com/Hawksight-AI/semantica/tree/main/docs) and [README](https://github.com/Hawksight-AI/semantica/blob/main/README.md) for guides and examples.
Check the [docs folder](https://github.com/semantica-agi/semantica/tree/main/docs) and [README](https://github.com/semantica-agi/semantica/blob/main/README.md) for guides and examples.
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — wait for the issue to be assigned to you before opening a PR, to avoid duplicate work.
## Description
<!-- Provide a clear description of your changes -->
# Local composite actions (./x) and Docker image refs (docker://...) use a
# different pinning mechanism and aren't in scope here.
[["$content"=~ uses:\ +\./ ]]&&continue
[["$content"=~ uses:\ +docker:// ]]&&continue
if[["$content"=~ $USES_PATTERN]];then
repo="${BASH_REMATCH[1]}"
ref="${BASH_REMATCH[3]}"
checked=$((checked +1))
if[[ ! "$ref"=~ ^[0-9a-fA-F]{40}$ ]];then
echo"::error file=$file,line=$lineno::$repo is pinned to '$ref', not a full commit SHA. Mutable tags/branches can be silently re-pointed (see the LiteLLM/Trivy 2026 incident) - pin to a commit SHA instead."
echo"::warning file=$file,line=$lineno::$repo@$sha has no trailing '# vX' comment recording which tag it corresponds to - add one for auditability."
continue
fi
resolved=$(gh api "repos/$repo/commits/$tag" --jq '.sha' 2>/dev/null)
if[[ -z "$resolved"]];then
echo"::error file=$file,line=$lineno::Could not resolve '$repo@$tag' via the GitHub API (rate limit, deleted tag, or typo). Treating as unverifiable = failure."
fail=1
continue
fi
if[["$resolved" !="$sha"]];then
echo"::error file=$file,line=$lineno::$repo is pinned to $sha but tag '$tag' now resolves to $resolved. Update the pin or the comment."
# Guard 1: fail loudly if Safety exited before writing a report at all
# (network error, API auth failure, tool crash). Without this check a
# missing or empty file causes jq to fall back to "0", making a broken
# scanner indistinguishable from a clean scan.
if [ ! -s safety-report.json ]; then
echo "::error::Safety scan produced no report (safety-report.json is missing or empty). Treating as failure — check for network errors, API auth failures, or Safety crashes in the logs above."
# Guard 2: ensure VULNS is a non-negative integer before the -gt
# comparison. "null" (missing/null key) or "" (jq parse failure) would
# cause bash's -gt to throw an arithmetic error and fall through to the
# success branch — the same silent-pass bug as a missing file.
if ! [[ "$VULNS" =~ ^[0-9]+$ ]]; then
echo "::error::Safety report exists but 'vulnerabilities' is missing or non-numeric (got: '${VULNS}'). The report may be malformed or Safety may have written an error-only JSON. Treating as failure."
exit 1
fi
if [ "$VULNS" -gt 0 ]; then
echo "❌ Security vulnerabilities found: $VULNS"
echo "CI will fail to prevent merging of vulnerable dependencies"
lines.push('<details>', '<summary>Show all findings</summary>', '');
lines.push(...items);
lines.push('', '</details>');
} else {
semgrepResults = '## No Security Patterns Found\\n';
lines.push(...shown);
}
} catch (e) {
semgrepResults = '## Semgrep scan completed\\n';
return lines.join('\n');
}
// Create summary comment
const comment = `# 🔒 Security Scan Results\\n\\n${safetyResults}\\n\\n${banditResults}\\n\\n${semgrepResults}\\n\\n---\\n\\n*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*\\n\\n📊 **Security Policy**: CI fails on vulnerabilities and HIGH severity issues.`;
.map((issue) => `- \`${issue.test_name}\` in \`${issue.filename}:${issue.line_number}\``)
);
const semgrepSection = renderSection(
'Semgrep — static analysis patterns',
'semgrep-report.json',
(data) => (data.results || []).map(
(issue) => `- \`${issue.check_id}\` in \`${issue.path}:${issue.start?.line ?? '?'}\``
)
);
const comment = [
'# 🔒 Security Scan Results',
'',
safetySection,
'',
banditSection,
'',
semgrepSection,
'',
'---',
'',
'*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*',
'',
'📊 **Security Policy**: CI fails on Safety vulnerabilities and Bandit HIGH-severity findings. Semgrep findings above are informational and do not block merge.',
- New `pip install semantica[langchain]` extra (`langchain-core>=0.3.0`), included in the `all` bundle
- `integrations/langchain/SemanticaRetriever` — LangChain `BaseRetriever` that seeds from `HybridSearch` then walks graph edges (`hops=2` default) for GraphRAG-style retrieval; falls back to `ContextGraph.query` when hybrid search is unavailable
- `integrations/langchain/SemanticaKGTool` / `SemanticaDecisionTool` — `BaseTool` subclasses with Pydantic `args_schema` (`semantica_query_graph`, `semantica_query_decisions`); `build()` returns the tool, or `None` when langchain-core is absent
- Retriever and VectorStore read HybridSearch nested `metadata` (`content`, `node_id`, `node_type`) rather than top-level fields that HybridSearch does not set
- All adapters remain importable without langchain-core (`LANGCHAIN_AVAILABLE` flag)
- Docs: `docs/integrations/langchain.md`, README native-integration matrix, and `docs.json` nav entry
## [0.6.6] - 2026-08-20
### Added
- **Semantica RDF vocabulary, and deterministic entity/relationship IRIs** (#1109, closes #1107, closes #1101) by @fabio-rovai, reviewed by @KaifAhmad1
- Every RDF/JSON-LD export mints terms in `https://semantica.dev/ns#`, and until now nothing declared what those terms meant — the namespace 404s and no vocabulary shipped with the package, so a consumer receiving an export had no way to tell `sem:text` from a typo of it, and no closed-world checker could validate an export at all
- `semantica/ontology/vocabulary/semantica-ns.ttl` declares the terms the exporters actually emit — drawn from the emitting call sites in `export/rdf_exporter.py`, `export/json_exporter.py` and `provenance/manager.py`, not from what a vocabulary "ought" to contain. Ships inside the package (`from semantica.ontology.vocabulary import vocabulary_turtle`) so it loads without a network round trip, and is the same document intended to be served at the namespace IRI once hosting/content-negotiation is sorted
- `tests/ontology/test_vocabulary.py` ties the document to the code: every term a serializer can write must be declared, so adding a term to an exporter without declaring it fails the build
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
- **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path
- `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI
- 236 export and ontology tests pass
- **First-class CrewAI integration** (#988, closes #962) by @Shindevrp
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
- `integrations/crewai/SemanticaDecisionTool` — a CrewAI `BaseTool` wrapping `AgentContext` with 5 decision-intelligence actions (`record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`)
- `integrations/crewai/SemanticaKnowledgeSource` — a CrewAI `BaseKnowledgeSource` that serializes a `ContextGraph` into crew knowledge storage; implements both the legacy `load_content()` and current `validate_content()`/`aadd()` contracts so it works across `crewai>=0.80.0`
- All three classes degrade gracefully when `crewai` is not installed (still importable, full Semantica API available)
- New `tests/integrations/crewai/`: 70 tests covering stub-based present-case behavior (Pydantic/BaseTool subclassing, every action, knowledge-source chunking/storage) plus a subprocess isolation test for the crewai-absent degradation path
- Docs: `docs/integrations/crewai.md` page, `docs.json` Integrations nav entry, and README integration-matrix/install updates
- **Hardened during code review**: live `graph`/`context`/extractor state is excluded from CrewAI JSON serialization (`model_dump(mode="json")`) with `model_post_init` self-healing defaults, so checkpoint/resume no longer raises `PydanticSerializationError`; `query_graph` now searches node content (not just ids/types); `trace_causal_chain` returns an explicit error instead of substituting similarity precedents when causal tracing is unavailable, and calls `trace_decision_causality(..., max_depth=...)` with the correct argument name; `find_precedents` propagates `max_precedents` as the backend `limit`; `add_to_graph` writes are serialized under a module lock so concurrent agents can't double-count duplicate adds; nameless entities are skipped instead of creating `repr()`-junk nodes
- **Hardened during second code review**: `check_policy` rules are now coerced type-aware — `bool("false")` was truthy, so `enabled == false` reported a violation for `enabled: false`, and string datums like `"0.90"` were compared lexicographically instead of numerically; `trace_causal_chain` no longer raises `AttributeError` (which escaped the tool) when the decision context has no `knowledge_graph`, returning honest error JSON instead; knowledge-source storage failures log an actionable ERROR (a missing crew embedder otherwise silently left agents with empty retrieval); `add_to_graph` uses a per-graph re-entrant lock instead of a process-global one (independent graphs no longer serialize each other, and re-entrant extractors can't deadlock); entity/relation `confidence=None` normalizes to `1.0` instead of failing the whole extraction; added a subprocess integration test against the real `crewai` package covering `Crew`-level serialization round-trip and restore
- **`ContextGraph` gains retraction and purge — the graph previously had no way to remove a node or edge without discarding everything via `clear()`** (#957, closes #955) by @pravit-amp, reviewed by @KaifAhmad1
- `retract_node()`/`retract_edge()` close an entity's validity window rather than deleting it, reusing the existing `valid_from`/`valid_until`/`state_at()` machinery: the entity drops out of `find_active_nodes()` and future `state_at()` queries going forward, but `state_at()` calls before the retraction time still return it, so decisions recorded against it stay explainable. A `("kind", id)`-keyed retraction record captures who/why/when, retrievable via `get_retraction()`/`list_retractions()`
- `purge_node()`/`purge_edge()` are the destructive counterpart: the entity is removed outright, from history as well as the active view, for erasure obligations retraction alone cannot satisfy (e.g. GDPR Article 17). Only a tombstone remains — that a purge happened, when, and why — deliberately never the purged content, via `get_tombstone()`/`list_tombstones()`. Purge is graph-scope only: `AgentMemory` and any bound vector store are not reached, so it is one step of an erasure workflow rather than the whole of it
- Both operations default to `cascade=True` (also touching every incident edge, and for `purge_node`, the marker node of any cross-graph link the node exits through) since leaving edges active around an inactive/removed node produces an inconsistent active view or dangling endpoints; both accept `cascade=False` for callers that want to handle edges themselves
- Both are idempotent: retracting/purging an already-retracted/purged entity returns `False` rather than raising, and a repeat retraction preserves the original record's reason rather than overwriting it
- Retraction/purge closing a validity window never widens an existing one — a node or edge added with `valid_until` already in the past keeps that earlier bound rather than being pushed later by a subsequent retraction time
- Reuses the existing audit-trail path with no changes to `change_management`: `MutationRecord` already documented `REMOVE_NODE`/`REMOVE_EDGE` in its operation vocabulary; retraction now emits `UPDATE_NODE`/`UPDATE_EDGE`, purge emits `REMOVE_NODE`/`REMOVE_EDGE`, matching the documented contract. Mutation payloads are snapshotted inside the lock and the callback fires after it is released, so a callback that itself mutates the graph (e.g. `clear()`) can't observe or lose in-flight records
- **Fixed during review** (@KaifAhmad1): `retract_edge()`/`purge_edge()` resolved "the edge" for a given `edge_id` via the first matching object only. `edge_id` is content-derived and, prior to #926, was not guaranteed unique — a graph holding two identical `add_edge()` calls had two edge objects sharing one id. A direct `retract_edge()`/`purge_edge()` call would silently leave the second duplicate untouched (still live, still active) while returning `True` and recording a tombstone/retraction that claimed the edge was fully handled; repeat `purge_edge()` calls also silently overwrote the tombstone's `reason`/`purged_at` on each partial attempt instead of no-op'ing. The same gap let `retract_node()`'s cascade skip a duplicate outright, since it checked the live `_retractions` dict mid-loop and treated the first duplicate's just-written record as proof the second was already handled. `#926` (merged) stops *new* duplicates from being created, but any graph already holding one — loaded from a save made before that fix, or built during the window before it landed — could still trigger this. Now `retract_edge()`/`purge_edge()` act on every edge matching the id under one record, and the cascade's dedup check is snapshotted before the loop starts so within-call duplicates are still closed rather than skipped. 5 new regression tests in `TestDuplicateEdgeId`
- New `tests/context/test_context_graph_retraction.py`: 49 tests, covering retraction/purge semantics, cascade, idempotency, validity-window narrowing, id-keyspace collisions between node and edge ids, cross-graph link teardown, `clear()`/`load_from_file()` resetting retraction/tombstone state, audit-trail integration against a real `TemporalVersionManager`, mutation-emission ordering under a concurrent `clear()`, and concurrent purges
- Full `tests/context/` suite: 533 passed
- **`DistanceExporter.compute_pairs()` gains an opt-in `metric_errors` column to distinguish legitimate `None` results from computation failures** (#960, follow-up to #879) by @Karunasagar12
- Previously, a `None` in `hop_count`/`weighted_distance`/`semantic_similarity`/betweenness could mean either "no path exists" or "the underlying computation raised" — logged as a warning per #879, but not otherwise surfaced, so the two cases were indistinguishable in exported CSV/JSONL/DataFrame data. `include=["metric_errors"]` now adds a `metric_errors` field per row: `""` when all requested metrics succeeded, or a comma-separated list of metric names that raised (e.g. `"hop_count,weighted_distance"`)
- Opt-in only — default `compute_pairs()`/`to_csv()`/`to_dataframe()`/`to_jsonl()` schema is unchanged unless `"metric_errors"` is explicitly requested
- The four metric helpers (`_betweenness`, `_hop_distance`, `_weighted_distance`, `_semantic_similarity`) now return `(value, error_name | None)` tuples internally; `compute_pairs()` aggregates the error names per row
- **Fixed during review** (Qodo): `_betweenness()` failures weren't tracked into `metric_errors` in the initial version — centrality computation could raise and the column would still report `""`. Now returns its error tuple like the other three helpers
- **Known limitation**: `include=["metric_errors"]` with no other metric names computes nothing, so the column is always `""` in that case — pass it alongside the metrics you want tracked, e.g. `include=["hop_count", "metric_errors"]`
- New `tests/export/test_distance_exporter_metric_errors.py`: 6 tests covering success, single/multiple failures, opt-out, the no-path-vs-error distinction, and default-schema stability; existing `tests/export/test_distance_exporter.py` updated for the new tuple return type
- Full `tests/export/` suite: 77 passed
- **`ContextGraph.to_kg_dict()`: an adapter converting a `ContextGraph`'s internal `nodes`/`edges`/`source` shape into the canonical `entities`/`relationships`/`source_id` shape `RDFExporter` and `TemporalGraphQuery` consume** (#1081) by @cxzg007
- Previously there was no supported way to feed a `ContextGraph` into those consumers without hand-rolling the field remapping; `to_kg_dict()` does it once, with an `entities_only` option that drops relationships left dangling by the filter
- **Fixed during review** (Qodo): null `properties`/`metadata` on a node loaded from JSON raised `TypeError` when copied — both are now guarded with `or {}`; entity ids are coerced to `str(node_id)` to match `ContextEdge`'s already-str-coerced endpoints, so valid relationships were no longer dropped by `entities_only` filtering
- `RDFExporter`'s validator and `TemporalGraphQuery` now also accept `source_id`/`target_id` endpoints, the shape `to_kg_dict()` emits
### Changed
- **`GraphBuilder`'s 6 public methods now have Google-style docstrings** (#878, closes #876) by @cakeni
- `semantica/kg/graph_builder.py`'s `build`, `build_single_source`, `add_temporal_edge`, `create_temporal_snapshot`, `query_temporal`, and `load_from_neo4j` — the core knowledge-graph construction API, imported directly by callers — previously had zero docstrings across all 6 methods, the only file in a 10-file audit sample with that gap, despite CONTRIBUTING.md requiring Google-style `Args`/`Returns`/`Raises`/`Example` docs for public methods. Added full docstrings for all 6, plus the previously undocumented `build_single_source`, with runnable (`# doctest: +SKIP`) usage examples
- **Corrected during review**: `query_temporal`'s docstring claimed the query text was used to filter the graph; the implementation only records it in the result (`results = {"query": query, ...}`) with no interpretation or filtering. Corrected to state that explicitly
- **Corrected during review**: `create_temporal_snapshot`'s docstring implied entities were filtered for validity at the snapshot timestamp like relationships are; the implementation copies all entities unfiltered and only filters `relationships` by `valid_from`/`valid_until`. Docstring now distinguishes the two
- **Corrected during review**: `add_temporal_edge`/`create_temporal_snapshot` docstrings overclaimed numeric-timestamp support; `_parse_time()` only special-cases `str` and `datetime`, falling back to a bare `str()` cast for anything else (not true numeric parsing). Narrowed to "datetime or ISO-formatted string"
- **Fixed along the way**: `build()`'s `**options` documented a default only for `extract`; `extract_relations`, `extract_triplets`, `ner_method`, `relation_method`, and `triplet_method` all have concrete defaults in `_extract_from_text()` (`True`, `True`, `"llm"`, `"llm"`, `"llm"`) that were left unstated, inconsistent with CONTRIBUTING.md's own docstring example of noting defaults inline
- **`GraphBuilder` raw-text extraction now defaults to local extractors instead of LLM extraction** (#941, closes #930) by @dex0shubham
- `GraphBuilder._extract_from_text()` defaulted `ner_method`, `relation_method`, and `triplet_method` to `"llm"`, and ran relation extraction unconditionally (`extract_relations` defaulted to `True`) — all four contradicting the defaults documented in the `build()` docstring at the time (`"ml"` / `"pattern"` / `False`), and diverging from the standalone extractors (`NERExtractor` defaults to `method="ml"`, `RelationExtractor` and `TripletExtractor` to `method="pattern"`). The practical effect was that any raw-text `build()` call silently required a configured provider, an API key, and network access
- Defaults are now `ner_method="ml"`, `relation_method="pattern"`, `triplet_method="pattern"`, and `extract_relations=False`, matching the docstring. LLM extraction remains fully available and is now opt-in
- **To restore the previous behaviour**, pass the methods explicitly:
```python
builder.build(
sources,
ner_method="llm",
relation_method="llm",
triplet_method="llm",
extract_relations=True,
)
```
- #878 landed in the meantime and resolved the same mismatch in the opposite direction, documenting the LLM values (`"llm"` / `"llm"` / `"llm"`, `extract_relations: True`) as the contract. Per the decision on #930 the code is the side that changes, so those docstring defaults are corrected here to `"ml"` / `"pattern"` / `"pattern"` / `False`, keeping #878's formatting
- Removed the stale `# Default to LLM methods as per requirement` comment, which read as an intentional decision but did not match the documented contract
- **Fixed along the way**: `_extract_from_text()` constructed a fresh extractor for every text, and `NERExtractor.__init__` loads its spaCy model eagerly when the method includes `"ml"` — so with the new default, a multi-document build would have reloaded the model once per source. Extractors are now built once per `(kind, method)` and reused for the lifetime of the builder, via `GraphBuilder._get_extractor()`. This path was previously unreachable by default because the old `"llm"` default never touched spaCy
- **Fixed along the way**: `_extract_from_text()` never forwarded its extracted relations to triplet extraction — it passed only `entities=`, so `TripletExtractor` re-derived relations itself (via a method taken from `triplet_method`) whenever `relations is None`, duplicating work and producing triplets that could disagree with the relations already extracted using `relation_method`. Relations are now passed through as `relations=`; when relation extraction is disabled or fails, `None` is forwarded and `TripletExtractor` keeps its existing self-derivation behaviour
- **Fixed along the way**: `GraphBuilder._extraction_stats` was only initialised inside `build()`, so calling `_extract_from_text()` directly raised an `AttributeError` that the extraction path's broad `except` swallowed and reported as `"Entity extraction failed"`. It is now seeded in `__init__` as well; `build()` still resets it per run
- New regression coverage in `tests/kg/test_graph_builder_extraction_defaults.py` pinning all four defaults, verifying that no default resolves to `"llm"`, confirming explicit LLM opt-in still routes correctly, asserting extractors are constructed once across repeated texts, covering fallback method lists (e.g. `ner_method=["pattern", "ml"]`) for all three extractors, asserting relations are forwarded to triplet extraction (and that `None` is forwarded when relation extraction is disabled or fails), and running the real default path end to end with no provider mocked. Verified to fail against the pre-fix code
- Full `kg` suite: 473 passed
- **Explorer graph canvas now renders edge labels** (#1013, closes #1009) by @yzxcj797
- `GraphCanvas.tsx` had no edge-label rendering path at all; Sigma's edge-label renderer draws `data.label`, but the graph state stored the relationship type under `edgeType`, so simply enabling the renderer would have left every edge blank. `graphSceneState`'s edge reducer now maps `edgeType` onto `label` (suppressed for hidden edges)
- Rendering is gated behind a new `edgeLabelsEnabled` entry in the Effects panel (default on), wired through the existing `GraphEffectToggle`/`GraphEffectsState` plumbing, so dense graphs can still turn labels off
- **Fixed during review** (Qodo): two follow-up passes closed gaps the first cut left — label rendering wasn't wired through `explorationEffectsPluginPhaseC.tsx`'s Phase C variant, and toggling the effect off mid-session didn't clear already-rendered labels
- New coverage in `explorer/tests/graphSceneState.display.test.ts`
- **Removed `GraphWorkspaceShell.tsx`, `GraphRuntimeStage.tsx`, and `useGraphData.ts` — a second, unused implementation of the graph-loading/error-handling logic already fixed in `GraphWorkspace.tsx`** (#984, resolves the cleanup tracked in #981 by #980's review note) by @lakshayxi
- 1,564 lines removed; the surviving `GraphWorkspace` path is now the only implementation, so the "two copies that drifted apart" root cause #980 fixed can't recur in the copy nobody was maintaining
- **Explorer README and `docs/explorer-setup.md` corrected to describe the authentication 0.6.5 actually shipped**, plus a documented `/ws/graph-updates` auth note (#1040, fixes #1028) by @Kyou12138
- Both docs still claimed the Explorer API had no built-in authentication after v0.6.5 added mandatory `SEMANTICA_API_KEY` enforcement with a `503` fail-closed default; corrected to describe the actual behavior, including that only protected routes require the key (`/api/health`/`/api/info` stay open), the non-loopback-bind CLI warning only fires in anonymous mode or when the key is unset, and `SEMANTICA_API_KEY`/`SEMANTICA_ALLOW_ANONYMOUS` are documented in the environment-variable table
- **CI: pinned `github/codeql-action` to current v4** (#986) by @ZohaibHassan16, and **pinned Python dependencies in `requirements-ci.txt` for reproducible CI runs** (#945) by @yunaremaia, closing the gap where an unpinned CI dependency could silently change behavior between runs
- **README now states up front that Semantica's explainability is system-level, not foundation-model-internal** (#1033, #1034) by @KaifAhmad1
- Nothing in the README previously scoped what "explainable" meant, leaving readers to assume Semantica could expose or reconstruct an LLM's internal reasoning. A callout now states explicitly that Semantica explains and audits what the AI *system* did — context fed in, decisions produced, provenance, relationships, policies applied — not the model's private internal reasoning, and moved the note near the top of the README rather than leaving it implicit
### Fixed
- **KG provenance tests asserted on generated ID strings instead of stored records, and `kg_provenance.py` was missed by the `utcnow` sweep** (closes #946) by @pravit-amp
- The KG workflow and integration suites checked that a tracker call returned an ID matching a prefix (`assert cent_id.startswith("centrality_")`) without ever reading the record back, so an ID generator that returned a well-formed string and wrote nothing would have passed. Worse, some of those calls named tracker methods that do not exist anywhere in `semantica/` (`track_layer_analysis`, `track_centrality_score`), so the assertions were satisfied with no real interaction behind them
- Those tests now read provenance back through `get_provenance()` and assert on algorithm metadata, and call the methods that actually persist records. Verified by mutation rather than by a green run alone: neutering the manager's storage write (`self.storage.store(...)` → no-op) fails 10 tests
- `GraphBuilderWithProvenance` in `semantica/kg/kg_provenance.py` still stamped `activity_started_at_time`/`activity_ended_at_time` with the deprecated `datetime.utcnow()`; it was outside the `export/`+`provenance/` scope of the #1114 sweep below and now uses the same `utc_now_iso()` helper. `docs/guides/provenance.md` and `docs/reference/provenance.md` were still documenting `utcnow()` and a naive timestamp example, and now show the helper and the offset-bearing form
- 16 tests across the affected suites ended in `return <value>` instead of asserting, which pytest reports as `PytestReturnNotNoneWarning`; now zero
- **The temporal-evolution `stability` metric was a hardcoded placeholder, not a duration**
- `TemporalGraphQuery.analyze_evolution()` documents `stability` as a "relationship duration/stability measure", but the implementation appended a constant `1` for every relationship with both `valid_from` and `valid_until` set (`durations.append(1) # Placeholder`). The reported stability was therefore always `1.0` when any bounded relationship existed and `0` otherwise — it never reflected how long relationships actually stayed valid, so it could not distinguish a graph of decade-long relationships from one of one-second relationships
- `stability` now computes the mean valid-time duration in seconds (`(valid_until - valid_from).total_seconds()`) across relationships that have both bounds set. Relationships with a missing or open `valid_from`/`valid_until` are skipped (their duration is unbounded), and non-positive intervals are clamped to `0`; an empty set still reports `0`
- New tests in `tests/kg/test_kg.py` assert the mean-duration result, the skipping of unbounded/half-open intervals, and the empty-graph zero case
- **Every timestamp an export or a provenance record wrote was timezone-naive** (closes #1114) by @fabio-rovai
- `semantica/export/` stamped with `datetime.now().isoformat()`, which reads the machine's **local** clock; `semantica/provenance/` stamped with `datetime.utcnow().isoformat()`, which reads **UTC**. Both produce a naive value and both serialize identically, so nothing downstream can tell which zone a given timestamp belongs to — the same string means two different instants depending on which module wrote it
- In RDF the consequence is silent rather than loud. Under XSD 1.1 a value with no timezone compared against one with a timezone is indeterminate whenever the two fall inside the ±14 hour window; SPARQL turns an indeterminate comparison into an error, and `FILTER` discards errors as non-matches. A timezone-qualified query over an Oxigraph store returns an answer with every Semantica-written record quietly absent from it, which is a poor property for `prov:generatedAtTime`, `prov:startedAtTime`, `prov:endedAtTime` and `prov:atTime` to have
- New `utc_now()`/`utc_now_iso()` in `semantica/utils/helpers.py`, exported from `semantica.utils`, and used at all 29 call sites in `export/` (`json_exporter`, `yaml_exporter`, `report_generator`, `export_provenance`) and `provenance/` (`manager`, `schemas`, `bridge_axiom`). Values now read `2026-08-19T14:19:04.229937+00:00`: one unambiguous instant, comparable against any correctly stamped value, and valid `xsd:dateTimeStamp`. `sem:exportedAt`'s range in `semantica/ontology/vocabulary/semantica-ns.ttl` is tightened from `xsd:dateTime` accordingly, and its comment no longer has to explain why the weaker range was necessary
- `datetime.utcnow()` is deprecated as of Python 3.12 and scheduled for removal; constructing a `ProvenanceEntry` under `-W error::DeprecationWarning` on 3.13 raised, and no longer does
- New `tests/export/test_timestamp_timezones.py` and `tests/provenance/test_timestamp_timezones.py`: offset presence on every export and provenance path, PROV-O literals valid as `xsd:dateTimeStamp`, comparison against a timezone-aware instant without `TypeError`, the Oxigraph filter that dropped the naive value (with a bound inside the indeterminate window, so the test cannot pass by accident), and the document `@id` remaining a valid IRI with `+00:00` in it. 11 of the 13 fail on the parent commit
- **Fixed during review** (Qodo): once new entries carry `+00:00` and stored ones do not, `ProvenanceManager.query_recorded_between` and `audit_log` compared ISO timestamps as raw strings, so they ordered by spelling rather than by instant — an inclusive naive bound naming a stored offset-bearing timestamp sorted *below* it and dropped the record, and a bound written in another offset landed wherever its digits fell (`19:45+05:30` is 14:15Z, but sorted after 14:19Z). Both now compare instants through a new `to_utc_datetime()` helper that reads a missing offset as UTC, which is what the values written before this change actually were; a bound that cannot be read as a timestamp keeps the historical string comparison rather than raising on a call that used to work
- The remaining 147 naive call sites are in `context/`, `vector_store/`, `seed/` and elsewhere, where timestamps are compared against values parsed from previously stored naive strings. Converting those without a read-side migration would raise `TypeError: can't compare offset-naive and offset-aware datetimes` on existing data, so they are deliberately left for a separate change
- **`SHACLGenerator` mangles `#`-terminated namespaces into `#/`, so generated shapes target nothing** (#1082) by @changshenhan
- `__init__` normalized `base_uri` with `rstrip("/") + "/"`, which turns `http://example.org/manufacturing#` into `...manufacturing#/` — the most common RDF namespace convention. Every generated URI (`sh:targetClass`, `sh:path`, shape URIs) then landed in a different namespace than the instance data, and SHACL validation silently passed because the shapes targeted nothing
- `__init__` now preserves a namespace already ending in `/` or `#`, matching the `#`-aware normalization `generate()` already applies; `shapes_uri` inherits the fix
- New `test_hash_namespace_base_uri_is_not_mangled` in `tests/ontology/test_ontology_advanced.py` fails on the pre-fix normalization and passes with it; full ontology suite (76 tests) green
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
- Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load
- **Fixed during review** (@Sameer6305): `NERExtractor.__init__()` still had a direct `spacy.load()` call site with the same cache-bypass issue, outside the two files named in #998 but sharing the same root cause; routed through the cache alongside stale test patch targets and a strengthened cache-configuration assertion
- **Fixed during review** (@KaifAhmad1): `SemanticChunker.__init__` only caught `OSError` around `load_spacy_model()`, while the sibling fix to `NERExtractor` in this same PR added a broader `except Exception` for a model that is installed but fails at runtime (e.g. a config incompatible with the installed spaCy version). A broken-but-present model crashed `SemanticChunker()` outright instead of degrading to fallback chunking like every other path in this PR. Added the matching `except Exception` branch, leaving `self.nlp` as `None`; new `test_semantic_chunker_falls_back_when_spacy_runtime_is_broken` mirrors the existing `NERExtractor` regression test for the same scenario
- New `tests/split/test_spacy_model_cache.py`: cache reuse across repeated calls/instances, shared cache between `split_by_sentences()`/`SemanticChunker`/`NERExtractor`, distinct model names loading separately, missing-model fallback without poisoning the cache, and the broken-runtime fallback added above
- `pytest tests/split/test_spacy_model_cache.py tests/split/test_splitter.py tests/split/test_chunkers.py`: all passing (3 pre-existing, unrelated `tests/test_ner_configurations.py` failures confirmed present on `main` before this PR)
- **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305
- Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result
- `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records
- `export_yaml({"data": [...]}, path)` previously wrote a structurally valid file with every collection empty, no exception, no warning, and the progress log reporting a completed export. `export_semantic_network`, `export_for_pipeline`, and `export_ontology_schema` now raise `ValidationError` when the payload shares no recognized key with what the method reads, or resolves to nothing while an unread key still holds records — an empty mapping is still accepted, since a genuinely empty graph has no records to lose
- **Breaking**: the two cases above, plus a bare list, now raise instead of returning cleanly with data silently dropped or a raw `AttributeError` from exporter internals. Migration: pass records under a recognized key (`{"entities": [...]}` / `{"nodes": [...]}` for `semantic_network`, `{"classes": [...]}` for `schema`)
- **Fixed during review** (Qodo): progress tracking could report a completed export before the output directory existed or the file was written; `export()` now creates the directory and serializes before starting tracking, so a rejected export leaves nothing behind
- **Fixed during review** (@Sameer6305, round 1): `normalize_graph_payload()`'s collection resolver treated any truthy value as a collection — `{"entities": "abc"}` silently became three single-character records, `{"entities": 42}` leaked a raw `TypeError` from inside `list()`. Collection values are now validated before conversion, rejecting strings/bytes/mappings/non-iterable scalars by name. Separately, `Neo4jCSVExporter._normalize_graph` called the shared resolver with `require_recognized=False`, so it alone kept accepting an unrecognized mapping as a silent empty export; the opt-out (introduced earlier in this same PR, with no other caller) was removed
- **Fixed during review** (@Sameer6305, round 2): `YAMLSchemaExporter`'s usable-schema check could treat scalar schema metadata (`version`, `uri`, `title`, `description`) as evidence records had been exported, letting records under an unread key drop silently; and `_is_record()` accepted modules and class/type objects through the generic `__dict__` path, which would have reached exporter internals instead of failing at the boundary. Both closed, with regression coverage
- **Fixed during final maintainer review** (before merge): four more gaps in the shared boundary that the earlier rounds didn't reach
- `LPGExporter`/`ArangoAQLExporter` called `normalize_graph_payload()` with no type guard, so non-mapping input raised `ValidationError` from inside the resolver — while YAML and `Neo4jCSVExporter` raised `ProcessingError` for the identical mistake, per this PR's own stated contract. The `_require_mapping()` guard that already existed in `yaml_exporter.py` is now shared from `utils/helpers.py` and used by all three
- `Neo4jCSVExporter._normalize_graph` checked `isinstance(graph, dict)`, so a non-dict `Mapping` (`MappingProxyType`, `ChainMap`) fell through to the object-attribute branch and was rejected, even though the identical payload exported fine via `LPGExporter`/`ArangoAQLExporter`/YAML. Now checks `isinstance(graph, Mapping)`
- `normalize_graph_payload()` accepts dataclass and attribute-bearing object records (`Neo4jCSVExporter._record_to_dict` reads them), but `LPGExporter`/`ArangoAQLExporter` call `.get(...)` directly on resolved entities — an object-shaped record passed validation only to crash with a raw `AttributeError` once used, the exact failure this PR's boundary exists to prevent. Records are now converted to plain dicts at the boundary (`_coerce_records` → new `_record_to_dict`), so every consumer gets a uniform shape regardless of which reading the caller used
- Two non-empty spellings of the same collection (e.g. `entities` and `nodes`) holding identical records in a different order were rejected as conflicting, since the check used plain list equality; a caller round-tripping through a dict-keyed cache or a set has no reason to preserve order. Comparison is now an order-independent multiset of each record's canonical JSON form
- New regression coverage in `tests/utils/test_normalize_graph_payload.py`: exception-type parity for non-mapping input across `export_lpg`/`export_arango`/`export_neo4j_csv`, dataclass-record conversion verified end-to-end through the same three exporters, `Neo4jCSVExporter` accepting a `MappingProxyType` payload, and reordered-alias equality (plus a duplicate-count case confirming the multiset check still catches real conflicts); 4 existing tests updated to assert the corrected dict-conversion behavior instead of the previous object passthrough
- `pytest tests/export tests/utils tests/context tests/test_export_module.py tests/test_export_methods_wrapper.py tests/test_notebooks_simulation.py`: 718 passed, 4 skipped (up from 641 passed, 62 subtests at PR submission); `black`/`isort`/`flake8 --max-line-length=88` clean on every line this PR touches; `python -m build`: succeeds
- **`ContextGraph.add_edge` had no dedupe — identical edges were stored repeatedly under one shared edge ID, and re-ingest doubled the edge set** (#926, closes #922) by @pravit-amp
- `_add_internal_edge` appended to `self.edges`, `edge_type_index`, and `_adjacency` unconditionally, with no check for an edge already present. Edge identity is content-derived (`_resolve_edge_identity` builds `edge_id` from `source_id`/`target_id`/`edge_type`/`weight`/`metadata`/`valid_from`/`valid_until`), so two identical `add_edge` calls produced two edge objects sharing one `edge_id` — the graph already considered them the same edge, it just kept both copies. `self.nodes` already deduped by ID; edges did not, so `stats()["edge_count"]` inflated, `density()` could exceed its mathematical maximum of `1.0`, and a refresh/restore job calling `build_from_entities_and_relationships()` (or reloading a saved graph) doubled the edge set on every cycle
- Added an `edge_id -> ContextEdge` index (`_edge_index`), mirroring how `self.nodes` dedupes by node ID. `_add_internal_edge` now returns `False` when the `edge_id` already exists, checked before touching `edges`/`edge_type_index`/`_adjacency` and before firing the mutation callback, so a repeat `add_edge` is a silent no-op with no phantom `ADD_EDGE` audit event
- Genuinely parallel edges are unaffected: differing type/weight/metadata/validity still produce distinct content-derived `edge_id`s, so multigraph semantics are preserved
- Both state-reset paths (`load_from_file()` and `clear()`) also clear `_edge_index`
- New tests: repeat `add_edge` is a no-op, parallel edges with distinct attributes are preserved, re-ingest via `build_from_entities_and_relationships()` stays at one edge, and `clear()` resets the dedupe index
- **`POST /api/enrich/extract` returned 503 on every request; the whole `/api/decisions*` family returned 500 as soon as a decision existed** (#886, closes #883, closes #884, closes #889) by @joseedson18jc, reviewed by @Sameer6305
- `semantica/explorer/routes/enrich.py` imported `extract_entities`/`extract_relations` from `semantica.semantic_extract.methods`, names that module never defined (only per-strategy variants like `extract_entities_ml` exist) — the `except ImportError` handler reported this as `"semantic_extract module not available"`, masking a wiring bug as a missing dependency. The route now calls `NamedEntityRecognizer`/`RelationExtractor` directly and forwards extracted entities into relation extraction instead of re-deriving them
- `ContextGraph.record_decision()` stores `timestamp` as `datetime.now().timestamp()` (a float), while `DecisionResponse.timestamp` was typed `Optional[str]`; passing the value through unconverted failed pydantic validation on every decision route (`/api/decisions`, `/{id}`, `/{id}/chain`, `/{id}/precedents`, `/{id}/compliance`). Added a `field_validator(mode="before")` on `DecisionResponse` normalizing float/int/datetime inputs to ISO-8601
- Folds in the fix for #889:`extract_entities_ml`/`extract_relations_similarity`/`extract_relations_dependency` called `spacy.load()` on every invocation (~120ms of a ~132ms call, ~60x the actual extraction work). Added a process-level, lock-guarded `load_spacy_model()` cache in `semantic_extract/methods.py`, keyed by model name; failed loads are not cached, and the separate `get_nlp_model()` cache (different `disable=` pipeline config for similarity work) is kept independent to avoid handing one caller's spaCy pipeline to another
- **Fixed during review** (@Sameer6305): capped previously-unbounded input text on `/api/enrich/extract`; tightened the route's exception handling
- **Fixed during review** (@KaifAhmad1): the timestamp validator's `math.isfinite()` guard only rejected NaN/inf — a finite-but-out-of-range epoch (e.g. milliseconds mistakenly stored instead of seconds, such as `1723600000000`) still raised an uncaught `OverflowError`/`OSError` from `datetime.fromtimestamp()`, reintroducing an unhandled 500 on `/api/decisions*` for exactly the class of bug this PR closes. Now caught and re-raised as a `ValueError`. Also excluded `bool` from the numeric branch (`isinstance(True, int)` is `True` in Python, so `timestamp=True` was silently coerced to epoch 1 instead of being rejected)
- New/updated tests: `tests/explorer/test_explorer_api.py` (`TestRecordedDecisions`, extraction coverage, 4 new `TestDecisionResponseTimestampValidator` cases for the range/bool fixes), `tests/semantic_extract/test_spacy_model_cache.py` (6 tests)
- `GraphWorkspace.tsx` only destructured `{ data, isLoading, isFetching }` from `useLoadGraph()`, ignoring the `isError`/`error`/`refetch` that `useQuery` (`retry: 0`) already returned. Combined with `GraphLoadingOverlay` having no error prop and `showLoadingOverlay` staying true whenever `loadingProgress` held a stale frame, a backend-down or failed fetch left the graph workspace stuck on the last progress frame indefinitely, with no error message and no way to recover short of a full page reload
- `GraphLoadingOverlay` now accepts `error`/`onRetry` and renders an error card with the real fetch error message and a Retry button (`refetch()`) instead of the stuck progress UI
- The landing page's `WelcomeScreen` replaced its hardcoded `ready: boolean` (and hardcoded "System Online" text) with a real `checking` / `online` / `offline` status derived from the same connectivity probe already driving the 4th metric card, so the status dot, text, and metric can no longer drift apart or lie about connectivity
- **Smaller fixes bundled in the same PR**: search results are now dismissible (previously stayed open indefinitely, pushing the graph down); relevance scores display as rounded whole numbers instead of `96.900`/`138.000`; added a debounced (250ms) typeahead combobox to graph search with arrow-key navigation, `aria-activedescendant`, and Escape-to-close, using the existing `/api/graph/search` endpoint
- **Fixed during review** (Qodo): the typeahead's debounced fetch had no `AbortController`, so a fast-typing user could have a stale suggestion response resolve after a newer one, replacing correct suggestions with outdated ones. In-flight requests are now aborted on every re-debounce and when the query is cleared after a selection
- **Noted during review** (@Sameer6305): `GraphWorkspaceShell.tsx` contains a third, unused implementation of the same graph-loading/error-handling logic this PR fixes — the issue itself named "two copies that drifted apart" as the root cause the original bug slipped through. Deliberately left out of this PR's scope and tracked separately in #981 rather than blocking this fix
- **Markdown import hardened against TOCTOU symlink races during file reads** (#932, closes #856) by @lakshanmuruganandam, with fixes by @Sameer6305
- `AgentMemory._read_markdown_path` read files via `Path.read_text()` after a `Path.is_symlink()` pre-check, leaving a time-of-check/time-of-use window: a path validated as a regular file could be swapped for a symlink before the actual read, causing the importer to follow the link and read an unintended target
- Reads now go through a new `_read_markdown_file_content()` helper: the path is opened via low-level `os.open()` with `os.O_NOFOLLOW` on platforms that support it (POSIX), so a symlink substituted after validation fails atomically with `ELOOP` instead of being followed; the resulting file descriptor is then verified with `os.fstat()`/`stat.S_ISREG()` to reject non-regular files (FIFOs, devices) even after a successful open
- Directory imports now also exclude symlinked entries from the file listing (`not file_path.is_symlink()`), consistent with the single-file path already rejecting them
- **Known limitation**: Windows has no `os.O_NOFOLLOW`, so on that platform the only defense is the earlier `is_symlink()` pre-check, leaving a narrow TOCTOU window; documented inline rather than implying a stronger cross-platform guarantee than the implementation provides
- New `tests/context/test_agent_memory_markdown.py` coverage: rejecting a symlinked path at both the private helper and the public `import_data()` API, silently excluding symlinked entries during directory import, and the `fstat()`/`S_ISREG` guard against non-regular files (mocked FIFO)
- `pytest tests/context/test_agent_memory_markdown.py`: 46 passed, 4 skipped (symlink-creation tests skip on Windows without `SeCreateSymbolicLinkPrivilege`)
- **`VectorManager.maintain_store()`/`collect_statistics()` crashed with `AttributeError` on persistent `VectorStore` backends** (#914, closes #855) by @yunaremaia, with fixes by @Sameer6305
- Both methods accessed `store.vectors`/`store.metadata` directly, which are only initialized for the `inmemory` backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus, SQLite, PgVector, Weaviate) crashed immediately. Same root cause as the #839/#843/#845/#848 cluster, but `VectorManager` operates on a `VectorStore` instance from the outside, so the fix needed a public accessor rather than another internal guard
- Added a backend-agnostic `VectorStore.count()`: the `inmemory` backend counts its local dict; persistent backends delegate to a `count()` on the wrapped backend store when one exists, or raise `NotImplementedError` — following the `get_vector()`/`get_metadata()` precedent from #843, a missing/uninitialized backend store is never silently reported as an empty, healthy store
- `maintain_store()` and `collect_statistics()` now go through `store.count()` instead of touching `.vectors`/`.metadata`
- **Fixed during review** (@Sameer6305): the initial version had `count()` implemented at the dispatch level only, with no shipped backend actually providing one, and `maintain_store()` manufactured a vacuous `metadata_count == vector_count` tautology for persistent backends (always reporting `healthy: True` without checking anything). Added real `count()` implementations to `FAISSStore` (`len(index.vector_ids)` — FAISS has no delete path, so this list is always consistent with the index), `SQLiteVecStore`, and `PgVectorStore` (both via `SELECT COUNT(*)`); `Qdrant`/`Pinecone`/`Milvus`/`Weaviate` continue to raise `NotImplementedError` since none of them guarantee a cheap, reliable synchronous count. `maintain_store()` now reports `metadata_count: None` for persistent backends instead of the fabricated equality, with `healthy` meaning "store is reachable," not "metadata verified"
- Two earlier Qodo findings (a count() path that silently returned 0 for a missing backend store, and an unvalidated `hasattr` check that could raise `TypeError` on a mis-shaped adapter) were fixed before this review — replaced with `NotImplementedError` and a `getattr`+`callable()` capability check, respectively
- New `tests/vector_store/test_vector_manager_persistent.py`: dispatch-level tests for `count()` (inmemory, delegation, missing backend, non-callable `count`, mis-shaped adapter), full `VectorManager` inmemory semantics including divergence detection, persistent-backend dispatch tests, and backend-specific tests against real/mocked FAISS, SQLite (`sqlite-vec`, skipped if unavailable), and PgVector stores
- `get_node_property` returned `None` for both "node missing" and "property missing" with no way to distinguish them, and `get_node_attributes` returned `{}` for a missing node while its siblings disagreed on the not-found signal (`get_node_property`/`find_node` → `None`, `get_edge_data` → `{}`). Both now accept a `default=` parameter matching `dict.get()`'s convention, defaulting to their historical return values (`None` and `{}` respectively) for backward compatibility. Callers that need to disambiguate "node missing" from "value legitimately absent" can pass a private sentinel as `default`
- Added Google-style docstrings to `get_node_property`, `get_node_attributes`, `get_edge_data`, and `find_node` documenting each method's not-found contract, addressing #877's "sibling not-found contract undocumented" gap
- **Corrected during review**: the PR as submitted claimed to fix `add_node_attribute` firing its `mutation_callback` "outside `with self._lock`, without holding the lock," but the diff only removed a stray blank line — the callback call remained outside the lock, unchanged. Further investigation found this was not actually a bug: `self._lock` is a `threading.RLock`, and the same release-the-lock-before-invoking-the-callback pattern is used deliberately in `_add_internal_node`/`_add_internal_edge` elsewhere in this class, avoiding holding the lock for the duration of an arbitrary user-supplied callback. The real inconsistency was that, unlike those two siblings, `add_node_attribute`'s callback call wasn't wrapped in `try/except` — a raising callback propagated uncaught here but was caught and logged there. Now wrapped the same way (`except Exception as e: self.logger.warning(...)`)
- 13 tests covering happy path, missing node, missing property, sentinel disambiguation, falsy-zero, callback firing/non-firing, and (added during review) a raising callback no longer propagating out of `add_node_attribute`
- **Three `tests/normalize/` tests failed for reasons unrelated to the normalize implementations: a missing optional-dependency skip guard, an incomplete chardet allowlist, and a UTC/local timezone mismatch** (#881, closes #860) by @aoright
- `test_detect_language`/`test_detect_with_confidence` in `tests/normalize/test_language_detector.py` asserted on real `langdetect` output with no skip guard, even though `langdetect` is an optional dependency absent from `pyproject.toml` that `LanguageDetector` already degrades gracefully without (`LANGDETECT_AVAILABLE = False`, falls back to `default_language`) — any environment without it failed both tests unconditionally, including a fresh CI run without optional extras installed. Both are now gated with `@unittest.skipUnless(LANGDETECT_AVAILABLE, ...)`
- `test_detect_encoding` in `tests/normalize/test_encoding_handler.py` asserted `chardet.detect()`'s result against a 3-name allowlist (`iso-8859-1`/`windows-1252`/`latin-1`); on a short Latin-1 sample, chardet is free to return other compatible single-byte codepages (e.g. `windows-1253`), which fails the allowlist and then cascades into `test_convert_to_utf8` decoding the bytes as Greek instead of the original text. The test now uses a longer, unambiguous Latin-1 corpus and asserts that the detected encoding round-trip-decodes the original text instead of matching a fixed name list; `test_convert_to_utf8` now passes `source_encoding="latin-1"` explicitly rather than relying on chardet's heuristic auto-detection
- `test_normalize_date_relative` in `tests/normalize/test_date_normalizer.py` compared `RelativeDateProcessor`'s local-clock-based `"today"` (`datetime.now()`, naive, UTC-normalized after the fact by `convert_to_utc()`) against a separately-computed UTC reference date — failing intermittently in any timezone east of UTC whenever the local and UTC dates diverge for part of the day. The test now patches `datetime.now()` to a fixed reference time, making the assertion independent of host timezone
- `pytest tests/normalize`: 77 passed, 2 skipped (`langdetect` not installed); `black`/`isort`/`flake8 --max-line-length=88` clean on all three changed files. Test-only change; no production code touched
- **MCP server reported a stale `0.4.0` version instead of the installed package version** (#870, closes #863) by @oiahoon
- `semantica/mcp_server/__init__.py` hardcoded `"version": "0.4.0"` in both the MCP `initialize` response (`SERVER_INFO`) and the `semantica://schema/info` resource, regardless of the actual installed `semantica` version — every MCP client (Claude Desktop, Windsurf, Cline, Continue, VS Code Copilot, etc.) showed the wrong server version. Both surfaces now derive from `semantica.__version__`, the package's authoritative version source, so they can no longer drift from `pyproject.toml`
- New regression coverage in `tests/test_mcp_server_version.py`, including `!= "0.4.0"` canaries and a cross-surface consistency check
- **Fixed along the way**: the separate root-level `mcp/` package (`mcp/__init__.py`, `mcp/server.py`, `mcp/resources/registry.py`) — a companion MCP server implementation not included in the built distribution, but documented in `mcp/__init__.py` as a supported way to run against Claude Desktop/Windsurf/etc. from a source checkout — had the same three hardcoded `0.4.0` literals; fixed the same way, with matching regression tests in `tests/test_mcp_package_version.py`
- **`VectorStore._filter_by_metadata()``AttributeError` on all persistent backends** (#857, closes #849) by @TaherTadpatri
- `_filter_by_metadata()` iterated `self.metadata` directly, which only exists on the `inmemory` backend — any persistent backend (`faiss`, `qdrant`, `pinecone`, `milvus`, `pgvector`, `sqlite`, `weaviate`) crashed with `AttributeError` on `filter_decisions(query=None, ...)` / metadata-only filtering. Filtering is now delegated to a native `filter_by_metadata()` implemented on each backend store, using backend-native payload/SQL/JSON filtering (Qdrant `scroll()`, Pinecone `query()`, Milvus expression filters, PostgreSQL JSONB, SQLite `json_extract()`, Weaviate collection filters)
- **Fixed along the way**: `PineconeStore.get_index()` and `filter_by_metadata()` called a nonexistent `self.describe_index_stats()` on the store itself (the method only exists on the `PineconeIndex` wrapper returned by `self.index`); the resulting `AttributeError` was silently swallowed, so dimension auto-detection always failed quietly. Now correctly calls `self.index.describe_index_stats()`
- **Fixed along the way**: `PineconeStore.filter_by_metadata()` probed for filter-only matches using an all-zero dummy query vector, which Pinecone rejects for cosine-metric indexes — the library's own default — making metadata-only filtering silently non-functional out of the box. Now uses a unit vector instead
- **Fixed along the way**: `PgVectorStore.filter_by_metadata()`'s list-filter branch formatted boolean values with `str(v)` (`'True'`/`'False'`), never matching PostgreSQL JSONB's lowercase `'true'`/`'false'` text rendering, even though the equivalent scalar-filter branch already handled this correctly
- **Fixed along the way**: list-valued metadata fields (e.g. `{"tags": ["python", "js"]}`) could never match a list filter on the SQLite or PostgreSQL backends, because both extracted the whole array as its JSON/text representation instead of matching individual elements — silently diverging from the in-memory backend's set-intersection semantics. SQLite now uses `json_each()` over a `json_type`-guarded array/scalar wrapper; PostgreSQL now uses the `?|` "any array element" operator alongside the existing scalar `= ANY(...)` path
- **Fixed along the way**: `FAISSStore.filter_by_metadata(limit=0)` returned one result instead of zero, because the limit check ran after appending the current match
- **Fixed along the way**: `MilvusStore`'s metadata expression builder rendered `NaN`/`Infinity` filter values as bare unquoted tokens, producing an invalid Milvus expression whose server-side rejection was then swallowed by a broad `except`, indistinguishable from "no matches"; these values are now rejected up front with a clear `ValidationError`
- New/expanded test coverage in `tests/vector_store/test_backend_metadata_filtering.py` (all 7 backends, including the Pinecone dimension/zero-vector, PgVector boolean-list, FAISS `limit=0`, and Milvus `NaN` regressions) and `tests/vector_store/test_sqlite_vec_store.py` (new `TestSQLiteVecStoreFilterByMetadata`, run against the real `sqlite-vec` extension, including the array-vs-scalar intersection case)
- **`DistanceExporter` silently swallowed metric computation failures, exporting `None` values indistinguishable from a legitimate "no path" result** (#879, closes #874) by @AmirF194
- `_betweenness`, `_hop_distance`, `_weighted_distance`, and `_semantic_similarity` each caught `Exception` and returned their sentinel (`None`/`{}`) with no logging; a failed computation and a real "no path exists" looked identical in exported CSV/JSONL/DataFrame data. All four now log a `warning` with `exc_info=True` before returning the sentinel; exported row shape and values are unchanged
- **Fixed along the way**: the module logger was built with `get_logger(__name__)`, which double-prefixed it to `semantica.semantica.export.distance_exporter` — a name `setup_logging()` never configures — so this module's logging (including a pre-existing `logger.debug` call) was silent regardless. Now uses `get_logger("export.distance_exporter")`, matching every other exporter in the module
- New regression coverage in `tests/export/test_distance_exporter.py`: warnings fire on exception for all four helpers, exported sentinel values/shape stay unchanged, and the legitimate "no KG backend" `None` path still logs nothing
- Full `tests/export/` suite: 71 passed
- **`explain_violations` rendered hardcoded placeholders (`min_count=1`, `max_count=1`) instead of the SHACL shape's real constraint values, and misused the violation message text as the datatype/class value** (#1094) by @cxzg007
- `_run_pyshacl` never read `sh:minCount`/`sh:maxCount`/`sh:datatype`/`sh:class` back from the violation's `sh:sourceShape`, so every plain-English explanation was wrong regardless of what the shape actually declared. `SHACLViolation` now carries those four fields (also exposed via `to_dict()`), populated by back-referencing `sh:sourceShape`; `explain_violations` renders the real values, falling back to `"?"` when a value is genuinely absent
- **Known limitation**: `sh:qualifiedMinCount`/`sh:qualifiedMaxCount` are not handled yet and still fall back to the `"?"` placeholder
- New regression tests cover both the rendering path and the `sh:sourceShape` back-reference (skipped when `pyshacl`/`rdflib` are absent)
- **Entity merging silently dropped `entity_id` aliases, and exact-match entity resolution had three correctness gaps** (#1086, #1026) by @T1mn
- `entity_merger.py`/`merge_strategy.py`/`entity_resolver.py` used inconsistent logic for extracting an entity's id across the merge path, so a merged entity could lose the `entity_id` aliases that let later lookups find it under its old identity. A new `semantica/utils/entity_ids.py` unifies id extraction across all three call sites
- `EntityResolver`'s exact-match path is now honored rather than silently falling through to fuzzy matching in some cases; entities with no identifier are preserved instead of being dropped, and blank exact-match names are ignored rather than matching every other blank name
- New/expanded coverage in `tests/kg/test_entity_pipeline.py` and `tests/kg/test_entity_resolver_exact.py`
- **`flatten_dict()` silently collided keys when a flattened path from one branch matched a literal key already present at the target depth** (#1062) by @shahzaib-ahmadcs
- Two differently-shaped inputs could flatten to the same output key, with the second write silently overwriting the first — no error, no warning, just a dropped value. Collisions are now detected and handled explicitly instead of overwriting
- **`ExcelParser.__init__` raised `NameError` on every instantiation — `get_progress_tracker()` was called but never imported** (#1016, closes #1014) by @pravit-amp
- Same defect as the one fixed for `SimilarityCalculator` in #530, this time in `semantica/parse/excel_parser.py`; the existing test imported the class but never constructed it, so nothing caught the missing import. Added construction coverage for every parser exported from `semantica.parse`, driven off `__all__` so future additions are covered automatically, living outside `test_parse_comprehensive.py` (whose `setUp` mocks `get_progress_tracker` into each module and would mock away the exact interaction under test)
- **Graph analytics (`centrality_calculator.py`, `community_detector.py`, `connectivity_analyzer.py`) dropped isolated nodes and diverged on how each computed its working view of the graph** (#1011) by @T1mn
- Each analyzer had its own ad hoc logic for building the node/edge set it operated over, and none of them included nodes with no edges — a node with zero connections simply vanished from centrality scores, community assignments, and connectivity reports instead of appearing with a zero/singleton value. A new shared `semantica/kg/_graph_view.py` centralizes graph-view construction (including node fallbacks and community payload shaping) for all three analyzers, which are now ~250 lines lighter combined
- New `tests/kg/test_analytics_node_scope.py` covering isolated-node presence across all three analyzers
- **Explorer fired temporal-bounds and snapshot requests before the graph itself had loaded, tripling failed requests when the backend was down and leaving the timeline scrubber with nothing to scrub** (#1003) by @lakshayxi
- Two new predicate functions gate the temporal effects on the graph having actually loaded (an empty graph still counts as loaded); confirmed against a downed backend that this cuts three failing requests per page load down to one
- **`SeedDataManager.load_from_database()` never actually reached the database, and connection failures were mislabeled as a missing optional dependency** (#995, closes #973) by @yzxcj797
- `DBIngestor.execute_query`/`export_table` need the connection string as their first positional argument; `load_from_database()` only passed it into the constructor's config dict, which those methods never read, so every call raised `TypeError` before connecting. Also split the combined `except (ImportError, OSError)` handling apart — a genuine connection failure was reported as `"module not available"`, sending debugging in the wrong direction; `OSError` now propagates as an actual failure, chained via `from e`
- **SPARQL `CONSTRUCT` detection matched inside a leading `#`-comment, misclassifying `SELECT`/`ASK` queries as `CONSTRUCT` across all four SPARQL backends** (#951) by @pravit-amp
- `CONSTRUCT_QUERY_RE` skipped comments with a bare `\#[^\n]*`, whose backtracking `*` let a `# CONSTRUCT ...` comment line "swallow" the real query-form keyword on the next line for a query like `# CONSTRUCT ...\nSELECT ...`. The mistaken `CONSTRUCT` classification sent `Accept: text/turtle` and tried to parse a SELECT/ASK response body as Turtle, failing with a misleading parse error. The regex now requires a comment to reach a line terminator (LF or CR, per the SPARQL grammar) before matching
- **`k_shortest_paths` mutated caller-visible graph state during traversal and ignored direction when excluding already-used edges** (#1000) by @T1mn
- `semantica/kg/path_finder.py`'s search left side effects behind after returning, and edge exclusion during Yen's-algorithm-style path removal didn't respect the traversal direction of directed graphs, letting a later search see edges that should have been available. Both fixed; new coverage in `tests/kg/test_path_finder.py`
- **`trace_decision_causality()` ignored explicitly recorded causal edges, inferring causes only from shared NER entities plus timestamp ordering** (#983) by @hsd2514
- A `CAUSED`/`INFLUENCED`/`PRECEDENT_FOR` edge added via `add_causal_relationship()` had no effect on the trace — when entity extraction found nothing in common between two decisions, `trace_decision_chain()` came back empty even with an explicit edge stored in the graph. Explicit causal edges are now traversed first as ground truth, with entity/timestamp inference kept as an additive fallback for pairs with no explicit link; edges whose source has no decision record (e.g. a graph restored via `from_dict`) are skipped so a stale edge can't abort the trace
- **`RepoIngestor`'s module-level DNS resolve cache had no lock, raising `RuntimeError: OrderedDict mutated during iteration` under concurrent `ingest_repository()` calls** (#979) by @manjunathbhaskar
- `_REPO_HOST_RESOLVE_CACHE` is a shared `OrderedDict` read, written, and pruned by every thread with no synchronization — reliably reproduced with 32 threads hammering resolution under a low TTL and small cache cap. Now guarded by a lock
- **`GraphBuilder` didn't remap relationship endpoints after entity resolution merged nodes, leaving relationships pointing at ids that no longer existed in the resolved graph** (#978) by @T1mn
- New coverage in `tests/kg/test_graph_builder_external.py`; a follow-up commit hardens the remapping against edge cases found during review
- **Explorer's dev server esbuild target didn't match the browser targets the production build declares**, occasionally producing dev-only syntax errors on older browsers (#966) by @le-czs
- `explorer/vite.config.ts` now sets the dev esbuild target explicitly to match
- **`normalize`'s number normalizer accepted currency symbols without validating them against the surrounding text, and an earlier fix's currency-code matching wasn't token-bounded** (#940) by @Mr-Neutr0n, reviewed by @ZohaibHassan16
- Symbol currencies are now validated before being accepted; currency codes are matched on token boundaries so a code embedded inside a longer token no longer false-positives
- **`ContextGraph.to_dict()` was the one reader on the class that didn't hold `self._lock`, raising `RuntimeError: dictionary changed size during iteration` under a concurrent writer and risking a torn snapshot otherwise** (#929) by @pravit-amp
- Every other reader (`stats()`, `density()`, `find_nodes()`, `find_edges()`, `get_neighbors()`, `get_nodes_by_label()`, `state_at()`, `save_to_file()`) already took the lock after it was introduced; `to_dict()` predated that change and was missed. `save_to_file()` was safe only incidentally, since it builds its payload inline under its own lock rather than delegating to `to_dict()`
- **`PipelineWithProvenance` had a broken import and no working `run()` method** (#862) by @Karunasagar12
- `from .pipeline import Pipeline` failed because `Pipeline` lives in `pipeline_builder.py`, not a nonexistent `pipeline.py` — fixed to `from .pipeline_builder import Pipeline`. The class also had no `run()`; it now delegates to `ExecutionEngine.execute_pipeline()`, the intended execution path for a built `Pipeline`. The constructor now accepts a built `Pipeline` instance directly
### Security
- **Tarball restore path traversal, latent SQL injection, DNS-rebinding TOCTOU in the shared SSRF guard, stored XSS in report generation, and unvalidated SPARQL object IRIs in AnzoStore** (#1079) by @KaifAhmad1
- `semantica backup restore`'s tar extraction (`cli.py`) stripped only the literal `semantica-backup/` prefix and called `tar.extract()` with no path-containment check, no symlink/hardlink validation, and (on Python <3.12) no extraction filter — a crafted archive member (`../../<file>`, or a symlink pointing outside the restore root) could write arbitrary files above the restore directory. Every member is now validated for resolved-path containment before extraction, symlink/hardlink targets are rejected both lexically (absolute path, `..` segments) and by resolution, and `filter="data"` is applied on Python ≥3.12
- `DataExporter.export_table_data()` (`db_ingestor.py`) was missing the `text` import from `sqlalchemy` — a `NameError` that made the method non-functional, but latently: the query it built from raw f-string interpolation of `table_name`/`schema`/`where`/`order_by` was already injectable, so fixing the import alone (without also fixing the injection) would have silently armed it. Both are fixed together: the import is restored, `table_name`/`schema` are now validated against a strict identifier allowlist, and `where`/`order_by` are checked against a blocklist (statement separators, comments, UNION, DDL/DML keywords, time-based blind-injection primitives, schema-enumeration terms). This is a blocklist, not a grammar — it closes the concrete UNION-exfiltration path and common injection primitives, but a boolean-blind subquery using none of the blocked keywords could still get through; `where`/`order_by` must be treated as trusted/operator input, not exposed to untrusted end users, and the docstrings now say so explicitly
- `request_with_ssrf_guard()` (`ssrf.py`) validated a hostname's resolved IPs, then let the underlying HTTP client re-resolve the same hostname independently at connect time — a low-TTL or DNS-rebinding answer could differ between the two lookups, so a hostname that validated as public could still connect to a private/internal address. Ported the IP-pinning pattern already used by `explorer/routes/ontology.py`'s `_make_pinned_session` into the shared ingest guard: the one resolution that decides accept/reject is now also the one the connection is pinned to, via a custom `HTTPAdapter` that presents the real hostname over TLS SNI / Host header while connecting only to the validated IPs. Also closes the RFC 6598 Carrier-Grade NAT gap noted as a known limitation in #905/#868:`100.64.0.0/10` is now in `BLOCKED_NETWORKS`
- `ReportGenerator._generate_html()` (`export/report_generator.py`) f-string-interpolated report title/summary/metrics into HTML with no escaping — an ingested entity or document whose content flowed into a report (e.g. `<img src=x onerror=...>`) executed as stored XSS when the report was opened. All interpolated values are now `html.escape()`d
- `AnzoStore._format_object_for_sparql()` (`triplet_store/anzo_store.py`) validated the subject/predicate of a triplet via `sparql_escaping.validate_uri()` before interpolating them into a SPARQL `INSERT DATA` clause, but delegated the **object** position to a separate formatter that wrapped it as `<{obj}>` without the same validation — an object value containing `>`/`}`/`{`/`"` could close the intended `<...>` token early and inject additional SPARQL Update operations. The Blazegraph/RDF4J backends were hardened for the equivalent gap previously; Anzo's object position now goes through the same `validate_uri()` check
- Also hardened in the same pass: Apache AGE's `create_index()``index_type` parameter is now allowlisted (was interpolated raw into a `USING` clause); Neo4j's `limit` is now explicitly validated (raises `ValidationError` for non-integer input instead of falling through to a generic `ProcessingError`); the `ffprobe` metadata-extraction subprocess call is guarded against a filename starting with `-` being parsed as an option; the MCP server no longer echoes raw exception text to JSON-RPC clients, logging full details server-side and returning a generic message plus the exception class name instead
- **Fixed during review** (@KaifAhmad1): the SSRF IP-pinning change introduced a connection-pool leak of its own — `requests.Session.mount()` silently drops whatever adapter it replaces without closing it, so a multi-hop redirect chain on a reused session leaked one pooled connection per hop. Pinned adapters are now tagged and explicitly closed before being replaced, both per-hop and on final restore
- **Fixed during review** (@KaifAhmad1): mounting a pinned adapter and setting a Host header on a caller-supplied `Session` is not inherently thread-safe — two guarded calls sharing the same session from different threads could interleave their mount/restore cycles. Added a per-session lock (`_get_session_lock`) so concurrent guarded calls on the same session now serialize instead of racing; verified with a two-thread test showing correct serialization and zero cross-contamination of per-request Host headers
- **Fixed during automated PR review** (Qodo): `export_table_data()`'s new identifier/fragment validation raised `ValidationError` from inside a `try` whose blanket `except Exception` re-wrapped it as `ProcessingError`, masking the distinction between "bad input" and "the export itself failed" that callers rely on elsewhere in this module. Added the `except ValidationError: raise` guard already used by its sibling methods
- **Fixed during automated PR review** (Qodo): on a hop where IP pinning doesn't apply (`allow_private_ips=True`), `_apply_connection_pin()` unconditionally popped the session's `Host` header instead of restoring whatever it was before pinning touched it — a caller-supplied session carrying its own legitimate `Host` override (e.g. fronting a private endpoint under a different name) had that override silently dropped for the in-flight request, only reappearing afterward via the outer `finally` restore. It now restores the session's own pre-call header state (set back if present, popped only if it was truly absent) instead of always popping
- **Fixed during automated PR review** (Qodo): the `where`/`order_by` blocklist matched keywords/punctuation inside properly quoted string literals and identifiers too, so legitimate data like `status = 'union'` or `name = 'a--b'` was rejected as if it were SQL syntax. The blocklist now runs against a copy with quoted-literal contents masked out (`_mask_sql_literals`) — a malformed/unterminated quote sequence doesn't match the masking pattern and is left fully exposed to the blocklist, so this closes false positives without opening a masking-based bypass; the fragment actually used in the query is unchanged
- Re-ran each finding's proof-of-concept (or an equivalent adversarial test) against the fix and confirmed it is blocked: tar path/symlink traversal (both lexical and resolved-path forms), SQL UNION exfiltration and identifier breakout, DNS-rebinding TOCTOU (including under a configured `HTTP_PROXY`, which the pinning adapter also rejects outright since a proxy would resolve DNS itself), stored XSS, and the AnzoStore SPARQL injection
- `pytest tests/ingest/`: 266 passed, 2 skipped (10 pre-existing failures unrelated to this change — identical failure set confirmed on unmodified `main`); full regression sweep across `graph_store`, `export`, `triplet_store`, `parse`, and backup/restore: 313 passed
- **`Authorization`/`Proxy-Authorization` credentials could leak to a different origin across HTTP redirects, and several ingest paths bypassed the shared SSRF/redirect guard entirely** (#1067, closes #947) by @Sameer6305, reviewed by @KaifAhmad1
- `request_with_ssrf_guard()` previously only stripped sensitive headers from per-request `kwargs["headers"]` on a cross-origin redirect; session-level `Authorization`/`Proxy-Authorization` headers, `session.auth`, and `session.trust_env` (`.netrc` lookup) could all still resurrect credentials on the hop to a foreign origin. All five credential sources are now stripped case-insensitively, kept stripped for the remainder of a multi-hop redirect chain (no resurrection even if a later hop returns to the original host), and unconditionally restored via `finally` — including on exceptions and redirect-limit errors
- `MCPClient._send_request_http()` and `PublicAPIIngestor.detect_public_api()`/`ingest_public_api()` called `httpx.post()`/`requests.post()`/`session.request()` directly, bypassing `request_with_ssrf_guard()` entirely. Both now route through the shared guard, including when `validate_no_auth=False`
- `SeedDataManager.load_from_api()` mutated the caller-supplied `headers` dict in place when adding an API-key `Authorization` header, silently leaking the key back into a dict the caller might reuse elsewhere. Now copies before modifying
- **Fixed during review** (@KaifAhmad1): `allow_private_ips=True` (used to let MCP servers run on localhost/internal networks) was applied to every redirect hop, not just the operator-configured host — a compromised or malicious MCP server could 302-redirect to an internal address (e.g. `169.254.169.254` cloud metadata) and the guard would follow it unchecked, defeating the SSRF protection this PR otherwise adds. Added `allow_private_ips_on_redirect` to `request_with_ssrf_guard()`: a redirect target inherits the original host's private-IP trust only when it matches that host; any other host falls back to strict validation. `MCPClient` now pins `allow_private_ips_on_redirect=False`, so only same-host redirects on a trusted MCP server keep working — a cross-host hop into private address space is blocked
- **Fixed during review** (@KaifAhmad1): `detect_public_api()` only caught `requests.exceptions.RequestException`, but `request_with_ssrf_guard()` raises `ValidationError` (a disjoint hierarchy) for SSRF-blocked hosts, blocked redirect targets, missing `Location`, or exceeded redirect limits — unlike its sibling `ingest_public_api()`, which already caught it. Callers (including `is_public_api()`) got an undocumented raw `ValidationError` instead of `ProcessingError`, and the error-logging call was skipped. Now catches `(ValidationError, ProcessingError)` and re-raises, matching the sibling method
- **Fixed during review** (@KaifAhmad1): `detect_public_api()`/`ingest_public_api()` forwarded `**options` into `request_with_ssrf_guard(..., session=self.session, allow_private_ips=self.allow_private_ips, **request_options)` without stripping `session`/`allow_private_ips` from `request_options` first — a caller passing either through the per-call `**options` (a plausible mistake, since `allow_private_ips` is also a documented constructor-level knob) got a raw `TypeError: got multiple values for keyword argument`. Both are now popped from `request_options` before the call
- New regression coverage added during review: `TestAllowPrivateIpsOnRedirect` (cross-host redirect into private space blocked, same-host redirect trust preserved, default behavior unchanged for existing callers that don't pass the new kwarg) and `TestMCPClientAuthRedirect::test_redirect_to_private_ip_is_blocked`/`test_same_host_redirect_on_private_mcp_server_is_not_blocked` in `tests/ingest/test_auth_header_redirect_security.py`; `test_detect_public_api_propagates_ssrf_validation_error` and duplicate-kwarg regression tests for both methods in `tests/ingest/test_public_api_ingestor.py`
- **`FeedIngestor`/`FeedMonitor` (RSS/Atom feed ingestion) had no SSRF protection, allowing requests to internal/private network targets** (#928, closes #927) by @ZohaibHassan16
- `FeedIngestor.ingest_feed()`, `discover_feeds()` (link-tag fetch, common-path HEAD probe, and feed-validation GET), and `FeedMonitor.check_updates()` all called `requests.get()`/`requests.head()` directly with default redirect-following and no scheme allowlist or private/loopback/link-local IP validation — despite `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()` already existing and being used by `web_ingestor.py`/`api_ingestor.py`. `ingest_feed()`'s own URL check only verified `urlparse(url).scheme`/`.netloc` were non-empty, never that the scheme was http/https or that the resolved target IP was safe. Reachable via the public `ingest_feed()`/`ingest()` entry points with any caller-supplied feed URL
- All 5 call sites now route through `request_with_ssrf_guard()`, which validates scheme (http/https only) and resolved IP before the request, and re-validates every redirect `Location` before following it — closing both the direct-IP and redirect-chain SSRF paths. Added an `allow_private_ips` config option to both `FeedIngestor` and `FeedMonitor`, consistent with the other ingestors
- **Fixed during review** (Qodo): `test_discover_feeds_empty` mocked `requests.get`, which no longer executes now that the code path goes through `request_with_ssrf_guard()` (backed by `requests.request`) — the test was passing without exercising the real code. Corrected to mock `requests.request` and `socket.getaddrinfo`
- `pytest tests/ingest/test_feed_ingestor.py`: 12/12 passed. Independently reproduced the issue's own PoC (`FeedIngestor().ingest_feed("http://127.0.0.1:8765/feed.xml")` against a live local server) and confirmed it now raises `ValidationError` instead of succeeding
- **Known limitation carried over from `discover_feeds()`'s pre-existing design**: its common-path and feed-validation loops use a blanket `except Exception: continue`, which now also silently absorbs `ValidationError` from a blocked candidate URL the same way it already absorbed network failures — the request is still correctly blocked before reaching the network, so this is not an SSRF bypass, just a missed opportunity to log "blocked as SSRF target" distinctly from "unreachable"
- **`RepoIngestor` clone surface hardened against GitPython URL/option injection** (#905, closes #868) by @pravit-amp
- `RepoIngestor.ingest_repository()` passed the caller-supplied repository URL and arbitrary `**options` straight through to `git.Repo.clone_from()` on a `GitPython>=3.1.50` floor predating hardening for `ext::`-style transport helpers and `$VAR`/`${VAR}` environment-variable expansion in clone URLs — unvalidated clone options (`upload_pack`, `multi_options`, `template`, `config`, `env`, ...) could be abused for command execution, and unvalidated hostnames allowed SSRF against internal services (e.g. cloud metadata endpoints)
- `GitPython` floor raised to `>=3.1.58`
- Clone options passed to `clone_from()` are now allowlisted to `{depth, branch, single_branch, no_tags}`; anything else raises `ValidationError` before the clone is attempted
- Repository URLs are validated before cloning: scheme allowlist (`https`, `http`, `git`, `ssh`), rejection of `$VAR`/`${VAR}` tokens, and hostname resolution with every returned address screened against private/loopback/link-local/unspecified ranges. scp-like SSH remotes (`user@host:path`) are recognized and normalized to `ssh://` before the clone call
- **Fixed during review** (@Sameer6305): the SSRF check originally used `ip.is_reserved`, which flags the NAT64 Well-Known Prefix (`64:ff9b::/96`, RFC 6052) as reserved — falsely blocking `github.com` and other public hosts on IPv6-only/dual-stack networks using NAT64. Narrowed the block list to private/loopback/link-local/unspecified only
- **Fixed during review** (@Sameer6305): local filesystem repository paths (`git clone /path/to/local/repo`) were being treated as remote URLs and rejected outright; local paths now bypass network validation entirely since they make no network requests and carry no SSRF risk
- **Known limitation**: the SSRF host check does not classify RFC 6598 Carrier-Grade NAT space (`100.64.0.0/10`) as blocked — Python's `ipaddress.IPv4Address.is_private` does not cover that range, so a hostname resolving into it (e.g. some Kubernetes/CNI pod networks) would not be caught. Follow-up recommended to add it explicitly alongside the existing private/loopback/link-local checks
- **HTTP response header injection via `node_id`, unbounded-memory DoS in link prediction, and unsanitized imported node IDs in the Explorer** (#912) by @Sunil56224972
- `semantica/explorer/routes/provenance.py`'s `GET /api/provenance/report` f-string-interpolated the `node_id` query parameter directly into the `Content-Disposition` response header; a `\r\n`-bearing `node_id` could inject arbitrary response headers (`Set-Cookie` session fixation, `Content-Type` override for reflected XSS). Fixed with `_safe_content_disposition_filename()`, which strips `\r`, `\n`, `\x00`, `"`, `\` and length-caps the value before interpolation
- `POST /api/enrich/links` (link prediction) loaded up to 999,999 nodes with no cap or concurrency guard, then scored every candidate — a single request could consume ~1.6 GB RAM, and concurrent requests compounded that with no limit. Capped the candidate pool at 10,000 nodes (`413` if exceeded) and added an `asyncio.Semaphore(2)`, mirroring the SPARQL DoS fix in #898
- `POST /api/import` stored uploaded JSON/CSV node IDs verbatim; since provenance reports reflect `node_id` into `Content-Disposition`, an attacker could upload a node with a CRLF-bearing ID once and trigger the header-injection chain above for every subsequent viewer. Added `_sanitize_import_node_id()`, applied to node and edge `source_id`/`target_id` fields on both the JSON and CSV import paths
- **Corrected during review**: the JSON import path had a second, unsanitized branch — any uploaded node object already carrying a `"properties"` key (the shape this app's own `/api/export` produces, and already used elsewhere in the test suite) was appended to the graph as-is, bypassing `_sanitize_import_node_id()` entirely and leaving the stored-header-injection chain open via a one-line payload (`{"id": "<crlf>", "properties": {}}`). That branch now sanitizes `id` before storing
- **Corrected during review**: the link-prediction cap checked `total` only *after* calling `session.get_nodes()`/`get_edges()`, which normalize the graph's *entire* matching node/edge set before applying `limit` — so the guard ran after the expensive work it was meant to prevent had already happened, on every request regardless of graph size. Added `GraphSession.get_raw_counts()`, an O(1) check against the raw `len(graph.nodes)`/`len(graph.edges)` collections, and moved the size check ahead of the normalizing calls
- **Corrected during review**: 5 of the original PR's 22 regression tests asserted that literal words like `"Set-Cookie"`/`"Content-Type"` disappeared from the sanitized value — the sanitizer only strips `\r\n\x00"\\`, not letters, so those assertions failed against the PR's own fix as submitted. Corrected to assert on the property that actually blocks header injection (no `\r`/`\n` survives), and added end-to-end tests that exercise the real `/api/import` → `/api/provenance/report` route chain (not just the standalone sanitizer function) so the `properties`-key bypass has regression coverage
- Full `explorer` suite: 241 passed; `tests/test_security_regression_pr2.py`: 30 passed
- **`fastapi`/`python-multipart` floors in the `explorer` extra allowed PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p, `python-multipart` ReDoS)** (#871, closes #869) by @agu2347
- `explorer` declared `fastapi>=0.100.0` and `python-multipart>=0.0.6`; both floors resolve to versions carrying a ReDoS in `python-multipart`'s `Content-Type` header option parser (`parse_options_header`), reachable by any endpoint that accepts form/multipart data — an attacker-crafted header option can stall the event loop for minutes
- **Corrected during review**: the original fix raised only `fastapi>=0.109.1`, leaving `python-multipart>=0.0.6` unchanged. `python-multipart` is declared as its own direct dependency in the `explorer` extra rather than pulled in transitively via `fastapi[all]`, so a bare `fastapi` install enforces no `python-multipart` floor at all — the vulnerable `0.0.6` could still resolve with `fastapi>=0.109.1` in place. Floors raised to `fastapi>=0.109.2` / `python-multipart>=0.0.7`, the first versions of each that exclude the vulnerable range
- **Fixed along the way**: the `Security` workflow's `pip-audit` job ran only on a weekly schedule with `continue-on-error: true`, against a bare Python environment with none of Semantica's optional extras installed — it would never have seen `fastapi`/`python-multipart` regardless of which floor was pinned. `security-scan.yml`'s Safety check has the same blind spot (`pip install -e ".[llm-litellm]"` only, never `[explorer]`). `pip-audit` now also runs on `pull_request` when `pyproject.toml` changes, installs `semantica[all]`, and fails the build on any finding for that trigger; the schedule/`workflow_dispatch` runs stay non-blocking pending a full pass over any pre-existing findings across the whole `[all]` tree
- **Caught by the new gate on its first run**: `python -m pip install -e ".[all]"` pulled in `setuptools==79.0.1`, vulnerable to CVE-2026-59890/GHSA-h35f-9h28-mq5c/PYSEC-2026-3447 (Unicode-normalization bypass of `MANIFEST.in` exclude/prune patterns on macOS APFS/HFS+, letting excluded files leak into a built sdist), fixed in `83.0.0`. `[build-system] requires` had the exact same too-permissive-floor pattern this whole entry is about (`setuptools>=61.0`), and `actions/setup-python`'s baked-in `setuptools` isn't governed by that pin at all since it's outside any isolated build. Bumped `[build-system] requires` to `setuptools>=83.0.0`, and the `Security` workflow now runs `pip install --upgrade pip setuptools` before auditing so the scanned environment can't have a stale ambient copy regardless of what governs it
- Full `explorer` suite: 241 passed
- **`SeedDataManager.load_from_api()` made unguarded HTTP requests, with no SSRF protection at all** (#942) by @ZohaibHassan16
- `load_from_api()` called `requests.get()` directly instead of going through `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()`, unlike every other ingestor in this module — a caller-supplied `api_url` could target internal/private network addresses with no validation. Now routes through the shared guard, gaining redirect validation and bounded DNS resolution for free
- **Follow-up** (#959, closes #943) by @yunaremaia: added an `allow_private_ips` opt-in (parsed via the shared `parse_bool` helper) for trusted internal deployments that legitimately need to load from a private-network API, while keeping the guard's block-by-default behavior for everyone else
## [0.6.5] - 2026-08-11
### Added
- **Embedded Oxigraph backend for `TripletStore`** (#838, closes #834) by @Linxiushen
- Added `OxigraphStore` (`semantica/triplet_store/oxigraph_store.py`), an in-process SPARQL 1.1 store via the optional `pyoxigraph` dependency — no external server (Blazegraph/Jena/RDF4J/Anzo) required, fixing the confusing plain connection-error failure `TripletStore` previously produced with no server running (no local Docker daemon, no Java, CI, or a fresh laptop)
- Runs fully in memory by default, or persists to a local directory via `TripletStore(backend="oxigraph", path=...)`; reopening the same directory resumes existing data
- Full CRUD, native batch loading (`Store.extend`), named-graph scoping (`graph=` on add/query), and SPARQL SELECT/ASK/CONSTRUCT/DESCRIBE result mapping matching the existing backend contract; reuses `sparql_escaping.py` for datatype-IRI resolution instead of reimplementing it, and preserves RDF literal datatype/language metadata across writes, reads, and query results
- New optional `semantica[tripletstore-oxigraph]` extra (`pyoxigraph>=0.5.0`), included in the `all` extra; the import is lazy, so `TripletStore` and the rest of Semantica keep working without `pyoxigraph` installed
- Wired into `TripletStore` (`backend="oxigraph"`, added to `SUPPORTED_BACKENDS` and `NAMED_GRAPH_CAPABLE_BACKENDS`) and exported from `semantica.triplet_store`; README, module reference, glossary, and usage guide updated with install/configuration examples
- **Fixed along the way**: a missing `pyoxigraph` install surfaced as a generic wrapped `ProcessingError` instead of the underlying `ImportError` and its install hint, because `TripletStore._initialize_store_backend()`'s broad `except Exception` caught and rewrapped it; `ImportError` is now re-raised as-is so the `pip install "semantica[tripletstore-oxigraph]"` hint reaches the caller
- New integration tests in `tests/triplet_store/test_oxigraph_store.py` covering persistence/reopen, named-graph isolation, SELECT/ASK/CONSTRUCT result shapes, and the missing-dependency error message; skipped automatically when `pyoxigraph` isn't installed, and not yet exercised in CI since it doesn't install the optional extra or run the Python test suite
- **PROV-O trust blockers and general spec completeness for `ProvenanceManager`** (#825) by @KaifAhmad1
- **Invalidation instead of hard delete**: new `ProvenanceManager.invalidate(entity_id, agent_id, reason=None)` tombstones an entry — archives its pre-invalidation state under a stable versioned key, then appends the invalidated entry (`invalidated`, `invalidated_at_time`, `invalidated_by`, `invalidation_reason`) — instead of mutating or deleting it, so an audit can prove a fact existed, was reviewed, and was retracted. `ProvenanceManager.clear()` remains the bulk dev/test store-reset utility it always was; it was not repurposed
- **Hash-chained integrity**: every entry now carries `sequence_id`/`previous_checksum`, chaining it to the entry immediately before it in insertion order. New `ProvenanceManager.verify_chain()` walks the chain and reports any break, including a row hard-deleted directly from the underlying table — something a lone per-row SHA-256 checksum can never detect on its own. `compute_checksum()` now also covers `agent_id`/`agent_type`, the lineage-link fields, and the invalidation fields, closing several fields that previously weren't tamper-evident
- **Typed Agent/Activity**: `agent_id` was a dead field — no `track_*` method read it from kwargs, so it was always the `"semantica"` default regardless of what callers passed; fixed, and paired with new `AgentRecord(id, agent_type, is_automated)` / `ActivityRecord(id, activity_type, started_at_time, ended_at_time)` dataclasses (pass via `agent=`/`activity=` kwargs) so a human reviewer, an LLM call, and an automated pipeline stage are now distinguishable, and activities carry real start/end timing. Wired through all 18 `*_provenance.py` wrapper modules and `track_entity`/`track_relationship`/`track_chunk`/`track_property_source`
- **Versioning vs. derivation split**: new `previous_version_id` ("this corrects a prior version of the same fact") and `derived_from_id` ("this was derived from a different source entity") fields, additive alongside the legacy combined `parent_entity_id` so existing readers are unaffected
- **Downstream lineage traversal**: new `get_descendants()`/`trace_descendants()` (reverse BFS in both `InMemoryStorage` and `SQLiteStorage`), closing the gap flagged in `semantica/explorer/routes/provenance.py` where `direction="downstream"` was dead code with no reverse lookup to feed it; the Explorer's `/api/provenance` lineage response now merges both directions
- **W3C PROV-O qualified relations**: `export_prov()` now emits `prov:qualifiedAssociation`/`hadRole` (distinguishing "approved by" from "generated by" for sign-off workflows), `qualifiedGeneration`/`Generation`, `qualifiedUsage`/`Usage`, `qualifiedDerivation`/`Derivation`, `qualifiedInvalidation`/`Invalidation`, `wasAssociatedWith` (Activity→Agent), `actedOnBehalfOf` (Agent→Agent delegation), and `wasInformedBy` (Activity→Activity, via a new `informed_by=[...]` kwarg), alongside the existing plain triples
- **Bitemporal + Bundle support**: `revision_type`/`supersedes`/`valid_from`/`valid_until` fields (plain caller-supplied passthrough, matching the deprecated `kg.ProvenanceTracker`'s actual contract) plus new `revision_history()` and `query_recorded_between()` methods, closing the two "no direct equivalent yet" rows in `docs/migration/kg-provenance-tracker.md`; `bundle_id` emits `prov:Bundle`/`hadMember` membership triples to partition provenance by source/dataset/ingestion-run
- **Configurable, interlinked namespace**: `export_prov(base_uri=...)` / `--base-uri` CLI flag, defaulting to a new `ProvenanceManager.DEFAULT_BASE_URI` (`https://semantica.dev/ns#`) that `RDFExporter`'s `NamespaceManager` and `OWLExporter`'s default `ontology_uri` now both reuse, so KG-exported, OWL-exported, and PROV-exported URIs for the same `entity_id` co-resolve instead of three independently-hardcoded placeholder domains
- New CLI commands: `semantica provenance invalidate|verify-chain|descendants`
- **Fixed along the way**: `track_entities_batch()` silently absorbed batch-level typed kwargs (`agent_id`, `entity_type`, `activity_id`) into the opaque `metadata` JSON blob instead of forwarding them, so the documented banking example in `docs/guides/provenance.md` never actually worked as written
- **Fixed along the way**: `compute_checksum()` had to exclude `entity_id` itself from the hash — `track_entity()`'s versioning archives a prior value by copying it to a new key (`"X"` → `"X:v:<timestamp>"`), and hashing `entity_id` meant that legitimate relabel permanently orphaned any other entry that had already chained its `previous_checksum` from the pre-relabel value, surfacing as a false-positive "broken chain." Archival and invalidation are now always a pure relabel (unchanged checksum/sequence position) followed by a fresh chained append, never an in-place mutation of an already-chained entry
- **Fixed along the way**: `InMemoryStorage.get_chain_head()` ignored the already-committed chain head whenever the current transaction had staged any entries, understating the head and corrupting the next append's chain link
- **Fixed along the way**: several new `ProvenanceEntry` fields were initially wired into the dataclass and `export_prov()` but not into `SQLiteStorage`'s DDL/INSERT/row-mapping — `InMemoryStorage` stores the dataclass directly so it masked the gap. Added a permanent regression test (`test_all_fields_round_trip_through_sqlite`) asserting every field survives a SQLite round trip, to catch this class of bug for any future field additions
- Flagged, not fixed (separate, pre-existing issues independent of #825): `semantica/pipeline/pipeline_provenance.py` imports a nonexistent module and wraps a `Pipeline` dataclass with no `run()` method, so `PipelineWithProvenance` has never worked; most of the 18 wrapper modules' backing classes are themselves missing or incomplete (e.g. `context.context_manager`, `deduplication.deduplicator`, `normalize.normalizer` don't exist; `EmbeddingGenerator` exists but has no `.embed()`); `kg_provenance.py` passes `entity_type` inside its `metadata={}` dict instead of as a top-level `track_entity()` kwarg across most of its ~30 call sites, so it never actually populates the real field
- Extensive new test coverage across `tests/provenance/test_manager.py`, `test_schemas.py`, and `test_storage.py` (invalidation, hash-chain verification including a simulated hard-delete-detection case and an interleaved-chaining stress test, agent/activity typing, versioning/derivation split, downstream lineage, qualified export triples, bitemporal methods, Bundle export, and namespace interlinking)
- **Altair Anzo triplet store backend** (#813) by @KaifAhmad1
- Added `AnzoStore` (`semantica/triplet_store/anzo_store.py`), a fourth peer to `BlazegraphStore`/`RDF4JStore`/`JenaStore` speaking plain SPARQL 1.1 over HTTP — no new dependency, since Anzo has no official Python SDK but needs none
- The one structural difference from the existing backends: Anzo addresses data by a dataset/graphmart **URI** (`dataset_uri`, required) rather than a short namespace/repository name, so the endpoint path (`<endpoint>/sparql/<store_type>/<url-encoded_dataset_uri>`) percent-encodes it; `store_type` defaults to `"graphmart"` and can be set to `"dataset"`
- Reuses the shared `sparql_escaping.py` literal-escaping, datatype-IRI resolution, and CONSTRUCT-detection helpers rather than reimplementing them, matching `BlazegraphStore`'s CONSTRUCT/bindings `execute_sparql` contract exactly
- Wired into `TripletStore` (`backend="anzo"`, added to `SUPPORTED_BACKENDS` and `NAMED_GRAPH_CAPABLE_BACKENDS`) and `config.py` (`TRIPLET_STORE_ANZO_ENDPOINT` env var / `anzo_endpoint` config key), and exported from `semantica.triplet_store`
- 32 new tests in `tests/triplet_store/test_anzo_store.py` (mocked HTTP, no live Anzo instance needed), including dataset-URI percent-encoding cases that don't apply to the other backends
- Bulk loading uses SPARQL `INSERT DATA` (the same approach `BlazegraphStore` uses) rather than Anzo's separate HTTP Client Interface, keeping the `bulk_load()` contract identical across backends
- **Comprehensive unit and security test suite for the `/api/sparql` Explorer route** (#773) by @Sameer6305
- Added `tests/explorer/test_sparql_route.py` (34 tests) covering the SPARQL Explorer route (`semantica/explorer/routes/sparql.py`), which executes arbitrary SPARQL queries against an in-memory rdflib projection of the live graph and previously had zero test coverage
- Verified read-only allowlist enforcement against write and mutation queries (`INSERT DATA`, `DELETE DATA`, `DELETE WHERE`, `DROP ALL`, `CLEAR ALL`, `LOAD`, `CREATE GRAPH`, `MODIFY`, comments, and multi-statement injections like `SELECT ... ; DROP ALL`), confirming rejected queries short-circuit before any graph is built or queried
- Verified resource-limiting behavior, confirming row capping (`_SPARQL_MAX_ROWS`) truncates results and sets `truncated: true`, query timeout (`_SPARQL_TIMEOUT_S`) returns a clean error message without crashing, and concurrency semaphore (`_SPARQL_MAX_CONCURRENT`) prevents thread starvation under load
- Verified RDF projection fidelity for node properties and edge relationships, and error formatting for malformed SPARQL syntax with line and column extraction
- Follow-up review fixes (#805): extracted the duplicated row-cap-and-truncate loop (previously copy-pasted between the `CONSTRUCT`/`DESCRIBE` and `SELECT` branches) into a shared `_cap_rows()` helper so the `_SPARQL_MAX_ROWS` cap is enforced identically by both; added `test_row_cap_truncates_construct_results`, since the truncation path for `CONSTRUCT`/`DESCRIBE` results had no direct test coverage even though `SELECT` truncation did
- **Global default persistent storage for `ProvenanceManager`, plus a working `provenance` CLI** (#795, #802) by @Sameer6305 and @KaifAhmad1
- Every ingestion/processing module (`kg_provenance.py`, `pipeline_provenance.py`, and 20+ other call sites) instantiated its own `ProvenanceManager()` with no `storage_path`, so all of them silently fell back to `InMemoryStorage` and the SQLite audit trail was never actually written. `ProvenanceManager.set_default_storage_path(path)` now sets a class-level default that every no-arg instantiation picks up, and `Semantica.__init__` wires `config.provenance.storage_path` into it automatically during orchestrator init
- Added the thread-safe `default_storage_path(path)` context manager (`semantica.provenance.default_storage_path`) for test isolation — it stacks nested overrides and guarantees restoration of the previous default on exit, even on exception, so tests can't leak global state into each other
- Fixed `ProvenanceManager.__init__` raising `TypeError` on the CLI's `config=` kwarg, and implemented the four methods the CLI already called but that didn't exist on the class: `lineage()`, `audit_log()`, `export_prov()` (W3C PROV-O turtle/ntriples/jsonld via `rdflib`), and `check()` — unblocking `semantica provenance lineage|audit|export|check` end-to-end
- Follow-up review fixes: `track_entity` no longer aliases a caller-supplied `used_entities` list (it copied the reference and later mutated it in place via `.append()`, which could corrupt a list the caller still held); removed dead fallback branches in `orchestrator.py`/`manager.py` left over from not realizing `Config.get()` already resolves dotted paths; added a `--dry-run` option to `provenance audit` to match `provenance export` (previously only the global `--dry-run` flag worked, not a local one); and `provenance check --strict` no longer prints a green "✓" success line immediately before failing — a failing check now renders as a warning before the `ClickException` is raised
- **Markdown round-trip export/import for `AgentMemory`** (#765, #786) by @SaurabhScripts and @Sameer6305
- `AgentMemory.export(format="markdown")` and `import_data(format="markdown")` add a human-editable, diff-friendly alternative to the existing JSON/dict serialization: one Markdown file per memory item, with `id`, `created_at`, `updated_at`, and `type`/`kind` in required YAML frontmatter and the memory content as the Markdown body
- Exporting without a `destination` returns a single memory as a Markdown string; exporting a set requires a destination directory and writes one stable, content-hashed filename per memory ID, so re-exporting an unchanged set is byte-for-byte idempotent
- Importing upserts by ID: unknown IDs create new memories, known IDs replace them atomically (local state and vector store are only mutated after the whole batch validates cleanly), and unchanged re-imports are a deterministic no-op
- Malformed frontmatter, duplicate IDs within an import batch, and duplicate YAML keys are all rejected before any memory is mutated, with actionable error messages
- Export refuses to overwrite symbolic links and replaces files atomically; import safely compares timezone-aware and timezone-naive timestamps so retention, recency sorting, and date filters stay correct across both
- Entities and relationships round-trip as memory-local provenance only — Markdown import intentionally does not write into `ContextGraph`, matching the MVP scope agreed on in #765
- Documented the file contract and workflow in `docs/reference/context.md`; 43 new tests in `tests/context/test_agent_memory_markdown.py` cover round-trip losslessness, idempotency, validation errors, rollback on failure, and vector-store sync ordering
- **Markdown directory round trips for `ContextGraph`** (#852) by @SaurabhScripts
- `ContextGraph.save_to_file(..., format="markdown")` and `load_from_file(..., format="markdown")` persist a deterministic `graph.md` relationship manifest plus one human-editable Markdown file per node, preserving graph, node, edge, family, temporal, and cross-graph link identities
- Imports validate the complete directory before replacing graph state, rebuild indexes and analytics state atomically, create JSON-compatible stub nodes for dangling edge endpoints, and emit the same granular node/edge audit events as JSON loading
- Existing exports are replaced atomically only after their complete canonical layout is validated; untracked files, renamed node files, symlinks, Windows directory junctions, and other reparse points cause a fail-closed error instead of authorizing directory deletion
- Added 30 focused tests covering deterministic round trips, manual edits, validation rollback, managed-directory identity, publish rollback, audit-manager compatibility, stale-cache clearing, mocked and real Windows junctions, and missing-path behavior
### Fixed
- **Markdown import followed filesystem links even though Markdown export already refused to overwrite them** (#851, follow-up to #765, #786) by @SaurabhScripts
- `AgentMemory._read_markdown_path()` now rejects symlink files, broken symlinks, symlinked directories, Windows directory junctions, and other Windows reparse points supplied directly; linked entries discovered inside an otherwise valid directory are safely skipped, preserving the current directory-import contract
- `_read_markdown_file_content()` re-checks the file and parent directory immediately before and after opening, uses `O_NOFOLLOW` where available, and verifies the resulting descriptor is a regular file via `fstat`/`S_ISREG`, so link swaps are rejected rather than silently followed
- Junction detection uses `os.path.isjunction()` where available and falls back to the Windows reparse-point file attribute on older Python versions; export applies the same link check before replacing a Markdown file
- Documented the import restriction in `docs/reference/context.md`; added 11 tests to `tests/context/test_agent_memory_markdown.py` covering file/directory/broken-symlink rejection, simulated open races, mocked and real Windows junctions, and the reparse-point fallback
- Any additional review follow-up commits land in this same PR/entry rather than as a separate changelog item
- **`PipelineWithProvenance` raised `ModuleNotFoundError` on import and `AttributeError` on `.run()`** (#858, closes #858) by @Karunasagar12
- `from .pipeline import Pipeline` failed because `semantica/pipeline/pipeline.py` does not exist; corrected to `from .pipeline_builder import Pipeline`
- `.run()` called `self._pipeline.run()` on the `Pipeline` dataclass, which has no such method; replaced with `self._engine.execute_pipeline(self._pipeline, ...)` delegating to `ExecutionEngine`
- Constructor now accepts a built `Pipeline` instance (from `PipelineBuilder.build()`) instead of `**config`; the old `Pipeline(**config)` internal construction was invalid and never functional
- Replaced deprecated `datetime.utcnow()` with `datetime.now(timezone.utc)` in `run()`
- **`VectorStore.search_vectors()` returned inconsistent result shapes across backend implementations** (#853, closes #845) by @Sameer6305, reviewed by @KaifAhmad1
- Every built-in backend (FAISS, Milvus, pgvector, Pinecone, Qdrant, SQLite-vec, Weaviate, in-memory) now returns the same canonical `SearchResult` shape (`id`, `score`, `metadata`, `vector`, `distance`), instead of some backends omitting `vector`/`metadata`/`distance` or, for Weaviate, returning a backend-specific `properties` key instead of `metadata`
- Added a `SearchResult``TypedDict` (`semantica/vector_store/vector_store.py`, exported from `semantica.vector_store`) documenting the contract; `metadata` now always defaults to `{}` rather than being absent, and `id` accepts `Union[str, int]` to accommodate Milvus/Qdrant's native integer IDs without casting
- **Review fix**: the score-normalization formula added for Pinecone and Qdrant (`1.0 / (1.0 + max(0.0, 1.0 - score))`) clamped every raw score `>= 1.0` to an identical `1.0`, silently collapsing result ranking whenever the raw score could exceed 1 — which happens routinely for dot-product-metric indexes (unbounded), as opposed to cosine (bounded to `[-1, 1]`). Replaced with `(score / (1 + |score|) + 1) / 2`, which is strictly monotonic and bounded in `(0, 1)` for any real input, so ranking order is preserved regardless of metric or vector normalization
- Added `test_qdrant_unbounded_dot_product_scores_preserve_ranking` and `test_pinecone_unbounded_dotproduct_scores_preserve_ranking` (`tests/vector_store/test_search_result_schema.py`) asserting normalized scores stay strictly ordered and bounded for raw scores well above 1.0, the case the original formula silently collapsed and the existing tests (which only used scores `< 1`) never exercised
- Left out of scope, per the original PR: Weaviate's `similarity_search()` still isn't wired into `VectorStore.search_vectors()`'s backend dispatch; Milvus's collection schema still has no metadata column so its results always return `metadata: {}`; and `include_vectors` support (populating the `vector` field) is not yet implemented for any backend
- **`DecisionEmbeddingPipeline.find_similar_decisions()` crashed with `AttributeError` for any `VectorStore` backend other than `inmemory`** (#842, closes #839) by @Sameer6305
- `_get_candidate_embeddings()` iterated `VectorStore.vectors`/`VectorStore.metadata` directly, internal dicts only populated for `backend="inmemory"`; every persistent backend (FAISS, Pinecone, Qdrant, Milvus, ...) raised `AttributeError`. It now fetches candidates via the backend-agnostic `VectorStore.search_vectors()`, reading metadata via a `res.get("metadata") or res.get("payload")` fallback for backends that key it differently
- Backends such as FAISS don't return the raw vector for each hit; `find_similar_decisions()` and `_find_semantic_similar()` now fall back to the search-provided score (normalized from `distance` when present) as the semantic similarity for those candidates instead of computing cosine similarity against a zero placeholder vector
- `get_decision_statistics()` had the identical bug iterating `store.metadata.values()`; it now returns a limited stats payload with an explanatory `warning` field for backends that don't expose a full in-memory metadata dict, instead of crashing
- **Fixed along the way**: `_get_candidate_embeddings()`'s expand-and-retry loop (which widens the search pool when post-filtering leaves too few matches) discarded every candidate it had found once the pool hit its cap (`limit * 10`) without ever collecting `limit` matches or getting a short page back from the backend — the loop fell through without executing the branch that assigns results, silently returning `[]` even when matching candidates existed. It now falls back to the last batch collected instead of dropping it
- Added end-to-end regression tests against real `inmemory` and `faiss` backends (no mocks) plus a targeted unit test for the expand-and-retry loop's fallback behavior
- **`QdrantStore.search_vectors()` returned results keyed by `"payload"` instead of `"metadata"`** (#841, closes #840) by @divyankshah
- `QdrantCollection.search_points()` built its result dicts as `{"id", "score", "payload"}`, while `PineconeStore.search_vectors()` and every other backend consumed by `HybridSearch` use `"metadata"`. This silently dropped Qdrant metadata from results and made `HybridSearch.filter_by_metadata()` reject every candidate whenever a filter was applied, since it looks up `result["metadata"]` and got nothing back
- Normalized `search_points()` to return `"metadata"` instead of `"payload"`, matching the existing convention; no other module reads the old key, so the rename is a straight fix rather than a partial one
- Extended `tests/vector_store/test_vector_store_deepdive.py::test_qdrant_store` to assert the returned key is `"metadata"` (not `"payload"`) and that `HybridSearch.filter_by_metadata()` correctly matches against Qdrant results end-to-end
- **Explorer Temporal panel never rendered after clicking the toolbar button** (#830, #836) by @Sameer6305
- The panel stayed permanently stuck on "Loading temporal…" in `npm run dev`, with repeating "Maximum update depth exceeded" errors in the browser console. Two independent render loops were responsible:
- **Diagnostics state churn**: `handleDiagnosticsChange` unconditionally called `setGraphDiagnosticsState` on every invocation. `buildEffectAvailability` (inside `GraphCanvas`'s diagnostics `useEffect`) always returns a new object, so each call scheduled a re-render that immediately retriggered the effect. Fixed by comparing the incoming snapshot field-by-field against the last accepted value via `lastDiagnosticsRef` before calling `setState`
- **scrubberTime churn**: React 18 concurrent mode re-ran `TimelinePanel`'s `useEffect` with a structurally-new `Date` object for the same timestamp when speculative renders discarded `useMemo` caches, causing repeated `setScrubberTime` calls that propagated into `temporalState` churn and retriggered the diagnostics effect. Fixed by deduplicating by millisecond value via `onTimeChange`/`lastScrubberMsRef`
- **Bonus**: `temporal-overlay`'s `shouldLoad` predicate was changed to gate strictly on `panelState["temporal-panel"]`, removing the `|| temporalState?.currentTime` branch that caused eager loading on every scrubber update and continuously cancelled in-flight `load()` completions
- **Bonus**: `temporalState` removed from the plugin-loading `useEffect` dependency array; predicates extracted into `pluginRegistryPredicates.ts` and wired through `GraphWorkspace.tsx` so regression tests exercise the production code rather than a local copy
- The `scrubberTime`-churn fix was also applied to the equivalent (but currently unused/unmounted) `GraphWorkspaceShell.tsx`, which shares the same `TimelinePanel` integration pattern but does not have the diagnostics-churn code path
- **Follow-up review fix**: the diagnostics dedup's `structureLayer` comparison now also covers `disabledReason`, `curveCount`, `bridgeCurveCount`, and `backboneCurveCount` (previously only `cacheKey`/`lastDrawAt`/`enabled` were compared, so a pure `disabledReason` transition could leave the dev-only diagnostics panel stale)
- **Follow-up review fix**: `test:graph-store`, `test:graph-workspace`, and the new `test:plugin-registry` regression test are now run in CI (`.github/workflows/ci.yml`) — previously none of the Explorer frontend's `node --test` suites executed anywhere in CI, only `npm run build`, so this fix's own regression coverage (and all prior frontend test coverage) provided no protection against silent regressions
- **`HybridSearch.search()` crashed with `AttributeError` for any `VectorStore` backend other than `inmemory`** (#833, #837) by @KaifAhmad1
- `HybridSearch.search()` read `self.vector_store.vectors` directly, an internal dict `VectorStore` only populates for `backend="inmemory"`; every other backend (faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised `AttributeError`, making `HybridSearch` unusable against any real store. It now delegates to `VectorStore.search_vectors()` (the backend-agnostic public API) for non-inmemory backends, applies `metadata_filter` as a post-filter over the returned candidates, and normalizes results to a consistent `{id, score, distance, metadata}` shape
- **Fixed along the way**: `vector_ids` could stay `None` when callers passed explicit `vectors`/`metadata` without `vector_ids`, crashing downstream list indexing — now defaulted to generated positional IDs
- **Fixed along the way**: a `query_vector` passed as a plain list crashed backend stores (e.g. `FAISSStore.search_similar`) that call `.ndim` on it — now normalized to a numpy array up front
- **Fixed along the way**: `VectorStore.store_vectors()` silently dropped metadata for FAISS (and any `add_vectors`-only backend) because it called `add_vectors(vectors, **options)` without forwarding `metadata`, even though `FAISSStore.add_vectors()` accepts it — this blocked `HybridSearch`'s metadata filtering from ever matching anything on FAISS
- **Follow-up review fixes**: the legacy `top_k` kwarg was read but left in `options`, then forwarded via `**options` into `VectorStore.search_vectors()`, colliding with backends (sqlite, pgvector) that pass an explicit `top_k=k` to their own `search()` and raising `TypeError: got multiple values for keyword argument 'top_k'` — now popped instead of just read; `VectorStore.search_vectors()`'s dispatch only recognized backend methods named `search`/`search_similar`, so delegation still hit `NotImplementedError` for qdrant/milvus/pinecone, which name their method `search_vectors()` with a differently-named count parameter (`limit` vs `k`) — added a third dispatch branch that binds the count positionally so it works regardless of the backend's parameter name; a missing `distance` in backend-delegated results defaulted to the raw `score`, silently reusing the local path's cosine-similarity convention (`distance = 1 - score`) even for backends using unrelated metrics (L2, inner product) — now left as `None` instead of a fabricated, metric-inconsistent value
- Verified across all 7 supported backends: `inmemory`/`faiss`/`sqlite` work live end-to-end; `pgvector`'s dispatch reaches `PgVectorStore.add()`/`.search()` (blocked only by no Postgres server in the verification sandbox); `qdrant`/`milvus`/`pinecone` now reach their real `search_vectors()` method instead of crashing, though their storage side (`store_vectors()`) still doesn't recognize `insert_vectors`/`upsert_vectors`, and `weaviate` remains entirely unwired (`add_objects`/`query_vectors`) on both sides — both are separate, pre-existing gaps independent of this fix, left for a follow-up
- **`VectorStore.store_vectors()` silently dropped metadata for FAISS (and any `add_vectors`-only) backend** (#832, #835) by @KaifAhmad1
- `store_vectors()` fell into a branch that called `self._backend_store.add_vectors(vectors, **options)` without `metadata` whenever the backend exposed `add_vectors()` but neither `add()` nor `store_vectors()` — true for `FAISSStore`, the backend most real usage configures for genuine ANN search. Every caller that stores vectors with metadata (e.g. `AgentMemory._store_memory_vector()`, used internally by `AgentContext.store()`) lost that metadata once it reached FAISS, with no error or warning
- Downstream, `ContextRetriever._retrieve_from_vector()` recovers a result's text via `metadata.get("content", "")`, which was always `""` for any vector stored this way; `_rank_and_merge()` then embedded that empty string, tripping `TextEmbedder.embed_text()`'s empty-text rejection and masking the real bug as a spurious `TextEmbedder` failure recorded by the progress tracker
- `store_vectors()` now forwards `metadata` to `add_vectors()`, but only when the backend's `add_vectors()` signature actually accepts it (checked via `inspect.signature`, accepting either an explicit `metadata` parameter or a `**kwargs` catch-all), so a future/custom backend with a stricter signature raises no `TypeError`
- **Follow-up review fix**: the `inspect.signature()` probe is wrapped in `try/except (ValueError, TypeError)`, consistent with the identical pattern already used in `ProvenanceManager.trace_lineage()`, so signature introspection failing on an unusual callable can no longer abort `store_vectors()` before it even attempts to call the backend
- **`AgnoDecisionKit.check_policy` silently treated unevaluable policy rules as compliant** (#778, #822) by @Sameer6305
- `_eval_rule()` previously `return`ed `True` when a rule referenced a field missing from the decision payload, or when the rule string didn't match the expected `<field> <op> <value>` format — the docstring's claim that exceptions never silently return `compliant=True` didn't cover this, since neither path raised
- Both cases now raise `ValueError` instead, which routes through `check_policy`'s existing exception handler and records a `warnings` entry (e.g. `"Could not evaluate rule 'minimum_score >= 0.9': rule references undefined field 'minimum_score'"`) instead of disappearing with no signal
- `violations`/`compliant` are unaffected — an unevaluable rule is not counted as a violation, since it's genuinely unknown whether it would have passed; this matches the existing `compliant`/`violations`/`warnings` shape already used by `ContextGraph.enforce_decision_policy`
- This is additive: `warnings` was already part of the return contract and populated for other exception cases, so no caller that only checks `compliant` is affected, and no existing test asserts `warnings == []` for a payload that hits either of these paths
- **Follow-up review fix**: `check_policy` decoded `policy_rules` with `json.loads` and iterated the result without checking it was actually a list; a JSON-encoded bare string (e.g. `policy_rules='"confidence >= 0.7"'`) decodes to a `str`, so iterating it evaluated one "rule" per character — combined with the fix above, an 18-character rule string produced 17 warnings instead of being treated as the single rule it was meant to be. A decoded string is now wrapped as a single-element rule list; any other non-list shape (number, object, etc.) or non-string list element now produces exactly one `warnings` entry instead of silently misbehaving or being iterated character-by-character
- **Follow-up review fix**: `_eval_rule` used `data.get(field) is None` to detect a missing field, which can't distinguish a genuinely absent key from a key explicitly present with a JSON `null` value — both produced the same "undefined field" warning, misdiagnosing nullable fields. Field presence is now checked with `field not in data` first; a present-but-`null` value now raises a distinct `"field {field!r} is null — cannot evaluate rule"` message instead of the misleading "undefined field" one
- **Follow-up review fix**: `check_policy` only checked that `decision_data` was valid JSON, not that it decoded to an object. When it decoded to a list, `field not in data` silently became list-*membership* testing instead of a key check (e.g. `"confidence" not in ["confidence", 0.95]` is `False`), so a matching rule fell through to `data["confidence"]`, which raised a raw, confusing `TypeError: list indices must be integers or slices, not str` instead of any meaningful diagnostic; numbers/strings/bools produced similarly opaque `TypeError`s. `check_policy` now rejects any `decision_data` that doesn't decode to a JSON object upfront with a single clear `violations` entry, the same way it already rejects malformed JSON
- Added 15 tests to `tests/integrations/agno/test_decision_kit.py` covering the missing-field case (the issue's traced example), the malformed-rule-string case, the bare-JSON-string `policy_rules` amplification case, non-list/non-string `policy_rules` shapes, the missing-key-vs-null-value distinction, non-object `decision_data` shapes (list/number/string/bool/null), and regression checks confirming normal rule evaluation on present fields is unchanged
- **No cycle detection for SKOS concepts at write time** (#774, #819) by @mikemikimike, reviewed by @Sameer6305 and @KaifAhmad1
- Added cycle detection (`validate_skos_hierarchy`) for `skos:broader` and `skos:narrower` relationships in `ContextGraph.add_edge()` and `ContextGraph.add_edges()`, preventing direct 2-node cycles, self-loops, and multi-hop hierarchy cycles
- Added `GraphSession.add_nodes_and_edges()` to validate SKOS hierarchy edges upfront under lock before node insertion, preventing partial-write leaks where nodes remain after a cyclic edge is rejected
- Updated vocabulary, ontology (`/api/ontology/load`, `/api/ontology/create`), and JSON/CSV import routes to use `add_nodes_and_edges()` and return HTTP 422 with actionable error messages when a cycle is detected
- Follow-up fix by @KaifAhmad1:`validate_skos_hierarchy()` previously re-walked *every* SKOS hierarchy edge already in the graph on each write, so one pre-existing cycle anywhere (e.g. legacy data) blocked all unrelated future writes; it now only traverses concepts touched by the edges being written, while still checking against existing edges for cycles that span old and new data
- Follow-up fix by @KaifAhmad1: in `/api/ontology/load`, `except HTTPException: raise` was unreachable because a broader `except Exception` clause above it already matched `HTTPException`, so a 422 raised after a successful `OntologyIngestor` parse was silently swallowed and reprocessed via the fallback RDF parser; reordered the clauses so the deliberate 422 always propagates
- Follow-up fix (#775): `/api/ontology/{uri}/refresh` was missed by the original sweep and still called `session.add_nodes()` then `session.add_edges()` as two independent operations, so a cyclic SKOS edge rejected by `add_edges()` left the nodes from the preceding `add_nodes()` call committed to the graph; switched to `session.add_nodes_and_edges()` with the same `except ValueError` → HTTP 422 handling already used by `/api/ontology/load` and `/api/ontology/create`. Audited every other `add_nodes()`/`add_edges()` pairing in the repo (`GraphStore`, `graph_builder.py`, `agent_memory.py`, `context_graph.py.load()`, `enrich.py`) — none share `GraphSession`'s SKOS-cycle-validation write path, so none were changed
- `upsert_memory()` now logs `logger.warning("[%s] record_decision failed: %s", self._role, exc, exc_info=True)` when `record_decision()` fails, matching the error-logging convention used for `store()` in the same method with traceback context preserved
- Preserves graceful fallback behavior: `record_decision()` remains optional and `upsert_memory()` continues without propagating the exception
- Added regression coverage in `tests/integrations/agno/test_shared_context.py` for both `store()` and `record_decision()` warning paths
- **`AgnoDecisionKit`/`AgnoKGToolkit` silently swallowed Agno tool registration failures** (#780, #818) by @Sameer6305 and @KaifAhmad1
- Removed the `try/except: pass` wrapped around `self.register(fn)` in both toolkits' `__init__`; when Agno is installed, a registration failure now propagates immediately instead of leaving the toolkit half-registered with no signal to the caller
- Graceful degradation when Agno isn't installed (`AGNO_AVAILABLE=False`) is unchanged — `_tools` is still populated so callers can introspect available tools without the package
- Fixed a related duplicate-entry bug: `self._tools` was appended to unconditionally *before*`register()` ran, which could double-count a tool when Agno's own `Toolkit.register()` also tracks it in `self._tools`
- This is a behavior change for callers that construct these toolkits expecting instantiation to always succeed — audited: no in-repo call site relies on the old silent-failure behavior
- Expanded `tests/integrations/agno/test_decision_kit.py` and `test_kg_toolkit.py` with coverage for registration invocation counts, failure propagation, graceful degradation, and no-duplicate-`_tools` assertions
- **`ProvenanceManager` tracking methods silently swallowed failures without logging and returned fabricated entries** (#783)
- `track_relationship()`, `track_chunk()`, and `track_property_source()` now return `Optional[ProvenanceEntry]` (`None` on storage failure, consistent with #782's `track_entity` fix) instead of a fabricated populated object
- `_save_entry()` now always logs on any storage failure, including previously-silent per-item batch failures
- `track_entities_batch()` and `track_chunks_batch()`'s rare block-level transaction failures are now logged too
- `source_tracker.py`'s `track_sources_batch()` no longer counts failed tracking calls in its stats
- **MCP `handle_get_causal_chain` returned an empty-but-valid-looking response when both `CausalChainAnalyzer` and the graph fallback were unavailable** (#781, #817) by @Sameer6305 and @KaifAhmad1
- Returns an explicit `{"error": "Causal chain analysis is not supported on this graph backend", "chain": []}` instead of `{"chain": [], "count": 0, "direction": ...}`, letting clients distinguish "unsupported" from a legitimately empty chain
- The fallback path now introspects `graph.get_causal_chain`'s signature to forward `direction`/`max_depth` (or a `depth` kwarg, or nothing, depending on what the backend accepts) instead of always calling with just `decision_id`, matching the primary analyzer path's behavior
- Hardened input handling: non-dict `args`, non-string `decision_id` (previously a latent `AttributeError` on `.strip()`), and `max_depth` clamped to `(0, 100]` with a safe default on invalid input
- Added `tests/test_mcp_decisions_causal_chain.py` (11 tests) covering the unsupported-backend, fallback-forwarding, and validation/exception paths across multiple backend signature shapes
- **Follow-up review fix**: the signature-detection try/except previously caught the *actual call*'s exceptions in the same block used for introspection failures, so a genuine bug inside a backend's `get_causal_chain` (raising an unrelated `TypeError`) was misread as a signature mismatch and the backend was invoked a second time with identical arguments before the real error surfaced. Signature introspection and the resulting call are now split into separate try/excepts so a successfully-introspected call is made exactly once; added `test_internal_typeerror_calls_backend_only_once` to lock this in
- **`ProvenanceManager.track_entity` persisted partial history and returned fabricated entries on storage failure** (#782, #816) by @Sameer6305 and @KaifAhmad1
- `track_entity()`'s two-step write (history archive + primary update) is now atomic — if either write fails, the whole operation rolls back via the existing #807`transaction()` mechanism, instead of silently persisting a partial state
- `track_entity()`'s return type is now `Optional[ProvenanceEntry]`: on failure it returns a safe deep copy of the pre-failure existing entry (if one existed) or `None` (if this was a brand-new, never-successfully-tracked entity) — never a fabricated object claiming values that were never actually persisted
- This is a behavior change for callers that inspect the return value without checking for `None` first — audited: 0 of 47 production call sites in the repo currently dereference the return value, so this is safe today, but any NEW caller must handle `None`
- `InMemoryStorage` gained real transactional rollback (staging-buffer based) to match this guarantee — previously `transaction()` was a no-op
- **`ProvenanceManager` duplicated the same checksum/persist/exception-swallow block across 4 tracking methods** (#784, #815) by @Sameer6305 and @KaifAhmad1
- Consolidated the repeated `entry.checksum = compute_checksum(entry)` / `try: self.storage.store(entry) except Exception: pass` block used by `track_entity`, `track_relationship`, `track_chunk`, and `track_property_source` into a single `ProvenanceManager._save_entry()` helper, preserving the existing graceful-failure behavior and the batch `_conn`/re-raise semantics from #807
- Added 4 regression tests (`tests/provenance/test_manager.py`) covering storage-failure swallowing for each of the four tracking methods, none of which had coverage for this path before
- **Follow-up review fix**: the initial refactor of `track_entity`'s exception fallback (the branch that runs when a failure happens *before* the entry is built, e.g. a retrieve error inside the atomic transaction) routed through `_save_entry()`, which made a new `self.storage.store(entry)` call outside the already-failed transaction — a real behavioral change from the original code (which only computed a checksum on that path) that could have reintroduced the exact race #807's `BEGIN IMMEDIATE` transaction serialization was meant to prevent. Reverted that branch to only compute the checksum, and added `test_track_entity_pre_build_failure_fallback_skips_store` asserting `storage.store` is never called on that path
- **`SQLiteStorage` and `ProvenanceManager` connection churn, non-atomic writes, and batch tracking overhead** (#807) by @Sameer6305
- Scoped a single SQLite connection to the full duration of each public storage method call (`track_entity()`, `store()`, `retrieve_all()`, `clear()`) instead of opening independent connections per internal SQL statement, reducing connection churn by ~67% while closing the handle before the public method returns to preserve Windows filesystem unlink safety
- Implemented the `SQLiteStorage.transaction()` context manager with Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `busy_timeout=5000`, `synchronous=NORMAL`, and immediate write transactions (`BEGIN IMMEDIATE`), ensuring concurrent read-modify-write sequences (including history version ID generation) are serialized without lock contention or data loss
- Added block-level transaction sharing to `track_entities_batch()` and `track_chunks_batch()`, reducing SQLite commit overhead by ~99.9% for large batches and deferring `tracked_count` increments until successful commit so rolled-back items are never reported as successes
- Preserved 100% backward compatibility for custom storage backends overriding `trace_lineage(self, entity_id)` by inspecting signatures dynamically before passing `max_depth`, and optimized BFS lineage queries with batched IN-clause lookups per frontier level
- **Follow-up fix**: `retrieve()` and `trace_lineage()` were initially routed through `transaction()` too, so plain reads took the same `BEGIN IMMEDIATE` writer lock as read-modify-write calls, serializing every read behind every other read/write and defeating the WAL concurrency this PR was meant to add. They now use a dedicated `_read_connection()` (configured, no explicit `BEGIN`) so reads no longer contend for the writer lock
- **Follow-up fix**: `track_entity()`/`track_chunk()` caught all internal storage exceptions unconditionally, so when called from `track_entities_batch()`/`track_chunks_batch()`'s shared per-block transaction, a single item's storage failure (e.g. non-JSON-serializable metadata) was swallowed inside the call and never surfaced to the batch loop's per-item `except`, inflating `tracked_count` for entries that were never persisted. Both methods now re-raise when invoked with a shared `_conn` (batch context) while still degrading gracefully on standalone calls, so batch counts match what's actually committed
- Added 8 dedicated regression tests in `tests/provenance/test_sqlite_storage_performance_807.py` covering PRAGMA configuration, Windows unlink safety, batch transaction sharing, BFS `max_depth`, rollback count accuracy, custom storage backward compatibility, concurrent read-modify-write serialization, and connection cleanup guards on configuration error
- **Closed remaining `ProvenanceManager` storage-failure test-coverage gaps identified by a #785 audit** (#785)
- An audit of `tests/provenance/` (filed against a claim that zero tests exercised `storage.store()` failures) found #782/#783/#784/#807 had already closed most of the gap, but two residual surfaces had no test: `track_relationship()`, `track_chunk()`, and `track_property_source()`'s storage-failure-swallowing contract (returns `None`, logs, persists nothing) was only verified against `InMemoryStorage`, never `SQLiteStorage`; and `track_chunks_batch()` had no test for per-item `_save_entry` failure logging or for the block-level transaction-failure log message, even though `track_entities_batch()` had both
- No production code changed — #782/#783/#784/#807 already implemented the correct behavior; this closes the coverage gap proving it holds on both backends
- Added `test_track_relationship_storage_error_swallowed_sqlite`, `test_track_chunk_storage_error_swallowed_sqlite`, `test_track_property_source_storage_error_swallowed_sqlite`, `test_chunks_batch_logs_per_item_failure_memory`, and `test_track_chunks_batch_block_level_transaction_failure_logs` to `tests/provenance/test_manager.py`
- Read-path failure coverage (`get_lineage()`/`trace_lineage()`/`get_provenance()`/`clear()` propagating a raised storage exception) remains untested and is a candidate for a follow-up issue, since none of those methods currently wrap the underlying storage call in a try/except
- **Explorer's Provenance UI used a naive 2-hop graph traversal instead of the audit-grade `ProvenanceManager` backend** (#792, #809) by @Sameer6305
- `semantica/explorer/routes/provenance.py` never imported or called `ProvenanceManager` (`semantica/provenance/manager.py`); `/api/provenance` and `/api/provenance/report` built their lineage response entirely from a naive 2-hop networkx traversal over the live graph instead of querying the SQLite-backed, checksummed audit log. Both endpoints now query `session.provenance_manager.get_lineage(node_id)` first, and a new `_transform_audit_lineage()` maps the W3C PROV-O entries into the exact `{"nodes": [...], "edges": [...]}` shape `LineageDiagram.tsx` already expects — no frontend changes required
- Falls back to the original 2-hop traversal, never a 500: no audit records for a node, a `ProvenanceManager` storage failure (corrupted DB, permissions), or a failed SHA-256 integrity check on any entry in the lineage chain all degrade cleanly to the naive path. A new `source: "audit" | "graph_traversal"` field on the response discloses which path actually served the data
- `ProvenanceManager.get_lineage()` now returns `integrity_verified`, computed by re-verifying every entry's checksum before it's trusted; a single tampered or corrupted entry anywhere in the lineage chain now falls the *entire* response back to graph traversal rather than serving partially-verified audit data
- Replaced an initial classmethod-based `ProvenanceManager.set_default_storage_path()` approach (caught in review before merge — it would have let any two sessions/apps in the same process silently share and overwrite each other's storage path, including across unrelated test runs) with `provenance_storage_path` threaded through `GraphSession.__init__` and `create_app(...)`, so each session's `ProvenanceManager` is independently scoped
- Disclosed limitation: `ProvenanceManager.trace_lineage()`/`get_lineage()` only walk `parent_entity_id`/`used_entities` backward, so the audit path currently surfaces upstream lineage only — the naive fallback remains the only source for downstream/descendant relationships until `ProvenanceManager` gains a reverse lookup
- New `tests/explorer/test_provenance_manager_wiring.py` (8 tests): the audit path via a real multi-hop `track_entity()` chain, empty-record fallback, simulated storage-failure degradation (asserts `200`, not `500`), checksum-tamper fallback, evidence-field preservation, `create_app()` storage-path wiring, and cross-session storage isolation, confirmed order-invariant across `tests/explorer/` and `tests/provenance/` in both execution orders
- **`POST /shacl/validate` and the `/health` SHACL dimension never ran live SHACL validation** (#772, #804) by @Sameer6305 and @KaifAhmad1
- `/shacl/validate` had no data graph to validate submitted shapes against — only a Turtle syntax check. Added `_data_graph_turtle_for_uri()`, which serializes the loaded ontology's nodes/edges into an RDF/Turtle instance graph (CURIE resolution across owl/rdfs/skos/dct/dc, arbitrary node-property projection, typed individuals) and wires both `/shacl/validate` and the `/health` SHACL dimension to `OntologyEngine.validate_graph()` via pySHACL, returning real `conforms`/violations instead of a hardcoded `status="unavailable"` stub
- Fixed a cross-ontology namespace leak in `_node_belongs_to_ontology`: its prefix fallback (`_extract_namespace()`) split only on the last `/`, so sibling ontologies sharing a domain (e.g. `.../onto-a` and `.../onto-b`) could match entities across ontologies that shouldn't be related; fixed by comparing against the full URI stem via the new `_ontology_namespace()` helper
- Added resource guardrails to `/shacl/validate` to close a DoS risk flagged in review: a submitted-Turtle byte cap (`SEMANTICA_MAX_SHACL_TURTLE_BYTES`, default 256 KB), a parsed-triple cap (`SEMANTICA_MAX_SHACL_TRIPLES`, default 1,000), a validation timeout (`SEMANTICA_MAX_SHACL_TIMEOUT`, default 15s), and a global concurrency semaphore (`SEMANTICA_MAX_SHACL_CONCURRENCY`, default 4)
- Fixed `HealthDimension.status` being set to `"error"` on a real (non-`ImportError`) validation exception, which isn't a valid value on that model — Pydantic construction raised and turned the whole `/health` endpoint into a 422 on any real bug; now reports `status="critical"` (already a valid value) with a regression test forcing this exact path
- Follow-up review fixes: reverted an unrelated regression that had crept into this PR — `POST /api/ontology/create` had gone back to silently swallowing `OntologyEngine.from_data`/`from_text` failures into a near-empty "minimal" ontology instead of raising `HTTPException(500)`, undoing the earlier #770/#787 fix for the same endpoint (and breaking `TestOntologyCreateFailures`, which wasn't run before this PR's initial merge request); `sh:Warning`/`sh:Info`-severity pySHACL results were silently dropped from the `/shacl/validate` response — a shape using non-`Violation` severities could report `conforms=False` with an empty `violations` list and no explanation, so warnings/infos are now folded into the response's `violations` array; and `/health` was independently re-fetching and re-truncation-checking the same ontology's nodes/edges once for the generated SHACL shapes and once for the data graph — both now share a single fetch via `_fetch_analysis_graph()`
- New regression tests: `TestOntologyCreateFailures` (pre-existing, now passing again), `test_shacl_validate_surfaces_warning_severity_results`, `test_health_dedupes_node_edge_fetch`, plus the existing 26-test `tests/explorer/test_ontology_subissue3.py` suite (28/28 passing) and the pre-existing `tests/ontology/` suite (83/83 passing)
- **Neptune cookbook CloudFormation stack exposed the database port to the entire internet and had no network audit trail** ([code scanning alert #28](https://github.com/semantica-agi/semantica/security/code-scanning/28), [#26](https://github.com/semantica-agi/semantica/security/code-scanning/26), [#27](https://github.com/semantica-agi/semantica/security/code-scanning/27), `AC_AWS_0276`/`AC_AWS_0369`/`AC_AWS_0148`) by @KaifAhmad1
- `cookbook/introduction/neptune-setup.yaml`'s security group let anyone on `0.0.0.0/0` reach the Neptune Bolt/OpenCypher port (8182); it now requires a `ClientCidr` parameter (CIDR-validated, no default) so the stack can't be created without the deployer explicitly scoping access to their own IP or VPN/office range
- Added `AWS::EC2::FlowLog` plus a dedicated CloudWatch Logs group and IAM role so all traffic in the stack's VPC is now logged
- Left the account-wide IAM password policy check (`AC_AWS_0148`) unimplemented as a stack resource on purpose: `AWS::IAM::AccountPasswordPolicy` is an account singleton, and wiring it into a disposable per-learner tutorial stack would mean creating or deleting this stack also mutates or removes the account's real password policy — suppressed with a documented `ts:skip=AC_AWS_0148` explaining why, rather than "fixed"
- Updated `21_Amazon_Neptune_Store.ipynb`'s `aws cloudformation create-stack` instructions, prerequisites, and cost table to match the new required `ClientCidr` parameter and flow-log line item
- **Follow-up to the knowledge-explorer Helm chart default-namespace/seccomp scanner findings reopening** ([code scanning alert #846](https://github.com/semantica-agi/semantica/security/code-scanning/846), [#847](https://github.com/semantica-agi/semantica/security/code-scanning/847), [#848](https://github.com/semantica-agi/semantica/security/code-scanning/848), [#68](https://github.com/semantica-agi/semantica/security/code-scanning/68), [#63](https://github.com/semantica-agi/semantica/security/code-scanning/63), `CKV_K8S_21`/`AC_K8S_0086`/`AC_K8S_0080`) by @KaifAhmad1
- The `checkov.io/skip1` metadata annotation added previously (see the `CKV_K8S_21` entry below) evidently isn't being honored by the Microsoft Defender for DevOps scan — the same finding reopened under new alert numbers on the current `main`. Added the more standard `# checkov:skip=CKV_K8S_21` and `# ts:skip=AC_K8S_0086` inline comments at the top of `templates/deployment.yaml`, `templates/service.yaml`, and `templates/configmap.yaml` as a second suppression path (matching the convention already used in `deploy/gcp/cloudrun-service.yaml`), plus `# ts:skip=AC_K8S_0080` on `templates/deployment.yaml` for the seccomp finding, which trips for the same root cause: terrascan's static template scan never resolves `{{ toYaml .Values.podSecurityContext }}`, even though `values.yaml` sets `seccompProfile.type: RuntimeDefault` correctly
- Confirmed the `deploy/kubernetes/*` (non-Helm) manifests already had TLS and seccomp configured correctly, so no code change was needed there for the corresponding alerts (#61 and the non-Helm seccomp finding) — expected to close on the next scan
- Documented both suppression mechanisms and the reasoning in `.checkov.yaml`
- Residual risk: this environment could not run checkov/terrascan locally to confirm the inline comments are actually honored during a Helm-rendered scan; if the alerts are still open after the next scan, the reliable fallback is splitting the CI checkov/terrascan invocation so `deploy/helm/` is scanned with these specific checks excluded via `--skip-check` instead of relying on in-file suppression
- **`react-hooks/set-state-in-effect` cascading renders across 12 Explorer workspace files** (#769, #796) by @Sameer6305 and @KaifAhmad1
- Replaced synchronous `setState` calls inside `useEffect` bodies with React's recommended "adjust state during render" pattern (`if (x !== prevX) { setPrevX(x); ...setState... }`) across `OntologyWorkspace`, `ManageWorkspace`, `LineageWorkspace`, and `GraphWorkspace`, and inlined async data-fetching effects with `ignore` flags to prevent race conditions and stale writes after unmount
- Fixed a regression the inlining itself introduced: `AlignmentsTab.tsx`, `KGOverviewTab.tsx`, `OntologyManager.tsx`, and `VersionsTab.tsx` each duplicated their existing fetch callback (`reload` / `fetchOverview` / `fetchRegistry` / `loadVersions`+`loadProposals`) into a second, inline copy for the mount effect, and the copy silently dropped the `setError`/`flashMsg` calls the original had — re-introducing, on the very first page load, the exact error-swallowing behavior that #767/#790 had already fixed for these same files. The inline copies now mirror the original's error handling (including `207` partial-success messages) exactly
- Fixed `LineageDiagram.tsx` only clearing the previously-rendered nodes/edges when the new `activeId` was falsy instead of on every id change, so switching directly between two lineage views briefly kept showing the *previous* view's stale diagram instead of clearing before the new fetch resolved
- `GraphWorkspace.tsx` and `GraphLoadingOverlay.tsx` still have unrelated `react-hooks/set-state-in-effect` violations outside this PR's 12-file scope (confirmed via `npx eslint .`); left as follow-up work rather than expanding this PR further
- **Checkov flagged the knowledge-explorer Helm chart for using the default Kubernetes namespace** ([code scanning alert #779](https://github.com/semantica-agi/semantica/security/code-scanning/779), [#778](https://github.com/semantica-agi/semantica/security/code-scanning/778), [#777](https://github.com/semantica-agi/semantica/security/code-scanning/777), `CKV_K8S_21`) by @KaifAhmad1
- `templates/service.yaml`, `templates/deployment.yaml`, and `templates/configmap.yaml` all already set `metadata.namespace` to `{{ .Release.Namespace }}`, which is only bound at `helm install`/`helm template` time; Checkov's helm framework renders the chart without a namespace override, so it always resolves to `default` and trips `CKV_K8S_21` even though the chart is namespace-agnostic by design
- Added a `checkov.io/skip1: CKV_K8S_21` metadata annotation to each of the three files to suppress the scanner artifact false-positive properly in Helm templates, and documented the reasoning in `.checkov.yaml`
- **No React error boundaries around lazy-loaded Explorer workspaces — a single render error crashed the whole app** (#768, #794) by @Sameer6305
- Added an `ErrorBoundary` class component (`explorer/src/ErrorBoundary.tsx`) and wrapped each lazy-loaded workspace's `<Suspense>` block in `App.tsx` with it, keyed on the active sub-view so navigating away from and back to a crashed tab remounts it cleanly
- Failed retries are capped at 3 before the fallback UI switches from "Try Again" to a "Reload Application" dead-end, preventing infinite retry loops on deterministic crashes; raw error/stack details are logged via `console.error` only and never rendered into the fallback UI
- Fixed the retry counter so it resets after a retry actually succeeds and stays error-free for a few seconds, instead of never resetting (which could permanently exhaust the retry budget on unrelated, individually-recoverable transient errors) or resetting on the very next commit (which could fire prematurely while `Suspense` was still showing its fallback)
- `ShaclStudio.tsx`, `VersionsTab.tsx`, `SKOSVocabularyManager.tsx`, `EntityResolutionTab.tsx`, `LineageDiagram.tsx`, `DecisionWorkspace.tsx`, `KGOverviewTab.tsx`, `OntologyManager.tsx`, `OntologySearch.tsx`, `ReasoningWorkspace.tsx`, and `SparqlWorkspace.tsx` now render a visible error banner instead of only `console.error()`-ing failed fetches
- Added explicit `response.status === 207` (Multi-Status) handling across these workspaces so partial backend failures surface a warning instead of reading as a full success (`response.ok` is `true` for all 2xx codes, including 207)
- Added defensive JSON parsing so an unexpected non-JSON (e.g. HTML 500) response body no longer crashes the app with `SyntaxError: Unexpected token < in JSON`
- Fixed `KGOverviewTab.tsx` dropping the `/api/graph/nodes` partial-success warning whenever `/api/graph/stats` also returned 207 — both warnings are now shown (appended) instead of one being silently discarded
- Fixed `HealthTab.tsx`'s registry load still using a bare `.catch(() => {})` that swallowed errors identically to the pattern fixed elsewhere in this same folder; failures now populate the existing error banner
- Fixed `AlignmentsTab.tsx`'s `reload()` using `Promise.allSettled` but never handling the `"rejected"` branches for the registry/alignments fetches, so both failures previously vanished with no error surfaced and no logging
- **`tests/explorer/test_explorer_api.py` failed with `TypeError: Client.__init__() got an unexpected keyword argument 'app'` on current httpx** (#788, #789) by @Sameer6305
- `httpx>=0.28.0` removed the `app=` kwarg that Starlette's `TestClient` relies on to wrap a FastAPI app for testing; `httpx` wasn't pinned anywhere in `pyproject.toml`, so different environments could independently resolve an incompatible transitive version and hit the same break
- Added an explicit `httpx<0.28.0` constraint to the main `[project.dependencies]` array (not just a dev extra), so it applies globally across production, dev, and CI installs
- Without the pin, the full test suite fails to even complete collection (fails immediately on `tests/explorer/test_vocabulary.py` with the same `TestClient` error); with it, `tests/explorer/test_explorer_api.py` goes from 7 failed/12 passed/58 errors to 77 passed, 0 errors
- **Explorer backend routes returned HTTP 200 with error/empty bodies on failure, defeating frontend error handling** (#770, #787) by @Sameer6305 and @KaifAhmad1
- `GET /api/temporal/patterns` now raises `HTTPException(500)` on a genuine computation failure instead of silently returning an empty-but-valid `TemporalPatternResponse`; the `ImportError` fallback (optional `kg` extra not installed) is unchanged and still degrades gracefully to an empty list
- `POST /api/ontology/create` now raises `HTTPException(500)` when ontology generation fails in either the `sample_data` or `schema_text` mode, instead of silently falling back to a partial/minimal ontology with a misleading `nodes_added` count
- `GET /api/analytics` sets `response.status_code = 207` (Multi-Status) when some, but not all, of the requested metrics fail, and raises `HTTPException(500)` when every requested metric fails — a plain 2xx (including 207) reads as success to callers that only check `response.ok`, so an all-failed request now surfaces as a hard error rather than a body full of `{"error": ...}`
- Added regression tests covering all three failure paths (`test_patterns_failure_returns_500`, `test_analytics_partial_failure_returns_207`, `test_analytics_total_failure_returns_500`, and two `TestOntologyCreateFailures` cases)
### Security
- **DNS check-then-use hardening for the ontology URL fetcher, and a remaining object-IRI validation gap** (#916, follow-up to GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236) by @KaifAhmad1
- **DNS check-then-use (TOCTOU) window**: GHSA-8c7v-62gr-hj6g's own fix description flagged this as a secondary gap — `_validate_fetch_url()` resolved and validated a hostname once, but `_fetch_url_sync()` then let `requests` resolve the same hostname again independently at connect time. A low-TTL or rebinding DNS answer could differ between the two lookups, reopening the SSRF window the validation exists to close
- `_validate_fetch_url()` now returns the validated IP, and a new `_make_pinned_session()` builds a per-hop `requests.Session` whose connection pool is pinned directly to that IP — bypassing DNS resolution for the connection entirely — while explicitly restoring the real hostname as the outgoing HTTP `Host` header and, for HTTPS, the TLS SNI `server_hostname`/`assert_hostname`, so the connection reaches the validated IP but still presents (and is verified against) the real hostname's identity, keeping virtual hosting and certificate validation correct
- Caught during implementation: an earlier draft set urllib3's `_dns_host` post-construction, assuming (as in some urllib3 releases) that it was decoupled from `host`. In the version this project installs (2.7.0), `host` is a property that reads/writes `_dns_host` directly, so that approach would have silently changed the Host header too — caught by an end-to-end test against a real local server before landing, rather than shipping. Verified with real (non-mocked) local HTTP and HTTPS servers, the latter using a generated self-signed certificate to prove SNI/cert-hostname verification checks the real hostname rather than the pinned IP, plus a negative control confirming a hostname/cert mismatch is still correctly rejected, not silently bypassed
- **Object-IRI validation gap** (GHSA-8vgg-8mr4-r236 follow-up, distinct from the object-branch fix already shipped in #911): a triplet object already wrapped in `<...>` skipped `sparql_escaping.validate_uri()` in both `blazegraph_store.py` and `rdf4j_store.py`'s `_format_object_for_sparql`/`_format_object_for_ntriples`, only checking the inner content for a literal space or `>` — the pre-wrapped and unwrapped branches now validate identically
- **Fixed along the way** (caught in automated review across two follow-up rounds): `_validate_fetch_url()` originally pinned to only the first resolved IP, so a hostname with multiple A/AAAA records would fail outright if that specific address was unreachable — it now returns every validated IP and `_make_pinned_session()` falls back through all of them, verified by pinning to a genuinely unreachable address followed by a working one and confirming the fetch still succeeds; the test HTTPS server allowed TLSv1/TLSv1.1 by not setting a minimum version, now pinned to TLSv1.2; and when an HTTP(S) proxy applied, pinning was silently skipped in favor of the unpinned path — proxies are now disabled outright for this fetcher (`session.trust_env = False`, so `HTTP_PROXY`/`HTTPS_PROXY` env vars are never consulted) with a fail-closed 502 backstop if a proxy is ever forced onto the session some other way, verified by pointing `HTTP_PROXY` at an address that would fail if actually used and confirming the fetch still succeeds directly
- New `tests/explorer/test_ontology_dns_pinning.py` (12 tests: real local HTTP/HTTPS servers including 2 real-TLS checks, multi-IP fallback success/failure, and no-proxy-trust verification — gracefully skipped without the optional `cryptography` package where applicable); updated `tests/explorer/test_ontology_ssrf.py` for the new per-hop session construction; 4 new tests in `tests/triplet_store/test_sparql_injection.py` for the object-IRI fix. Full `explorer` + `triplet_store` suite: 572 passed
- **Missing Origin validation on the `/ws/graph-updates` WebSocket handshake** (#917, GHSA-4643-wpgq-w329) by @KaifAhmad1
- `CORSMiddleware` doesn't cover WebSocket handshakes at all (Starlette's CORS support only wraps HTTP), so under `SEMANTICA_ALLOW_ANONYMOUS=true` — the mode `docker-compose.dev.yml` ships — the anonymous-mode key bypass accepted a `/ws/graph-updates` connection from any origin. Loopback binding isn't a boundary against a browser: any page the operator has open can still reach `ws://localhost:8000/ws/graph-updates` directly, and `ConnectionManager.broadcast` sends every `graph_mutation` to every connected socket with no per-connection scoping. Combined with `/api/import` accepting `multipart/form-data` (a CORS-safelisted content type that skips preflight), a hostile page could write to the graph over REST and read the result back over the unauthenticated WebSocket
- Not affected: any deployment with `SEMANTICA_API_KEY` configured — the handshake already rejects without a valid key in that mode. This was an anonymous-mode-only, development-configuration exposure
- Fix: check the handshake's `Origin` header against `app.state.explorer_settings['allowed_origins']` — the same list `CORSMiddleware` already enforces for HTTP — before the key check. A missing `Origin` (native/CLI clients, which never set the header) is still allowed through, since the browser is the only threat this closes
- 4 new tests in `tests/explorer/test_explorer_auth.py`: hostile Origin rejected under anonymous mode; hostile Origin rejected even with a correct key (Origin is checked first, so a leaked key alone can't hijack the socket); an allowlisted Origin still connects; a missing Origin still connects. Full `explorer` suite: 226 passed
- **Polynomial-time ReDoS in the SPARQL route's `_PREFIX_DECL` regex** (#915, CodeQL `py/polynomial-redos`) by @Sameer6305
- The prior pattern's trailing `\s*` overlapped with the preceding `<[^>]*>` IRI-body match on inputs containing no closing `>` (e.g. `base<` followed by thousands of `!<` repetitions), forcing the regex engine to explore every possible split between the two quantifiers — O(n²) backtracking reachable from `req.query` via `_is_read_only_query()`
- Fixed by making the two quantifiers character-disjoint: horizontal whitespace only (`[ \t]`, never overlapping the IRI body) instead of `\s*`, and excluding CR/LF from the IRI body (`[^>\r\n]*`) so it can never span a line boundary. Independently verified: the exact pathological payload (`base<` + `!<`× 5,000/20,000) scales linearly (0.238ms → 0.841ms for 4x input, not the ~16x a surviving quadratic blowup would show)
- Added `_SPARQL_MAX_QUERY_LEN = 10_000` as defense-in-depth, checked in `execute_sparql()` before any regex work so a future pattern regression stays bounded regardless
- Two correctness regressions raised in review were checked and did not reproduce: comment-then-prefix stripping order means an inline comment after a `PREFIX` line (`PREFIX ex: <...> # comment`) is already gone by the time `_PREFIX_DECL` runs, verified directly against the pipeline; and the allowlist's `.sub()`-based cleaning only ever affects the yes/no decision, never the query actually sent to `graph.query()` — so even the narrow case of a multi-line string literal that happens to start a line with the literal text `PREFIX` or `BASE` can only cause a legitimate query to be wrongly rejected, never let something malicious through, since rdflib's parser still gates whatever actually executes
- 20 new/updated tests in `tests/explorer/test_sparql_route.py` and `tests/test_security_regression.py` (inline prologues, CRLF line endings, multi-line CRLF prefix chains, oversized-query rejection). 225 `explorer` + 82 SPARQL-specific tests passing
- **SPARQL injection via unvalidated triplet IRIs** (#911, GHSA-8vgg-8mr4-r236) by @KaifAhmad1
- `Triplet.subject`/`.predicate` (and, in some builders, `.object`) were interpolated directly into SPARQL update/query strings in the Blazegraph and RDF4J stores, and into a SELECT filter in the Jena store. A subject containing `>` closes the `<...>` IRI token early, so the rest of the value is parsed as more SPARQL. Entity names are document text in the normal ingest pipeline, so anyone whose content gets processed could append operations like `CLEAR ALL`, running with the application's store credentials
- Applied the existing `sparql_escaping.validate_uri` (already used by `anzo_store.py`, the one backend that was already hardened — this generalizes its approach rather than inventing a new one) at every subject/predicate/object interpolation site: `blazegraph_store.py`'s `_build_insert_data`, `_triplets_to_rdf`, `bulk_load`'s `graph` option, `get_triplets`'s filter, and `delete_triplet`; `rdf4j_store.py`'s `_triplets_to_ntriples`, `get_triplets`'s filter, and `delete_triplet`; `jena_store.py`'s `get_triplets`'s filter (the only vulnerable site there — `add_triplets`/`delete_triplet` already use rdflib's native `Graph.add`/`.remove` with `URIRef` rather than building query strings)
- **Fixed along the way** (caught in review, by @ZohaibHassan16): `_format_object_for_sparql`'s URI branch — used when a triplet's *object* is itself a URI rather than a literal — only checked for spaces and `>` inline instead of running the same `validate_uri` check applied to subject/predicate, leaving the object position as a narrower but real gap in both Blazegraph and RDF4J. Also fixed test flakiness in `RDF4JStore`'s test fixtures, which weren't mocking `_connect()` and so were making real network calls
- New `tests/triplet_store/test_sparql_injection.py` (12+ tests) reproducing the advisory's own injection payload (`http://example.com/a> ... ; CLEAR ALL ; INSERT DATA { ...`) against all three backends' write and read paths, asserting the malicious query is never built or sent. Full triplet_store suite: 330+ tests passing
- Side note, not part of this fix: found that `jena_store.py`'s `get_triplets()` builds syntactically invalid SPARQL for its WHERE-clause filters (missing a `FILTER()`/separator before the equality conditions) — a pre-existing correctness bug, unrelated to the injection fix, left alone here and worth a separate follow-up
- **Cypher injection via unvalidated node labels, relationship types, and property keys** (#910, GHSA-482h-hw99-h62p) by @KaifAhmad1
- Node labels and property keys passed to `create_node`/`create_relationship` were interpolated directly into Cypher strings in the Neptune, Neo4j, and FalkorDB graph stores. Property *values* are parameterized, but labels and keys can't be bound as query parameters, and nothing validated them — so a document-derived entity type or property name (the normal ingest path) could close the current Cypher token early and append arbitrary statements (e.g. `DETACH DELETE`), running with the application's database credentials
- New shared `semantica/graph_store/query_sanitize.py`: `sanitize_identifier()` generalizes `age_store.py`'s existing `_sanitize_label`/`_sanitize_rel_type` (the only backend that already validated this) into a helper the other backends import without an import cycle with `graph_store.py`/`methods.py`
- Applied at every label/relationship-type/property-key interpolation site in `amazon_neptune.py`, `neo4j_store.py`, `falkordb_store.py`, `graph_store.py` (`degree_centrality`'s own query builder), and `methods.py` (`update_relationship`'s own query builder) — covers `create_node`, `create_nodes`, `create_relationship`, `get_nodes`, `get_relationships`, `get_neighbors`, `shortest_path`, `update_node`, `create_index`, and all relationship-type filters across the three backends
- **Fixed along the way** (caught in review, by @Sameer6305): `depth`/`max_depth` path-length parameters are meant to be integers, but `Neo4jStore.get_neighbors()`/`shortest_path()` interpolated them into the Cypher variable-length-path syntax (`*1..{depth}`) without coercion — unlike the Neptune/FalkorDB equivalents, which already cast to `int()`. A string `depth` (e.g. `"1]->(x) DETACH DELETE x //"`) reached the query verbatim. Added the same `int()` coercion Neptune/FalkorDB already had, plus `GraphStore.get_neighbors()`'s `hops`/`depth` alias resolution
- New `tests/graph_store/test_cypher_injection.py` (unit tests on `sanitize_identifier` plus the labels/keys/rel-types injection payload run against Neptune/Neo4j/FalkorDB `create_node`/`create_relationship`, asserting the malicious query is never built or sent) and the depth-coercion regression above; plus additions to `tests/test_graph_store.py` (`degree_centrality`) and `tests/test_graph_store_methods.py` (`update_relationship`). Full graph_store suite: 224+ tests passing
- **4 critical/high vulnerabilities in the Explorer API and vector store: RCE, SSRF, XXE, and DoS, plus Cypher/SPARQL injection hardening found along the way** (#898) by @Sunil56224972
- **[CWE-502] Arbitrary code execution via `pickle.load()`**: `VectorStore.save()`/`load()` used `pickle` for the on-disk `store_data.pkl`; a crafted `.pkl` file placed in the store directory (file upload, shared filesystem, or supply-chain compromise) could execute arbitrary code on deserialization. Replaced with JSON — vectors and metadata are fully JSON-serializable, so nothing is lost — and `load()` now refuses any legacy `.pkl` file it finds with a migration error rather than deserializing it
- **[CWE-918] SSRF via redirect bypass in `ontology.py`'s URL fetcher**: `_validate_fetch_url()` correctly blocked private/loopback/reserved addresses on the caller-supplied URL, but `_fetch_url_sync()` fetched with `allow_redirects=True`, so a validated *public* first hop could 302 to `http://169.254.169.254/...` (cloud instance metadata) or an internal service, and `requests` followed it with no re-check. Redirects are now followed manually, capped at 5 hops, with `_validate_fetch_url()` re-run against every hop's target — including relative `Location` headers, resolved via `urljoin()` before validation — and every response (redirect or final) is explicitly closed to avoid leaking connections back to the pool
- **[CWE-611] XXE injection in the RDF/XML parser**: `_safe_parse_rdf()` depended on `defusedxml` for XXE protection, but `defusedxml` wasn't declared in `pyproject.toml`'s `explorer` extra, so it was silently absent in normal installs and the code fell back to a bare warning plus unsafe parsing — a crafted RDF/XML ontology with an external entity could read arbitrary server files. Added `defusedxml>=0.7.1` to the extra, and `_safe_parse_rdf()` now fails closed: it raises rather than parsing untrusted RDF/XML if `defusedxml` isn't importable, replacing an earlier regex-based DOCTYPE-stripping fallback that was reviewed and rejected as bypassable
- **[CWE-770] DoS via unbounded SPARQL graph materialization**: `_build_rdflib_graph()` loaded up to 999,999 nodes and 999,999 edges into memory per query, and with up to 4 concurrent SPARQL requests permitted, an attacker could exhaust server memory. Added a 50,000 node/edge cap (`_SPARQL_MAX_GRAPH_NODES`); oversized graphs now return a clean error instead of attempting materialization
- **Cypher injection via Apache AGE's `graph_name` and `$$`-delimiter breakout**: `graph_name` was interpolated unvalidated into `cypher('{graph_name}', $$ ... $$)`, and raw Cypher query text containing `$$` could close AGE's dollar-quoted string delimiter early and append arbitrary SQL. `graph_name` is now validated against the same identifier allowlist `age_store.py` already used for labels/relationship types, and any query containing `$$` is rejected outright
- **SPARQL Explorer route (`/api/sparql`) hardened against comment/PREFIX-hiding bypass**: `_is_read_only_query()` now strips comments and PREFIX/BASE declarations before checking the leading keyword, and additionally scans the full query body for SPARQL Update keywords (INSERT/DELETE/DROP/LOAD/CLEAR/CREATE/COPY/MOVE/ADD) — so `SELECT ... ; DROP ALL` is now rejected by the keyword scan itself rather than relying solely on rdflib's parser
- **Fixed along the way** (maintainer follow-up, addressing automated review findings and a regression introduced across several rounds of iteration on the original fix):
- `VectorStore.save()`'s numpy handling used `list(v)` for the JSON fallback path, which produces `numpy.float32` elements that `json.dump()` can't serialize — changed to `v.tolist()`
- the SPARQL graph-size `ValueError` was raised outside `execute_sparql()`'s exception handling and surfaced as an unhandled 500 instead of a clean API error — moved inside
- every streamed `requests` response in the ontology redirect loop, including the one actually read and returned, is now closed in a `finally` block — a connection-pool leak that a rework of the redirect logic had briefly reintroduced after an earlier fix
- a later commit meant to add opt-in API-key auth (`explorer/auth.py`, gated on `EXPLORER_API_KEY`) instead **replaced and silently disabled** the `Depends(require_auth)` enforcement already merged into `main` for GHSA-j4mq-hprp-987v (Critical — unauthenticated Explorer API), removed the `/ws/graph-updates` handshake check, and — unlike `require_auth` — failed *open* (allowed all requests) whenever its key was unset. Merging that version would have silently reverted an already-fixed Critical CVE the moment this branch landed. Removed `explorer/auth.py`; restored the per-router `Depends(require_auth)` wiring and the WebSocket auth check; kept the one genuine improvement in that commit (adding `X-API-Key` to the CORS `allow_headers` list) by folding it into the existing CORS config
- the new SPARQL keyword-scan's comment-stripping regex (`#[^\n]*`) also matched the `#` inside standard RDF namespace IRIs (e.g. `.../1999/02/22-rdf-syntax-ns#`), corrupting any query with a normal `rdf:`/`rdfs:`-style `PREFIX` declaration — caught because the hardening's own bundled tests failed against two of its own cases. Fixed by only treating `#` as a comment-start at line-start or after whitespace; the companion `PREFIX`/`BASE` regex was also fixed to accept bare `BASE <...>` declarations, which have no prefix-name token between the keyword and the IRI
- New/updated regression tests: `tests/explorer/test_ontology_ssrf.py` (redirect re-validation, relative-redirect resolution, response closing, redirect-cap enforcement), `tests/test_security_regression.py` (Cypher/SPARQL injection, XXE, numpy serialization, SSRF redirect handling), plus additions to `tests/explorer/test_sparql_route.py`, `tests/vector_store/test_vector_store.py`, and `tests/explorer/test_explorer_auth.py`
- Note: the Cypher-injection hardening here is scoped to `age_store.py`'s `graph_name`/`$$` breakout, found while reviewing this PR. The broader label/property-key/relationship-type injection across the Neptune, Neo4j, and FalkorDB backends (GHSA-482h-hw99-h62p, #910) and the triplet-store SPARQL injection across Blazegraph/RDF4J/Jena (GHSA-8vgg-8mr4-r236, #911) are covered by separate, still-open PRs, as is the unauthenticated-Explorer-API fix referenced above (GHSA-j4mq-hprp-987v, #909, already merged)
- **CI/CD supply-chain hardening against mutable-tag Action compromise (LiteLLM/Trivy-class attack)** (#824) by @KaifAhmad1
- Every third-party GitHub Action across all 8 workflows is now pinned to a full commit SHA instead of a mutable tag (`@v7` → `@3d3c42e... # v7`), closing the exact vector used against LiteLLM in March 2026 (a compromised Trivy Action tag stole a long-lived publishing token)
- Added `verify-action-pins.yml` + `.github/scripts/verify-action-pins.sh`: a CI check that fails closed on any `uses:` reference that isn't a full SHA (catching a newly introduced mutable tag, not just auditing existing pins) and re-verifies every pin against the GitHub API on each workflow change, on push to `main`, and weekly; an unresolvable API lookup is treated as a failure rather than a silent skip
- `release.yml`: scoped `permissions` to the job level (workflow default is now `contents: read`), added a `concurrency` group so simultaneous tag pushes can't race the publish job, and added SLSA build provenance attestation (`actions/attest-build-provenance`) for every released wheel
- Created a protected `pypi` GitHub Environment (required reviewer, restricted to `v*` tag deployments) and enabled branch protection on `main` (required PR review with stale-approval dismissal, required status checks, no force-push/deletion, required conversation resolution) — PyPI publishing already used Trusted Publishing (OIDC) with no long-lived token
- Grouped Dependabot's `github-actions` updates into a single PR
- **`security-scan.yml`'s Safety dependency-vulnerability check was silently non-functional** (#824) by @KaifAhmad1
- `safety check --json --output safety-report.json` is invalid in Safety 3.x (`--output` now selects a console format, not a file path); the command errored on every run, swallowed by `|| true`, so no report was ever produced and the job always fell back to a generic "scan completed" message with the vulnerability count hardcoded to 0
- Switched to `--save-json`, the correct flag for writing a JSON report to disk; also fixed `vuln.package` → `vuln.package_name` and Semgrep's `issue.rule_id` → `issue.check_id` (both produced `undefined` in the PR comment)
- The job never installed Semantica's own dependencies before scanning, so Safety was auditing the scanner tools' own transitive deps, not the project's; added `pip install -e ".[llm-litellm]"` so the actual dependency tree — including the LiteLLM extra — is what gets scanned
- Rewrote the PR-comment builder: every line previously used `\\n` inside JS template literals, which renders as the literal text `\n` rather than a newline, producing an unreadable wall of text; now builds real line arrays and collapses long finding lists into a `<details>` block
- Added the `pull-requests: write` permission the comment-posting step was missing (silently failing via its own try/catch on every prior run)
- **`pypdf2==3.0.1` removed (CVE-2023-36464)** (#824) by @KaifAhmad1
- Surfaced by the Safety fix above: PyPDF2 is a discontinued project (merged into `pypdf`) permanently frozen at the vulnerable 3.0.1 with no patched release possible. `grep -rn "import PyPDF2"` found zero real usages anywhere in the codebase — it was only referenced in docstrings describing a `PyPDF2.PdfReader()` fallback for PDF parsing that was never actually implemented (`pdfplumber` does the real work). Removed the dependency and corrected the stale docstrings in `parse/__init__.py`, `parse/methods.py`, `parse/pdf_parser.py`, and `ingest/email_ingestor.py`
- Surfaced by the same Safety fix restoring a working CI gate: Bandit's HIGH-severity check was blocking on 10 pre-existing `hashlib.md5()` calls, all generating short deterministic cache keys, entity IDs, or IRI suffixes from non-secret input — none used for passwords, tokens, or verifying untrusted data
- Bandit's own message suggests `usedforsecurity=False`, but that keyword argument needs Python 3.9+ and `pyproject.toml` declares `requires-python = ">=3.8"`; used a targeted `# nosec B324` with a one-line justification instead, which suppresses only this check with no runtime behavior change on any supported Python version
## [0.6.0] - 2026-07-21
### Added
- **Named-graph support for `JenaStore` via `Dataset` migration** (#756, #757) by @Sameer6305 and @KaifAhmad1
- `JenaStore` now backs onto `rdflib.Dataset(default_union=False)` instead of `rdflib.Graph`, closing #756 and fully closing out the #754/#756 cross-backend named-graph parity effort across Blazegraph, RDF4J, and Jena
- `default_union=False` is explicitly set so existing `execute_sparql()`/`get_triplets()` calls that don't pass `graph=` keep seeing only the default graph, not a union across all named graphs
- `add_triplets()` accepts a `graph=` option: when supplied, triples are written to that named graph (4-tuple add via `Dataset.graph(uri)`); when omitted, behavior is unchanged (3-tuple add routes to the default graph)
- Fixed a pre-existing bug where the remote-endpoint path instantiated the read-only rdflib `SPARQLStore` instead of `SPARQLUpdateStore`, so every `add_triplets()` call against a remote Fuseki endpoint silently failed (`TypeError` swallowed, `success=True`/`added=0` returned); also fixed a constructor bug where `self.endpoint` was always `None` regardless of how `JenaStore` was called, making the remote path unreachable in practice
- `serialize()` now logs a warning instead of silently dropping named-graph content when the requested format (`turtle`, `xml`, `n3`, …) can only serialize the default graph; use `format="trig"` or `format="nquads"` to include all graphs
- `create_model()`'s `triplet_count` now documented as counting across all graphs (default + named), not just the default graph, matching the `Dataset`-wide semantics
- `delete_triplet()` remains scoped to the default graph only (named-graph parity for delete is an explicit follow-up, matching the maintainer's scoping of this migration to `add_triplets`); the removal is passed `self.graph.default_graph` explicitly as its context, since `Dataset.remove()` on a bare 3-tuple resolves to a wildcard context internally and would otherwise delete matching triples out of every named graph too — a follow-up fix to the initial PR #757 for a bug that had no test coverage
- 9 new tests covering `Dataset` construction, `default_union=False` confirmation, named-graph write isolation, `serialize()` warning behavior, and `delete_triplet()`'s default-graph scoping
- Extended CONSTRUCT execution support from Blazegraph-only to the RDF4J and Jena backends (#755), closing #754
- `RDF4JStore.execute_sparql` gains a CONSTRUCT-aware path (`Accept: text/turtle`, rdflib Turtle parsing, the same `(s, p, o, metadata)` 4-tuple contract) and named-graph writes via RDF4J's REST `context` parameter
- `JenaStore.execute_sparql` gains the equivalent CONSTRUCT-aware path over its in-process `rdflib.Graph`
- `_CONSTRUCT_QUERY_RE` moved to `sparql_escaping.py` as a shared, backend-agnostic constant used by all three backends
- Added pipeline integration via the `construct_template` step type
- **Databricks Connector (Unity Catalog + Delta Lake ingestion)** (#747) by @KaifAhmad1
- Added `DatabricksIngestor` (`semantica/ingest/databricks_ingestor.py`), mirroring `SnowflakeIngestor`'s structure and public API shape: a `DatabricksConnector` connection handler, a `DatabricksData` dataclass, and an optional-import guard for `databricks-sdk`/`databricks-sql-connector`
- Supports personal access token and OAuth M2M (service principal `client_id`/`client_secret`) authentication, configurable via constructor args or `DATABRICKS_*` environment variables
- `ingest_table()`/`ingest_query()` run against a SQL warehouse or cluster via `databricks-sql-connector`, with `where`/`order_by`/`limit`/`offset` support and the same identifier-escaping and unsafe-`ORDER BY` rejection as `SnowflakeIngestor`; each call closes the SQL connection it opened unless one is already open (e.g. via the `with DatabricksIngestor(...)` context manager), which reuses and closes it exactly once instead of leaking a second connection per call
- `get_table_schema()`, `list_catalogs()`, `list_schemas()`, and `list_tables()` introspect Unity Catalog via `databricks-sdk`'s `WorkspaceClient`, validating both catalog and schema are resolved before calling the SDK; `get_table_lineage()` calls Unity Catalog's table-lineage REST API for upstream/downstream `Table --DEPENDS_ON--> Table` dependencies, plus an opt-in `include_column_lineage=True` that resolves per-column lineage via the column-lineage API
- `export_as_documents()` converts ingested rows into Semantica document dicts for KG construction, matching `SnowflakeIngestor.export_as_documents()`'s shape
- Registered as a lazy export in `semantica.ingest` (`DatabricksIngestor`, `DatabricksData`, `DatabricksConnector`) and as the `db-databricks` optional extra (`pip install "semantica[db-databricks]"`) in `pyproject.toml`, included in `db-all`
- New `docs/integrations/databricks.md` page modeled on `docs/integrations/snowflake.md`, plus a `DatabricksIngestor` section and table row in `docs/reference/ingest.md` and cross-links between the two integration pages
- 35 unit tests in `tests/test_databricks_ingestor.py` covering both auth methods, table/query ingestion, connection lifecycle (including reuse under the context manager), pagination, unsafe `ORDER BY` rejection, catalog/schema validation, schema/catalog/table listing, table and column lineage, document export, and the missing-dependency error path, closing #747
- **SQLite Vector Store Backend (`sqlite-vec`)** (#726) by @Luffy2208 and @KaifAhmad1
- Added `SQLiteVecStore` (`semantica/vector_store/sqlite_vec_store.py`), a disk-backed local vector store using the `sqlite-vec` extension's `vec0` virtual tables, closing #240
- Supports Cosine and L2 distance metrics, dynamic JSON metadata filtering, read-only mode, and an in-memory (`:memory:`) mode
- Registered as the `"sqlite"` backend in `VectorStore.SUPPORTED_BACKENDS`, with `db_path`/`sqlite_path` config and a `VECTOR_STORE_SQLITE_PATH` environment variable
- Batched `add`/`delete`/`get` and `executemany`-based `update` to avoid per-row round trips; optional `use_wal=True` enables `journal_mode=WAL` + `synchronous=NORMAL` for improved write concurrency
- Lazy-imports `sqlite-vec` so the dependency stays fully optional (`pip install semantica[vectorstore-sqlite]`); table names and metadata filter keys are validated against a strict identifier pattern before SQL interpolation
- Fixes `VectorStore.update_vectors`/`delete_vectors` to delegate to the active backend store instead of only mutating in-memory state, correcting existing behavior for all non-`inmemory` backends
- 25 unit and integration tests in `tests/vector_store/test_sqlite_vec_store.py` covering init, add, search, get, update, delete, read-only mode, and stats
### Fixed
- **`kg.ProvenanceTracker` compatibility wrapper out of sync with `ProvenanceManager`, causing 9 pre-existing test failures** (#744, #751) by @Sameer6305 and @KaifAhmad1
- `kg.ProvenanceTracker` was a standalone in-memory implementation that never delegated to the unified `ProvenanceManager` backend; its own test suite asserted the existence of `get_lineage`, `track_relationship`, `track_entities_batch`, `get_provenance`, and `_use_unified`, none of which were ever implemented, plus a stale `get_all_sources()` assertion expecting `"timestamp"` instead of the actual `"recorded_at"` key
- Rather than completing the abandoned compatibility layer, `kg.ProvenanceTracker` and its remaining supported methods (`track_entity`, `get_all_sources`, `query_recorded_between`, `revision_history`, `export_audit_log`) now emit `DeprecationWarning`s pointing callers to `semantica.provenance.ProvenanceManager`
- Removed/rewrote the 9 tests that only exercised the never-implemented compatibility methods to instead verify the observable behavior of the still-supported API, and corrected the stale `get_all_sources()` assertion
- Added the previously-missing `docs/migration/kg-provenance-tracker.md` migration guide referenced by every new deprecation warning, with a method-mapping table to `ProvenanceManager` and a before/after example, closing #744
- **`ProvenanceManager.track_entity` silently overrides an explicit `parent_entity_id`/`derived_from` on re-track** (#742) by @Sameer6305
- `track_entity()` resolved `parent_id` via a documented precedence chain (`parent_entity_id` kwarg > `metadata["derived_from"]` > source-as-known-entity-id fallback), but the history-preservation block that runs afterward unconditionally overwrote that resolved value with an auto-generated `f"{entity_id}:v:{existing.last_updated}"` history pointer whenever the entity was being re-tracked, discarding whatever parent the caller had just explicitly supplied with no warning
- `track_entity()` now records whether the precedence chain already resolved an explicit parent (`parent_entity_id` kwarg, `metadata["derived_from"]`, or the source-as-known-entity-id fallback) before the history block runs, and only falls back to the auto-generated history pointer when the caller supplied no explicit parent on that call
- The archived history entry for the previous version is still kept reachable in `get_lineage()` via `used_entities` (BFS-traversed by `InMemoryStorage.trace_lineage()`) even when an explicit parent is supplied, so re-tracking with a new parent no longer orphans the prior version from the lineage chain; when no explicit parent is supplied, `used_entities` is left alone since `parent_entity_id` already points at the same history id, avoiding a duplicate self-reference
- **`ProvenanceManager.get_lineage` does not link entities that share a source URL** (#735) by @KaifAhmad1
- `track_entity()`'s only auto-linking logic looked up `source` as if it were an existing entity's `entity_id`, so passing the same real URL/DOI as `source` for two conceptually linked entities (e.g. a document and a decision derived from it) never produced a parent link, leaving `get_lineage()` returning a chain of length 1
- `metadata["derived_from"]` was preserved and echoed back in the output JSON but was never consulted by any linking or traversal code, so the caller's explicit relationship was silently inert
- `track_entity()` now treats `metadata["derived_from"]` as an explicit parent link (unless `parent_entity_id` was already passed directly), so `InMemoryStorage.trace_lineage()`'s existing BFS over `parent_entity_id` picks it up for free
- `metadata["derived_from"]` is now recognized on any `collections.abc.Mapping`, not just a concrete `dict`, so e.g. `types.MappingProxyType` metadata still creates the parent link
- `get_lineage()`'s metadata aggregation now applies the queried entity's own metadata last so it wins over ancestor metadata on conflicting keys, matching the documented "most recent entry's metadata takes precedence" behavior — previously `trace_lineage()`'s BFS order caused ancestor metadata (now reachable via `derived_from` chains) to silently overwrite the queried entity's own values
- Added 9 regression/edge-case tests in `tests/provenance/test_manager.py` covering the happy path, explicit `parent_entity_id` precedence over `derived_from`, precedence over the `source`-as-known-entity-id fallback, a `derived_from` pointing at a never-tracked entity, non-string/empty-string `derived_from` values being ignored, a self-referencing `derived_from` not hanging traversal, multi-hop `derived_from` chains, metadata precedence between a queried entity and its ancestors, and non-`dict``Mapping` metadata, closing #735
- **`Reasoner.add_rule` had no deduplication, doubling rules and silently emptying `forward_chain()` on rerun** (#732) by @KaifAhmad1
- `add_rule()` unconditionally appended to `self.rules`, so re-running the same setup code on an existing `Reasoner` instance (e.g. re-executing a Jupyter cell) duplicated every rule; since `forward_chain()` only records a conclusion if it isn't already in `self.facts`, the second run's duplicated rules matched but produced no new results, with no error or warning
- `add_rule()` now compares an incoming rule's `rule_type`, `conditions`, and `conclusion` against existing rules and returns the existing `Rule` instead of appending a duplicate, keeping repeated `add_rule()` calls with the same definition idempotent
- Added `test_add_rule_deduplicates_identical_rule`, `test_add_rule_deduplication_is_idempotent_across_forward_chain`, and `test_add_rule_does_not_dedupe_distinct_rules` regression tests
- **`InferenceResult.premises` always empty from `forward_chain`/`backward_chain`** (#739) by @Sameer6305
- `_match_rule()` discarded matched facts and returned only instantiated conclusions, so `ExplanationGenerator` always produced empty premises lists regardless of which facts actually satisfied a rule, closing #733
- `_match_rule()` now returns `(conclusion, matched_facts)` tuples; `forward_chain()` threads those facts into `InferenceResult(premises=...)`, merging premises when the same conclusion is derived more than once within a pass
- `_prove_goal()`'s base cases (goal already a known fact; goal matched via pattern unification) now return `premises=[goal]`/`premises=[fact]` instead of `[]`
- Facts are matched against a `sorted()` snapshot instead of the raw `set` so rule matching and premise selection are deterministic
- Added `test_forward_chaining_premises` regression test mirroring the existing backward-chaining premises test
- **Missing `shacl` optional-dependency extra** (#736) by @Sameer6305
- `pip install semantica[shacl]` referenced no matching extra in `pyproject.toml`, so `pyshacl` was never installed despite being documented as the fix in `ontology_validator.py`'s `ImportError` message, the Explorer API, the healthcare cookbook notebook, and the changelog
- Added `shacl = ["pyshacl>=0.25.0"]` to `[project.optional-dependencies]` and folded `shacl` into the `all` extra
- **`NodeEmbedder``AttributeError` masked in `ContextGraph.analyze_graph_with_kg`** (#734) by @Sameer6305
- `analyze_graph_with_kg()` called a non-existent `NodeEmbedder.generate_embeddings()`, and the surrounding broad `except Exception` swallowed the resulting `AttributeError`, silently returning `{"error": "Graph analysis failed due to an internal error"}` from `get_causal_chain()`'s supporting analytics and `get_decision_insights()`
- Rewired the call site to the real `NodeEmbedder.compute_embeddings(graph_store, node_labels, relationship_types)` API, deriving `node_labels`/`relationship_types` from `self.node_type_index`/`self.edge_type_index`
- Added a dedicated `except AttributeError` branch that logs distinctly and re-raises, so a broken internal method call surfaces as a diagnosable error instead of being indistinguishable from a legitimately empty analysis result
---
## [0.5.1] - 2026-06-29
@@ -788,4 +1552,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
For detailed release notes, see [GitHub Releases](https://github.com/Hawksight-AI/semantica/releases).
For detailed release notes, see [GitHub Releases](https://github.com/semantica-agi/semantica/releases).
Thank you for your interest in contributing! Every contribution, no matter how small, is valuable. 🎉
⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/sV34vps5hH)**
⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/semantica-agi/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/sV34vps5hH)**
> **New to contributing?** Start with a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/sV34vps5hH) community.
> **New to contributing?** Start with a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/sV34vps5hH) community.
---
## 🚀 Quick Start
1. Find a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue)
2. [Fork Semantica](https://github.com/Hawksight-AI/semantica/fork) & clone the repository
1. Find a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue)
2. [Fork Semantica](https://github.com/semantica-agi/semantica/fork) & clone the repository
3. Make your changes
4. Submit a pull request!
**Need help?** Join [Discord](https://discord.gg/sV34vps5hH) or [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
**Need help?** Join [Discord](https://discord.gg/sV34vps5hH) or [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions)
---
## 🗂️ Working on an Existing Issue
If you want to work on an open GitHub issue, please follow these steps to keep things coordinated and avoid duplicate effort:
1.**Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome.
2.**Comment if you'd like the issue reserved.** Leaving a comment like *"I'd like to take this on"* is the fastest way to get assigned, but it isn't required — maintainers can also assign an issue directly to a contributor (e.g., based on recent activity in the repo) without waiting for a comment first.
3.**Wait for assignment.** A maintainer will assign the issue when appropriate, whether or not a comment was left. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
4.**Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work.
```bash
git checkout -b fix/short-description # or feature/short-description
```
5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue.
> **Why this matters:** Assignment (with or without a comment) helps maintainers track who is working on what and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
Not sure where to start? Try a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH).
---
## 🔀 Duplicate PRs & Issue Priority
When more than one pull request targets the same issue, maintainers triage using this order of priority. These rules decide between PRs that are otherwise following the [assignment workflow above](#-working-on-an-existing-issue) — opening a PR before being assigned doesn't grant priority on its own, and an unassigned PR can still be closed as a duplicate once someone else is assigned to the issue.
1. **Contributor-raised issue with an existing PR.** If the person who opened the issue has also opened a PR for it, that PR is prioritized (they still need to be assigned before it's merged).
2. **Maintainer-raised issue with a claim comment.** If we opened the issue and someone has commented asking to work on it, we assign it to them and check their PR before picking up any other PR for the same issue.
3. **No prior assignment or comment.** If multiple PRs exist and no one was assigned or claimed the issue first, priority goes to whichever contributor has the most consistent activity in the repo over the last 60 days (e.g., merged PRs, substantive reviews, or issue triage participation) — not just PR volume.
4. **Late duplicate PRs.** If a PR is opened after another contributor has already been assigned to the issue, we close the duplicate early rather than let it sit open, and point the author to another open issue (or ask them to check `main` for newly opened ones). This avoids contributors spending time updating a PR that won't be merged.
5. **Overlapping scope.** If a PR covers multiple issues, or there's genuine overlap between competing PRs, maintainers discuss it on [Discord](https://discord.gg/sV34vps5hH) before deciding rather than resolving it unilaterally.
**Why this matters:** it keeps triage predictable, avoids wasted contributor effort on PRs that won't merge, and helps retain active contributors.
---
@@ -78,7 +116,7 @@ Thank you for your interest in contributing! Every contribution, no matter how s
**What:** Report bugs you find
**How:** Use the [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
**How:** Use the [bug report template](https://github.com/semantica-agi/semantica/issues/new?template=bug_report.md)
**Include:** Description, steps to reproduce, expected vs actual behavior, environment details
@@ -88,7 +126,7 @@ Thank you for your interest in contributing! Every contribution, no matter how s
**What:** Suggest new features or improvements
**How:** Use the [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
**How:** Use the [feature request template](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md)
**Include:** Problem statement, proposed solution, use cases
@@ -108,7 +146,7 @@ Thank you for your interest in contributing! Every contribution, no matter how s
@@ -24,7 +24,7 @@ Security vulnerabilities should be reported privately to prevent potential explo
### 2. Report Security Issue
Create a [GitHub Security Advisory](https://github.com/Hawksight-AI/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix.
Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us via the security email listed in `SUPPORT.md`.
Include the following information:
@@ -37,7 +37,7 @@ Include the following information:
### 3. Response Timeline
- **Initial Response**: Within 48 hours
- **Initial Response**: Within 24 hours for critical issues; within 48 hours for non-critical issues
- **Status Update**: Within 7 days
- **Resolution**: Depends on severity and complexity
@@ -112,6 +112,101 @@ We regularly update dependencies to address security vulnerabilities. However, y
- Be cautious with external API calls
- Implement proper authentication and authorization
## CI/CD Supply-Chain Security
Semantica's build and release pipeline is explicitly hardened against
CI/CD supply-chain attacks — the class of attack behind the March 2026
LiteLLM/Trivy incident, where a compromised third-party Action with a
**mutable tag** was used to steal a long-lived publishing token, after which
malicious packages were pushed straight to PyPI without ever touching the
source repository. Every control below maps directly to closing one step of
that attack chain.
### Immutable build inputs
- **Risk**: a tag (`@v4`, `@release/v1`) is re-pointed by a compromised upstream maintainer or account, silently changing what every consumer's CI runs.
**Control**: every third-party GitHub Action in every workflow is pinned to a full 40-character commit SHA, with the human-readable tag kept only as a trailing comment (e.g. `actions/checkout@3d3c42e... # v7`).
- **Risk**: a SHA pin drifts out of sync with its own comment over time, or is mistyped.
**Control**: `verify-action-pins.yml` fails closed on any `uses:` reference that isn't a full commit SHA (catching a newly added mutable tag, not just auditing existing pins), resolves every pinned tag via the GitHub API on each workflow change, on every push to `main`, and weekly, and fails if the SHA no longer matches the tag it claims to be — an API lookup that can't be resolved is treated as a failure, not a silent skip.
- **Risk**: manually re-pinning ~15 actions across 8 workflow files on every upstream release is error-prone.
**Control**: Dependabot (`github-actions` ecosystem) opens a grouped PR that bumps the SHA *and* the tag comment together whenever an action releases — pins never require hand-editing.
### Publishing pipeline (highest-privilege path)
- **Risk**: a long-lived `PYPI_TOKEN` sitting in repo/org secrets is exfiltrated by any compromised step.
**Control**: PyPI publishing uses Trusted Publishing (OIDC) (`id-token: write`) — there is no long-lived PyPI credential anywhere in this repository to steal.
- **Risk**: a compromised CI run publishes to PyPI with no human in the loop.
**Control**: the publish job runs only inside a protected `pypi` GitHub Environment with a required human reviewer — every release needs manual approval in the Actions UI before it runs.
- **Risk**: the release job could be triggered from an arbitrary branch/ref.
**Control**: the `pypi` environment's deployment-branch policy is restricted to `v*` tags only.
- **Risk**: a scanner or unrelated job inherits publish-level credentials.
**Control**: `release.yml` sets `permissions: contents: read` at the workflow level; `contents: write` / `id-token: write` / `attestations: write` are granted only to the release job, never workflow-wide.
- **Risk**: two tag pushes race through the publish pipeline simultaneously.
**Control**: `concurrency: group: release-${{ github.ref }}` serializes releases per tag.
- **Risk**: a consumer can't verify a wheel on PyPI actually came from this repo's CI.
**Control**: SLSA build provenance is attested for every release via `actions/attest-build-provenance`, producing a signed, verifiable record of the exact commit and workflow run that produced the artifact (checkable with `gh attestation verify`).
### Repository controls
- **Risk**: unreviewed or force-pushed changes land on `main`.
**Control**: `main` requires 1 approving PR review (stale approvals dismissed on new pushes), resolved conversations, and blocks force-pushes and branch deletion.
- **Risk**: a PR merges without its security/CI checks passing.
**Control**: merges require the `build`, `Analyze Python` (CodeQL), and `security-scan` checks to pass, in strict mode (checks must be re-run against the latest `main`).
- **Risk**: a compromised scanner job reaches secrets or write access.
**Control**: scanning jobs (`CodeQL`, `security-scan.yml`, `security.yml`, `defender-for-devops.yml`) run with read-only, least-privilege permissions (typically `contents: read` + `security-events: write` only) and never share a job, environment, or secret scope with the publish job.
- **Risk**: secrets are committed accidentally.
**Control**: GitHub secret scanning and push protection are both enabled at the repository level, rejecting pushes that contain recognizable credential patterns before they land in history.
## Automated Security Scanning
Every scan below runs continuously in CI, not just at release time:
- **CodeQL** (`security-and-quality` query pack) — Python source: injection, unsafe deserialization, and other code-level vulnerability classes. Runs in `codeql.yml` on every push/PR to `main` and weekly.
- **Bandit** — Python-specific security anti-patterns (hardcoded secrets, unsafe `eval`/`pickle`, weak crypto, etc.); CI fails on any HIGH-severity finding. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **Semgrep** (`p/security` ruleset) — cross-language static-analysis security patterns. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **Safety** — known CVEs in Semantica's own installed dependencies, including optional LLM-provider extras such as LiteLLM; CI fails on any match. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **pip-audit** — independent, PyPA-maintained vulnerability database cross-check against installed dependencies (Safety and pip-audit use different advisory sources, so both run). Runs in `security.yml` weekly.
- **Microsoft Defender for DevOps** (`eslint`, `templateanalyzer`, `terrascan`) — JavaScript/TypeScript lint-security rules and infrastructure-as-code misconfigurations. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
- **Checkov** — Kubernetes, Helm, Dockerfile, GitHub Actions, and secrets-pattern IaC scanning; results upload to the same Security tab as CodeQL. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
- **GitGuardian** — secret-detection check on every pull request, installed as a GitHub App integration (not a repo-local workflow). Runs on every PR.
- **GitHub secret scanning + push protection** — blocks known credential patterns before they're pushed, and continuously scans existing history. Platform-level, continuous.
- **Dependabot** — version/security PRs for Python, Docker, and GitHub Actions dependencies, grouped where relevant to reduce review noise. Configured in `.github/dependabot.yml`, runs weekly for security-relevant packages and monthly for docs dependencies.
- **`verify-action-pins.yml`** — enforces that every Action reference is a full commit SHA (failing on a newly introduced mutable tag) and confirms each SHA still matches the tag it claims to be. Runs on every workflow change, every push to `main`, and weekly.
All SARIF-producing scanners (CodeQL, Checkov, Microsoft Defender) publish
findings to the repository's **Security → Code scanning alerts** tab, giving
a single audit trail across tools rather than scattered per-tool reports.
### Adopting this posture in a fork or downstream deployment
Teams standing up their own instance of Semantica, or forking it for an
internal/regulated deployment, can reuse this posture directly:
1. Keep Dependabot's `github-actions` ecosystem entry — it is what keeps
SHA pins current without manual maintenance.
2. Re-run `verify-action-pins.yml` after re-pointing the repository's Actions
at your own mirrors, if you do so.
3. If you publish your own PyPI package from a fork, configure your own
Trusted Publishing trust relationship on PyPI (Trusted Publishing is
scoped to a specific `owner/repo` + workflow filename) and your own
protected environment with your own required reviewers — these are not
transferable from this repository.
4. Branch protection, environment protection, and repository secret
scanning are repository *settings*, not workflow files — cloning or
forking the repo does **not** copy them. They must be re-applied via
the GitHub UI or API on the new repository.
5. GitHub secret scanning and push protection are repository settings that
don't carry over to a fork either — re-enable both under the new
repository's Security settings, not just Dependabot.
6. GitGuardian runs as a GitHub App installation scoped to this specific
repository, not a workflow file — a fork gets no secret-detection
coverage from it until the app is installed separately on the new repo.
7. CodeQL's `upload-sarif` step in `codeql.yml` only runs meaningfully if
Default Setup is *not* already enabled for the repository (it's designed
to skip gracefully otherwise) — check whether Default Setup or Advanced
Setup is active on the new repository and adjust expectations for where
CodeQL findings show up accordingly.
## Dependency Security Policy
### Regular Updates
@@ -156,8 +251,8 @@ We appreciate responsible disclosure. Security researchers who help us improve t
For security-related questions or concerns:
- **GitHub Issues**: [Create an issue](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/09_Semantic_Layer_Construction.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/09_Semantic_Layer_Construction.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/12_Unstructured_to_Ontology.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/13_Manual_Ontology_Snowflake_Mapping.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"\n",
"# Advanced Vector Store - Made Easy\n",
"\n",
@@ -352,7 +352,7 @@
"- Build a multi-user application\n",
"- Explore the [introduction notebook](../introduction/13_Vector_Store.ipynb) for more basics\n",
"\n",
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/Hawksight-AI/semantica)."
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/semantica-agi/semantica)."
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
"\n",
"Semantica is a **semantic intelligence and knowledge engineering framework**. It helps you:\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/04_Document_Parsing.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Data_Normalization.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/04_Data_Normalization.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
"\n",
"# Entity Extraction - Comprehensive Guide\n",
"\n",
@@ -622,7 +622,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
"\n",
"# Relation Extraction - Comprehensive Guide\n",
"\n",
@@ -599,7 +599,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Building_Knowledge_Graphs.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Graph_Analytics.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
"\n",
"# Chunking and Splitting - Comprehensive Guide\n",
"\n",
@@ -817,7 +817,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Embedding_Generation.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/12_Embedding_Generation.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"\n",
"# Vector Store - Comprehensive Guide\n",
"\n",
@@ -492,7 +492,7 @@
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/15_Export.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/17_Visualization.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb)\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb)\n",
"This notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n",
"\n",
"### Key Features\n",
"\n",
"- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n",
"- **OpenCypher Support**: Query using standard OpenCypher syntax\n",
"- VPC with public subnets and Internet Gateway\n",
"- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
"- IAM user with least-privilege access for OpenCypher queries\n",
"- Security group allowing Bolt protocol (port 8182) access\n",
"\n",
"> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
"\n",
"**Outputs:**\n",
"- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
"- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
"- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
"- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
"- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
"| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
"| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
"\n",
"> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
"\n",
"---"
]
"source": "# Amazon Neptune Graph Store\n\n## Overview\n\nThis notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n\n### Key Features\n\n- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n- **OpenCypher Support**: Query using standard OpenCypher syntax\n- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n- **Native ~id Support**: Leverages Neptune's native element ID handling\n- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n\n### Prerequisites\n\n- An Amazon Neptune Database cluster\n- AWS credentials configured (boto3, environment variables, or IAM role)\n- Network access to your Neptune cluster (VPC, security groups)\n- Your public IP address or VPN/office CIDR (run `curl ifconfig.me` to find your public IP), used below to restrict database access\n\n#### Quick Setup with CloudFormation\n\nIf you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n\n```bash\n# Deploy the Neptune stack (takes ~15-20 minutes)\n# Replace 203.0.113.25/32 with your own public IP (run `curl ifconfig.me` to find it)\n# or your office/VPN CIDR. This restricts who can reach the database on the\n# network level - never widen it to 0.0.0.0/0 outside of a short-lived local experiment.\naws cloudformation create-stack \\\n --stack-name semantica-neptune \\\n --template-body file://neptune-setup.yaml \\\n --parameters ParameterKey=ClientCidr,ParameterValue=203.0.113.25/32 \\\n --capabilities CAPABILITY_NAMED_IAM\n\n# Wait for stack creation to complete\naws cloudformation wait stack-create-complete --stack-name semantica-neptune\n\n# Get the outputs (endpoint, port, credentials)\naws cloudformation describe-stacks --stack-name semantica-neptune \\\n --query 'Stacks[0].Outputs' --output table\n```\n\nThe template creates:\n- VPC with public subnets, Internet Gateway, and VPC Flow Logs (to CloudWatch Logs)\n- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n- IAM user with least-privilege access for OpenCypher queries\n- Security group allowing Bolt protocol (port 8182) access only from the `ClientCidr` you specify\n\n> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager. The `ClientCidr` parameter is required (no default) precisely so the database is never silently exposed to the whole internet.\n\n**Outputs:**\n- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n\n**Cleanup:**\n```bash\naws cloudformation delete-stack --stack-name semantica-neptune\n```\n\n**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n\n| Resource | Cost (USD) |\n| --- | --- |\n| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n| Storage (10 GB) | ~1/month |\n| I/O requests | ~1-5/month |\n| Public IPv4 address | ~3.60/month (0.005/hr) |\n| VPC Flow Logs (CloudWatch Logs) | ~1-2/month depending on traffic |\n| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n\n> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n\n---"
"In high-stakes domains — healthcare, legal, finance, research — a Knowledge Graph is only as trustworthy as its ability to answer **\"where did this fact come from?\"**. Semantica's `provenance` module provides audit-grade, W3C PROV-O-aligned tracking for every entity, relationship and chunk that flows through your pipeline.\n",
"\n",
"In this cookbook you will learn how to:\n",
"\n",
"- Track entities and relationships with **source details** (DOI, page, verbatim quote, confidence)\n",
"- Walk the full **lineage** of a fact (document → chunk → entity → KG)\n",
"- Audit **revision history** and **all sources** behind an entity\n",
"- **Invalidate** a fact without deleting it (prov:Invalidation) — corrections stay provable\n",
"- Verify **tamper-evidence** with chained SHA-256 checksums\n",
"\n",
"**The Scenario:** a research team ingests findings from two scientific papers (with DOIs) into a Knowledge Graph. A regulator later asks: *\"Which paper, which figure, and which exact sentence supports the claim that fish biomass increased by 463%? And was that fact ever corrected?\"*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"from semantica.provenance import (\n",
" ProvenanceManager,\n",
" compute_checksum,\n",
" verify_checksum,\n",
")\n",
"\n",
"# In-memory storage for this demo; pass storage_path=\"provenance.db\"\n",
"# (or a config with provenance.storage_path) for a persistent SQLite backend.\n",
"## Step 1: Track Entities with Audit-Grade Source Details\n",
"\n",
"Every fact we ingest carries its evidence with it: the **source identifier** (a DOI here), the **location** inside the source (a figure), the **verbatim quote**, and the extractor's **confidence**."
"## Step 2: Track the Relationship Between Facts\n",
"\n",
"Facts rarely stand alone. The claim about biomass increase is *about* the marine reserve — that relationship is a first-class provenance-tracked object too.\n",
"\n",
"`track_relationship()` has no dedicated subject/object fields, so by convention we record which two entities it connects inside `metadata`."
"`get_lineage` reconstructs everything known about a fact; `trace_lineage` returns the ordered chain of `ProvenanceEntry` records — every version, every activity, every agent that touched it."
"## Step 4: Audit Sources and Revision History\n",
"\n",
"When the regulator asks *\"has this fact ever been corrected?\"*, `revision_history` answers with the full version chain, and `get_all_sources` lists every source document that ever supported the entity."
"print(f\"{len(revisions)} revision(s) on record\")\n",
"\n",
"for s in prov.get_all_sources(\"claim_biomass_increase\"):\n",
" print(\"source:\", s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Invalidate — Correct Without Deleting\n",
"\n",
"Suppose paper #1 is retracted in part. An audit trail must **not** silently delete the fact: `invalidate` archives the pre-invalidation state and appends a fresh `prov:Invalidation` entry naming **who** retracted it and **why**."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"invalidated = prov.invalidate(\n",
" entity_id=\"claim_biomass_increase\",\n",
" agent_id=\"reviewer_dr_chen\",\n",
" reason=\"Partial retraction: Figure 2 statistics corrected by publisher (see erratum).\",\n",
"Each entry carries a deterministic SHA-256 checksum chained to the previous entry. Recompute and compare to detect any after-the-fact corruption of the provenance record."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# entry_biomass was returned by track_entity in Step 1\n",
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)\n",
"\n",
"# Reasoning Module — Practical Guide\n",
"\n",
"Semantica's `reasoning` module derives new knowledge from existing facts and knowledge graphs. It ships several strategies behind one facade:\n",
"\n",
"- **`Reasoner`** — unified facade with forward chaining, backward chaining, and one-shot `infer_facts`\n",
"- **`DatalogReasoner`** — semi-naive Datalog fixpoint evaluation with variable queries\n",
"- **`ExplanationGenerator`** — human-readable explanations and reasoning paths for inferred conclusions\n",
"- Plus lower-level engines: `ReteEngine`, `SPARQLReasoner`, `GraphReasoner`, temporal reasoning\n",
"\n",
"This notebook walks through the facade, the Datalog engine, and explanations. All APIs are verified against `semantica/reasoning/`."
"## 1) Forward chaining with the `Reasoner` facade\n",
"\n",
"Facts are simple `Predicate(args)` strings. Rules use `IF <conditions> THEN <conclusion>` with `?x`-style variables. `forward_chain()` derives everything possible and returns a list of `InferenceResult` objects."
"`infer_facts(facts, rules)` **adds** the given facts and rules to this `Reasoner` instance, runs forward chaining to fixpoint, and returns the derived facts as strings. It does not reset the instance's existing state — create a fresh `Reasoner()` first if you need isolation between runs."
"print(proof.conclusion if proof else \"not provable\")\n",
"print(\"premises:\", proof.premises if proof else None)"
]
},
{
"cell_type": "markdown",
"id": "b245581d",
"metadata": {},
"source": [
"## 4) Re-run safety\n",
"\n",
"`add_rule` deduplicates rules with identical conditions and conclusion, so re-executing a setup cell (the common Jupyter re-run) does not duplicate rules — see issue #732."
"Skipping duplicate rule (same conditions/conclusion as 'rule_1'): IF Person(?x) THEN Human(?x)\n"
]
},
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"\n",
"# Simulate a Jupyter cell re-run: add the same rule twice\n",
"r1 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"r2 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"len(reasoner.rules)"
]
},
{
"cell_type": "markdown",
"id": "ba2e5c4a",
"metadata": {},
"source": [
"## 5) Datalog reasoning\n",
"\n",
"`DatalogReasoner` uses classic Datalog syntax (`head :- body.`) and semi-naive fixpoint evaluation. Queries return variable bindings as a list of dicts — use uppercase variables to ask *which* facts hold."
"`ExplanationGenerator` turns `InferenceResult` objects into structured `Explanation` and `ReasoningPath` records, so agents can show *why* they believe a derived fact."
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)\n",
"\n",
"# Change Management — Practical Guide\n",
"\n",
"Semantica's `change_management` module provides versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies:\n",
"This notebook runs a complete save → tag → verify → tamper-detect cycle. All outputs are real executed results verified against the repository's `semantica/change_management/` source at the time of writing (the `pip install` cell may fetch a newer release with slightly different behavior)."
"A snapshot is a dict with a required `label` plus your payload. Here we attach the KG data, the change log, and a SHA-256 `checksum` computed over everything except the checksum field itself."
"storage.get_tag(\"release\"), [s[\"label\"] for s in storage.list_all()]"
]
},
{
"cell_type": "markdown",
"id": "96df12da",
"metadata": {},
"source": [
"## 4) Verify integrity — and catch tampering\n",
"\n",
"`verify_checksum(snapshot)` recomputes the SHA-256 over the snapshot (minus its `checksum` field) and compares. A single mutated character in the data flips the result to `False`."
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)\n",
"\n",
"# Seed Data — Practical Guide\n",
"\n",
"The `seed` module bootstraps a knowledge graph from **trusted, pre-known data** (CSV/JSON/database/API sources) before any extraction runs. This gives extraction a foundation to link against instead of starting from an empty graph.\n",
"\n",
"Key pieces:\n",
"\n",
"- **`SeedDataManager`** — registers data sources and builds foundation graphs\n",
"## 1) Prepare a seed CSV and register the source\n",
"\n",
"`register_source(name, format, location, entity_type=...)` records where trusted data lives. `verified=True` (the default) marks the source as pre-validated."
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is seeding</td><td>🌱 seed</td><td>SeedDataManager</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"🔄 Semantica is seeding: Loading seed data from CSV: /var/folders/7s/bvvstgs10y963tz6_4bbnklr0000gn/T/semantica-seed-eu9__ep1/companies.csv 🌱 seed SeedDataManager |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"loaded 2 records\n"
]
},
{
"data": {
"text/plain": [
"{'id': 'c1',\n",
" 'name': 'Acme',\n",
" 'type': 'Company',\n",
" 'industry': 'robotics',\n",
" 'entity_type': 'Company',\n",
" 'source': 'companies'}"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"records = manager.load_source(\"companies\")\n",
"print(f\"loaded {len(records)} records\")\n",
"records[0]"
]
},
{
"cell_type": "markdown",
"id": "f2ebce64",
"metadata": {},
"source": [
"## 3) Build the foundation graph\n",
"\n",
"`create_foundation_graph()` converts every registered source into graph-ready entities and relationships. Entities carry `confidence: 1.0` — seed data is trusted by definition."
"`validate_quality(foundation_graph)` returns `valid`, `errors`, `warnings`, and `metrics` so you can gate bad seed data before it pollutes the graph."
Change `app` in `fly.toml` before launch if the default app name is already taken.
Fly apps get a public `*.fly.dev` URL by default, so `SEMANTICA_API_KEY` is required — without it the Explorer refuses every protected route (503) rather than serving anonymously. Pass the same value as the `X-API-Key` header from any client that talks to the deployed API.
{{- include "knowledge-explorer.labels" . | nindent 4 }}
annotations:
runterrascan.io/skip:'[{"rule": "AC_K8S_0086", "comment": "Namespace is bound via .Release.Namespace at helm install time"}, {"rule": "AC_K8S_0080", "comment": "seccompProfile RuntimeDefault is set in values.yaml (podSecurityContext)"}]'
checkov.io/skip1:CKV_K8S_21=Namespace bound via .Release.Namespace at helm install/template time
checkov.io/skip2:CKV_K8S_31=seccompProfile RuntimeDefault set in values.yaml
spec:
{{- if not .Values.autoscaling.enabled }}
replicas:{{.Values.replicaCount }}
@@ -19,8 +23,10 @@ spec:
{{- include "knowledge-explorer.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
runterrascan.io/skip:'[{"rule": "AC_K8S_0080", "comment": "seccompProfile RuntimeDefault is set in values.yaml (podSecurityContext)"}]'
checkov.io/skip1:CKV_K8S_31=seccompProfile RuntimeDefault set in values.yaml
The Redis plugin variables are wired to the requested FalkorDB env names for deployment compatibility. The Explorer currently reads these settings but does not persist graph state to FalkorDB.
Railway exposes this service on a public domain, so `SEMANTICA_API_KEY` is required — without it the Explorer refuses every protected route (503) rather than serving anonymously. Pass the same value as the `X-API-Key` header from any client that talks to the deployed API.
After creation, update `ALLOWED_ORIGINS` in the Render dashboard if you attach a custom domain.
`SEMANTICA_API_KEY` is auto-generated by the blueprint (`generateValue: true`) since this service gets a public `onrender.com` URL — without it the Explorer refuses every protected route (503) rather than serving anonymously. Find the generated value in the Render dashboard's environment tab and pass it as the `X-API-Key` header from any client that talks to the deployed API.
Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.5.1) \[Computer software\]. https://github.com/semantica-agi/semantica
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* \[Computer software\]. https://github.com/semantica-agi/semantica
</Tab>
<Tab title="MLA">
Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.1, GitHub, 2026, https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. GitHub, 2026, https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="Chicago">
Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.1. GitHub, 2026. https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. GitHub, 2026. https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="IEEE">
Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.5.1, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
</Tab>
</Tabs>
## Acknowledgment Text
> "This work uses Semantica (Hawksight AI, 2026), an open-source framework for semantic layer construction and knowledge engineering."
> "This work uses Semantica (2026), an open-source graph-native infrastructure framework for context and accountable AI systems, providing Context Graphs, knowledge graphs, and full decision provenance."
@@ -16,6 +16,9 @@ At its core, Semantica adds a **context and accountability layer** on top of you
- **Accountability Layer** — Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable.
- **Extension Layer** — `PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code.
<Warning>
**This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits *what the AI system did*, not the foundation model's private internal reasoning.
@@ -35,6 +35,7 @@ Essential guides to master the Semantica framework.
- **[Vector Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)** — Setting up vector stores for similarity search and retrieval. *Intermediate*
- **[Graph Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Graph_Store.ipynb)** — Persisting knowledge graphs in Neo4j or FalkorDB. Topics: Neo4j, Cypher, Persistence · *Intermediate*
- **[Ontology](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)** — Defining domain schemas and ontologies to structure your data. Topics: OWL, RDF, Schema Design · *Intermediate*
- **[Seed Data](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)** — Bootstrapping a knowledge graph from trusted CSV, JSON, database, and API sources before extraction runs. Topics: SeedDataManager, Foundation Graphs · *Intermediate*
## Advanced Concepts
@@ -50,6 +51,9 @@ Deep dive into advanced features, customization, and complex workflows.
- **[Multi-Source Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)** — Merging data from disparate sources into a unified graph. Topics: Entity Resolution, Merging, Fusion · *Advanced*
- **[Reasoning and Inference](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)** — Using logical reasoning to infer new knowledge from existing facts. Topics: Logic Rules, Inference Engines · *Advanced*
- **[Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)** — Modeling and querying data that changes over time. Topics: Time Series, Temporal Logic, Allen Algebra · *Advanced*
- **[Provenance Tracking](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/22_Provenance_Tracking.ipynb)** — Audit-grade, W3C PROV-O-aligned tracking of where every entity, relationship, and chunk came from. Topics: PROV-O, Lineage, Checksums, Invalidation · *Advanced*
- **[Reasoning Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)** — Deriving new knowledge from existing facts with forward chaining, backward chaining, and Datalog strategies. Topics: Reasoner, Datalog, Explanations · *Advanced*
- **[Change Management](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)** — Versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies. Topics: ChangeLogEntry, Version Storage, Data Integrity · *Advanced*
## How to Run
@@ -80,6 +84,6 @@ Deep dive into advanced features, customization, and complex workflows.
You can also run the cookbook using Docker:
```bash
docker run -p 8888:8888 hawksight/semantica-cookbook
docker run -p 8888:8888 semantica/semantica-cookbook
`--host 0.0.0.0` makes Explorer reachable on every network interface. The server has no built-in authentication. Only use this on a trusted private network.
`--host 0.0.0.0` makes Explorer reachable on every network interface. Since v0.6.5 the Explorer API requires `SEMANTICA_API_KEY` (sent as the `X-API-Key` header) and fails closed with `503` when unconfigured; unauthenticated access is only possible when `SEMANTICA_ALLOW_ANONYMOUS=true` is set explicitly. Only use this on a trusted private network.
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
@@ -52,6 +52,16 @@ Semantica works alongside these frameworks, not against them.
</Accordion>
<Accordion title="Does Semantica explain an LLM's internal reasoning or chain-of-thought?" icon="triangle-exclamation">
No. This is **system-level explainability, not foundation-model explainability**. Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system.
What Semantica explains is *outside* the model: what context and data were used, what decision was produced, the provenance behind it, the relevant relationships, the policies applied, and the resulting decision trail.
In short: Semantica explains and audits *what the AI system did* — not the foundation model's private internal reasoning.
</Accordion>
<Accordion title="Is Semantica free?" icon="tag">
Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source.
@@ -129,7 +139,7 @@ If you're on an older version, install extras individually: `pip install "semant
@@ -149,7 +149,7 @@ A database optimized for storing and querying graph-structured data using node a
A retrieval strategy combining vector similarity search with keyword or metadata filtering: higher accuracy than either approach alone.
**Triplet Store**
A database designed specifically for storing and querying RDF `(subject, predicate, object)` triples. Semantica supports Blazegraph, Apache Jena, and RDF4J.
A database designed specifically for storing and querying RDF `(subject, predicate, object)` triples. Semantica supports embedded Oxigraph as well as Blazegraph, Apache Jena, and RDF4J.
**Vector Store**
A database optimized for storing and searching high-dimensional embedding vectors by similarity. Semantica supports FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector.
> Semantica is maintained by Hawksight AI with community contributions under an open governance model.
> Semantica is maintained by the Semantica team with community contributions under an open governance model.
## Roles
- **Maintainers** — Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards.
- **Maintainers** — Semantica team: review and merge PRs, manage releases and code quality, set project direction and community standards.
- **Contributors** — Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md).
- **Community Members** — Use Semantica, provide feedback, share use cases, and participate in GitHub Discussions and Discord.
`AgentContext` maintains a persistent memory layer for LLM agents — storing observations as vector embeddings, retrieving them by semantic similarity, and optionally blending graph proximity into the ranking. Use it when your agent needs to recall past findings across sessions without re-reading source material on every restart.
## What Is Agent Memory?
Agent Memory provides persistent storage and intelligent retrieval of information across multiple agent sessions. `AgentContext` is the core component that orchestrates memory storage, retrieval, and management by combining three key systems:
**VectorStore** handles semantic search using vector embeddings. It stores text as high-dimensional vectors and retrieves similar content through cosine similarity or other distance metrics.
**ContextGraph** maintains structured knowledge as nodes (entities) and edges (relationships). This enables multi-hop traversal and graph-aware retrieval that follows connections between related entities.
**AgentContext** orchestrates both components, providing a unified interface for storing memories, retrieving relevant context, and managing conversations across sessions.
**Persistent memory vs stateless retrieval:** Traditional RAG systems lose context between sessions. Agent Memory persists learned information, conversation history, and accumulated knowledge across restarts, enabling long-term memory and cross-session recall.
## Why Use Agent Memory?
**Cross-session recall.** Agents remember previous interactions, findings, and decisions without re-processing source material after restarts.
**Long-term knowledge accumulation.** Information builds up over time as agents process more documents, creating increasingly rich knowledge bases for future queries.
**Conversation history.** Agents maintain context within conversations and can reference earlier parts of extended interactions or investigations.
**Graph-aware retrieval.** Beyond simple semantic similarity, retrieval follows entity relationships to find connected information that pure vector search would miss.
**Decision tracking.** Record decisions with full context and reasoning paths, enabling audit trails and precedent matching for similar future scenarios.
## When To Use / When Not To Use
**Use Agent Memory for:**
- Long-running agents that need to accumulate knowledge over time
- Research assistants that build understanding across multiple sessions
- Investigation workflows where context builds incrementally
- Systems that must remember prior interactions and decisions
- Scenarios requiring audit trails and decision precedents
**Do not use when:**
- Building simple stateless RAG systems for one-time document queries
- Performing one-off document searches without need for persistence
- Running temporary experiments that don't require knowledge retention
- Simple retrieval tasks where relationships between entities don't matter
<Info>
This guide covers the memory layer. For graph-enriched traversal and entity linking, see [Context Graphs](context-graphs). For decision accountability — recording, auditing, and causally tracing what the agent chose — see [Decision Intelligence](decision-intelligence).
</Info>
@@ -18,11 +57,10 @@ Configure the vector store, knowledge graph, and `AgentContext` together at star
**Forgetting to persist memory before shutdown.** Agent Memory is stored in memory during execution. Without calling `save()` before process termination, all accumulated memories, graph relationships, and conversations are lost.
**Using the same conversation namespace for unrelated tasks.** Conversation IDs should scope related interactions. Using a single conversation for multiple unrelated investigations pollutes retrieval results and makes context less focused.
**Storing excessive low-value information.** Not every observation needs permanent storage. Focus on storing insights, decisions, and significant findings rather than verbose raw logs or temporary calculations.
**Using Agent Memory when simple retrieval would be sufficient.** For one-time document lookups or stateless queries, traditional retrieval is simpler and more efficient than setting up persistent memory infrastructure.
**Retrieving too much context and increasing latency.** Large `max_results`, high `max_hops`, or broad queries can retrieve excessive context, increasing LLM token usage and response latency. Start with focused retrieval parameters.
## Related Guides
- [Context Graphs](context-graphs) — How the underlying `ContextGraph` stores entity nodes and decision nodes; temporal interval reasoning; deduplication before node insertion; ontology from graph.
@@ -4,12 +4,99 @@ description: "Snapshot, version, diff, and migrate knowledge graphs and ontologi
icon:"clock-rotate-left"
---
Knowledge graphs change constantly — threat actors get re-attributed, CVE scores update when exploits drop, clinical trial endpoints shift between phases. `TemporalVersionManager` gives your graph a verifiable history: named snapshots before every consequential change, diffs between any two states, one-call rollback, and SHA-256 checksum verification before publishing downstream.
## What Is Change Management & Versioning?
Knowledge graphs change constantly. `TemporalVersionManager` gives your graph a verifiable history by capturing complete state snapshots at specific points in time. It allows you to take named snapshots before consequential changes, generate detailed diffs between any two states, roll back to previous versions with a single call, and verify SHA-256 checksums before publishing data downstream.
## Storage Behavior
Pass `storage_path`, e.g. `TemporalVersionManager(storage_path="versions.db")`, to persist snapshots to a SQLite database on disk. Omit `storage_path` and it defaults to an in-memory store that vanishes when your script finishes.
## Why Use Change Management?
Change Management acts as your safety net and audit trail. Use it to:
- **Safeguard Ingestion**: Take a snapshot before a large batch ingestion so you can instantly roll back if the data is corrupted.
- **Audit Trails**: Maintain a verifiable log of when a change occurred, who authorized it, and exactly what nodes/edges were modified.
- **Release Gating**: Compare staging and production graphs and verify checksums before signing off on a release.
## Which Tool Do I Need?
Semantica offers multiple tracking features. It is critical to choose the right one:
- **Change Management** (this guide): Use for **whole-graph snapshots**, state diffs, and full rollbacks.
- **Provenance**: Use for granular **source and lineage tracking**. It answers *"Which specific document did this node come from?"*
- **Agent Memory**: Use for **conversational and context state**. It answers *"What decisions did the AI agent make during this session?"*
## When To Use / When Not To Use
- **When to Use**: You have critical checkpoints (like daily feeds, partner merges, or regulatory submissions) where you need to freeze the entire state of the graph and potentially revert it.
- **When NOT to Use**: You have a massive, multi-million node graph and want to track every minor edit. Because `TemporalVersionManager` snapshots the entire graph dictionary, snapshotting huge graphs too frequently will cause severe storage bloat. Use Provenance for granular tracking instead.
<Info>
`TemporalVersionManager` integrates with `AgentContext.flush_checkpoint()` — agent checkpoints and manual snapshots share the same storage format, so diffs work across both.
`TemporalVersionManager` integrates directly with `AgentContext.flush_checkpoint()` — agent checkpoints and manual snapshots share the same storage format, allowing diffs across both automated and manual workflows.
</Info>
---
## Typical Workflow
A standard change management cycle follows this progression:
1.**Snapshot**: Capture the baseline graph state.
2.**Modify**: Run your ingestion, mutations, or analysis.
3.**Compare**: Generate a diff to see what changed.
4.**Verify**: Check the SHA-256 hash to ensure data integrity.
5.**Tag**: Apply a human-readable tag (e.g., `approved`).
6.**Rollback**: Revert the graph state if the modifications were incorrect.
---
## Universal Example: Employee Profile Update
Let's look at a universally understood example: tracking an employee's department transfer.
- **Snapshotting huge graphs too frequently**: `TemporalVersionManager` snapshots the entire graph structure. Doing this on every minor edit for a massive graph will cause severe storage bloat. Use it for milestone gating, not event sourcing.
- **Forgetting `attach_to_graph` before mutation tracking**: If you want to use `get_node_history()`, you must call `vm.attach_to_graph(graph)`*before* any mutations happen. Otherwise, the events will not be captured.
- **Confusing provenance with versioning**: Do not use version snapshots to answer "Where did this specific node's data come from?". That is the role of the Provenance module. Versioning tracks the state of the *entire* graph at a point in time.
- **Forgetting rollback confirmation requirements**: Calling `restore_snapshot` in automated scripts will raise a `ProcessingError` and crash your pipeline unless you explicitly pass `require_confirmation=False`.
- **Storage growth from excessive snapshots**: Over time, SQLite databases can grow large if you never prune old snapshots or if you snapshot unnecessarily.
Run conflict detection after deduplication and before SHACL validation. Deduplication removes duplicate nodes; conflict resolution reconciles disagreeing property values on the same canonical entity. Running them out of order — detecting conflicts before deduplication — will produce spurious conflicts between entities that should have been merged first.
</Info>
## Detecting the disagreement
## What Is Conflict Resolution?
Start by loading your multi-source records for the same entity. `ConflictDetector` groups them by entity ID, then compares the values each source reports for a given property. Any entity where two or more sources report different values for the same property produces a `Conflict` object.
When you merge data from multiple sources, the same real-world entity — a customer, a product, a threat actor, a drug compound — often appears with contradictory property values. One database says a customer's email is `alice@example.com`; another says `alice.smith@example.com`. One security feed rates a CVE at 10.0; two others rate it 9.1 and 9.5.
**Conflict resolution** is the systematic process of deciding which value is most trustworthy and recording that decision with evidence, so the canonical entity ends up with one defensible, auditable value per property.
### Key Concepts
**Canonical entity** — The single authoritative record for a real-world thing. After deduplication, each entity has exactly one canonical node in your graph. Conflict resolution determines which property values belong on that node.
**Conflicting values** — Two or more different values asserted for the same property on the same canonical entity, each reported by a different source.
**Credibility score** — A number between 0.0 and 1.0 you attach to each source record, indicating how reliable that source is. A government registry might carry 0.99; a scraped blog might carry 0.30. You supply these; Semantica uses them during `CREDIBILITY_WEIGHTED` resolution.
**Confidence score** — A number between 0.0 and 1.0 the resolver *computes* after resolution, reflecting how certain the outcome is. A unanimous vote produces high confidence; a close split among equally credible sources produces lower confidence. This appears on `ResolutionResult.confidence` and should be read as a signal, not a guarantee that the resolved value is correct.
**Resolution strategy** — The rule for picking the winning value: majority vote, credibility-weighted average, latest timestamp, and so on. See [Resolution strategies at a glance](#resolution-strategies-at-a-glance) for the full list.
**Audit trail** — The complete record of every resolution decision: conflict ID, strategy used, resolved value, sources consulted, and confidence score. Returned by `resolver.get_resolution_history()`.
**Provenance-aware resolution** — Resolution that records not just the winning value but which source it came from. Every `ResolutionResult` carries a `sources_used` field, so you can always trace a canonical value back to its origin — critical in regulated environments.
## Why Use Conflict Resolution?
- **Multi-source pipelines always produce disagreements.** Differences in update cadence, data-entry conventions, and source reliability are unavoidable. Without an explicit resolution step, you silently favor one source over another with no record of the choice.
- **You get a defensible, auditable decision log.** Compliance teams, auditors, and domain experts need to know which source won and why. The audit trail provides exactly that.
- **Easy cases are automated; hard cases are escalated.** Routine disagreements — slightly different name spellings, stale timestamps — are resolved algorithmically. Genuinely ambiguous cases — competing legal classifications, different clinical endpoints — are flagged for expert review without blocking the rest of the pipeline.
## When To Use / When Not To Use
**Use conflict resolution when:**
- You are merging two or more independent sources for the same entity.
- Sources disagree on property values and you need a single canonical value.
- You need an auditable record of every resolution decision.
- Some conflicts require domain-expert review before they can be resolved.
**Skip conflict resolution when:**
- **A single authoritative source already exists.** If one system is always correct for a given property, read from it directly. Adding resolution machinery around a single source creates complexity without benefit.
- **All sources are always in agreement.** Verify this empirically before skipping; silent disagreements are common in practice.
- **You want to preserve all conflicting values.** If retaining every source's assertion matters more than picking one, model provenance directly in your graph schema instead of resolving to one winner.
## Typical Workflow
```mermaid
flowchart TD
A[Raw Sources] --> B[Deduplication]
B --> C[Conflict Detection]
C --> D{Auto-resolvable?}
D -- Yes --> E[Apply Resolution Strategy]
D -- No --> F[Expert Review Queue]
E --> G[Persist Canonical Values]
F --> G
G --> H[SHACL Validation]
```
1.**Deduplication** — Merge duplicate nodes so each entity has exactly one canonical record. Conflict resolution operates on a single canonical entity; you must identify it before comparing what different sources say about it. See [Deduplication](deduplication).
2.**Conflict Detection** — Call `detect_entity_conflicts()` to surface all property disagreements at once, or `detect_value_conflicts()` to target a specific property.
3.**Resolution** — For each conflict, apply a strategy (`CREDIBILITY_WEIGHTED`, `MOST_RECENT`, `VOTING`, etc.) or route it for expert review (`EXPERT_REVIEW`).
4.**Persist Canonical Values** — Write resolved values back to your canonical entities or graph store. See [Persisting resolved values](#persisting-resolved-values).
5.**SHACL Validation** — Enforce structural constraints on the resolved graph to confirm it satisfies your ontology. See [SHACL Validation](shacl-validation).
## Quick Start: A Beginner Example
Before diving into domain-specific scenarios, here is the shortest path through the API. Three systems — a CRM, an ERP, and an LDAP directory — hold slightly different contact details for the same customer. Two of the three agree that the canonical email is `alice.smith@example.com`; the CRM has an older value.
`detect_entity_conflicts()` scanned both `email` and `phone` automatically — you did not name them. Because `phone` is identical across all three records, no conflict was detected for it. The email disagreement resolves to `alice.smith@example.com` because two of three sources agree on that value.
When every conflict in a batch should use the same strategy, pass `strategy=` directly to `resolve_conflicts()`. Use `set_resolution_rule()` when different entity-property pairs need different strategies — explained in [Setting per-property resolution rules](#setting-per-property-resolution-rules).
## Detecting Conflicts
`ConflictDetector` provides three methods. Choose the one that fits your situation:
| Method | What it scans | When to use |
| :--- | :--- | :--- |
| `detect_entity_conflicts(entities)` | Every property on each entity at once | First pass; you do not know in advance which properties conflict |
| `detect_value_conflicts(entities, property_name)` | One named property across all entities | Targeted check for a known hot-spot property |
| `detect_relationship_conflicts(relationships)` | Edge types between the same node pair | Structural disagreements in graph edges |
### Scanning All Properties at Once — `detect_entity_conflicts`
`detect_entity_conflicts()` is the recommended starting point for a new pipeline. It inspects every property found on your entity records and returns a single flat list of all conflicts — without you having to enumerate properties in advance.
Without `entity_type`, the detector checks every key found on your entity dicts (excluding bookkeeping fields such as `id`, `source`, and `metadata`). Start here to get a complete picture, then decide which conflicts need which resolution strategy.
### Scanning a Specific Property — `detect_value_conflicts`
Use `detect_value_conflicts()` when you already know which property to check, or when you want to apply different detection logic to each property. `ConflictDetector` groups the records by entity ID, then compares each source's value for that property. Any entity where two or more sources report different values produces a `Conflict` object.
Each `Conflict` captures the full picture: which entity, which property, every disagreeing value, and which source reported each. This is already enough to build a review queue — but the goal is to resolve these automatically according to rules you set.
## Setting per-property resolution rules
## Setting Per-Property Resolution Rules
The key method is `set_resolution_rule(entity_id, property_name, strategy)`. It takes three arguments: which entity, which property, and which `ResolutionStrategy` to apply when that combination appears in a conflict. Rules are stored in the resolver and automatically applied when you call `resolve_conflicts()` without passing an explicit strategy.
`set_resolution_rule(entity_id, property_name, strategy)` registers a strategy for a specific entity-property combination. The resolver stores the rule under the key `entity_id.property_name` and applies it automatically when you call `resolve_conflicts()`.
Because rules are keyed by both entity ID and property name, `set_resolution_rule()` is entity-specific. There is no wildcard that applies a rule to all entities or all properties at once.
**When to use `set_resolution_rule()`:** Use it when different entity-property combinations need different strategies. For example, an entity's `legal_name` might use `CREDIBILITY_WEIGHTED` while its `last_updated` uses `MOST_RECENT`. Registering a rule per combination lets the single `resolve_conflicts()` call handle all of them correctly in one pass.
**When to pass `strategy=` directly to `resolve_conflicts()`:** If every conflict in a batch should use the same strategy, pass it directly to `resolve_conflicts()` instead of registering a rule for each entity-property pair:
```python
# Same strategy for every conflict — no per-property rules needed
# For this CVE, NVD is the most authoritative source on scoring.
# CREDIBILITY_WEIGHTED will use the credibility_score field on each source record
# CREDIBILITY_WEIGHTED uses the registered source credibility
# to weight the vote — NVD at 0.98 will dominate over the commercial feed at 0.91.
resolver.set_resolution_rule(
"cve-2024-3400",
@@ -108,9 +271,9 @@ resolver.set_resolution_rule(
You can set rules before or after detection — the resolver applies them lazily when `resolve_conflicts()` is called.
## Resolving the batch
## Resolving the Batch
Pass all detected conflicts to `resolve_conflicts()`. For each conflict, the resolver looks up whether a property-specific rule is set for that entity and property combination. If one is found, it applies that strategy. If none is set, it falls back to the default strategy (voting, unless you override it in the constructor).
Pass all detected conflicts to `resolve_conflicts()`. For each conflict, the resolver looks up whether a rule is registered for that entity-property combination. If one is found, it applies that strategy. If none is set, it falls back to the default strategy (voting, unless you override it in the constructor).
```python
all_conflicts=score_conflicts+exploit_conflicts
@@ -132,7 +295,7 @@ for r in results:
[RESOLVED] cve-2024-3400_cvss_score_conflict
Resolved value : 10.0
Strategy used : credibility_weighted
Confidence : 72%
Confidence : 36%
Sources used : ['nvd', 'commercial_feed', 'vendor_paloalto']
Notes : Resolved by credibility-weighted voting (weight: 0.98)
@@ -146,7 +309,7 @@ for r in results:
NVD wins the CVSS score — its credibility weight (0.98) edges out the commercial feed (0.91) and the vendor (0.87), so 10.0 becomes the canonical score. The exploitation status resolves to `in_wild` — the commercial feed and vendor advisory are both more recent than NVD's initial triage, and both report active exploitation.
## Handling conflicts that need human judgment
## Handling Conflicts That Need Human Judgment
Not every conflict can be auto-resolved. A disagreement about the legal classification of a financial instrument, or about a patient's current medication list, is too consequential to resolve by algorithm. Flag these for review without blocking the rest of the batch:
@@ -156,14 +319,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
# Drug trial data: efficacy agreed, primary endpoint disputed
`EXPERT_REVIEW` sets `resolved=False` on the result. The conflict stays in the graph unresolved, the metadata field carries `requires_expert_review: True`, and the review queue JSON gives your clinical team exactly what they need to make the call.
## Reviewing the full audit trail
## Persisting Resolved Values
`resolve_conflicts()` returns `ResolutionResult` objects — it does not automatically write resolved values back to your graph or entity store. That step is yours to implement using whatever storage layer your pipeline uses.
The most direct approach is to pair each `ResolutionResult` with its original `Conflict` object — the two lists are returned in the same order — and write the winning value onto your canonical entity:
```python
# canonical_entity is your authoritative record — a dict, graph node, database row, etc.
- **Conflicts with `resolved=False`** — flagged for expert or manual review — should not be written to the canonical record until a human has made the call. Keep them in the review queue.
- **Confidence is a signal, not a guarantee.** A 72% confidence score means the resolver had reasonable but not unanimous evidence for its decision. Treat low-confidence results with additional scrutiny before writing them to production.
- **Track provenance.** `result.sources_used` tells you which source's value won. Store this alongside the canonical value if your compliance requirements demand a full evidence chain.
## Reviewing the Full Audit Trail
After a resolution run, `get_resolution_history()` returns every decision made since the resolver was instantiated. This is your compliance log:
The report aggregates every conflict the detector has seen across its lifetime — useful for pipeline monitoring and for identifying which entity types or data sources generate the most disagreements.
## Detecting relationship conflicts
## Detecting Relationship Conflicts
Value conflicts live on properties. Relationship conflicts live on edges — two sources asserting contradictory connections between the same node pair:
@@ -267,7 +463,7 @@ for c in rel_conflicts:
Relationship conflicts typically require expert review rather than voting, because conflicting edge types often reflect genuinely different intelligence assessments rather than data entry errors.
## Domain examples
## Domain Examples
<Tabs>
@@ -282,11 +478,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
| `EXPERT_REVIEW` | Flags for domain expert queue; `resolved=False` | Scientific or legal disambiguation required |
## Common Pitfalls
**Running conflict resolution before deduplication**
If duplicate nodes for the same real-world entity still exist, `ConflictDetector` treats each duplicate as a separate entity disagreeing with the others — producing spurious conflicts that should never have existed. Always run deduplication first.
**Forgetting to persist resolved values**
`resolve_conflicts()` returns `ResolutionResult` objects; it does not write them anywhere. Inspecting the results and moving on without updating your canonical entity means nothing has actually changed in your data. See [Persisting resolved values](#persisting-resolved-values).
**Scanning properties one at a time across a large entity set**
Calling `detect_value_conflicts()` for every property in a manual loop produces redundant passes over your data. Use `detect_entity_conflicts()` instead — it handles all properties in a single call and is the recommended starting point for bulk detection.
**Misunderstanding credibility scores**
Credibility scores are weights you assign based on your prior knowledge of source reliability — not ground truth. A source registered with `set_source_credibility("source", 0.99)` can still be wrong. `CREDIBILITY_WEIGHTED` resolution amplifies your beliefs about source quality; if those beliefs are miscalibrated, the resolutions will be too. Validate scores against known ground truth before relying on them in production.
**Treating resolved values as guaranteed truth**
A resolved value is the most defensible answer given your sources and strategy — not necessarily the correct one. Low confidence scores and `EXPERT_REVIEW` flags are signals to scrutinize results before writing them to a canonical record or downstream system.
**Using conflict resolution when a single authoritative source already exists**
If one system is always correct for a given property, read from it directly. Layering conflict resolution over a single source adds complexity, introduces unnecessary doubt, and produces an audit trail that adds no real information.
**Registering rules in a loop to apply one strategy uniformly**
Calling `set_resolution_rule()` for every entity-property pair just to apply the same strategy to all of them creates O(N) setup for no benefit. Pass `strategy=` directly to `resolve_conflicts()` when one strategy covers the whole batch.
## Related Guides
- [Deduplication](deduplication) — remove duplicate nodes before running conflict detection
The directory contains a versioned `graph.md` manifest for graph identity,
relationships, and cross-graph link descriptors, plus one file per node under
`nodes/`. A node's content is its Markdown body; its ID, type, properties,
metadata, and temporal validity are YAML frontmatter. Node, edge, family, graph,
and cross-graph link IDs are preserved across round trips.
Markdown loading uses replacement semantics, like `from_dict()`: it parses and
validates the complete directory before replacing the current graph. Invalid YAML,
duplicate IDs, unsupported versions, and unsafe filesystem links fail without
partially mutating the graph. As with JSON loading, an edge endpoint without a node
file creates an `entity` stub node. Symlinks, Windows directory junctions, and other
Windows reparse points are rejected.
Re-exporting to an existing managed directory atomically replaces it, removing stale
node files. Before replacement, Semantica validates the complete canonical export
layout, not just the manifest header. Untracked files, assets, extra directories, or
renamed node files therefore cause the export to fail closed instead of being deleted.
Keep attachments and hand-written indexes outside the managed export directory.
If the graph had cross-graph links created with `link_graph()`, call `resolve_links()` after loading to restore live navigation — object references cannot be serialized, so they must be reconnected manually:
`AgentContext.record_decision()` stores every AI decision as a node in the knowledge graph, linked by causal edges to the decisions that preceded it and the outcomes that followed. Use it to build an auditable reasoning trail — one that lets you reconstruct, six months later, exactly which classification caused which escalation, and which policy was checked before it was recorded.
## What Is Decision Intelligence?
Decision Intelligence records and analyzes an agent's own decisions as structured data that can be queried, analyzed, and reused. Instead of decisions disappearing after execution, they become persistent graph nodes with searchable metadata, reasoning chains, and causal relationships.
**Decision Intelligence records decisions** by capturing the scenario, reasoning, outcome, confidence, and decision maker for each choice the agent makes. These decisions become queryable nodes in your knowledge graph.
**Decisions become graph nodes** that can be linked causally (Decision A caused Decision B), searched by similarity (find decisions like this scenario), and analyzed statistically (confidence trends, common outcomes).
**The goal is auditability, explainability, precedent search, and causal tracing.** You can trace why decisions were made, find similar past decisions for consistency, and understand the full causal chain from initial detection to final action.
**Decision Intelligence vs. Reasoning:** Reasoning derives new facts from existing data using logical rules. Decision Intelligence records the choices and judgments the agent made during problem-solving.
**Decision Intelligence vs. Graph Analytics:** Graph Analytics analyzes the structural properties of your knowledge graph. Decision Intelligence focuses specifically on the decision-making process and its audit trail.
## Why Use Decision Intelligence?
**Auditable AI actions.** Every decision is recorded with reasoning, confidence, and timestamp, creating a complete audit trail for AI behavior in production systems.
**Explainability.** When stakeholders ask "why did the system do X?", you can trace the exact decision chain that led to that action, including intermediate reasoning steps.
**Precedent reuse.** Before making new decisions, agents can search for similar past scenarios and their outcomes, promoting consistency and learning from previous experience.
**Causal analysis.** Understand how early decisions cascade into later outcomes by following causal relationships between linked decision nodes.
**Governance and compliance.** Policy engines can gate decisions against compliance rules, and all policy applications are recorded for regulatory audit.
## When To Use / When Not To Use
**Use Decision Intelligence when:**
- Building autonomous agents that make consequential choices
- Operating under compliance requirements (financial services, healthcare, defense)
- Building approval systems with multiple decision points
- Working in risk-sensitive environments where decisions must be explainable
**Do not use when:**
- Building stateless chatbots that only retrieve information
- Implementing simple RAG systems without decision-making
- Creating read-only information retrieval applications
- Building applications that never make actionable decisions requiring audit trails
## API Architecture Overview
Decision Intelligence coordinates three main components:
**AgentContext** serves as the high-level orchestration layer. It provides `record_decision()`, `find_precedents()`, and causal chain methods while managing the underlying storage and retrieval systems.
**PolicyEngine** handles policy evaluation and compliance checking. It stores policy rules as graph nodes and validates decisions against those rules before they're recorded.
**DecisionRecorder** specializes in recording structured decision data, managing approval chains, and handling policy exceptions when decisions need to bypass normal rules.
<Info>
Decision tracking requires both a `VectorStore` (for embedding-based precedent search) and a `ContextGraph` (for causal graph storage). Set `decision_tracking=True` on `AgentContext` — omitting either component raises `RuntimeError`at call time.
Decision tracking requires both a `VectorStore` (for embedding-based precedent search) and a `ContextGraph` (for causal graph storage). Set `decision_tracking=True` on `AgentContext` — omitting `ContextGraph` raises a `RuntimeError` at call time. `VectorStore` is required by `AgentContext` itself: leaving the argument out raises a `TypeError`from Python's argument binding, while passing `vector_store=None` raises a `ValueError` during initialization.
The `decision_maker` field identifies the component, workflow, agent, or system that produced this decision. Use consistent identifiers like `"cti_pipeline_v2"`, `"analyst_chen"`, or `"risk_model_v3"` to enable filtering and analysis by decision source.
The `Decision` dataclass that backs this node has the following fields — these are what get stored and searched:
**Recording decisions without linking causal relationships.** Isolated decision nodes provide less insight than connected decision chains. Use `add_causal_relationship()` to link related decisions and enable causal tracing.
**Creating isolated decision nodes.** Decisions gain value when connected to entities, other decisions, or outcomes in your graph. Link decisions to relevant entities using the `entities` parameter.
**Recording too many low-value decisions.** Not every minor choice needs permanent recording. Focus on consequential decisions that affect outcomes, require audit trails, or benefit from precedent search.
**Treating precedent similarity as proof.** High similarity scores indicate related scenarios, not identical situations. Use precedents as guidance while considering the specific context of each new decision.
**Using Decision Intelligence when simple retrieval is sufficient.** If your system only retrieves information without making actionable choices, traditional search or Agent Memory may be more appropriate than decision tracking.
## Related Guides
- [Context Graphs](context-graphs) — how `ContextGraph` stores decision nodes and causal edges
description: "Detect duplicate entities using multi-factor similarity, merge them with configurable strategies, and keep your knowledge graph clean at scale."
---
## What Is Deduplication?
Deduplication is the process of identifying entities that refer to the same real-world object but appear as separate records in your data, then merging them into a single canonical representation. This process resolves aliases, spelling variations, and formatting differences that occur when data comes from multiple sources.
**Key deduplication concepts:**
**Canonical entities** are the single, authoritative representation of a real-world object after merging all duplicate records. The canonical entity becomes the node that all relationships point to in your knowledge graph.
**Aliases** are alternative names or identifiers for the same entity. For example, "APT29", "Cozy Bear", and "Midnight Blizzard" are all aliases for the same threat actor.
**Entity resolution** is the broader process of determining when different records refer to the same entity, including the similarity calculation, duplicate detection, and merging steps.
**Similarity algorithms:**
- **Jaro-Winkler** measures string similarity with higher scores for shared prefixes, ideal for names with common beginnings
- **Levenshtein** distance counts character edits needed to transform one string into another, good for catching typos and variations
**Clustering** groups related duplicates together using algorithms like Union-Find, ensuring that if A matches B and B matches C, all three are grouped together even if A and C don't directly match.
## Why Use Deduplication?
**Data quality and consistency.** Eliminate duplicate nodes that fragment relationships and create inconsistent query results across different names for the same entity.
**Accurate analytics and metrics.** Get correct counts, centrality measures, and relationship analysis when entities aren't artificially split across multiple nodes due to naming variations.
**Relationship consolidation.** Merge scattered relationships onto single canonical entities, enabling complete analysis of connections and patterns that would be missed with fragmented data.
**Source integration.** Seamlessly combine data from multiple feeds, systems, and databases where the same entities appear under different identifiers and naming conventions.
**Graph efficiency.** Reduce graph size and improve query performance by eliminating redundant nodes while preserving all information through proper merging strategies.
**Provenance preservation.** Maintain complete audit trails showing which source contributed each piece of information to the final canonical entity.
## When To Use / When Not To Use
**Use deduplication for:**
- Multi-source data integration where entities appear under different names or identifiers
- Entity types prone to aliases and variations (organizations, people, products, geographic locations)
- Knowledge graphs where relationship accuracy depends on entity consolidation
- Data quality workflows requiring canonical entity management
- Analytics requiring accurate entity counts and relationship metrics
- Scenarios where the same real-world objects appear across multiple systems or databases
**Do NOT use deduplication for:**
- Single-source data with consistent entity identifiers and naming conventions
- High-throughput streaming scenarios where deduplication latency is unacceptable
- Data with reliable primary keys where duplicates are impossible by design
- Cases where entity variations should be preserved as separate nodes (different product versions, time-based entity states)
- Simple exact-match scenarios where basic database constraints handle uniqueness
**Be cautious with:**
- Large datasets where O(n²) pairwise comparison becomes computationally expensive
- Fuzzy matching when deterministic primary keys (LEI, CVE-ID, ISIN) are available
- Very low similarity thresholds that may merge genuinely different entities
## Typical Workflow
The deduplication workflow follows a systematic process from detection through merging:
**1. Detect** → Use `detect_duplicates()` or `DuplicateDetector` to identify potential matches using multi-factor similarity scoring
**2. Group** → Apply clustering algorithms to collect transitively related duplicates into groups (A matches B, B matches C → group A,B,C)
**3. Select Canonical** → Choose representative entity for each group based on completeness, source authority, or confidence scores
**4. Merge** → Combine duplicate entities using strategies like `keep_most_complete` or `merge_all` while preserving provenance
**5. Validate** → Review merge results and adjust thresholds or strategies based on precision/recall analysis
**6. Update Graph** → Replace duplicate nodes with canonical entities and transfer all relationships
This pipeline transforms fragmented multi-source data into clean, consolidated knowledge graphs ready for analytics and reasoning.
## API Patterns: Functional vs Class-Based
Semantica provides both simple functional wrappers and comprehensive class APIs for different use cases:
**Functional wrappers for simple workflows:**
- `detect_duplicates()` — one-shot duplicate detection with minimal configuration
- `calculate_similarity()` — compare two entities with detailed similarity breakdown
- `merge_entities()` — convenience wrapper around merge_duplicates() for quick merging
**Class APIs for complex workflows:**
- `DuplicateDetector` — configurable duplicate detection with clustering, incremental processing, and advanced similarity options
- `EntityMerger` — sophisticated merging with multiple strategies, provenance tracking, and merge history
**Usage guidelines:**
- Use `merge_duplicates()` when you have a raw collection of entities and need automatic duplicate detection
- Use `merge_entity_group()` when you already know which entities are duplicates and just need to merge a pre-determined group
- Don't mix functional wrappers with class APIs in the same workflow—choose one approach and stick with it
The deduplication module detects duplicate entities across multi-source knowledge graphs using six complementary similarity algorithms — exact match, Levenshtein, Jaro-Winkler, cosine, property comparison, and vector embedding — then merges them into a single canonical entity while preserving full provenance. Use it to collapse alias clusters (e.g. "APT29", "Cozy Bear", "Midnight Blizzard") before running graph analytics or conflict resolution.
<Info>
@@ -11,7 +101,9 @@ Run deduplication after ingestion and before conflict resolution. Deduplication
## Finding your duplicates: the first scan
Start with `detect_duplicates()`. Point it at your threat actor entities and let the pairwise algorithm compare every pair. For a dataset of a few thousand nodes this runs in seconds — the O(n²) cost only matters above ten thousand entities.
Start with `detect_duplicates()` for straightforward duplicate detection on smaller datasets. Point it at your entities and let the pairwise algorithm compare every pair using multiple similarity signals.
**Scaling consideration:** For datasets of a few thousand nodes, this runs in seconds. The O(n²) pairwise comparison cost only becomes problematic above ten thousand entities—for larger sets, see the clustering section below.
```python
from semantica.deduplication import detect_duplicates
@@ -64,11 +156,11 @@ for c in candidates:
signals: ['property'] # alias "APT29" in Midnight Blizzard record
```
The scores tell a clear story. "APT29" and "APT-29" score 0.89 — the hyphen is the only difference, pure edit-distance signal. "Cozy Bear" and "The Dukes" score lower (0.61) because the names are completely dissimilar, but the property signal fires because both records carry `"APT29"` in their aliases list. "APT28" never appears in the results because it shares only the country field — not enough to cross the 0.6 threshold.
The scores tell a clear story. "APT29" and "APT-29" score 0.89 — the hyphen is the only difference, producing strong string similarity signals. "Cozy Bear" and "The Dukes" score lower (0.61) because the names are completely dissimilar, but the property signal fires because both records carry `"APT29"` in their aliases list. "APT28" never appears in the results because it shares only the country field — not enough to cross the 0.6 threshold.
## Understanding the candidate object
Each `DuplicateCandidate` carries the two entities, their scores, and a `reasons` list explaining which signals fired. This is your audit trail for the detection decision:
Each `DuplicateCandidate` carries the two entities, their similarity scores, and a detailed breakdown of which similarity algorithms contributed to the match. This provides full transparency for audit and threshold tuning:
```python
from semantica.deduplication import calculate_similarity
@@ -98,11 +190,11 @@ Components :
embedding 0.78 # semantic vectors land in the same cluster
```
The property component (0.94) is doing most of the work here. "Cozy Bear"'s record carries `aliases: ["APT29"]`, which creates an almost-definitive signal. When you see a pattern like this — a weak name score but a strong property score — you're looking at a real alias relationship, not a false positive.
The property component (0.94) is doing most of the work here. "Cozy Bear"'s record carries `aliases: ["APT29"]`, which creates an almost-definitive signal that these entities refer to the same threat actor. When you see a pattern like this — weak name similarity but strong property matching — you're typically looking at a genuine alias relationship rather than a false positive.
## Grouping duplicates before merging
For a small dataset you can merge pairs directly. For a larger graph where the same entity might appear under six different names across twelve feeds, use `detect_duplicate_groups()`. It runs Union-Find clustering to collect all aliases of the same underlying entity into a single group, regardless of whether every pair individually crosses the threshold:
For small datasets, you can merge candidate pairs directly. For larger graphs where the same entity might appear under six different names across twelve feeds, use duplicategrouping with Union-Find clustering. This ensures that if A matches B and B matches C, all three entities are grouped together even if A and C don't directly meet the similarity threshold:
```python
from semantica.deduplication import DuplicateDetector, EntityMerger
@@ -132,11 +224,11 @@ Found 2 duplicate groups:
Representative: 'APT28'
```
The group result shows the problem clearly: five separate nodes that should be one. The `representative` field is the entity the merger will use as the base — the one with the most filled properties, in this case "APT29" from the MISP feed which carries the fullest attribute set.
The group result shows the consolidation clearly: five separate nodes that should be one canonical entity. The `representative` field identifies the entity the merger will use as the base — typically the one with the most complete attribute set, in this case "APT29" from the MISP feed.
## Merging: collapsing the group without losing data
Now merge. The `keep_most_complete` strategy keeps the entity with the highest property count as the canonical node and fills in any missing fields from the other sources. With `preserve_provenance=True`, the merge operation records which source contributed every field in the merged result:
Once you have identified duplicate groups, the merging process consolidates them into canonical entities. The `keep_most_complete` strategy selects the entity with the highest property count as the canonical node and enriches it with any missing fields from the other sources:
The five source entities are replaced by one. Every relationship those five nodes carried — to campaigns, malware families, TTPs, infrastructure — now attaches to the canonical "APT29" node. Nothing is lost; the provenance records show exactly which feed contributed which attribute.
The five source entities are replaced by one canonical representation. Every relationship those five nodes carried — to campaigns, malware families, TTPs, infrastructure — now attaches to the canonical "APT29" node. The merge operation preserves all information while eliminating redundancy, and the provenance records show exactly which feed contributed each attribute.
## Reviewing merge history for audit
After a batch merge, pull the full history to review every decision made:
After batch merging operations, you can retrieve the complete history to review every decision made. This audit trail is essential for understanding merge decisions and explaining them to stakeholders:
```python
history = merger.get_merge_history()
@@ -175,14 +266,14 @@ history = merger.get_merge_history()
This history is what you present when a feed owner asks why their entity was merged into another one. Every decision is recorded.
This history provides complete transparency about merge decisions. When a feed owner asks why their entity was merged into another one, you have the documented evidence and reasoning for the decision.
## Streaming ingestion: incremental deduplication
When your pipeline is ingesting continuously — new STIX bundles arriving hourly — you don't want to re-run pairwise comparison over the entire graph on every batch. Use `incremental_detect()` to compare only the new entities against the existing set:
When your pipeline processes continuous data streams — new threat intelligence arriving hourly — you don't want to re-run pairwise comparison over the entire graph on every batch. Use incrementaldetection to compare only new entities against the existing canonical set:
```python
# Existing graph entities (already deduplicated)
@@ -212,11 +303,13 @@ New duplicates found in this batch: 1
score=0.67 # alias field carries "APT29" — property signal fires
```
NOBELIUM goes to the merge queue. Scattered Spider scores below threshold against every existing actor and gets added to the graph as a new node.
NOBELIUM gets queued for merging with the existing APT29 canonical entity. Scattered Spider scores below threshold against every existing actor and gets added to the graph as a new, unique node.
## Scaling to large entity sets
For graphs above ten thousand nodes, pairwise comparison becomes too slow. Use `build_clusters()` to run vectorized batch comparison, then merge each cluster:
For graphs above ten thousand nodes, pairwise comparison becomes computationally expensive due to its O(n²) complexity. Use `build_clusters()` to run more efficient vectorized batch comparison, then merge each resulting cluster:
**Performance warning:** Always profile your similarity operations on representative data sizes. What works for 1,000 entities may become unacceptably slow at 10,000+ entities without appropriate scaling strategies.
```python
from semantica.deduplication import build_clusters
For even larger sets, switch to `method="hierarchical"` which uses agglomerative bottom-up clustering and scales to hundreds of thousands of entities at the cost of some precision.
## A Simple Example: Customer Deduplication
Before exploring domain-specific cases, let's walk through a straightforward customer deduplication scenario. A company's CRM system has accumulated duplicate customer records from web signups, sales team entries, and support tickets:
```python
from semantica.deduplication import detect_duplicates, merge_entities
print(f" Merged from {len(op.source_entities)} records")
# Result: One John Smith record with email, phone, and company information
# from all three original records, with full provenance tracking
```
This example demonstrates the core concepts: similarity detection finds related records, and merging consolidates them into canonical entities that preserve all available information.
## Common Pitfalls
**Threshold tuning without validation.** Setting thresholds too low creates false positive merges between genuinely different entities. Always manually review a sample of detected duplicates before running large-scale merging operations.
**Pairwise scaling problems.** The O(n²) cost of comparing every entity pair becomes prohibitive above 10,000 entities. Use clustering methods (`build_clusters`) or switch to vectorized similarity for large datasets.
**Using fuzzy matching when primary keys exist.** If your entities have reliable unique identifiers (LEI codes, CVE IDs, ISBN numbers), use exact matching on those fields instead of computationally expensive similarity algorithms.
**Mixing wrapper and class APIs inconsistently.** Don't call `detect_duplicates()` then manually instantiate `EntityMerger`—choose either the functional approach or class-based approach and use it consistently throughout your workflow.
**Ignoring merge strategy implications.** `keep_first` overwrites later records completely, `merge_all` can introduce conflicting values, and `keep_most_complete` may not respect source authority. Choose the strategy that matches your data quality requirements.
**Skipping provenance tracking.** Without `preserve_provenance=True`, you lose visibility into which source contributed each field in the canonical entity, making audit trails impossible.
**Inadequate similarity algorithm selection.** Pure string similarity fails for alias relationships ("APT29" vs "Cozy Bear"), while property matching may be too aggressive for entities with shared attributes but different identities.
`ContextGraph` distance intelligence answers the structural question that pure semantic similarity cannot: given two nodes, what is their precise relationship in terms of graph topology, path weight, and inferential confidence? Use it to annotate attribution chains with hop counts and confidence decay, rank retrieval results by structural proximity to an anchor node, and surface implied connections for analyst review.
## What Is Distance Intelligence?
Distance intelligence quantifies and analyzes the structural relationships between nodes in your knowledge graph. It provides detailed metadata about graph paths including hop counts, distance bands, confidence decay, and path analysis.
**Distance metadata** includes hop counts (number of edges between nodes), distance bands (semantic categories like "direct", "near", "distant"), confidence decay (accumulated trust along paths), and path analysis (finding optimal routes between nodes).
**Hop counts** measure the number of edges you must traverse to reach one node from another. A hop count of 1 means direct connection; 3 means you traverse through 2 intermediate nodes.
**Distance bands** convert raw hop counts into meaningful categories: "direct" (0-1 hops), "near" (2-3 hops), "mid-range" (4-6 hops), and "distant" (7+ hops). These categories help interpret the semantic meaning of graph distances.
**Confidence decay** multiplies edge weights along a path to compute accumulated trust. If each edge has weight 0.8, a 3-hop path has confidence decay of 0.8³ = 0.512, indicating moderate confidence in the connection.
**Path analysis** finds optimal routes between nodes using algorithms like Dijkstra's shortest path or Yen's k-shortest paths algorithm.
**Distance intelligence vs. graph analytics:** Analytics computes statistical measures like centrality and communities across the entire graph. Distance intelligence focuses on specific paths and relationships between particular nodes.
**Distance intelligence vs. graph traversal:** Simple traversal follows edges to find neighbors. Distance intelligence quantifies the quality and confidence of those connections using weights, paths, and decay metrics.
## Why Use Distance Intelligence?
**Confidence-aware retrieval.** Instead of treating all graph connections equally, distance intelligence weights results by path confidence, giving higher rankings to nodes connected through stronger, more direct relationships.
**Relationship discovery.** Find not just whether two entities are connected, but how they're connected, through which intermediaries, and with what level of confidence across the full path.
**Causal analysis.** Trace cause-and-effect chains through your knowledge graph with quantified confidence at each step, essential for decision tracking and audit trails.
**Precedent search.** Find similar past cases by analyzing structural similarity and path patterns, not just content similarity.
**Graph-aware ranking.** Blend semantic similarity with graph proximity to surface contextually relevant results that pure vector search would miss.
- Proximity-weighted retrieval from specific anchor nodes
- Finding alternative connection routes for verification
- Ranking results by both content relevance and structural proximity
**Simple graph traversal may be sufficient for:**
- Finding direct neighbors of a node
- Basic graph exploration without confidence weighting
- Cases where all edges have equal importance
- Simple reachability queries (can A reach B?)
**Distance intelligence may be unnecessary for:**
- Single-hop neighbor lookups
- Graphs where edge weights don't represent meaningful confidence
- Simple existence queries rather than quality assessment
- Scenarios where path analysis adds unnecessary complexity
<Info>
Distance Intelligence feeds into proximity-blended retrieval (`proximity_weight` on `retrieve()`), causal chain analysis (`trace_decision_causality()`), and advanced precedent search (`find_precedents_hybrid()`). Enable it by passing `include_distance_metadata=True` on neighbor queries or `proximity_weight > 0` on retrieval calls.
</Info>
## Distance Bands: Turning Hop Counts into Meaning
The first tool in distance intelligence is `classify_path_distance` — it maps any BFS depth to a human-readable band that carries semantic meaning.
The first tool in distance intelligence is `classify_path_distance` — it maps any Breadth-First Search (BFS) depth to a human-readable band that carries semantic meaning.
```python
from semantica.utils.helpers import classify_path_distance
@@ -38,6 +90,14 @@ These bands appear automatically on every result that uses `include_distance_met
Each hop along a path multiplies the accumulated confidence by the edge weight. The product — `confidence_decay` — is the single most useful signal for deciding whether a multi-hop inference is trustworthy.
<Info>
**Confidence Decay and Edge Weights:** Confidence decay depends directly on edge weights in your graph. Weights should represent confidence, trust, relevance, or similar domain-specific signals where higher values indicate stronger relationships. Unweighted graphs (all edges weight 1.0) produce no meaningful decay analysis.
</Info>
<Info>
**Dense Graph Warning:** Very dense graphs can make path analysis computationally expensive and results harder to interpret. Dense connectivity creates many possible paths with similar weights, making distance-based rankings less discriminating.
**K-shortest paths — Yen's algorithm.** Use when you need alternative attribution chains, redundancy analysis, or corroboration routes. Finding the three shortest paths and showing they all converge on the same target is stronger evidence than a single path.
**K-shortest paths — Yen's algorithm.** Yen's algorithm finds multiple alternative paths between two nodes, ranked by total path cost. Use when you need alternative attribution chains, redundancy analysis, or corroboration routes. Finding the three shortest paths and showing they all converge on the same target is stronger evidence than a single path.
**Treating confidence decay as statistical probability.** Confidence decay is a heuristic measure based on edge weights, not a statistical probability. A decay value of 0.6 doesn't mean "60% probability" — it means the path strength based on your domain-specific weight assignments.
**Using unweighted graphs and expecting meaningful decay.** If all edges have weight 1.0, confidence decay will always be 1.0 regardless of path length, providing no useful discrimination between paths. Assign meaningful weights that reflect relationship strength.
**Excessive path exploration on dense graphs.** Dense graphs with many interconnected nodes can generate exponentially large numbers of paths. Limit `max_hops`, use `min_confidence` thresholds, and consider whether simple neighbor lookup would be sufficient.
**Overusing distance analysis when simple neighbor lookup is enough.** If you only need direct neighbors or one-hop connections, basic graph traversal is simpler and faster than full distance intelligence analysis.
**Retrieving excessive graph neighborhoods.** Large `max_hops` values can retrieve massive subgraphs that overwhelm downstream processing. Start with 2-3 hops and increase only when needed for your specific use case.
description: "Export knowledge graphs to RDF (Turtle, JSON-LD, N-Triples), GraphML, Cypher (Neo4j), ArangoDB AQL, CSV, Parquet, OWL, and more."
---
## What Is Export?
Export converts Semantica graph data into formats used by external tools and systems. Unlike internal persistence mechanisms that keep data within Semantica, export is specifically designed for interoperability with external consumers.
**Export vs. internal persistence:**
- **`AgentContext.store()`** and graph persistence keep data inside Semantica for continued processing, retrieval, and reasoning
- **Export functions** serialize graph data into standardized formats that external systems can consume directly
Export enables integration with analytics platforms, graph databases, RDF triple stores, semantic web systems, data warehouses, business intelligence tools, and downstream consumers that need access to your knowledge graph data in their native formats.
## Why Use Export?
**Build once, export many.** Create your knowledge graph through Semantica's extraction and reasoning workflows, then export the same graph data to multiple formats for different consumers without rebuilding or reprocessing.
**Interoperability with existing ecosystems.** Connect Semantica graphs to established tools and workflows in your organization, from Neo4j graph databases to Gephi visualizations to pandas data analysis pipelines.
**Analytics and reporting workflows.** Feed graph data into business intelligence tools, statistical analysis platforms, and machine learning pipelines that require specific data formats like CSV, Parquet, or RDF.
**Graph database migration and deployment.** Move graphs from Semantica's in-memory representation to production graph databases like Neo4j, ArangoDB, or triple stores for scalable query performance.
**RDF and semantic web integration.** Export to semantic web standards (Turtle, JSON-LD, N-Triples) for integration with ontology tools, SPARQL endpoints, and semantic reasoning systems.
**Data lake and warehouse integration.** Export to columnar formats like Parquet for integration with modern data stack tools including DuckDB, Apache Spark, and cloud data warehouses.
**Compliance and archival workflows.** Generate standardized exports for regulatory submission, long-term archival, and audit trail requirements that mandate specific data formats.
## When To Use / When Not To Use
**Use export when:**
- Integrating Semantica graphs with external systems and tools
- Sharing graph data with teams using different technology stacks
- Building analytics pipelines that consume graph data in downstream processing
- Working with RDF and ontology workflows requiring semantic web standards
- Creating reports, visualizations, and business intelligence dashboards
- Migrating graphs to production databases for scalable query performance
- Meeting compliance requirements for specific data format submissions
**Do not use export when:**
- You simply want to save and reload Semantica state—use built-in persistence mechanisms instead
- Agent persistence and memory continuity are your primary goals
- Internal retrieval, reasoning, and graph operations are sufficient for your use case
- Export would add unnecessary complexity to workflows that operate entirely within Semantica
- You need real-time access to evolving graph data—export creates static snapshots
**Consider internal persistence instead when:**
- Your workflow involves iterative graph building, querying, and reasoning within Semantica
- You need to maintain agent memory, conversation history, and decision tracking
- Graph data will continue to be processed and enriched within Semantica workflows
`export_rdf`, `export_graph`, `export_lpg`, and related functions serialize a `ContextGraph` to any of ten formats in a single call, preserving node types, edge weights, and metadata faithfully. Use them when downstream consumers — triple stores, graph databases, visualization tools, ML pipelines, or spreadsheet auditors — each expect a different format from the same in-memory graph.
<Info>
All export functions take `graph.to_dict()` as their first argument — the same dict produced by `ContextGraph.to_dict()`. Build the graph once, export it to as many formats as you need without re-serializing.
All export functions take `graph.to_dict()` as their first argument — the same dict produced by `ContextGraph.to_dict()`. Build the graph once, export it to as many formats as you need without re-serializing. Note that `graph.to_dict()` materializes the entire graph in memory, so very large graphs may require additional memory planning.
</Info>
## Building the Graph to Export
@@ -36,6 +85,8 @@ graph_data = graph.to_dict() # single dict, reused across all exports below
## RDF Formats — For Triple Stores and Semantic Reasoners
**RDF (Resource Description Framework)** is the foundational data model for the semantic web, representing information as subject-predicate-object triplets. RDF formats are essential for integration with semantic web technologies, ontology tools, and systems requiring formal knowledge representation.
When your consumers are triple stores (GraphDB, Stardog, Apache Jena) or OWL reasoners (HermiT, Pellet), you want RDF. Semantica exports to all five standard RDF serializations through a single `export_rdf` call.
```python
@@ -58,6 +109,8 @@ The format to reach for depends on your consumer. Turtle is ideal for human revi
## Graph Formats — For Gephi, Maltego, and Network Analysis
**Labeled Property Graph (LPG)** formats represent networks with typed nodes and edges that carry attributes and metadata. These formats are optimized for graph visualization tools and network analysis platforms that focus on exploring relationships and structural patterns.
GraphML, GEXF, and DOT are the native formats of graph analysis and visualization tools. They preserve node attributes, edge weights, and type labels, so the graph you built in Semantica renders immediately in Gephi or NetworkX with full attribute data.
```python
@@ -77,6 +130,8 @@ The GEXF format is worth knowing about if you use Gephi for analyst briefings
## Neo4j Cypher — For Graph-Pattern Threat Hunting
**Cypher** is Neo4j's declarative graph query language that uses pattern matching to find and manipulate graph data. Cypher exports enable teams to run complex graph queries, pattern detection, and graph analytics using Neo4j's optimized query engine.
When the SOC team wants to run Cypher queries against the graph — finding threat actors that share infrastructure, or tracing multi-hop attack paths — you export to Cypher and load the result into Neo4j Desktop or Memgraph with a single command.
```python
@@ -116,6 +171,8 @@ The `include_collection_creation=True` flag means the AQL file is self-contained
## CSV — For Spreadsheet Audits and Statistical Analysis
**CSV (Comma-Separated Values)** is a simple tabular format universally supported by spreadsheet applications, statistical tools, and data analysis platforms. CSV export flattens graph data into rows and columns for teams that work primarily with tabular data.
The compliance team lives in Excel. The data science team lives in pandas. Both of them need CSV. `export_csv` writes the graph as flat rows — entities and relationships as separate files when you pass a base path.
```python
@@ -136,6 +193,8 @@ The split form is more useful for downstream tools: the entities CSV feeds a piv
## Parquet — For Data Lakes and ML Pipelines
**Parquet** is a columnar storage format optimized for analytics workloads, offering efficient compression and fast query performance. Parquet files integrate seamlessly with modern data stack tools and machine learning frameworks.
When the data science team runs feature engineering over graph attributes in DuckDB, Spark, or a lakehouse, Parquet is the format they want. It is columnar, compressed, and readable by every major ML framework.
```python
@@ -148,6 +207,10 @@ Once in Parquet, the graph entities become a DataFrame that can be joined agains
## OWL — For Ontology-Based Reasoning
**OWL (Web Ontology Language)** is a semantic web standard for representing rich ontologies with classes, properties, and logical constraints. OWL enables automated reasoning, consistency checking, and inference over formal knowledge models.
**OntologyGenerator** creates formal ontologies from graph data by analyzing entity types, relationships, and patterns to generate class hierarchies, property definitions, and logical constraints. This enables schema validation, automated reasoning, and integration with semantic web tools.
When you have generated an OWL ontology from your graph using `OntologyGenerator`, you can export it for Protégé, HermiT reasoning, or regulatory submission.
**Confusing export with persistence.** Export creates external snapshots for interoperability, while persistence maintains Semantica's internal state. Don't use export when you need to save and reload agent memory or continue graph-based workflows—use built-in persistence mechanisms instead.
**Exporting stale graph data after graph changes.** Always call `graph.to_dict()` after your final graph modifications. If you store `graph_data` early in your workflow and then modify the graph, exports will reflect the outdated state, not your latest changes.
**Re-running expensive extraction instead of reusing existing graph data.** Build your graph once through entity extraction and relationship inference, then export to multiple formats using the same `graph_data` dict. Don't rebuild the graph for each export format.
**Choosing overly complex formats when CSV is sufficient.** If downstream consumers work with tabular data and don't need graph structure preservation, CSV is simpler, faster, and more universally supported than RDF or GraphML formats.
**Assuming provenance and history automatically appear in exports.** Standard export formats capture the current graph state but don't include provenance chains, version history, or audit trails. Use dedicated provenance export mechanisms if you need full lineage information.
**Ignoring downstream schema requirements.** Different systems expect different identifier formats, attribute schemas, and relationship representations. Validate that your exported data matches the expectations of consuming systems before deploying to production workflows.
**Exporting extremely large graphs without memory planning.** The `graph.to_dict()` operation materializes the entire graph in memory. For very large graphs, monitor memory usage and consider chunking or streaming approaches for resource-constrained environments.
GraphRAG combines vector similarity with knowledge graph traversal so retrieval finds structurally connected facts, not just text that sounds related. When a `ContextGraph` is attached to `AgentContext`, every retrieval call automatically blends semantic search with multi-hop graph expansion — and `query_with_reasoning()` returns an auditable reasoning path alongside the LLM answer.
## What Is GraphRAG?
GraphRAG (Graph-Augmented Retrieval-Augmented Generation) enhances traditional RAG by combining vector similarity search with knowledge graph traversal. Instead of retrieving only semantically similar text, GraphRAG follows relationships between entities to find connected evidence across multiple documents.
**GraphRAG vs. traditional vector-only RAG:** Vector RAG finds documents similar to your query text. GraphRAG finds documents similar to your query AND documents connected to those through entity relationships, even if they don't mention your query terms directly.
**The role of graph traversal:** Starting from entities found in vector-similar documents, GraphRAG expands outward through relationship edges to discover related facts. This reveals connections that pure text similarity would miss — like finding that a threat actor targets healthcare by following the path: Actor → Tool → Victim Organization → Industry Sector.
## Why Use GraphRAG?
**Multi-hop discovery.** Find facts that are 2-3 relationship steps away from your query. A question about "APT29 healthcare targeting" can surface evidence about specific hospitals by traversing: APT29 → HAMMERTOSS → LifeCare → Healthcare Sector.
**Connected evidence.** Instead of isolated document fragments, retrieve coherent chains of related entities and their relationships. This provides richer context for LLM responses and human analysis.
**Investigation workflows.** Follow evidence trails by expanding from known entities through their connections. Start with a suspicious IP and discover the full infrastructure chain, or trace a drug interaction through metabolic pathways.
**Richer retrieval context.** Graph expansion surfaces relevant context that keyword or semantic search alone would miss, leading to more complete and accurate LLM responses.
**Explainability.** GraphRAG provides audit trails showing exactly which entities and relationships led to each piece of retrieved evidence, making the retrieval process transparent and verifiable.
## When To Use / When Not To Use
**GraphRAG adds value when:**
- Your domain has rich entity relationships (threat intelligence, clinical data, regulatory documents)
- Questions require connecting facts across multiple documents
- Investigation workflows benefit from following entity connections
- Explainability and audit trails are important
- You have well-structured knowledge graphs with meaningful relationships
**Simple vector search may be sufficient for:**
- Document retrieval based on topic similarity
- Single-document question answering
- Exploratory search where you don't know what you're looking for
- Domains with few meaningful entity relationships
**Latency and complexity considerations:**
- GraphRAG adds computational overhead from graph traversal
- Multi-hop expansion increases retrieval time and token usage
1. **Ingest** your documents using `AgentContext.store()` with entity extraction enabled
2. **Build Graph** through Named Entity Recognition (NER) and relationship extraction to populate the `ContextGraph`
3. **Retrieve** semantically similar documents and identify seed entities for graph expansion
4. **Expand Context** by following entity relationships within your specified hop limit
5. **Reason** (optional) using the expanded context with reasoning engines
6. **Answer** by providing the enriched context to an LLM through `query_with_reasoning()`
<Info>
**Graph Quality Dependency:** GraphRAG retrieval quality depends heavily on graph quality, consistent entity linking, and meaningful relationships. Poor entity extraction, duplicate entities, or weak relationships directly impact retrieval effectiveness.
</Info>
<Info>
**Context Expansion Warning:** Larger hop counts exponentially increase the amount of retrieved context, which can significantly increase LLM token usage and processing time. Start with 2-3 hops and monitor context size for your use case.
</Info>
<Info>
GraphRAG activates automatically when you pass `knowledge_graph=` to `AgentContext`. There is no separate mode to switch on. The `hybrid_alpha` parameter and `proximity_weight` argument control how much influence graph structure has relative to vector similarity.
</Info>
@@ -18,7 +83,7 @@ from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
# FAISS runs locally with no external dependencies
vs = VectorStore(backend="faiss", dimension=768, index_path="intel.faiss")
vs = VectorStore(backend="faiss", dimension=768)
graph = ContextGraph(advanced_analytics=True)
context = AgentContext(
@@ -31,7 +96,7 @@ context = AgentContext(
)
```
Now ingest your documents. `store()` with `extract_entities=True` runs the full extraction pipeline internally — NER, relation extraction, and entity linking — and populates both the vector index and the graph simultaneously:
Now ingest your documents. `store()` with `extract_entities=True` runs the full extraction pipeline internally — Named Entity Recognition (NER), relation extraction, and entity linking — and populates both the vector index and the graph simultaneously:
```python
intel_documents = [
@@ -82,27 +147,24 @@ With the graph populated, a plain `retrieve()` call already does more than vecto
results = context.retrieve(
"APT29 tactics against healthcare",
use_graph=True,
proximity_weight=0.5, # blend structural proximity into the final score
# [score=0.798] Healthcare critical infrastructure has been a high-priority...
```
Notice the third and fourth results: their vector scores are modest (0.623 and 0.590) — neither document mentions APT29 or TTPs. But their proximity scores are high because they are structurally adjacent to the seed nodes in the graph. Pure vector retrieval would have ranked them much lower or excluded them entirely. GraphRAG surfaces them because the graph knows they are connected.
Notice the top results: while pure vector search might rank connected facts lower because they lack keyword overlap, GraphRAG boosts their final `score` because they are structurally adjacent to the seed nodes in the graph. The returned `score` is a transparent blend of vector relevance and graph connectivity.
When you know specifically which entity you want to anchor the traversal to, pass `anchor_node`:
**Excessive hop counts.** Setting `max_expansion_hops` too high (>4) creates exponentially large context that overwhelms LLMs and increases costs. Start with 2-3 hops and increase only if needed.
**Poor graph quality.** GraphRAG amplifies graph quality issues. Duplicate entities, inconsistent naming, and weak relationships produce poor retrieval results. Clean your graph data before relying on GraphRAG for important queries.
**Duplicate entities.** Having "APT-29", "APT29", and "Cozy Bear" as separate nodes breaks relationship traversal. Entity linking during ingestion helps, but manual deduplication may be necessary.
**Using GraphRAG for simple lookup queries.** If you know the answer exists in a specific document and just need to retrieve it, traditional vector search is faster and simpler than GraphRAG.
**Assuming graph expansion is always beneficial.** More context isn't always better. Sometimes precise, focused retrieval outperforms broad graph expansion. Test both approaches for your specific use cases.
## Tuning the vector-graph balance
The `hybrid_alpha` parameter set in the `AgentContext` constructor establishes a default blend between vector similarity and graph influence. `0.0` is pure vector retrieval; `1.0` is pure graph traversal. The recommended starting point is `0.5`.
You can override this per call using `proximity_weight` in `retrieve()` without changing the constructor default:
When targeting a specific `anchor_node`, you can apply `proximity_weight` in `retrieve()` to dynamically blend structural distance from the anchor into the final score:
```python
# Exploratory query — let semantics lead, graph confirms
@@ -50,6 +50,7 @@ Use the ingest module when your data lives outside Semantica and you need to bri
- **Web content** — public documentation sites, regulatory publication pages, news feeds, or any URL you can crawl.
- **REST APIs** — internal platforms (SIEM, EDR, ITSM, CRM), threat intelligence feeds, or any paginated HTTP endpoint.
- **Databases** — existing SQL databases where relevant records can be fetched with a targeted query.
- **Enterprise data platforms** — tables already living in a Databricks lakehouse (Unity Catalog + Delta Lake) or a Snowflake warehouse, without exporting to CSV first.
- **Live streams** — Kafka or other message brokers where you need to process events as they arrive.
- **Git repositories** — source code, documentation, or configuration files tracked in version control.
@@ -298,6 +299,89 @@ for bundle in stix_xml_files:
print(f"{bundle.source_path}: {len(bundle.elements)} elements parsed")
```
## Source 6 — Enterprise Data Platforms (Databricks & Snowflake)
`DatabricksIngestor` and `SnowflakeIngestor` return wrapper objects (`DatabricksData` / `SnowflakeData`) whose `.data` field is `List[Dict]` — the same list-of-dicts row shape that `DBIngestor.execute_query()` returns directly, without a wrapper. The same "transform to text, then store" pattern from Source 3 applies: pull only the tables and columns you need with a targeted query, then build a sentence per record before handing it to `AgentContext.store()`.
```python
from semantica.ingest import DatabricksIngestor
# Unity Catalog + Delta Lake — PAT or OAuth M2M auth
databricks = DatabricksIngestor(
host="https://adb-xxx.azuredatabricks.net",
token="dapi-xxxxxxxx",
http_path="/sql/1.0/warehouses/xxxxxxxx",
catalog="main",
)
# .data is List[Dict] — one dict per row, same shape as DBIngestor.execute_query()
customers = databricks.ingest_query(
"SELECT customer_id, name, industry, arr FROM main.default.customers "
f"Order {r['ORDER_ID']}: {r['PRODUCT']} in {r['REGION']}, ${r['AMOUNT']}"
for r in orders.data
]
```
Feed the resulting text lists into `AgentContext.store()` exactly like any other structured source:
```python
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
graph = ContextGraph(advanced_analytics=True)
context = AgentContext(
vector_store = VectorStore(backend="faiss"),
knowledge_graph = graph,
)
context.store(
customer_texts + lineage_texts + order_texts,
extract_entities=True,
extract_relationships=True,
)
print(f"Enterprise data graph: {graph.stats()['node_count']} nodes")
```
For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-pair vs. OAuth for Snowflake), schema/catalog introspection, and troubleshooting, see the dedicated [Databricks Integration](../integrations/databricks) and [Snowflake Integration](../integrations/snowflake) guides.
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
## Combining All Five Sources
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
Semantica exposes a unified provider interface — a single `.generate()` method — across Groq, OpenAI, Anthropic Claude, HuggingFace, Novita AI, and 100+ providers via LiteLLM. Use it when you need to swap providers for latency, accuracy, cost, or data-residency reasons without touching application code.
## What Are LLM Integrations?
The `semantica.llms` module provides a unified interface for connecting to Large Language Model providers. Instead of learning different APIs for each provider, you use the same methods (`.generate()`, `.generate_structured()`) regardless of whether you're calling Groq, OpenAI, Anthropic, or local HuggingFace models.
**Unified interface across providers:** All LLM providers in Semantica expose identical methods, so switching from OpenAI to Anthropic requires changing only the provider constructor, not your application code.
**Provider wrappers vs semantic extraction provider strings:** The `semantica.llms` classes (`Groq`, `OpenAI`, `LiteLLM`, `HuggingFaceLLM`) are Python objects for text generation. The `semantica.semantic_extract` module accepts provider names as strings for entity and relationship extraction. Both approaches are covered in this guide.
## Why Use LLM Integrations?
**Provider portability.** Test with one provider, deploy with another. Switch from Groq for prototyping to Anthropic for production without code changes.
**Reduced vendor lock-in.** Avoid tying your application to a single LLM provider's API. If pricing changes or service availability issues arise, switching providers is straightforward.
**Consistent APIs.** Use the same `.generate()` and `.generate_structured()` methods across all providers instead of learning provider-specific interfaces.
**Multi-provider workflows.** Run fast models for initial classification and expensive frontier models for complex reasoning in the same pipeline.
**Local vs cloud deployment flexibility.** Use cloud providers during development and switch to local HuggingFace models for air-gapped production environments.
## When To Use / When Not To Use
**Use LLM integrations for:**
- Text generation, summarization, and question-answering tasks
- Complex reasoning that requires natural language understanding
- Structured data extraction from unstructured text
- Multi-step analysis requiring interpretation and synthesis
- Tasks where context, ambiguity, or domain knowledge matter
**Deterministic tools may be better for:**
- Pattern matching that regular expressions can handle
- Simple rule-based classification with clear criteria
- Mathematical calculations or statistical analysis
- Graph traversal and relationship queries
- Data transformations with known logic
**A full LLM may be unnecessary for:**
- Simple keyword search or exact string matching
- Deterministic workflows with predefined decision trees
- High-frequency, low-latency operations where inference overhead matters
- Tasks where explainability requires transparent rule-based logic
<Info>
The providers in `semantica.llms` (`Groq`, `OpenAI`, `LiteLLM`, `HuggingFaceLLM`) are for text generation and `query_with_reasoning()`. For structured entity and relation extraction, `semantica.semantic_extract` accepts provider names as strings. Both patterns are covered here.
</Info>
## Choosing a Provider
Four factors drive provider selection. **Latency** matters most in real-time SOC triage loops where an analyst is waiting on a triage verdict — Groq's inference server typically returns 8B model responses in under 300ms. **Accuracy** matters most in high-stakes decisions: clinical contraindication checks, credit committee reasoning, and legal document analysis reward the frontier models available via `LiteLLM`. **Data residency** constraints eliminate cloud providers for classified or HIPAA-regulated workloads — `HuggingFaceLLM` with a local model path covers those cases. **Cost at scale** favors high-throughput open-model providers like Novita AI for bulk extraction pipelines where you are processing thousands of documents per hour.
Four factors drive provider selection, each optimized for different use cases:
The good news: because Semantica's interface is identical across providers, you can prototype with Groq for speed, validate accuracy with Claude, and deploy to Azure OpenAI for compliance — without changing a single line of your application code. Only the provider constructor changes.
**Latency** matters most in real-time SOC triage loops where an analyst is waiting on a triage verdict. Groq's inference infrastructure typically returns 8B model responses in under 300ms, making it ideal for interactive workflows.
**Accuracy** matters most in high-stakes decisions: clinical contraindication checks, credit committee reasoning, and legal document analysis. Frontier models like Claude or GPT-4 available through `LiteLLM` provide the strongest reasoning capabilities.
**Data residency** constraints eliminate cloud providers for classified or HIPAA-regulated workloads. `HuggingFaceLLM` with local model paths enables fully air-gapped deployments without network calls.
**Cost at scale** favors high-throughput providers like Novita AI for bulk extraction pipelines processing thousands of documents per hour where per-token costs accumulate quickly.
The unified interface means you can prototype with Groq for speed, validate accuracy with Claude, and deploy to Azure OpenAI for compliance — without changing application code.
## The Shared Interface
@@ -31,6 +81,8 @@ This means every place in Semantica that accepts an LLM — `query_with_reasonin
## Groq — Fast Inference for Real-Time Agents
**Groq** is a cloud provider that specializes in ultra-fast language model inference using custom hardware called Language Processing Units (LPUs). Their infrastructure delivers sub-300ms response times for smaller models, making them ideal for real-time applications where speed matters more than maximum reasoning capability.
Groq Cloud runs open models on purpose-built Language Processing Units that deliver sub-300ms latency for 8B parameter models. This makes Groq the right default for any agent loop where the LLM is in the hot path — SOC triage, real-time alert classification, conversational agents.
```python
@@ -64,6 +116,8 @@ Groq model selection comes down to the speed-vs-capability tradeoff: `llama-3.1-
## OpenAI — Function Calling and Vision
**OpenAI** provides access to the GPT model family, including GPT-4o with advanced capabilities like function calling (structured tool use) and vision processing for images and documents. OpenAI models are well-suited for complex reasoning tasks that require strong language understanding and generation capabilities.
The `OpenAI` provider wraps the OpenAI API. Use it when you need GPT-4o's function-calling precision, vision capabilities for document screenshots, or when your team already has an OpenAI contract and wants to stay there.
```python
@@ -91,6 +145,8 @@ The default model `gpt-3.5-turbo` is fine for classification and light extractio
## LiteLLM — One Interface, 100+ Providers
**LiteLLM** is a universal adapter that provides a single interface to over 100 different LLM providers, including Anthropic Claude, Azure OpenAI, AWS Bedrock, Google Vertex AI, and local Ollama instances. It acts as a translation layer, converting your unified API calls into provider-specific requests, enabling easy switching between providers without code changes.
`LiteLLM` is the Swiss Army knife. It wraps the `litellm` library, which speaks to every major provider using a unified completion API. The model string encodes both provider and model name: `"anthropic/claude-sonnet-4-20250514"`, `"azure/gpt-4o"`, `"bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"`, `"ollama/llama3.2"`. Change the string, change the provider — no other code changes needed.
**HuggingFaceLLM** provides access to open-source models from the HuggingFace ecosystem, either downloaded from the HuggingFace Hub or loaded from local file paths. This is the only option for completely offline deployments where no network access is available during inference, such as classified environments or air-gapped systems.
`HuggingFaceLLM` loads a model from the HuggingFace Hub or from a local directory path. No network calls during inference. This is the only option for classified environments, HIPAA-constrained clinical deployments, and any network segment without outbound internet access.
```python
@@ -302,9 +360,8 @@ from semantica.vector_store import VectorStore
**Choosing expensive frontier models for simple extraction tasks.** GPT-4o or Claude Sonnet for basic entity extraction is overkill — Groq's Llama models handle straightforward NER and classification at a fraction of the cost and latency. Reserve frontier models for complex reasoning that requires nuanced interpretation.
**Ignoring latency differences between providers.** Groq typically responds in under 300ms, while Anthropic Claude can take 2-3 seconds for the same query. For real-time agents or interactive workflows, latency differences compound across multiple LLM calls. Profile your provider performance under realistic load.
**Using LLMs for deterministic pattern matching that regex can handle.** If your task is extracting email addresses, phone numbers, or other pattern-based entities, regular expressions are faster, cheaper, and more reliable than LLM extraction. Use LLMs when context, ambiguity, or domain knowledge matter for correct interpretation.
**Not validating structured outputs.** The `generate_structured()` method returns parsed JSON, but LLMs can still produce malformed or incomplete structures. Always validate the returned dictionary against your expected schema before using the data downstream.
**Switching providers without testing prompt behavior.** Different models respond differently to the same prompt. A prompt optimized for GPT-4 may produce poor results with Llama or Claude. When switching providers, test your prompts and adjust temperature, instructions, or examples as needed.
**Overusing local HuggingFace models for tasks requiring latest knowledge.** Local models have a knowledge cutoff from their training date and cannot access current information. For tasks requiring up-to-date knowledge (recent CVEs, current regulations, latest threat intelligence), cloud providers with more recent training data may be necessary.
## Related Guides
- [Agent Memory](agent-memory) — using `query_with_reasoning()` with any LLM provider for graph-grounded retrieval
@@ -4,15 +4,50 @@ description: "Connect Semantica's knowledge graph, decision intelligence, and re
icon: "plug"
---
The Semantica MCP server exposes your knowledge graph as 12 callable tools so any compatible AI client — Claude Desktop, Windsurf, VS Code extensions — can traverse the graph live, record decisions, run analytics, and export results during a conversation. Use it to give LLM agents direct, real-time access to graph data without writing custom tool wrappers.
## What Is MCP?
MCP stands for the Model Context Protocol. It is an open standard that allows external AI assistants (like Claude Desktop, Cursor, or Windsurf) to securely access local tools and data sources.
The Semantica MCP server exposes your knowledge graph as 12 callable tools. By connecting it, any compatible AI client can traverse the graph live, record decisions, run analytics, and export results during a conversation — without you having to write custom tool wrappers.
<Info>
The Semantica MCP server exposes 12 tools and 3 read-only resources. All tools accept and return JSON. No configuration beyond an optional environment variable for graph persistence is required.
</Info>
## Architecture & Communication
It is important to understand how MCP works under the hood. **The Semantica MCP server is not a REST API.** There are no network ports, no HTTP endpoints, and no API keys required.
Instead, the AI client launches `semantica-mcp` locally as a subprocess. All communication between the AI and Semantica happens securely through standard input and output (`stdio`). Because the server runs locally under your user account, it inherently has your local file permissions.
## Why Use MCP With Semantica?
- **Zero-Code Integration**: Instantly connect Semantica's graph capabilities to your favorite AI IDE or desktop chat app without writing any glue code.
- **Real-Time Graph Updates**: Chat with an AI to extract entities from documents and watch them populate your live knowledge graph instantly.
- **Auditable AI**: Use the AI to make decisions and have it automatically record the reasoning and causal chain directly into the graph via Semantica's decision intelligence tools.
## When To Use / When Not To Use
- **When to Use**: You want to use a third-party AI interface (like Claude Desktop or Windsurf) to manipulate, query, and reason over a Semantica knowledge graph on your local machine.
- **When NOT to Use**: You are building an autonomous Python script or backend service. If you are writing Python code to build an agent, use `semantica.context.AgentContext` natively instead of spinning up an MCP server. The MCP server does not support remote hosting over HTTP/SSE.
---
## Typical Workflow
Connecting your AI client follows a standard progression:
1. **Install**: Install Semantica in your Python environment.
2. **Configure Client**: Add the `semantica-mcp` command and absolute graph paths to your AI client's JSON configuration.
3. **Start Client**: Launch Claude Desktop or Windsurf, which automatically spawns the MCP server.
4. **Tool Calls**: Prompt the AI in natural language. The AI autonomously chains the 12 available tools.
5. **Graph Updates**: The AI directly modifies your local graph, adding entities, edges, and decisions.
---
## Starting the Server
Install Semantica, then launch the MCP server. It starts in stdio mode by default — the protocol used by Claude Desktop, Windsurf, VS Code extensions, and most MCP clients.
Install Semantica, then configure your client to launch the MCP server. The server runs using the `stdio` transport.
```bash
pip install semantica
@@ -26,14 +61,14 @@ semantica-mcp
python -m semantica.mcp_server
```
Startup info prints to stderr. Without `SEMANTICA_KG_PATH` the server initialises an empty in-memory graph — sufficient for testing. For a persistent graph that survives restarts, set the path:
By default, the server logs at `WARNING` level and produces no startup output. Set `SEMANTICA_LOG_LEVEL=INFO` (or `DEBUG`) to see startup messages on stderr. Without `SEMANTICA_KG_PATH` the server initialises an empty in-memory graph — sufficient for testing. For a persistent graph that survives restarts, set the path:
Without `SEMANTICA_KG_PATH`, the graph resets when the server process exits. Always set this path for any session whose data should survive a restart.
Without `SEMANTICA_KG_PATH`, the graph resets when the server process exits. Always set this path using an absolute file path for any session whose data should survive a restart.
</Info>
## Connecting to Claude Desktop
@@ -46,7 +81,7 @@ Edit the Claude Desktop config file — on macOS at `~/Library/Application Suppo
@@ -56,7 +91,7 @@ Edit the Claude Desktop config file — on macOS at `~/Library/Application Suppo
Restart Claude Desktop after saving. The Semantica tools appear in the tool palette automatically — Claude can now call them during any conversation.
If `semantica-mcp` is not on your system PATH (for example, if it is installed in a virtualenv), use the full binary path in `"command"`: `"/path/to/venv/bin/semantica-mcp"`.
If `semantica-mcp` is not on your system PATH (for example, if it is installed in a virtualenv), use the full absolute binary path in `"command"`: `"/path/to/venv/bin/semantica-mcp"`.
## Connecting to Other Clients
@@ -66,7 +101,7 @@ If `semantica-mcp` is not on your system PATH (for example, if it is installed i
@@ -93,7 +128,7 @@ If `semantica-mcp` is not on your system PATH (for example, if it is installed i
```bash
docker run --rm -i \
-e SEMANTICA_KG_PATH=/data/kg.json \
-v /local/path:/data \
-v /local/absolute/path:/data \
ghcr.io/semantica-agi/semantica-mcp:latest
```
@@ -109,15 +144,42 @@ Once connected, the LLM can call any of these tools during a conversation. The a
**Reasoning** — `run_reasoning` applies forward-chaining IF/THEN rules over a set of facts and returns derived conclusions.
**Analytics and export** — `get_graph_analytics` computes PageRank centrality and community detection. `get_graph_summary` returns node count, decision count, and server status. `export_graph` serializes the current graph to Turtle, JSON-LD, N-Triples, or plain JSON.
**Analytics and export** — `get_graph_analytics` computes PageRank centrality and community detection. `get_graph_summary` returns node count, decision count, and server status. `export_graph` serializes the current graph to Turtle (`"turtle"` / `"ttl"`), RDF/XML (`"xml"`), N-Triples (`"nt"`), JSON-LD (`"json-ld"`), or plain JSON (`"json"`).
## Universal Example: Employee Directory
Before diving into complex domain examples, here is a simple, universally understood session. An HR manager types a prompt into Claude Desktop:
> "Extract entities from this meeting transcript about Alice transferring to Engineering, add them to the graph, and record a promotion decision."
Claude chains four tool calls automatically:
```text
1. extract_entities(text="Alice is transferring to Engineering...")
The graph is updated instantly with the new organizational structure and a fully auditable decision trail.
## Watching a Real Agent Session
Here is what happens when an analyst types a prompt into Claude Desktop and the graph is live. The prompt is:
Here is what happens when a cybersecurity analyst types a prompt into Claude Desktop and the graph is live. The prompt is:
> "Extract entities and relationships from this OSINT report, add them to the knowledge graph, then record an attribution decision for APT29 with confidence 0.88 and export the full graph as Turtle."
Claude chains five tool calls automatically:
Claude chains six tool calls automatically:
```text
1. extract_entities(text="<report text>")
@@ -157,7 +219,7 @@ Resources expose graph state without a tool call — the client can read them at
| URI | Description |
| :-- | :---------- |
| `semantica://graph/summary` | Node count, edge count, server status |
| `semantica://graph/summary` | Node count, decision count, server status |
| `semantica://decisions/list` | Up to 50 most recent recorded decisions |
| `semantica://schema/info` | Server version, capabilities, available tool list |
@@ -254,11 +316,23 @@ The result is a fully auditable credit decision trail with precedent links, read
</Tabs>
---
## Common Pitfalls
- **Treating MCP as an HTTP server**: Do not try to `curl` the MCP server or look for a port number. It communicates via `stdin/stdout` and waits for JSON-RPC messages from the parent AI client.
- **Using relative paths for `SEMANTICA_KG_PATH`**: Because the AI client spawns the server as a subprocess, the working directory can be unpredictable. Always use absolute paths (e.g., `C:\Users\Name\graph.json` or `/Users/name/graph.json`) to avoid losing your data.
- **Virtual environment PATH issues**: If you installed Semantica inside a Python virtual environment, Claude Desktop will not automatically find `semantica-mcp` on the global system PATH. You must provide the absolute path to the binary in the `"command"` field.
- **Expecting remote hosting support**: Stdio-based MCP servers must run on the same local machine as the AI client. Remote execution over a network is not supported.
- **Confusing MCP integration with `AgentContext`**: If you are writing your own Python code to orchestrate an LLM, do not use the MCP server. Use the `AgentContext` class natively within your code.
---
## Troubleshooting
**Server does not appear in Claude Desktop** — fully quit and reopen Claude Desktop after editing the config (close the window is not enough). Verify the binary is on PATH: `which semantica-mcp` on Unix, `where semantica-mcp` on Windows. If using a virtualenv, use the absolute binary path in `"command"`. Set `SEMANTICA_LOG_LEVEL=DEBUG` and check stderr for startup errors.
**Server does not appear in Claude Desktop** — fully quit and reopen Claude Desktop after editing the config (closing the window is not enough). Verify the binary is on PATH: `which semantica-mcp` on Unix, `where semantica-mcp` on Windows. If using a virtualenv, use the absolute binary path in `"command"`. Set `SEMANTICA_LOG_LEVEL=DEBUG` and check stderr for startup errors.
**Graph data not persisting between sessions** — set `SEMANTICA_KG_PATH` to an absolute file path. Without it the graph is in-memory only and resets on every server restart.
**Graph data not persisting between sessions** — set `SEMANTICA_KG_PATH` to an absolute file path. Without it, the graph is in-memory only and resets on every server restart.
**Tool calls returning empty results** — `get_graph_summary` returning `"node_count": 0` means the graph is empty. Populate it via `add_entity` and `add_relationship`, or run `extract_entities` on text first and then `add_entity` for each result.
description: "Coordinate multiple AI agents through shared memory, knowledge graphs, and decision history — without a message broker."
---
## What Is Multi-Agent Coordination?
A multi-agent system is a software architecture where multiple autonomous agents work together to accomplish complex tasks that would be difficult or impossible for a single agent to handle effectively. Instead of building one monolithic agent that tries to do everything, developers split work across specialized agents that each focus on specific responsibilities.
**Why split work across multiple agents:**
- **Separation of concerns** — each agent specializes in one domain (ingestion, analysis, reporting) rather than trying to master everything
- **Independent reasoning** — different agents can use different models, prompts, and reasoning strategies optimized for their specific tasks
- **Parallel processing** — multiple agents can work simultaneously on different aspects of the same problem
- **Human-like workflow decomposition** — mimics how human teams naturally divide complex analytical work
**Semantica's coordination approach:**
Semantica coordinates agents through shared context (memory and knowledge graphs) rather than message brokers or API calls between services. Agents read and write to the same underlying data structures, enabling seamless information sharing without complex middleware.
**Single-agent vs multi-agent architectures:**
- **Single-agent** — one `AgentContext` handles all tasks from ingestion through final output
- **Multi-agent** — multiple `AgentContext` instances or namespaced workflows, each responsible for specific pipeline stages or analytical roles
## Why Use Multi-Agent Systems?
**Separation of responsibilities.** Divide complex workflows into focused, manageable stages where each agent excels at its specific domain without being overwhelmed by tangential concerns.
**Scalability of complex workflows.** Handle sophisticated analytical pipelines that require different expertise areas, processing speeds, and reasoning approaches without creating unwieldy monolithic agents.
**Independent reasoning stages.** Enable different agents to use different LLMs, prompts, confidence thresholds, and reasoning strategies optimized for their specific tasks rather than compromising on a one-size-fits-all approach.
**Specialized agent roles.** Create agents tailored for ingestion, enrichment, analysis, synthesis, and reporting—each with role-appropriate configurations and capabilities.
**Shared knowledge and evidence.** Multiple agents contribute to and benefit from the same knowledge graph and memory stores, creating a cumulative evidence base that improves as more agents contribute their findings.
**Human-like workflow decomposition.** Mirror natural human team structures where analysts, researchers, and decision-makers each contribute specialized expertise to collaborative analytical processes.
- Multi-stage processing pipelines with distinct phases that benefit from specialized approaches
- Research and investigation workflows where different agents handle different information sources or analytical methods
- Teams of specialized agents with different roles (OSINT collector, enrichment analyst, fusion officer)
- Long-running workflows where different agents may operate at different times or schedules
- Scenarios requiring different LLMs, reasoning approaches, or confidence thresholds for different analytical stages
**Do NOT use multi-agent systems for:**
- Simple document summarization or single-step information retrieval tasks
- Linear workflows where one agent can handle all steps effectively without specialization benefits
- Small, straightforward tasks where the coordination overhead exceeds the complexity of the core work
- Cases where a single agent with appropriate configuration can handle the entire workflow efficiently
**Important consideration:** Multi-agent systems introduce additional architectural complexity including state management, coordination patterns, and debugging challenges. Only choose multi-agent approaches when the benefits of specialization and separation of concerns outweigh this added complexity.
Semantica coordinates multiple agents through a shared `ContextGraph` — agents read and write to the same graph, or hand off serialized state via `save()` and `load()`, with no message broker required. Use this pattern when splitting work across ingestion, enrichment, reasoning, and reporting roles that must share a single evidence base.
<Info>
@@ -13,17 +62,17 @@ Semantica coordinates multiple agents through a shared `ContextGraph` — agents
Before writing any code, choose the right coordination pattern for your pipeline.
**Shared graph** works when all agents run in the same process. They hold references to the same `ContextGraph` object — thread-safe by default — so every `store()` from one agent is immediately visible to every `retrieve()` from another. This is the lowest-latency option and the right default for in-process pipelines.
**Shared Graph Pattern:** Multiple agents share references to the same `ContextGraph` and `VectorStore` objects within a single process. This provides the lowest latency since all agents see changes immediately, with built-in thread safety for concurrent access. Choose this when agents run simultaneously in the same application and need real-time access to each other's contributions.
**Save / load handoff** works when agents run in different processes, on different machines, or at different times. Agent A finishes its work, calls `context.save(path)`, and Agent B calls `context.load(path)` to pick up exactly where A left off — full memory, full graph, full vector index. This is how you implement shift handoffs, async pipelines, and cross-service orchestration.
**Save / Load Handoff Pattern:** Agents run in different processes, containers, or at different times. The first agent completes its work and calls `context.save(path)` to serialize its complete state. The next agent calls `context.load(path)` to restore exactly where the previous agent left off, including full memory, graph data, and vector indices. Choose this for distributed systems, scheduled workflows, or when agents run on different machines that require shared storage access.
**Namespaced memories** works when you have a single `AgentContext` instance serving multiple logical agents, each scoping its reads and writes with a`conversation_id`. Agents are isolated by tag, not by instance — useful for lightweight role separation without the overhead of multiple contexts.
**Namespaced Memory Pattern:** A single `AgentContext` serves multiple logical agents, with each agent scoping its reads and writes using unique`conversation_id` values. Agents remain isolated by namespace rather than by separate context instances. Choose this for lightweight role separation without the resource overhead of maintaining multiple complete contexts.
The pipeline in this guide uses all three.
## Pattern 1 — Shared Graph for Concurrent Ingestion
The OSINT collector and the enrichment agent run concurrently. They share a single `ContextGraph` and a single `VectorStore` — the graph's internal `RLock` makes concurrent writes safe.
The OSINT (**Open Source Intelligence** — publicly available information) collector and the enrichment agent run concurrently. They share a single `ContextGraph` and a single `VectorStore` — the graph's internal `RLock` makes concurrent writes safe.
```python
import threading
@@ -71,7 +120,7 @@ def osint_collection():
],
extract_entities=True,
extract_relationships=True,
conversation_id="osint-pipeline",
conversation_id="osint-pipeline", # namespace acts as agent identifier
)
```
@@ -93,7 +142,7 @@ def enrichment():
],
extract_entities=True,
extract_relationships=True,
conversation_id="enrichment-pipeline",
conversation_id="enrichment-pipeline", # separate namespace from OSINT agent
)
```
@@ -113,6 +162,8 @@ t1.join(); t2.join()
The reasoning agent runs after ingestion completes. In a production pipeline this might be a separate process, a different container, or a scheduled job. The ingestion agents save their shared state; the reasoning agent loads it.
**Important deployment note:** When agents run in different containers or on different machines, they must have access to the same saved state location through shared storage (network file systems, cloud storage, or shared volumes).
```python
# After ingestion: save the combined graph and vector index
osint_agent.save("./pipeline/enriched_intel/")
@@ -131,7 +182,7 @@ from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
from semantica.llms import LiteLLM
# Create a fresh context before loading — load() merges into the existing context
# Create a context to load the checkpoint into — load() will overwrite existing state
`load()`merges into the existing context — it does not wipe it first. Always create a fresh `AgentContext` before calling `load()` if you want a clean restore from a handoff checkpoint.
`load()`overwrites the existing context — it clears current memory, graph, and vector state before loading. Any unsaved data in the context prior to calling `load()` will be lost.
</Info>
## Pattern 3 — Namespaced Memories for Role Separation
The reporting agent does not need its own graph instance. It shares the reasoning agent's context but scopes its writes to its own namespace — the `conversation_id` acts as an agent identifier.
The reporting agent does not need its own graph instance. It shares the reasoning agent's context but scopes its writes to its own namespace — the `conversation_id` acts as an agent identifier to separate memory streams and prevent contamination between different logical agents.
**Namespace isolation with conversation_id:**
- `conversation_id` creates separate memory namespaces within the same `AgentContext`
- Each agent's memories remain isolated unless explicitly queried across namespaces
- Prevents accidental memory contamination when different logical agents work on related but distinct tasks
```python
# The reporting agent loads the synthesis output
@@ -215,7 +271,7 @@ for item in synthesis_items:
# Store the final report under the reporting agent's own namespace
@@ -227,11 +283,27 @@ print("Pipeline produced {} traceable context items".format(len(full_trail)))
Each agent's contributions are retrievable individually by filtering on `conversation_id`, or collectively by querying without a filter.
## Common Pitfalls
**Forgetting conversation_id namespaces.** Without unique `conversation_id` values, different agents' memories mix together, making it impossible to trace which agent contributed which insights. Always use distinct, meaningful conversation IDs for each logical agent.
**Accidental state loss with load().** The `load()` function overwrites existing context rather than merging it. If you have unsaved state in an `AgentContext`, calling `load()` will wipe it. Always save your current state or use a fresh context before loading a checkpoint.
**Using Shared Graph across separate processes.** The Shared Graph pattern only works within a single process where agents share object references. For distributed agents running in different containers or machines, use the Save/Load Handoff pattern instead.
**Assuming save/load works without shared storage.** Agents in different processes, containers, or machines must have access to the same filesystem location for save/load handoffs. Ensure shared storage (NFS, cloud storage, shared volumes) is properly configured.
**Overengineering simple workflows with multiple agents.** Multi-agent systems add coordination complexity and potential failure points. For straightforward single-step tasks, a simple single-agent approach is often more reliable and easier to debug.
**Mixing agent responsibilities excessively.** Each agent should have a clear, focused role. Agents that try to do too many different tasks lose the benefits of specialization and become harder to optimize, debug, and maintain.
**Ignoring memory isolation boundaries.** When using namespaced memories, be careful about queries that span multiple `conversation_id` values. Unscoped queries can accidentally retrieve memories from other agents, breaking logical isolation.
## Domain Examples
<Tabs>
<Tab title="Defense — CTI/Threat">
A three-agent intelligence fusion cell: an OSINT collector ingests public feeds, a HUMINT analyst loads classified summaries, and a fusion officer synthesizes both streams into a Priority Intelligence Requirement answer. The OSINT and HUMINT agents run concurrently on a shared graph; the fusion officer loads the combined state in a separate process on an air-gapped network segment.
A three-agent intelligence fusion cell: an OSINT collector ingests public feeds, a HUMINT (**Human Intelligence** — information gathered from human sources) analyst loads classified summaries, and a fusion officer synthesizes both streams into a PIR (**Priority Intelligence Requirement** — critical information needed for decision-making) answer. The OSINT and HUMINT agents run concurrently on a shared graph; the fusion officer loads the combined state in a separate process on an **air-gapped environment** (isolated network with no internet connectivity for security).
@@ -4,17 +4,92 @@ description: "Define, version, and enforce governance policies over knowledge gr
icon: "scale-balanced"
---
## What Is Policy Engine?
Policy evaluation is the systematic checking of decisions against predefined governance rules and constraints. Unlike application enforcement that automatically blocks non-compliant actions, policy evaluation provides compliance status that can trigger different workflows—approval processes, exception handling, or audit requirements.
**Key policy concepts:**
**Policy evaluation** checks whether decisions meet defined criteria without automatically preventing actions, enabling flexible governance workflows.
**Governance and compliance workflows** use policy evaluation results to route decisions through appropriate approval chains, exception processes, or audit trails.
**Approval processes** can be triggered by policy violations, creating documented exception paths with justification and approver accountability.
**Difference from enforcement:** Policy evaluation returns compliance status (`True`/`False`) but does not automatically block actions. Your workflow determines what happens next—immediate approval, escalation, exception handling, or rejection.
## Why Use Policy Engine?
**Governance and accountability.** Create auditable decision workflows where every policy evaluation, exception, and approval is permanently recorded in the knowledge graph with full provenance tracking.
**Compliance verification.** Systematically check decisions against regulatory requirements, internal policies, and risk management rules before they are finalized or acted upon.
**Approval workflow orchestration.** Route non-compliant decisions through structured approval processes with documented justifications and multi-level sign-offs.
**Regulatory compliance.** Meet audit requirements by maintaining complete policy version histories, exception records, and compliance checking trails that regulators can inspect.
**Risk management.** Flag high-risk decisions for additional review while allowing routine compliant decisions to proceed with minimal friction.
**Policy evolution tracking.** Maintain version histories of policy changes with impact analysis, enabling evidence-based policy refinement and regulatory reporting.
## When To Use / When Not To Use
**Use Policy Engine for:**
- Governance workflows requiring structured approval processes and audit trails
- Regulatory compliance where policy adherence must be documented and verifiable
- Regulated environments where policy violations trigger specific escalation procedures
- Risk management workflows where non-compliant decisions require additional oversight
- Audit requirements demanding complete policy application and exception tracking
**Do NOT use Policy Engine for:**
- Simple form validation or basic input checking—use standard validation libraries instead
- Basic business rules that don't require audit trails or governance workflows
- Low-stakes, high-throughput checks where policy evaluation overhead would impact performance
- Deterministic rule checking that doesn't benefit from version tracking and approval processes
- Real-time operational decisions where policy evaluation latency is unacceptable
**Warning:** Policy Engine adds governance overhead and requires careful workflow design. Only use when the benefits of structured policy management outweigh the additional complexity.
`PolicyEngine` enforces named policies against recorded decisions, returning `True` if the decision satisfies all policy rules. Use it to gate AI decisions at runtime — attributions requiring dual-source confirmation, escalations requiring senior approval, or any decision category where compliance must be verified before the outcome is recorded. Policies are versioned graph nodes, so every check, exception, and approval chain is part of the permanent audit trail.
<Info>
The Policy Engine sits above `AgentContext` and `ContextGraph`. Policies are stored as nodes in the same graph as decisions, giving them the same causal tracing, provenance, and temporal validity as any other knowledge graph entity. `PolicyEngine` and `Policy` import from `semantica.context`. `Decision` imports from `semantica.context` (it is a dataclass defined in `semantica.context.decision_models`). `DecisionRecorder` imports from `semantica.context.decision_recorder`.
The Policy Engine sits above `AgentContext` and `ContextGraph`. Policies are stored as nodes in the same graph as decisions, giving them the same causal tracing, provenance, and temporal validity as any other knowledge graph entity.
**Key objects:** `PolicyEngine` and `Policy` import from `semantica.context`. `Decision` is a dataclass with fields like `decision_id`, `category`, `scenario`, `reasoning`, `outcome`, `confidence`, `timestamp`, `decision_maker`, and `metadata`. `DecisionRecorder` imports from `semantica.context.decision_recorder` for approval workflow tracking.
</Info>
## Supported Rule Types
The PolicyEngine implementation supports specific rule patterns that evaluate decision attributes and metadata:
- `allowed_outcomes: ["approved", "approved_with_conditions"]` — `decision.outcome` must be in the list
**Category validation:**
- `required_categories: ["credit_risk", "operational_risk"]` — `decision.category` must be in the list
**Metadata field rules:**
- `min_*: value` — metadata field must be `>= value` (e.g., `min_credit_score: 680`)
- `max_*: value` — metadata field must be `<= value` (e.g., `max_ltv: 0.85`)
- `required_*: value` — metadata field must equal `value` (string) or contain all items (list)
**Field lookup behavior:** For rule `min_credit_score`, the engine checks `metadata["credit_score"]`, then `metadata["*_credit_score"]` (suffix match), then `decision.credit_score` attribute.
**Important:** The following rule types are NOT supported and will cause unexpected behavior:
# - missing required_source_b and required_approver fields
```
The engine returns `False`. The decision has not been rejected — it has been flagged. What happens next depends on your workflow. In some organisations, a non-compliant result simply blocks the write to the authoritative graph. In others, it triggers an exception process where a human approver reviews the evidence and signs off.
@@ -174,7 +260,7 @@ The impact dict contains per-decision detail, not just the count. You can inspec
The lead decides to proceed with the threshold increase. She updates the policy to version 1.1.0, recording her reason. The old version is preserved in the history.
print(f"Policy updated: {updated_policy_id} -> version 1.1.0")
# Policy updated: pol-attr-001 -> version 1.1.0
print(f"Policy updated: {policy_id} to version 1.1.0")
# Policy updated: pol-attr-001 to version 1.1.0
# Find all decisions that were evaluated under v1.0.0 —
# these need to be re-reviewed to confirm they still meet the new standard.
@@ -225,6 +311,24 @@ for version in history:
---
## Common Pitfalls
**Assuming failed compliance automatically blocks actions.** PolicyEngine returns compliance status but does NOT automatically prevent actions. Your workflow must check the returned boolean and decide what happens next—approval, rejection, exception handling, or escalation.
**Using unsupported rule keys.** The implementation only supports specific patterns: `min_*`, `max_*`, `required_*`, `min_confidence`, `allowed_outcomes`, and `required_categories`. Any other rule key falls back to a key-presence check: it passes only if that exact key exists in `decision.metadata`, regardless of its value. This means keys like `disallowed_outcomes` will silently **fail** compliance whenever that literal key is absent from metadata (the common case), and will silently **pass** — regardless of the actual outcome — if a `disallowed_outcomes` key happens to exist in metadata with any value. Neither behavior matches the intended "outcome must not be in this list" semantics — use `allowed_outcomes` instead.
**Treating exceptions as approvals.** Recording a policy exception with `record_exception()` does NOT automatically make a non-compliant decision compliant. Exceptions are audit trail entries—your workflow must still decide whether to proceed with the non-compliant decision.
**Assuming PolicyEngine modifies graph state automatically.** PolicyEngine only evaluates compliance and records policy applications, exceptions, and approval chains. It does not modify decision outcomes, metadata, or prevent actions—that is your workflow's responsibility.
**Using complex nested rule structures.** The implementation does not support complex conditional logic, nested operators, or arbitrary expressions. Keep rules simple: single field comparisons, list membership checks, and threshold validations only.
**Missing metadata for rule evaluation.** Rules like `min_credit_score` require the corresponding metadata field (`credit_score`) to be present in `decision.metadata`. Missing metadata fields cause rule evaluation to fail, making the decision non-compliant.
**Forgetting to check rule evaluation results.** Always handle both compliant and non-compliant cases explicitly. Non-compliant decisions that proceed without proper exception handling create audit gaps and governance risks.
---
## Domain Examples
<Tabs>
@@ -247,10 +351,11 @@ opsec_policy = Policy(
name = "TLP:RED — Restricted Dissemination",
description = "TLP:RED intelligence must not be shared outside the originating organisation",
@@ -4,6 +4,59 @@ description: "How Semantica tracks the origin and lineage of every entity, relat
icon: "file-certificate"
---
## What Is Provenance?
Provenance is the systematic recording of where data came from, how it was transformed, and who was responsible for each step in its lifecycle. Unlike ordinary graph metadata that simply describes entities, provenance creates an immutable audit trail that tracks the complete history of every piece of information in your system.
**Key provenance concepts:**
**Lineage** traces the chain of custody from original source through all transformations to the current state, showing exactly how data evolved over time.
**Source attribution** records the specific document, database, API call, or human input that produced each data element, enabling precise citation and verification.
**Integrity verification** uses cryptographic checksums to detect any unauthorized changes to provenance records after they were created.
**Audit trails** provide regulatory compliance by maintaining tamper-evident logs of all data operations, transformations, and decisions.
Provenance differs from simple metadata by creating legally defensible, cryptographically verifiable records that answer critical questions: "Where did this come from?", "Who processed it?", "When did it change?", and "Has it been tampered with?"
## Why Use Provenance?
**Compliance with regulatory requirements.** Meet FDA 21 CFR Part 11, ICH E6(R2) GCP, Basel III BCBS 239, and defense intelligence sharing agreements that mandate complete data traceability and electronic record integrity.
**Source attribution and citation.** Trace every entity, relationship, and property value back to its exact source document, API response, or human input for scientific reproducibility and legal defensibility.
**Auditability and transparency.** Provide auditors, regulators, and stakeholders with complete visibility into data processing workflows, including who performed each operation and when changes occurred.
**Conflict resolution and data quality.** When multiple sources provide different values for the same property, provenance records enable evidence-based conflict resolution by comparing source credibility, recency, and confidence levels.
**Tamper detection and forensics.** Cryptographic integrity verification detects unauthorized modifications to data records, supporting incident response and forensic analysis in security-sensitive environments.
**Traceability for data lineage.** Answer complex questions about data ancestry, especially in multi-stage processing pipelines where entities undergo extraction, enrichment, fusion, and analysis transformations.
- Data processing is transparent and reproducible through version control alone
- Regulatory compliance does not require cryptographic integrity verification
`ProvenanceManager` records a W3C PROV-O compliant entry for every entity, relationship, document chunk, and property value — with a SHA-256 checksum for tamper detection and automatic version chaining on every `track_entity()` call. Use it when you need to answer regulatory questions about where a value came from, who wrote it, and whether it has changed since first ingestion.
For any regulated deployment — security operations, clinical data, financial risk — use `storage_path`. A SQLite file can be backed up, versioned, and queried with standard tools without requiring a server.
<Note>
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes read-modify-write operations (like `track_entity()`) in atomic immediate transactions (`BEGIN IMMEDIATE`); plain reads (`retrieve()`, `trace_lineage()`) use a separate connection without an explicit write lock so they don't serialize behind writers. Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
</Note>
## Recording provenance when ingesting data
The moment data enters your graph is the moment provenance must be recorded. `track_entity()` captures the source document, the timestamp, the operator or pipeline that ran the extraction, a verbatim quote from the source, and a confidence score. It returns a `ProvenanceEntry` with a SHA-256 checksum computed automatically.
The moment data enters your graph is the moment provenance must be recorded. `track_entity()` captures the source document, the timestamp, the operator or pipeline that ran the extraction, a verbatim quote from the source, and a confidence score. It returns an`Optional[ProvenanceEntry]` (`ProvenanceEntry` on success, or `None` if storage fails on a brand-new entity) with a SHA-256 checksum computed automatically.
```python
# Ingesting CVE-2024-3400 from NVD and a commercial feed
# The NVD entry is now archived as cve-2024-3400:v:2024-04-12T14:22:07
@@ -97,6 +152,8 @@ This version chaining happens automatically. You do not need to manage history e
When the same property appears in multiple sources with different values — exactly the CVE score situation — use `track_property_source()` to record each attribution separately. This feeds directly into conflict detection downstream: the conflict module can compare all tracked values for a property and surface disagreements with full source metadata attached.
**SourceReference** is a structured metadata container that captures exactly where a piece of information came from within a document. It includes the document identifier, specific location (page, section, byte range), confidence level, and custom metadata fields for domain-specific attribution requirements.
```python
from semantica.provenance.schemas import SourceReference
@@ -134,7 +191,7 @@ When the regulator asks "where did the 9.8 come from?", this is the answer: `com
## Tracing the lineage of a node
Six months after ingestion, run a lineage trace. `get_lineage()` returns the full version chain — every state the entity has passed through, oldest to newest — along with summary metadata:
Once you have multiple provenance entries for an entity, you can trace its complete history to understand how it evolved over time. Six months after ingestion, run a lineage trace. `get_lineage()` returns the full version chain — every state the entity has passed through, oldest to newest — along with summary metadata:
```python
lineage = prov.get_lineage("cve-2024-3400")
@@ -161,16 +218,16 @@ Sources seen : ['NVD_feed_2024-04-12', 'commercial_feed_2024-04-12',
@@ -179,7 +236,9 @@ The chain answers all three of the regulator's questions. The 9.8 came from `com
## Verifying integrity
Every `ProvenanceEntry` carries a SHA-256 checksum computed at write time. If any field is modified after the fact — by a misconfigured pipeline, a database migration, or deliberate tampering — the checksum will not match on recomputation. Run integrity checks as part of any compliance audit:
Every `ProvenanceEntry` carries a SHA-256 checksum computed at write time. If any field is modified after the fact — by a misconfigured pipeline, a database migration, or deliberate tampering — the checksum will not match on recomputation.
Integrity verification is critical for regulatory compliance and forensic analysis. Run integrity checks as part of any compliance audit:
```python
from semantica.provenance.integrity import compute_checksum
@@ -206,7 +265,9 @@ A `TAMPERED` status means the stored hash does not match what would be computed
## Tracking document chunks and their children
Provenance is not just for entities. When a document is split into chunks for RAG or NLP processing, each chunk needs its own provenance record linking it to the source file and byte range. Child chunks (from recursive splitting) link to their parent via `parent_chunk_id`, which maps to `prov:wasDerivedFrom` in the W3C model:
Provenance is not just for entities. When a document is split into chunks for retrieval-augmented generation (RAG) or natural language processing workflows, each chunk needs its own provenance record linking it to the source file and byte range.
Child chunks (from recursive splitting) link to their parent via `parent_chunk_id`, which maps to `prov:wasDerivedFrom` in the W3C PROV-O standard:
```python
# Track the parent chunk (a section of an advisory PDF)
@@ -260,6 +321,22 @@ Unique sources : 12
This summary is the starting point for a compliance attestation: you can state the total number of tracked records, the number of distinct data sources, and the breakdown by record type.
## Common Pitfalls
**Provenance does not guarantee truth.** Provenance records faithfully track where information came from and how it was processed, but it cannot verify that the original sources were accurate. A perfectly documented chain from a flawed or malicious source still produces unreliable data.
**Reusing generic source identifiers.** Using non-specific source IDs like "daily_feed" or "batch_001" makes it impossible to trace individual records back to their exact origins. Always include timestamps, version numbers, or unique batch identifiers in source document names.
**Bypassing provenance workflows.** Manually inserting data or using ad-hoc scripts that skip `track_entity()` calls creates gaps in the audit trail. Ensure all data entry points—automated pipelines, manual corrections, and administrative operations—record appropriate provenance.
**Ignoring lineage verification.** Provenance chains can become complex in multi-stage processing pipelines. Regularly verify that `get_lineage()` and `trace_lineage()` return complete, logical chains without missing links or circular references.
**Overusing provenance in low-value scenarios.** Recording provenance for every intermediate calculation or temporary variable creates storage overhead without compliance benefit. Focus provenance tracking on entities, relationships, and properties that have legal, regulatory, or business significance.
**Failing to validate integrity checksums.** Cryptographic integrity verification only works if you actually check it. Include regular `compute_checksum()` validation in audit workflows and incident response procedures.
**Mixing provenance granularities.** Tracking some entities at the document level and others at the sentence level creates inconsistent audit trails. Establish consistent granularity standards for each data type and processing workflow.
## Domain examples
<Tabs>
@@ -297,7 +374,6 @@ prov.track_entity(
entity_type="threat_actor",
activity_id="ner_extraction",
source_location="paragraph_3",
agent_id="analyst_ALPHA",
)
# Tier 3: Campaign relationship from all-source fusion
# Tier 4: Property from two independent INT sources
@@ -361,7 +436,6 @@ prov.track_entity(
confidence=0.98,
entity_type="vulnerability",
activity_id="nvd_feed_ingestion",
agent_id="ingest_pipeline_v2",
)
# Six weeks later: NVD revised the score after PoC publication
@@ -372,7 +446,6 @@ prov.track_entity(
confidence=0.98,
entity_type="vulnerability",
activity_id="nvd_feed_update",
agent_id="ingest_pipeline_v2",
)
# Track CISA KEV addition as a separate property source
@@ -433,7 +506,6 @@ prov.track_entity(
entity_type="clinical_endpoint",
activity_id="structured_data_extraction",
source_quote="Vaccine efficacy against COVID-19 was 95.0% (95% CI, 90.3–97.6)",
agent_id="meddra_extraction_pipeline_v3",
)
# Multi-study property tracking for meta-analysis
@@ -533,7 +605,6 @@ prov.track_entity(
confidence=0.89,
entity_type="credit_decision",
activity_id="automated_underwriting",
agent_id="underwriting_model_v4",
)
# SR 11-7 audit output
@@ -562,12 +633,29 @@ Every `ProvenanceEntry` maps directly to W3C PROV-O terms. If your compliance te
| :--- | :--- | :--- |
| `prov:Entity` | `entity_id` | The tracked object — entity, chunk, relationship, or property |
| `prov:Activity` | `activity_id` | The process that produced it — `"ner_extraction"`, `"bureau_parsing"` |
| `prov:Agent`| `agent_id` | Who ran the activity — pipeline name, analyst ID |
| `prov:wasDerivedFrom` | `parent_entity_id` | The previous version of this entity — enables version chaining |
| `prov:Agent`/ `prov:Person` / `prov:SoftwareAgent` / `prov:Organization` | `agent_id`, `agent_type`, `is_automated` | Who — or what — ran the activity, and whether a human was directly accountable |
| `prov:qualifiedAssociation` + `prov:hadRole` | `role` | The agent's role for this specific entity — `"generator"` (default), `"approver"`, `"reviewer"` — for sign-off/four-eyes workflows |
| `prov:wasDerivedFrom` | `parent_entity_id` (legacy combined field) | The previous version or source of this entity |
| — | `previous_version_id` | This entry corrects/replaces a prior version of the *same* fact |
| `prov:wasDerivedFrom` | `derived_from_id` | This entry was derived from a *different* source entity |
| `prov:used` | `used_entities` | Entity IDs consumed to produce this one |
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `datetime.utcnow()` at write time |
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `utc_now_iso()` at write time |
| `prov:qualifiedInvalidation` | `invalidated`, `invalidated_at_time`, `invalidated_by`, `invalidation_reason` | A retraction/correction recorded as a tombstone via `ProvenanceManager.invalidate()`, never a hard delete |
| `prov:startedAtTime` / `prov:endedAtTime` | `activity_started_at_time`, `activity_ended_at_time` | Typed Activity timing — pass an `ActivityRecord` via the `activity=` kwarg to set these together with `activity_id` |
| `prov:qualifiedGeneration`/`Generation`, `qualifiedUsage`/`Usage`, `qualifiedDerivation`/`Derivation` | (derived from the fields above) | Additive qualified forms of `wasGeneratedBy`/`used`/`wasDerivedFrom`, emitted automatically alongside the plain triples |
| `prov:wasAssociatedWith` | (derived from `agent_id`) | Direct Activity→Agent link, distinct from the Entity→Agent `wasAttributedTo` |
| `prov:actedOnBehalfOf` | `acted_on_behalf_of` | Agent→Agent delegation — e.g. an automated agent acting on behalf of the human/organization that authorized it |
| `prov:wasInformedBy` | `informed_by_activities` (pass as `informed_by=[...]`) | Chains this entry's activity to prior activities it was informed by (e.g. a pipeline stage informed by the stage before it) |
| `prov:Bundle` + `prov:hadMember` | `bundle_id` | Groups entries by source/dataset/ingestion-run (membership triples, not true RDF named-graph partitioning) |
| — | `valid_from`, `valid_until`, `revision_type`, `supersedes` | Bitemporal fields merged from the deprecated `kg.ProvenanceTracker` — always caller-supplied (never auto-computed), surfaced via `ProvenanceManager.revision_history()`, which falls back to timestamp-based derivation for entries that don't set them explicitly |
The `checksum` field is not part of the PROV-O standard — it is Semantica's tamper-detection extension. Every entry's SHA-256 is computed from its content fields at write time and can be recomputed at any time to verify the record has not been modified.
`previous_version_id` and `derived_from_id` are additive alongside `parent_entity_id` — existing code reading `parent_entity_id` keeps working unchanged, while new code gets the two relations disambiguated.
The `checksum` field is not part of the PROV-O standard — it is Semantica's tamper-detection extension. Every entry's SHA-256 now also incorporates `previous_checksum` (the prior entry's checksum, by insertion order via `sequence_id`), chaining every entry to the one before it. `ProvenanceManager.verify_chain()` walks the full chain and reports any break — including a row that was hard-deleted from the underlying table, which a lone per-row checksum can't detect on its own.
Note: the banking example above passes `agent_id="credit_data_service_v2"` to `track_entities_batch()` — this now actually populates the entry's `agent_id` field (previously a bug caused batch-level typed kwargs like `agent_id`/`entity_type`/`activity_id` to be silently absorbed into the opaque `metadata` blob instead).
`export_prov()` mints entity/agent/activity URIs under `ProvenanceManager.DEFAULT_BASE_URI` (`https://semantica.dev/ns#` by default — the same namespace `RDFExporter`'s `NamespaceManager` uses for its `"semantica"` prefix, so KG-exported and PROV-exported URIs for the same `entity_id` co-resolve) unless overridden via `export_prov(base_uri=...)` or the CLI's `--base-uri` option.
DELTA-3 is flagged even though no document described it that way — the system traced: DELTA-3 supplied GAMMA-7, and GAMMA-7 exploits critical CVEs. For rules that need priority ordering or graded confidence, use the `Rule` dataclass:
If a rule has side-effecting actions, one concrete activation runs those
actions at most once on a Reasoner instance. Re-running `forward_chain()` is
therefore safe: already-attempted actions are not repeated. Use
`reasoner.reset_action_history()` when you intentionally want to replay them;
`reasoner.clear()` and `reasoner.reset()` also clear the history.
```python
# Higher priority rules fire first; confidence propagates into InferenceResult.confidence
reasoner.add_rule(Rule(
@@ -269,7 +275,7 @@ print("Loaded {} facts from graph".format(count))
## Step 5 — SPARQL queries over enriched working memory
After forward chaining has derived new facts, `SPARQLReasoner`lets you query the enriched working memory using SPARQL triple-pattern matching with optional inference expansion:
After forward chaining has derived new facts, `SPARQLReasoner`prepares SPARQL queries over the enriched working memory with optional inference expansion:
# metadata shows how many results came from inference vs ground facts
print("Original: {} Inferred: {}".format(
result.metadata.get("original_count", 0),
result.metadata.get("inferred_count", 0),
))
# expand_query() applies inference rules to the query text:
expanded = sparql.expand_query(query)
print(expanded)
```
`execute_query()` is not implemented yet: no triplet-store execution path exists, so it raises `NotImplementedError` rather than returning an empty result set that callers would misread as "no matches". Until execution lands, run the expanded query against your RDF store directly (for example with `rdflib`).
Inspect the expanded query before running it:
```python
@@ -369,6 +366,13 @@ engine.reset()
The rule network is compiled once by `build_network()`. Each subsequent `add_fact()` call propagates incrementally through only the nodes whose conditions it satisfies — not the full rule set — which keeps evaluation cost proportional to the number of new activations rather than the total rule count.
With a Reasoner bound, Rete action side effects are attempted once per rule,
bindings, and matched fact identity. Passing the same match to
`execute_matches()` again still returns the same conclusion, but does not repeat
its actions. Call `engine.reset_action_history()` to replay actions without
clearing working memory. `engine.reset()` and `engine.build_network()` also
clear the action history.
## Step 7 — Temporal interval reasoning
`TemporalReasoningEngine` computes Allen interval relations between time windows, letting you identify whether two events overlap, one contains the other, they meet at a boundary, and so on across your graph:
Semantic extraction is the process of automatically identifying meaningful information from unstructured text and converting it into structured, machine-readable formats. Unlike simple keyword search or pattern matching, semantic extraction understands context, relationships, and implicit connections between concepts in natural language.
**Key differences from basic text processing:**
- **Regex matching** finds exact patterns but misses contextual meaning
- **Keyword search** locates terms but ignores relationships between them
- **Manual annotation** captures semantic meaning but doesn't scale
- **Semantic extraction** automatically identifies entities, relationships, and events while preserving contextual understanding
When you extract entities like "APT29" and "NATO" from intelligence text, semantic extraction also captures that APT29 "targets" NATO networks, creating structured knowledge that feeds directly into graph databases, reasoning systems, and retrieval workflows.
## Why Use Semantic Extraction?
**Knowledge graph population.** Transform unstructured documents into interconnected knowledge graphs where entities become nodes and relationships become edges, enabling sophisticated graph traversal and reasoning.
**GraphRAG preparation.** Extract structured facts from raw text so that graph-grounded retrieval can find precise, contextually relevant information instead of just similar document chunks.
**Turning unstructured text into structured data.** Convert intelligence reports, clinical notes, legal documents, and regulatory filings into databases, RDF triples, and JSON schemas that downstream systems can query and process.
**Downstream retrieval and reasoning benefits.** Enable precise entity-based search, relationship discovery, causal analysis, and multi-hop reasoning that would be impossible with document-level retrieval alone.
**Automated knowledge discovery.** Surface hidden connections and patterns across large document collections that human analysts would miss due to volume and complexity.
## When To Use / When Not To Use
**Use semantic extraction for:**
- Converting intelligence reports, clinical notes, and regulatory documents into structured knowledge
- Building knowledge graphs from unstructured text corpora
- Preparing text for graph-based reasoning and GraphRAG workflows
- Discovering relationships and connections across document collections
- Creating structured datasets for downstream analysis and reporting
**Deterministic parsing may be better for:**
- Highly structured identifiers like email addresses, UUIDs, hashes, and log IDs where regex patterns are sufficient
- Simple data extraction from standardized formats (CSV, JSON, XML)
- Known patterns with fixed formats that don't require contextual understanding
- High-frequency operations where extraction speed is critical and semantic understanding unnecessary
**Consider simpler alternatives when:**
- Documents are already structured and don't require natural language understanding
- Simple keyword search or document retrieval meets your requirements
- Text quality is too poor for reliable semantic analysis (heavily corrupted OCR, fragmentary data)
## Typical Workflow
The semantic extraction workflow follows a structured sequence that transforms raw text into graph-ready knowledge:
**Ingest** → Load documents from various sources (files, databases, APIs) and prepare text for processing
**Extract** → Apply Named Entity Recognition (NER), relation extraction, event detection, and coreference resolution to identify meaningful information
**Resolve** → Consolidate entity mentions ("APT29", "the group", "they") into canonical references and disambiguate overlapping entities
**Relate** → Connect extracted entities through relationships, creating a web of structured connections between concepts
**Serialize** → Convert the extracted knowledge into RDF triplets, JSON-LD, or other structured formats
**Store** → Load structured output into knowledge graphs, vector databases, or agent memory systems
**Retrieve** → Query the structured knowledge through graph traversal, semantic search, and reasoning workflows
This pipeline transforms documents like "APT29 deployed HAMMERTOSS malware targeting NATO networks" into structured triplets like `(APT29, deployed, HAMMERTOSS)` and `(HAMMERTOSS, targets, NATO_networks)` that enable sophisticated downstream analysis.
`semantica.semantic_extract` turns unstructured text into structured graph-ready output: it identifies named entities, extracts relationships between them, detects time-anchored events, resolves coreferences, and serialises everything as RDF triplets. Use it to populate a `ContextGraph` from raw documents — intelligence reports, clinical notes, regulatory filings, or any free-text corpus.
<Info>
@@ -12,6 +76,8 @@ icon: "magnifying-glass"
## Step 1 — Named Entity Recognition: who and what is in the text
**Named Entity Recognition (NER)** identifies and classifies meaningful nouns and noun phrases in text, such as people, organizations, locations, products, and domain-specific entities like threat actors or drug names. NER forms the foundation of semantic extraction by identifying the key participants and objects in your documents.
`NamedEntityRecognizer` extracts meaningful nouns from a document and lets you choose the underlying method depending on your latency budget and domain requirements:
## Step 2 — Relation Extraction: how the entities connect
**Relation Extraction** identifies semantic relationships between entities, capturing not just what entities exist in text but how they interact, influence, or connect to each other. This creates the edges that link entity nodes in your knowledge graph.
`RelationExtractor` produces the web of connections between entities — who deployed what, who supplied whom, which CVE targets which product:
```python
@@ -111,6 +179,8 @@ The `context` field on each `Relation` stores the surrounding sentence. This let
## Step 3 — Event Detection: what happened, when, and to whom
**Event Detection** identifies discrete occurrences or actions described in text, capturing not just static relationships but dynamic processes that unfold over time. Events include participants, temporal boundaries, locations, and outcomes.
`EventDetector` surfaces structured time-anchored events — discrete occurrences with participants, time windows, and locations:
```python
@@ -155,6 +225,8 @@ for doc_idx, doc_events in enumerate(batch_events):
## Step 4 — Coreference Resolution: one entity, many names
**Coreference Resolution** identifies when different text spans refer to the same real-world entity, consolidating mentions like "APT29", "the group", "they", and "the threat actor" into unified references. This prevents downstream processing from treating the same entity as multiple separate objects.
`CoreferenceResolver` collapses references like "GAMMA-7", "the group", "they", and "the threat actor" into canonical chains so downstream extraction doesn't treat them as separate entities:
```python
@@ -180,6 +252,8 @@ With coreference resolved, you can now replace pronouns and aliases with canonic
**Triplet Extraction** converts semantic knowledge into subject-predicate-object triplets, the fundamental building blocks of knowledge graphs and RDF databases. This structured representation enables graph queries, reasoning, and integration with semantic web technologies.
`TripletExtractor` converts everything into subject-predicate-object triplets and serialises them as RDF, ready for graph ingestion and SPARQL queries:
**Treating extraction as guaranteed truth.** Semantic extraction produces confidence scores for a reason — even high-confidence extractions can be incorrect. Always validate critical extractions, especially for high-stakes decisions in security, clinical, or financial contexts.
**Ignoring confidence thresholds.** Low-confidence extractions often indicate ambiguous text, poor model fit, or noisy input. Setting appropriate thresholds (typically 0.65-0.85) filters unreliable results before they pollute downstream processing.
**Skipping entity resolution.** Different mentions of the same entity ("NATO", "North Atlantic Treaty Organization", "the alliance") will create duplicate nodes in your knowledge graph. Always run coreference resolution and entity deduplication.
**Poor OCR or poor input quality.** Semantic extraction depends on readable text. Documents with OCR errors, encoding issues, or heavy redaction will produce unreliable extractions. Clean and validate input text before extraction.
**Using LLM extraction where regex is sufficient.** For highly structured patterns like CVE identifiers (CVE-YYYY-NNNN), IP addresses, email addresses, or UUIDs, regular expressions are faster, cheaper, and more reliable than semantic extraction.
**Processing too much text at once.** Very long documents (>10,000 words) can overwhelm extraction models and produce inconsistent results. Segment long documents into logical chunks (sections, paragraphs) and process them separately.
**Mixing incompatible extraction methods.** Different methods produce different entity label schemas. LLM extraction might return "THREAT_ACTOR" while spaCy returns "PERSON" for the same entity. Normalize labels across methods or use consistent method chains.
## Choosing your extraction method
The six extraction methods trade off speed, accuracy, and infrastructure:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
