Moves a concise version of the system-level vs. foundation-model
explainability clarification up next to the opening pitch, so it's
visible before readers scroll to the high-stakes-domains section.
Adds a consistent scope note to README and docs (concepts, FAQ, index)
stating Semantica does not expose or reconstruct an LLM's internal
reasoning/chain-of-thought. It explains and audits the AI system
around the model: context, provenance, policies, decisions, and
execution history.
* docs(contributing): formalize issue assignment and duplicate-PR triage workflow
Comments are no longer required before an issue can be assigned - maintainers
may assign directly based on recent activity. Also documents the duplicate-PR
priority order for triage (contributor PR, claimed issue, activity tiebreak,
late duplicates, overlapping scope).
* docs(contributing): clarify assignment precedence and define activity tiebreak
Addresses Qodo review feedback on PR #1030: the duplicate-PR priority list
now states these rules apply on top of the assignment workflow (opening a PR
pre-assignment doesn't grant priority), and the "most active" tiebreak now
specifies a concrete 60-day window and signals instead of being subjective.
ExcelParser.__init__ called get_progress_tracker() without importing it,
so every instantiation raised NameError and the class was unusable. The
existing test imported ExcelParser but never constructed it, so nothing
caught it. Same defect as #530 in SimilarityCalculator, which was fixed
without sweeping the rest of the codebase.
Add construction coverage for every parser exported from semantica.parse,
driven off __all__ so later additions are covered automatically. These
live outside test_parse_comprehensive.py, whose setUp patches
get_progress_tracker into each parse module and would mock away the
interaction under test.
Closes#1014
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
* feat(crewai): add first-class CrewAI integration (#962)
Add native CrewAI support so Crew agents can share a ContextGraph and
AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching
the existing agno integration pattern.
- SemanticaKGTool: 5 KG actions (extract_entities, extract_relations,
add_to_graph, query_graph, find_related) with sync run()/async arun()
- SemanticaDecisionTool: 5 decision-intelligence actions
(record_decision, find_precedents, trace_causal_chain,
analyze_impact, check_policy) over AgentContext
- SemanticaKnowledgeSource: serializes a ContextGraph into crew
knowledge storage; bridges legacy load_content() and current
validate_content()/aadd() contracts for crewai>=0.80.0
- All classes degrade gracefully when crewai is absent
- New pip extra crewai=... included in the all bundle
- 70 new tests (stub-based present-case + subprocess degradation path)
- Docs: integrations/crewai.md, docs.json nav, README matrix updates
* fix(crewai): harden tools against real Semantica dataclass shapes (#962)
Bugs found during live testing with crewai 1.15.16:
- SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses
('str' object has no attribute 'end_char'): string names were passed to
extract_relations(entities=...), which requires Entity objects, and the
tool read .name/.source/.target instead of Entity's .text/.label and
Relation's .subject/.object. Add shape-agnostic field helpers.
- SemanticaDecisionTool() created an AgentContext without a knowledge_graph,
so _decision_backend was never set and record_decision raised 'Decision
tracking is not enabled'. Wire in a ContextGraph.
- record_decision hard-failed when the agent omitted optional fields; fall
back to category='general', reasoning='agent decision',
outcome='recorded'.
Add tests covering real Entity/Relation dataclass shapes and the live
auto-created AgentContext path (now 77 crewai tests, 212 total).
* fix(crewai): make find_related traverse edges undirected (#962)
ContextGraph.get_neighbors only follows outgoing edges, so a node whose
only edge is incoming (A -> B) reported no related concepts. Rebuild a
bidirectional adjacency from find_edges() in SemanticaKGTool._find_related
so 'related' honors both directions.
* fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962)
- Exclude live graph/context/extractor state from JSON serialization
(model_dump(mode="json")) so CrewAI checkpointing no longer raises
PydanticSerializationError; model_post_init self-heals defaults on restore
- query_graph now searches node content via graph.query() plus id/type
- trace_causal_chain returns an explicit error when causal tracing is
unavailable instead of substituting similarity precedents; call
trace_decision_causality(..., max_depth=...) with the correct kwarg name
- find_precedents propagates max_precedents/limit to the backend instead of
being silently capped at 10
- Serialize add_to_graph batches under a module lock to prevent concurrent
double-counting; skip nameless entities instead of creating repr()-junk nodes
- aadd() runs CPU-bound serialization in a thread executor
- Mirror crewai args_schema serialize/restore in the conftest stub and add
serialization regression tests (crewai: 92 tests)
* fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962)
- _eval_rule now coerces rule values type-aware: bool("false") was truthy, so
'enabled == false' reported a violation for enabled=false, and string datums
like "0.90" were compared lexicographically instead of numerically
- _trace_causal_chain no longer raises AttributeError (which escaped _run) when
the decision context lacks knowledge_graph; returns honest error JSON
- SemanticaKnowledgeSource storage failures log an actionable ERROR; without a
configured crew embedder agents previously retrieved nothing silently
- add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a
process-global one: independent graphs no longer serialize each other and
re-entrant extractor callbacks cannot deadlock
- entity/relation confidence=None normalizes to 1.0 instead of failing the
whole extraction with float(None)
- add subprocess integration test against real crewai covering Crew-level
serialization round-trip and checkpoint restore (stub tests cannot see it)
- docs: embedder requirement for SemanticaKnowledgeSource; resume contract note
* fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962)
Re-verification against real crewai showed the embedder-missing failure raises
ValueError even though storage IS wired, so the old except-ValueError branch
mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure.
Distinguish by storage presence instead of exception type: storage is None ->
DEBUG keep-in-memory (legitimate standalone use); storage wired but save()
raises -> actionable ERROR. Add regression test mirroring real crewai's
ValueError-on-missing-embedder behavior.
* fix(crewai): expose run()/arun() entry points in degraded mode (#962)
The public crewai contract is run()/arun(); without crewai installed they were
missing (only the private _run existed), so the documented 'usable without
crewai' path raised AttributeError at the entry point. Define them in degraded
mode only, leaving crewai's BaseTool implementations untouched when present.
Extend the degradation subprocess test to exercise run() and arun().
* fix(crewai): standardize query shape, field-name rules, and restore-state flag
- _query_graph: id/type matches now return the same schema as content
matches (id/type/label/content/score) instead of a bare list
- _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys
(e.g. "risk-score >= 0.9") are addressable in policy rules
- add had_live_state/reconstructed_state so checkpoint-restored tools
and knowledge sources signal that their live graph/context was lost
and an empty one reconstructed; knowledge source no longer hides the
loss by eagerly rebuilding its graph inside __init__ (pydantic calls
__init__ during model_validate)
* fix(crewai): address Qodo review — confidence errors, string trim, holistic availability
- record_decision: stop calling float() in _run, so malformed confidence
values surface as JSON errors (via _record_decision's handling) instead
of crashing the tool
- _coerce_value: return the stripped string for non-numeric literals so
whitespace-padded decision_data fields match policy rules
- centralize crewai availability in _availability.py so the exported
CREWAI_AVAILABLE flag is holistic across tools and knowledge source
(previously each module probed crewai independently and the package
flag came from decision_tool only)
* ci: regenerate requirements-ci.txt for the crewai extra
The crewai extra in pyproject.toml brings in crewai, crewai-tools and
transitive deps (chromadb, lancedb, ...). Recompile with
uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes.
* ci: keep crewai out of the locked CI dependency set
crewai (all versions) hard-requires chromadb~=1.1.0, which carries a
pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c)
with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in
the 'all' extra failed pip-audit and the safety check on requirements-ci.txt.
- drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is
unchanged and still installs crewai)
- stop listing crewai-tools in the extra: the integration only uses crewai core
(BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps
- regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0
vulnerabilities, staleness check matches
* docs(crewai): document crewai extra scope and chromadb CVE-2026-45829
- CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not
part of the 'all' bundle, with the chromadb CVE-2026-45829 reason
- integrations/crewai/README.md: add a security warning that installing
the extra pulls chromadb~=1.1.0, which is affected by the unpatched
pre-auth code-injection CVE-2026-45829
---------
* refactor(export): centralize graph-payload key normalization
Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.
Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.
Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.
Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.
CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.
* fix(export): reject non-mapping input to the YAML exporters
export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.
Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.
Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.
* fix(export): address YAML exporter review findings
- semantica/export/yaml_exporter.py — import Sequence from typing
instead of collections.abc. `Sequence[str]` in _require_mapping's
annotation is evaluated at function-definition time; collections.abc.Sequence
only became subscriptable in Python 3.9, so on the 3.8 this project
declares support for, importing this module raised TypeError.
typing.Sequence has supported subscripting since 3.5.3. Mapping stays
imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
and read exported YAML through a context manager instead of an
unclosed yaml.safe_load(open(...)).
* fix(export): reject YAML export payloads with no recognized key
Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.
- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
collections through normalize_graph_payload(), which raises rather than
returning empty collections for an unrecognized mapping. Adopting the
shared resolver rather than repeating the check locally also brings the
'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
from this library's own graph type to YAML, used in
examples/capability_gap_context_graphs_example.py — exports its records
instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
defaulted lookups and had the same defect; it goes through the resolver
too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
its own key set. Schemas are a separate vocabulary with no aliasing, so
_require_recognized_keys lives in this module rather than in the shared
graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
An export_json envelope carries one, so accepting it would readmit the
case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
output directory, so a rejected export leaves nothing behind.
The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.
Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.
Closes#953.
* fix(export): reject payloads whose records resolve to nothing
Addresses the Qodo findings on #958.
Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.
- utils/helpers.py — split the check in two. _require_recognized_keys keeps
the presence rule; _require_nothing_dropped runs after resolution and
refuses a payload that resolved to nothing while an unread key still holds
records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
always carries a populated 'statistics' dict, and an empty graph must stay
exportable, so 'metadata', 'statistics' and 'count' are named as context
rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
both checks through the shared helpers rather than its own copy, so the
two vocabularies cannot drift apart in what counts as a silent-empty
export.
Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.
- export/yaml_exporter.py — the serialization span now says it serialized,
not that it exported, and export() opens its own span around the
filesystem work that stops as failed on error. Nothing reports a completed
export until the bytes are on disk.
Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.
Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.
* fix(export): validate collection values and make Neo4j mappings strict
Two gaps at the boundary the shared normalizer is supposed to own.
_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.
Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.
Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.
* fix(export): close YAML schema and record validation gaps
Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived. Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.
Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects. Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary. Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.
Tests: 101 tests pass across
tests/utils/test_normalize_graph_payload.py
tests/export/test_yaml_exporter_key_recognition.py
tests/export/test_yaml_exporter_input_validation.py
tests/export/test_neo4j_csv_exporter.py
* fix(export): close exception-type and record-shape gaps in normalize_graph_payload
LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).
Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).
normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.
Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.
* docs(changelog): add entry for #958 YAML export input hardening
Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes#956, #952, #953.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Explorer was firing temporal requests before the graph even loaded.
When the backend is down, /api/graph/nodes fails but the temporal
bounds and snapshot effects didn't care , they fired anyway, off in
their own corner, ignoring whether the graph actually came up. Every
page load with no backend meant three failed requests instead of one,
and a scrubber that had nothing to scrub.
Added two small predicate functions and gated the temporal effects on
them. Basically: don't ask for time-based data until you know the
graph itself loaded. An empty graph still counts as loaded, so that
case isn't broken.
Confirmed with the backend down, before and after: three failing
requests down to one.
Fixes#982.
Fix DBIngestor calls in load_from_database , it was never actually reaching the db.
execute_query/export_table need the connection string as their first arg,
but we were only passing it to the constructor's config dict, which
those methods don't read. Every call blew up with a TypeError before
connecting.
Also split the ImportError/OSError handling , they were caught together
so a real connection failure got reported as "module not available",
which sent people looking in the wrong place. OSError now surfaces as
an actual failure with the original exception chained via `from e`.
Fixes#973.
* feat(context): add retraction and purge to ContextGraph
ContextGraph had 56 public methods and none that removed anything: the only
option was clear(), which discards the whole graph. Removing one entity meant
exporting to a dict, filtering by hand and rebuilding, losing provenance.
Add two operations with deliberately different contracts.
retract_node/retract_edge close the entity's validity window. The entity stops
being active going forward, but state_at() before the retraction still returns
it, so decisions recorded against it remain explainable. This reuses the
valid_from/valid_until machinery already present rather than adding a new
subsystem.
purge_node/purge_edge remove the entity outright, from history as well as from
the active view, leaving a tombstone that records that a purge happened and why
but never the purged content. Scope is this graph only; copies in AgentMemory
or a bound vector store are not reached, so it is one step of an erasure
workflow rather than the whole of it.
Both record themselves through the existing mutation_callback path.
MutationRecord already documented REMOVE_NODE/REMOVE_EDGE in its operation
vocabulary, so retraction emits UPDATE_NODE and purge emits REMOVE_NODE with no
changes required to change_management.
Incident-edge lookup scans self.edges rather than _adjacency, which is keyed by
source only and would otherwise leave inbound edges pointing at a removed node.
Purge updates edges, edge_type_index and _adjacency together so the indexes
cannot drift, and clear() now resets the retraction and tombstone records.
* fix(context): address review findings on retraction and purge
* fix(context): close every duplicate when retracting/purging by edge_id
edge_id is content-derived and not yet guaranteed unique (#922, fix
pending in #926): two identical add_edge() calls produce two edge
objects sharing one id. retract_edge()/purge_edge() resolved "the
edge" via the first matching object only, so a duplicate was silently
left untouched (still live, still active) while the call returned
True and recorded a tombstone/retraction claiming it was fully
handled. Repeat purge_edge() calls also silently overwrote the
tombstone's reason/purged_at on each partial attempt instead of
no-op'ing once nothing remained to purge.
retract_node()'s cascade had the same root cause from the other
direction: it checked the live _retractions dict mid-loop, so the
first duplicate's just-written record made the second look already
handled and it was skipped outright, left permanently active.
retract_edge()/purge_edge() now act on every edge matching the id
under a single record; the cascade's dedup check is snapshotted
before the loop starts so within-call duplicates are still closed
rather than skipped.
Adds TestDuplicateEdgeId (5 tests) reproducing all three paths.
* docs(changelog): document retraction/purge feature
Adds an Unreleased/Added entry for #955/#957 covering retract_node,
retract_edge, purge_node, purge_edge and the get/list accessors, plus
the duplicate-edge_id fix caught and applied during review.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
CONSTRUCT_QUERY_RE skipped comments with a bare \#[^\n]*, whose trailing *
backtracks. For '# CONSTRUCT ...\nSELECT ...' the engine gave back everything
after the '#', so the CONSTRUCT inside the comment satisfied the query-form
keyword and a SELECT/ASK was reported as a CONSTRUCT.
All four SPARQL backends delegate to this regex, so such a query took the
CONSTRUCT branch of execute_sparql, which sends Accept: text/turtle and parses
the body as Turtle — failing with a misleading 'Failed to parse CONSTRUCT
response as Turtle'.
Require a comment to reach a line terminator. Both LF and CR are accepted
because the SPARQL grammar ends a comment at either; matching only LF would
regress CR-terminated comments into false negatives.
Add regression tests covering both directions across all four backends.>
* fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#922)
* docs(changelog): document add_edge dedupe fix
Adds an Unreleased/Fixed entry for #922/#926 so the ContextGraph
edge-dedupe bug and its fix are recorded per Keep a Changelog format.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes
Two Explorer API endpoints fail on every install.
/api/enrich/extract imported extract_entities and extract_relations from
semantic_extract.methods, where neither name is defined — that module ships
only the per-strategy variants (extract_entities_ml, extract_relations_regex,
...), and nothing re-exports a plain facade. The resulting ImportError was
caught and reported as "semantic_extract module not available. Ensure spacy
and transformers are installed.", so a wiring bug looked like a missing
dependency. The route now calls NamedEntityRecognizer and RelationExtractor
directly, the classes the README documents, and feeds the extracted entities
into relation extraction rather than re-deriving them. The 503 branch stays
for a genuinely absent module.
Every /api/decisions* route returned 500 once the graph held a decision:
record_decision() stores timestamp as datetime.now().timestamp(), a float,
while DecisionResponse types the field as str, so pydantic rejected the value
the library itself wrote. A before-mode field validator on DecisionResponse
normalizes float, int and datetime inputs to ISO-8601, covering every route
that builds the model instead of only the list endpoint.
The existing tests missed both: test_extract accepted 503 as a pass, and the
decision fixtures are hand-built nodes carrying no timestamp at all. Both are
tightened, and a TestRecordedDecisions class exercises the routes against
decisions created through record_decision().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(semantic_extract): cache spaCy models instead of loading one per call
extract_entities_ml(), extract_relations_similarity() and
extract_relations_dependency() called spacy.load() on every invocation, so the
model was re-read from disk and re-initialized per call. On a short sentence
that is ~120 ms of loading around ~2 ms of work, and successive calls never got
cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the
pipeline ner_extract step and POST /api/enrich/extract, and process_batch()
multiplies it by the number of documents.
The module already had a cached loader for one code path — get_nlp_model() and
its _nlp_cache global — but the extraction functions bypassed it.
Adds load_spacy_model(), a process-level cache keyed by model name behind a
lock so concurrent callers do not each start a load, and routes the five call
sites through it. Errors are left uncached and propagate unchanged, so the
existing OSError fallbacks to pattern extraction still fire. get_nlp_model()
keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for
similarity work, so its model is not interchangeable with the NER one.
Cache entries record the spacy module object they came from. Several tests
patch methods.spacy with a mock and assert on load calls; without that guard a
name-keyed cache would hand a previous test's mock to a later one.
Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm:
extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(explorer): harden extraction and timestamp handling
* fix(explorer): catch OverflowError/OSError in decision timestamp validator
DecisionResponse._normalize_timestamp only guarded against NaN/inf via
math.isfinite(), but datetime.fromtimestamp() raises OverflowError or
OSError for finite epoch values outside the platform's representable
range (e.g. milliseconds stored where seconds were expected). Those
exceptions escaped the pydantic validator unhandled, reintroducing an
unhandled 500 on /api/decisions* for exactly the bug class this PR
closes. Also exclude bool from the numeric branch, since bool is an
int subclass and was being silently coerced to epoch 0/1.
* docs: add changelog entry for PR #886 (explorer extract/decisions fixes)
Documents the extraction 503, decisions timestamp 500, and folded-in
spaCy caching fixes, plus the review-round hardening from Sameer6305
and the timestamp overflow/bool fix from this follow-up commit.
---------
Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
The pin was 5595ccaf..., but upstream has since moved the v4 tag to
ff2f1c62.... The Verify Action Pins workflow flags this drift on every
PR that touches any workflow file, regardless of whether that PR
changed codeql.yml or defender-for-devops.yml.
Verified the new SHA against the GitHub API directly (not just the CI
error text) and confirmed .github/scripts/verify-action-pins.sh passes
clean locally (40/40 action references OK, exit 0).
* ci: pin Python dependencies in requirements-ci.txt for reproducible CI
Adds a committed lockfile pinning all transitive dependencies at exact
versions (uv pip compile, Python 3.11, all extras — 1581 lines), the
Python equivalent of explorer/package-lock.json + npm ci.
- CI installs from requirements-ci.txt before building the wheel
- CI verifies the lockfile is byte-identical to a fresh compile (fails
on staleness after pyproject.toml changes)
- CONTRIBUTING documents the regeneration command
Closes#938
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: address Qodo review — security scans use pinned deps, exclude gpu extras
- security-scan.yml installs from requirements-ci.txt instead of
"./[llm-litellm]" so Safety scans the exact CI/release dependency tree
- security.yml runs pip-audit -r requirements-ci.txt for the same parity
- lockfile regenerated with --extra all (the cross-platform set) instead
of --all-extras, which pulled faiss-gpu/cupy from the Linux-only gpu
extra and co-installed faiss-cpu + faiss-gpu in CI
- uv pinned to 0.12.1 (the version that generated the lockfile) in CI and
CONTRIBUTING so regeneration is deterministic
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: make lockfile staleness check immune to upstream releases
The previous check re-resolved pyproject.toml without constraints, so any
upstream package release (e.g. boto3 1.43.69 -> 1.43.70) failed CI even
when nothing in the repo changed — exactly the time-dependent drift Qodo
flagged. The check now re-resolves with requirements-ci.txt as a
constraint and compares only version lines, so it detects intentional
pyproject.toml changes but ignores upstream releases. CONTRIBUTING
updated to match.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: fix security workflows — install pip-audit; order tooling after pinned deps
Security workflow: the pip-audit install step was lost in the rebase
conflict merge — pip-audit was invoked but never installed (exit 127).
Security-scan workflow: installing safety first let the pinned
requirements-ci.txt overwrite its transitive deps (rich), breaking the
safety CLI at runtime (RuntimeError: Type not yet supported). Tooling is
now installed AFTER the pinned set.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): address review — hashes, build isolation, release builds, docs (4/4)
ZohaibHassan16's review flagged 4 supply-chain gaps; all addressed:
1. **Release builds now use the lockfile**: release.yml installs
requirements-ci.txt and runs `python -m build --no-isolation` so the
sdist/wheel is built against the exact tested dependency set.
2. **Build isolation pinned**: [build-system].requires is now
setuptools==84.0.0 + wheel==0.48.0 (exact pins, no ranges).
3. **Hashes**: requirements-ci.txt regenerated with --generate-hashes
(5,708 sha256 hashes, verified against PyPI). Staleness check updated
to strip the `\` line continuations hashes introduce.
4. **CONTRIBUTING.md documents the separate environment**: hashes,
never-install-into-dev note, build-system pins, --no-isolation release
builds.
Validated: stale-check diff clean, hash spot-check matches PyPI.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): apply --no-isolation to CI build + align benchmark to Python 3.11
Follow-up to ZohaibHassan16's second review round:
1. ci.yml was still running `python -m build` with build isolation
(unpinned setuptools/wheel from PyPI) — now `python -m build
--no-isolation` against the pinned deps, matching release.yml.
2. benchmark.yml was on Python 3.12 while the lockfile is compiled for
3.11 — aligned to 3.11 so every workflow runs the same environment.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): install pinned wheel before --no-isolation build
python -m build --no-isolation failed with 'Missing dependencies:
wheel==0.48.0' because wheel is build-time only — uv's lockfile
excludes it, so installing requirements-ci.txt alone left the build
env without it. Both ci.yml and release.yml now install wheel==0.48.0
(the same pin [build-system] declares) before building. Validated
locally: wheel builds clean with --no-isolation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(context): honor explicit causal edges in decision tracing
trace_decision_causality() inferred causes purely from shared NER entities
plus timestamp ordering, so relationships recorded through
add_causal_relationship() never affected the trace. When entity extraction
returned nothing, trace_decision_chain() came back empty even though an
explicit CAUSED edge was stored in the graph.
Traverse the explicit CAUSED/INFLUENCED/PRECEDENT_FOR edges first, since
they are the ground truth the caller recorded, and keep the entity and
timestamp inference as an additive fallback for pairs with no explicit
link. Edges whose source has no decision record (for example a graph
restored via from_dict) are skipped so a stale edge cannot abort the trace.
analyze_decision_influence() now reports explicitly linked decisions as
direct influence rather than surfacing them only as indirect, and no
longer lists the same decision under both direct and indirect.
Closes#975
* fix(context): address review feedback on causal edge tracing
Follow-up to the explicit causal edge fix, covering the issues raised in
review.
A stored edge weight of 0.0 was coerced to the 1.0 default by a truthiness
check, inflating confidence_decay in the causal chain report. add_edge() is
public and can create causal edges with any weight, so use an explicit None
check instead.
Explicit causes were collected into a dict keyed by source_id, so multiple
causal edges between the same pair of decisions overwrote each other and
only the last was traced. Collect every edge instead, keeping a separate set
of source ids for the entity fallback exclusion.
Cycle detection used a single traversal-wide visited set, so a decision
reached through one branch became unreachable through another and branching
graphs silently lost valid chains. Detect cycles per path instead; max_depth
still bounds the traversal.
Build a reverse index of causal edges once per call rather than scanning the
edge list at every visited node, and use edge_type_index in the influence
analysis. The three causal edge types are now a shared constant.
Adds regression tests for zero weights, parallel edges, branching graphs and
cycle termination.
* fix(context): bound causal trace and report truncation
Per-path cycle detection keeps branching graphs correct but makes the
traversal combinatorial in max_depth: on a densely connected graph the
number of distinct causal paths grows by roughly the branching factor per
level, so a raised max_depth could return hundreds of thousands of chain
reports and take seconds of CPU.
Add a max_chains bound, defaulting to 10000. Rather than dropping chains
silently, which is the exact failure this fix set out to eliminate, the
traversal stops at the bound and appends a {"truncated": True, ...} marker
so callers can always tell the trace is incomplete. A warning is logged with
the same detail. Pass max_chains=None for the previous unbounded behaviour.
Graphs that fit within the bound are unaffected.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): lock the repo host DNS resolve cache against concurrent mutation
_REPO_HOST_RESOLVE_CACHE is a module level OrderedDict shared by every
RepoIngestor instance and thread. _resolve_repo_host_ips and
_prune_repo_host_resolve_cache read, wrote, and iterated it with no lock,
so concurrent ingest_repository() calls (e.g. from a thread pool) could
mutate the dict while another thread was iterating it during pruning.
This reliably raised RuntimeError: OrderedDict mutated during iteration
under ordinary concurrent usage, not just adversarial input.
Reproduced with 32 threads hammering _resolve_repo_host_ips with a low
TTL and small cache cap so pruning and eviction happen on nearly every
call; the crash showed up within the first few hundred iterations on
every run before the fix and did not reproduce at all after it.
Fix adds a threading.Lock guarding every read, write, and prune of the
cache. The blocking socket.getaddrinfo call stays outside the lock so a
slow DNS lookup for one host cannot stall cache access for other hosts.
Added a regression test, TestRepoHostResolveCacheThreadSafety, that
drives 32 threads through _resolve_repo_host_ips with a short TTL and
small cache cap and asserts no exception is raised.
Full test suite: 4088 passed, 332 failed, 140 errors both before and
after this change (same counts on main), all from missing optional
dependencies in this local environment (snowflake, sqlite-vec, spaCy
models, faiss/torch version mismatches), not from this fix. The ingest
and SSRF focused test files pass cleanly: 106 passed, 0 failed.
* test(ingest): fail fast on the first hung thread in the resolve-cache race test
join(timeout=30) alone doesn't fail the test if a worker hangs -- it
just returns after the timeout with the thread still running, and the
test falls through to the errors check, which trivially passes since
a hung thread never got far enough to append one. A future deadlock
could slip past this test looking green.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s instead. Worker threads are daemon=True so a genuine hang can't
also block the test process from exiting.
Verified the assertion is load-bearing, not cosmetic: temporarily
injected an artificial 9999s sleep into the first worker in a
throwaway copy of the test and confirmed the test now fails in ~31s
with a clear message, instead of the ~16 minutes a mass hang would
otherwise cost. That copy was never committed.
Addresses the review comment on #979 from ZohaibHassan16 and Qodo's
automated review.
* test(ingest): fail fast on the first hung thread, for real this time
The previous commit (f94e3b38) claimed to check is_alive() right after
each individual join, but a git staging mistake meant it actually
committed the old batched version instead (checking all 32 threads
only after the whole join loop finished) -- ZohaibHassan16 caught this
by timing it directly, 5 hanging threads took ~5x longer than 1
hanging thread, which the per-thread version would not do.
This commit was built by resetting to the current branch tip, verifying
byte-for-byte against a separately saved copy of the intended fix, and
confirming the actual committed git object (not just `git diff`) has
the inline check before pushing anything.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s regardless of how many threads hang.
---------
* fix(explorer): show a retryable error when the graph fails to load
The dependency-pre-bundle overlay had no failure path: on a fetch
error it kept rendering the last progress frame forever with no
retry. Route isError/error out of the load query, surface a real
error card with the underlying message, and let retry re-fetch
without a full page reload.
* fix(explorer): reflect real backend connectivity on the landing page
The status dot and 'System Online' text were static, so a dead
backend still looked healthy. Track checking/online/offline explicitly
and drive both off the same state so they can't disagree.
* feat(explorer): let search results be dismissed, round relevance scores
The results strip had no close affordance and stayed pinned until the
next search. Add a header row with a dismiss button, and round scores
to whole numbers instead of showing three decimals of a raw relevance
value nobody can act on.
* feat(explorer): add typeahead suggestions to graph search
Typing in the search box now debounces a query against the existing
search endpoint and shows a combobox dropdown, with arrow-key
navigation, Enter/click to jump straight to a node, and Escape to
dismiss. Previously nothing happened until the full form was
submitted.
* fix(explorer): abort stale typeahead requests and clear suggestions on error
Clearing the search box while a suggestion fetch was in flight never
aborted it, so a late response could reopen the dropdown with results
for a query that was no longer typed. A non-OK response also left
whatever suggestions were already on screen untouched instead of
clearing them. Abort on every effect cleanup (not just unmount) and
clear suggestions on any non-abort failure.
* docs(changelog): add entry for Explorer backend failure states fix
Documents the (#980, closes#977) fix in the Unreleased/Fixed section.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* test(semantic_extract): skip openai-dependent tests when the SDK is absent, assert logs not stdout
* test(semantic_extract): pass logger name to assertLogs to match suite convention
All 11 existing assertLogs call sites in the suite pass a logger name
string rather than a Logger instance; tests/reasoning/test_reasoner.py
uses this exact .logger.name form. Behaviour is unchanged.
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
* feat(seed): add allow_private_ips opt-in for trusted internal API sources (Closes#943)
SeedDataManager.load_from_api now delegates to the shared SSRF guard
(semantica/ingest/ssrf.py, added in #906) instead of raw requests.get,
gaining redirect validation and bounded DNS resolution for free.
New config option allow_private_ips (parsed via the shared parse_bool
helper) lets trusted internal deployments load from private APIs while
the secure default (block private/loopback/link-local) is unchanged.
Tests updated to mock request_with_ssrf_guard; new tests cover the
block-by-default behavior and the opt-in flag reaching the guard.
19/19 green in test_seed_manager.py, 25/25 across both seed suites.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ssrf): strip sensitive headers on cross-host redirects (Qodo finding)
request_with_ssrf_guard reused the caller's headers on every redirect hop,
so an Authorization bearer token from load_from_api could leak to a
different redirect target host. Now strips Authorization and
Proxy-Authorization when the redirect origin (netloc) changes, while
keeping them for same-host hops (matching requests semantics).
2 new tests: cross-host redirect drops the credential; same-host keeps it.
37/37 green in test_ssrf_protection.py. load_from_api docstring now also
documents cloud-metadata blocking and per-hop redirect validation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ssrf): strip credentials on https->http downgrade redirects (review feedback)
_should_strip_auth now mirrors requests' should_strip_auth semantics:
strip on hostname change, port change, or scheme downgrade; keep the
credential only for the safe http->https upgrade on default ports.
Previously only netloc was compared, so an https->http redirect on the
same host replayed the Authorization header in cleartext.
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
* security: apply SSRF guard to feed ingestion requests
FeedIngestor and FeedMonitor fetched feed and website URLs with plain
requests.get/head calls, bypassing the SSRF validation already used by
web_ingestor.py and api_ingestor.py. This allowed feed URLs pointing at
loopback, link-local, or other private network addresses to be fetched
directly.
Route all outbound requests in feed_ingestor.py through
request_with_ssrf_guard, gated by the same allow_private_ips config
option the other ingestors expose.
* test: mock the correct request boundary in test_discover_feeds_empty
The test still patched requests.get after discover_feeds() moved to
request_with_ssrf_guard(), which calls requests.request and performs
real DNS resolution. That left the test hitting live network/DNS.
* docs(changelog): document FeedIngestor SSRF guard fix (#928, closes#927)
Records the SSRF guard applied to all 5 feed-ingestion request sites,
the Qodo-flagged test-mock fix, independent PoC verification, and the
carried-over exception-swallowing behavior in discover_feeds().
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(vector_store): make VectorManager methods work on persistent backends (#855)
maintain_store() and collect_statistics() reached into VectorStore
internals (.vectors/.metadata), which only exist for the inmemory
backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus,
...) crashed with AttributeError.
Add a public backend-agnostic VectorStore.count() accessor following
the get_vector()/get_metadata() precedent (#843) and the
NotImplementedError-on-unsupported-capability precedent of
_filter_by_metadata() (#848): inmemory counts its dict, persistent
backends delegate to count() when available, and raise
NotImplementedError otherwise. VectorManager methods now go through
count(); maintain_store() keeps the exact inmemory semantics (separate
vector/metadata dict counts) and reports a 1:1 count for persistent
backends, where metadata is stored alongside each vector.
Tests: 10 hermetic unit tests covering inmemory, delegation and the
NotImplementedError path. Core vector_store suite: 40 passed.
* fix(vector_store): raise NotImplementedError when count() unavailable
Address Qodo review findings on #914:
- Persistent backend with no wrapped store no longer silently returns 0
(which masked a missing initialization as an empty, healthy store);
it now raises NotImplementedError like get_vector()/get_metadata().
- A mis-shaped adapter exposing a non-callable 'count' attribute now
surfaces a clean NotImplementedError instead of a TypeError, via a
getattr + callable() capability check.
Adds regression tests for both cases.
* fix(vector_store): implement count() on FAISS/SQLite/PgVector backends (#914)
- FAISSStore.count(): returns len(index.vector_ids); 0 when no index exists yet
- SQLiteVecStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- PgVectorStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- VectorStore.count(): fix misleading NotImplementedError message; now describes
how to add count() support to a backend adapter rather than claiming only the
inmemory backend can ever support counting
- VectorManager.maintain_store(): split inmemory and persistent paths:
* inmemory: independently reads len(vectors) and len(metadata) and compares
them as an integrity check (original semantics preserved)
* persistent: calls store.count(); returns metadata_count=None because
metadata is co-located with vectors in the backend and cannot be counted
independently; never manufactures metadata_count=vector_count as a vacuous
tautology (#914 Qodo review)
- Tests: rewrite test_vector_manager_persistent.py with 31 tests covering
dispatch logic, inmemory divergence detection, persistent metadata_count=None
invariant, FAISSStore/PgVectorStore via mocks, and SQLiteVecStore via real
in-memory SQLite (skipped when sqlite-vec absent)
* docs(changelog): document VectorManager persistent-backend count fix (#914, closes#855)
Records the VectorStore.count() accessor, the FAISS/SQLite/PgVector
implementations added during review, and the maintain_store()
metadata_count fix (no longer fabricates equality for persistent backends).
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
* fix(ingest): harden RepoIngestor against GitPython URL and option injection
Bump GitPython to >=3.1.58, allowlist clone kwargs, and validate repo URLs
before clone_from to close env-var exfiltration and option-injection paths.
* fix(ingest): accept scp-like SSH remotes in RepoIngestor URL validation
* fix(ingest): resolve repo hostnames to block SSRF via private IPs
* fix(ingest): map malformed repo URL parse errors to ValidationError
* fix(ingest): bound and prune repo host resolve cache
Cap the repository host DNS cache, prune expired entries on access, and evict the oldest entries so long-running processes cannot accumulate unbounded host lookups from user-supplied repo URLs.
* fix(ingest): cap host resolve cache and tighten env-var token checks
Bound the repo host DNS cache with pruning and oldest-entry eviction, and narrow URL env-var blocking to actual $VAR/${VAR} tokens so literal dollar signs are not rejected.
* fix(ingest): preserve repo path compatibility and NAT64 support
* docs(changelog): document RepoIngestor GitPython hardening (#905, closes#868)
Records the clone-surface hardening (GitPython floor, clone-option
allowlist, URL/SSRF validation), the two fixes made during review
(NAT64 false-positive, local-path regression), and a known residual
gap: the SSRF host check doesn't classify RFC 6598 CGNAT space
(100.64.0.0/10) as blocked since ipaddress.is_private doesn't cover it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@wifi-10-43-175-99.wifi.berkeley.edu>
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(context): clarify get_node_property not-found contract (#877)
Add default= param to get_node_property and get_node_attributes so
callers can distinguish node-missing from property-missing using a
sentinel. Fix add_node_attribute calling mutation_callback outside
the lock. Tests added for all cases.
* fix(context): address Qodo review findings (#877)
* fix(context): wrap add_node_attribute mutation_callback in try/except (#877)
The PR claimed to move the callback back inside `with self._lock`, but
the diff only dropped a stray blank line -- the call stayed outside the
lock, unchanged. That's actually correct: self._lock is an RLock, and
_add_internal_node/_add_internal_edge deliberately release the lock
before invoking the callback too, so a slow/misbehaving callback never
holds up other threads. The real gap was that, unlike those two
siblings, this call site didn't catch exceptions from the callback.
Wrapped it the same way, with a regression test.
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(export): log DistanceExporter metric computation failures instead of swallowing them
The four private metric helpers in DistanceExporter (_betweenness,
_hop_distance, _weighted_distance, _semantic_similarity) each catch a bare
Exception and return None/{} with no signal. That makes an exported None
indistinguishable from a legitimate "no path exists" result, corrupting
downstream CSV/JSONL/DataFrame exports with no way to tell a real gap from a
swallowed error.
Log each caught exception at warning level with the offending source/target
before returning the existing sentinel. The exported row shape and values are
unchanged; only the observability of the failure changes.
Fixes#874
* fix(export): route DistanceExporter warnings through the semantica logger tree
get_logger(__name__) doubled the semantica. prefix (__name__ is already
semantica.export.distance_exporter), so the warnings this PR adds landed on
semantica.semantica.export.distance_exporter, a branch setup_logging() never
configures and does not reach the app's log handler. Also reworded the three
except-Exception log messages: they said "recording as no path", which
overclaims what a generic exception means.
Addresses review feedback from @KaifAhmad1 on #879.
* docs(changelog): add DistanceExporter logging fix entry
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract
_extract_from_text() defaulted ner_method, relation_method and
triplet_method to "llm" and ran relation extraction unconditionally,
contradicting the build() docstring ("ml"/"pattern"/False) and the
standalone extractor defaults. Any raw-text build() therefore required a
provider, an API key, and network access without saying so.
Defaults are now ml/pattern/pattern with extract_relations=False. LLM
extraction is unchanged and now opt-in via explicit kwargs.
Also documents relation_method and extract_triplets, which the docstring
never listed, and drops the stale "Default to LLM methods as per
requirement" comment.
Closes#930
* perf(kg): reuse extractors across texts instead of rebuilding per source
Addresses review feedback on #941. NERExtractor.__init__ loads its spaCy
model eagerly when the method includes "ml", so switching the default
from "llm" to "ml" made _extract_from_text() reload the model once per
source in a multi-document build.
Extractors are now cached per (kind, method) on the builder. Adds tests
asserting single construction across repeated texts, that distinct
methods still get distinct extractors, and that the default path runs
end to end without any provider call.
* fix(kg): keep fallback method lists working with the extractor cache
The extractor cache keyed directly on `method`, but all three extractors
accept a list for fallback ordering (e.g. ner_method=["pattern", "ml"]),
so a list argument raised TypeError: unhashable type: 'list' before
extraction started. Lists are now converted to tuples for the cache key
only; the extractor still receives the original value.
Also seeds _extraction_stats in __init__. It was previously created only
in build(), so calling _extract_from_text() directly — as the report's
repro does — raised an AttributeError that the broad except swallowed and
logged as "Entity extraction failed".
Adds coverage for list methods on all three extractors, cache reuse for
equal lists, and distinct entries for different orderings.
* fix(kg): forward extracted relations into triplet extraction
_extract_from_text() passed only entities= to extract_triplets(), so
TripletExtractor re-derived relations itself whenever relations is None,
using a method taken from triplet_method rather than relation_method.
That duplicated work and could yield triplets inconsistent with the
relations already extracted.
relations is now initialized to None, holds the extracted list when
extract_relations=True succeeds, and is forwarded to extract_triplets().
When extraction is disabled or fails, None is passed and
TripletExtractor's existing self-derivation is unchanged.
Folded in at maintainer request rather than tracked as #944.
* docs(changelog): note that #878 documented the LLM defaults before this landed
#878 merged while this was in review and resolved the same code/docstring
mismatch in the opposite direction. Records that #930's decision makes
the code the side that changes, and that #878's docstring formatting is
retained.
* docs(kg): document GraphBuilder public methods
* test(kg): skip module-level doctest to fix suite run
* docs(kg): restore GraphBuilder option documentation
* docs(kg): document default values for build() extraction options
extract_relations, extract_triplets, ner_method, relation_method, and
triplet_method all have concrete defaults in _extract_from_text(), but
the build() docstring only stated a default for extract, inconsistent
with CONTRIBUTING.md's docstring convention of noting parameter
defaults.
* docs: add changelog entry for GraphBuilder docstrings (#878, #876)
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(security): sanitize node_id in Content-Disposition to prevent header injection (CWE-113)
* fix(security): cap link prediction at 10k nodes with semaphore to prevent OOM DoS (CWE-770)
* fix(security): sanitize imported node IDs to prevent stored header injection chain (CWE-20)
* test(security): add self-contained PoC runner with real measured output
* test(security): add regression tests for header injection, DoS cap, import sanitization
* fix(security): comprehensive fix for header injection, DoS, and import ID sanitization
* fix: move semaphore to wrap entire data-load+scoring region, use node-specific edge queries (Qodo #2, #3)
* fix: sanitize edge source/target IDs to match sanitized node IDs (Qodo #4)
* fix: scope 999_999 check to predict_links function via AST (Qodo #1)
* fix: add explicit None guard to _sanitize_import_node_id
* fix(security): close import-sanitizer bypass, enforce link-prediction cap before the expensive scan
Follow-up to the fixes in this PR, found in review:
- export_import.py's "properties" in raw_node fast path stored the id
verbatim, completely skipping _sanitize_import_node_id() -- a node
payload of {"id": "<crlf>", "properties": {}} (the shape this app's
own /api/export produces) bypassed the VULN-3 fix entirely. That
branch now sanitizes id before storing.
- The link-prediction 10k-node cap checked `total` only after calling
session.get_nodes()/get_edges(), which normalize the graph's entire
matching set before applying `limit` -- so the DoS guard ran after
the expensive work it exists to prevent had already happened, on
every request regardless of graph size. Added
GraphSession.get_raw_counts(), an O(1) check against the raw
len(graph.nodes)/len(graph.edges), and moved the size check ahead of
the normalizing calls (also added an edge-count cap).
- 5 of the existing regression tests asserted that literal words like
"Set-Cookie"/"Content-Type" disappear from the sanitized value -- the
sanitizer strips \r\n\x00"\ , not letters, so those assertions failed
against this PR's own fix as submitted. Corrected to assert on the
actual security property (no \r/\n survives), and added end-to-end
tests that exercise the real /api/import -> /api/provenance/report
route chain so the properties-key bypass has regression coverage.
Full explorer suite: 241 passed. tests/test_security_regression_pr2.py: 30 passed.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(context): take the lock in ContextGraph.to_dict()
to_dict() iterated self.nodes.values() and self.edges without holding
self._lock, so a concurrent writer raised "RuntimeError: dictionary changed
size during iteration". It was the only reader on the class that did not take
the lock -- stats(), density(), find_nodes(), find_edges(), get_neighbors(),
get_nodes_by_label(), state_at() and save_to_file() all hold it.
Commit 1d1ae398 introduced the RLock and added 26 "with self._lock:" blocks;
to_dict already existed and was not among them. save_to_file is safe only
incidentally -- it holds the lock and builds its payload inline rather than
delegating to to_dict, so it never reaches the unguarded loops.
Beyond the RuntimeError, the unguarded body could also return a torn snapshot:
the statistics block reads len(self.nodes)/len(self.edges) after building the
node and edge lists, so a write landing in between yields counts that
contradict the payload they describe.
self._lock is an RLock, so this composes with the callers that already hold it
(build_from_conversation and build_from_documents both return self.to_dict()
from inside a locked block). Neither external caller -- agent_context's
_capture_checkpoint_state nor triplet_store's knowledge-graph conversion --
defines a lock of its own, so there is no ordering inversion.
Add tests/context/test_context_graph_thread_safety.py: a deterministic check
that to_dict() blocks while another thread holds _lock (no race window
needed), a reentrancy check, and three checks under concurrent writes covering
the RuntimeError, statistics/payload agreement, and duplicate node ids. Four
of the five fail against the unfixed method.
Closes#923
* test(context): make to_dict lock tests deterministic and hang-proof
Wait for the worker thread to actually start before asserting to_dict()
blocks on _lock, and run the reentrancy check in a joined worker so a
non-reentrant lock fails the test instead of hanging CI.
* test(context): assert worker threads actually stopped after timed joins
A join(timeout=...) on a daemon thread returns even if the thread is
still running, so a deadlock would leak a live thread into subsequent
tests instead of failing. Assert not is_alive() after each timed join.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* security(deps): bump fastapi floor to >=0.109.1 (PYSEC-2024-38)
The [explorer] extra declared fastapi>=0.100.0, which allows the
vulnerable 0.109.0 (PYSEC-2024-38, HTTP response splitting). Raise the
floor to 0.109.1, the patched release. One-line change, no functional
impact -- the 0.109.x API is stable and backward-compatible.
Fixes#869
* fix(deps): bump fastapi to >=0.109.2 and python-multipart to >=0.0.7 for PYSEC-2024-38
PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p) is a ReDoS in
python-multipart < 0.0.7: an attacker sends a crafted Content-Type header
that causes catastrophic backtracking in the multipart regex, stalling the
event loop and causing a DoS on any endpoint that parses form data.
The original PR bumped fastapi to >=0.109.1, but that version pins
starlette<0.36.0,>=0.35.0 and cannot install starlette 0.36.2+ (which
contains the fix via python-multipart>=0.0.7). FastAPI 0.109.2 is the
first version that pins starlette>=0.36.3 (verified against PyPI metadata).
Two changes are necessary:
1. fastapi>=0.109.1 -> fastapi>=0.109.2: ensures starlette>=0.36.3 is
installed as a transitive dependency, which in turn pulls the fixed
python-multipart>=0.0.7.
2. python-multipart>=0.0.6 -> python-multipart>=0.0.7: closes the direct
dependency path. python-multipart is listed explicitly in the explorer
extra, so without this floor a resolver could still install 0.0.6 and
leave the vulnerability present even with the fastapi bump.
The fix targets only the 'explorer' optional dependency group, which is
the only code surface where FastAPI and form-data parsing are used.
No functional API changes between 0.109.1 and 0.109.2; 239 Explorer tests
pass without modification.
* ci(security): gate pip-audit on explorer-extra dependency PRs, add changelog entry for PYSEC-2024-38
The Security workflow's pip-audit job ran weekly against a bare Python
env with none of Semantica's optional extras installed, and always
continue-on-error'd -- it would never have flagged the vulnerable
fastapi/python-multipart floors this PR fixes, or the first attempt at
the fix that left python-multipart>=0.0.6 in place. security-scan.yml's
Safety check has the same blind spot (only installs [llm-litellm]).
pip-audit now also runs on pull_request when pyproject.toml changes,
installs semantica[all] so it can actually see extras like [explorer],
and fails the build on findings for that trigger. Scheduled/dispatch
runs stay non-blocking pending a full pass over the [all] tree.
Also documents the fix (#871, closes#869) in CHANGELOG.md, including
the correction made during review after the original fastapi-only bump
turned out not to close the vulnerability.
* fix(deps): raise setuptools floor to >=83.0.0 (CVE-2026-59890), harden audit env
The new pull_request pip-audit gate (previous commit) caught this on its
first run: pip install -e ".[all]" resolved setuptools==79.0.1, vulnerable
to CVE-2026-59890 / GHSA-h35f-9h28-mq5c / PYSEC-2026-3447 (Unicode
normalization lets a MANIFEST.in exclude/prune pattern be bypassed on
macOS APFS/HFS+, leaking excluded files into a built sdist). Fixed in
setuptools 83.0.0.
[build-system] requires had the same too-permissive floor this whole PR
is about (setuptools>=61.0). Raised to >=83.0.0. Also upgrade pip/
setuptools explicitly in the Security workflow before running pip-audit,
since [build-system] requires only governs isolated build environments,
not the ambient one actions/setup-python provisions and pip-audit scans.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
semantica/mcp_server/__init__.py was fixed to stop hardcoding 0.4.0,
but the separate top-level mcp/ package (run via `python -m
mcp.server`, documented in mcp/__init__.py as a supported way to
configure Claude Desktop/Windsurf/etc. from a source checkout) still
hardcoded 0.4.0 in three places: mcp/__init__.py, mcp/server.py, and
mcp/resources/registry.py.
Reuses semantica.__version__ directly, matching the pattern just
adopted in semantica/mcp_server/__init__.py, so both implementations
stay in sync with the package version going forward.
The previous implementation used importlib.metadata.version('semantica') as
the primary version source with a PackageNotFoundError fallback to
semantica.__version__. This caused two of the three new regression tests to
fail in editable/development installs, where dist-info (egg-info) is written
at install time and is not automatically updated on subsequent version bumps.
In this repo, pyproject.toml declares version as a static field (not dynamic),
and semantica/__init__.py maintains __version__ in sync with it by convention.
semantica.__version__ is therefore the authoritative source of truth and is
always present whenever semantica.mcp_server is importable -- the importlib
.metadata indirection adds no value and can return a stale value.
Changes:
- semantica/mcp_server/__init__.py: replace the importlib.metadata try/except
block with a direct 'from semantica import __version__ as _SEMANTICA_VERSION'
- tests/test_mcp_server_version.py: rewrite tests to assert both MCP version
surfaces (SERVER_INFO['version'] and semantica://schema/info) against
semantica.__version__ as the single ground truth; add 0.4.0 regression
canaries and a cross-surface consistency assertion; remove the mirrored
importlib.metadata resolution that masked the staleness problem
The root-level mcp/ directory (a separate unpublished companion implementation
not included in the built package) is intentionally left unchanged -- it is
outside the scope of issue #863 which targets the semantica-mcp entry point.
- pinecone_store: call self.index.describe_index_stats() instead of the
nonexistent self.describe_index_stats(), and use a unit query vector
instead of an all-zero vector so filter_by_metadata() works on
cosine-metric indexes (the library's own default)
- pgvector_store: apply the existing lowercase true/false bool handling
to the list-filter branch too, and use the jsonb ?| operator so
list-valued metadata fields match on intersection instead of being
compared as a single JSON-text blob
- sqlite_vec_store: use json_each() with a json_type guard so list-valued
metadata fields match on intersection, mirroring the in-memory
backend's set-intersection semantics
- faiss_store: filter_by_metadata(limit=0) now returns [] instead of one
result
- milvus_store: reject NaN/Infinity filter values up front with a clear
ValidationError instead of building an invalid expression that gets
silently swallowed
- update the #848 FAISS NotImplementedError test to reflect that FAISS
now implements real filter_by_metadata() (this PR's whole point)
- add regression tests for each fix; sqlite tests run against the real
sqlite-vec extension
* fix(pipeline): resolve broken import and missing run() in PipelineWithProvenance
Fix two bugs in pipeline_provenance.py:
1. Wrong import path: `from .pipeline import Pipeline` fails because
`semantica/pipeline/pipeline.py` does not exist. Pipeline lives in
`pipeline_builder.py`. Fixed to `from .pipeline_builder import Pipeline`.
2. Pipeline dataclass has no run() method. PipelineWithProvenance.run()
now delegates to ExecutionEngine.execute_pipeline(), which is the
intended execution path for built pipelines.
Additional changes:
- Constructor now accepts a built Pipeline instance (breaking the previous
unusable API that tried to instantiate a dataclass with **config).
- Replace deprecated datetime.utcnow() with datetime.now(timezone.utc).
- Add test suite covering import, instantiation, execution, attribute
delegation, and provenance graceful degradation.
Fixes#858
* test: address Qodo review findings
- Remove redundant test_import_succeeds (module-level import already
guards against import regression at collection time).
- Fix test_provenance_disabled_when_import_fails to deterministically
simulate ImportError via sys.modules patch and assert provenance is
actually toggled off (runner.provenance is False).
* fix(pipeline): update provenance callers for Pipeline API
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Russell Jurney <russell.jurney@gmail.com>
CORSMiddleware doesn't cover WebSocket handshakes at all (Starlette's CORS
support only wraps HTTP), so under SEMANTICA_ALLOW_ANONYMOUS=true --
the mode docker-compose.dev.yml ships -- is_valid_api_key's anonymous
bypass accepted a /ws/graph-updates connection from any origin. Loopback
binding isn't a boundary against a browser: any page the operator has
open can still reach ws://localhost:8000/ws/graph-updates directly, and
ConnectionManager.broadcast sends every graph_mutation to every
connected socket with no per-connection scoping. Combined with
/api/import accepting multipart/form-data (a CORS-safelisted content
type that skips preflight), a hostile page could write to the graph
over REST and read the result back over the unauthenticated WebSocket
-- demonstrated end-to-end in the report with a real client.
Not affected: any deployment with SEMANTICA_API_KEY configured -- the
handshake already rejects without a valid key in that mode. This is an
anonymous-mode-only, development-configuration exposure.
Fix: check the handshake's Origin header against
app.state.explorer_settings['allowed_origins'], the same list
CORSMiddleware already enforces for HTTP, before the key check. A
missing Origin (native/CLI clients, which never set the header --
only browsers do) is still allowed through, since the browser is the
only threat this closes.
4 new tests in test_explorer_auth.py: hostile Origin rejected under
anonymous mode; hostile Origin rejected even with a correct key
(Origin is checked before the key, so a leaked key alone can't
hijack the socket); an allowlisted Origin still connects under
anonymous mode; a missing Origin still connects under anonymous mode
(native clients keep working). Full explorer suite: 226 passed.
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Qodo's re-review confirmed the multi-IP fallback fix but kept the proxy
finding open: logging-and-falling-back when a proxy applies still let
the DNS-pinning protection be silently skipped under proxy
configuration, rather than enforcing a clear policy either way.
Implemented Qodo's preferred option: proxies are now disabled outright
for this SSRF-sensitive fetcher via session.trust_env = False, so
HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars are never consulted in the
first place (a configured proxy would perform its own DNS resolution
of the target host outside this process's control, reopening the
DNS check-then-use race pinning exists to close). The adapter also
keeps a fail-closed backstop: if a proxy is somehow still configured
despite trust_env=False (e.g. set explicitly by future code), it now
raises a clear 502 instead of silently connecting through the proxy
unpinned.
_validate_fetch_url's destination classification (blocking private/
internal targets) is unaffected either way — it runs before any of
this and doesn't depend on proxy configuration.
4 new tests: trust_env is disabled on every pinned session; an
HTTP_PROXY env var pointed at an address that would fail if contacted
is confirmed genuinely unused (real local-server fetch still succeeds
directly); and the fail-closed backstop actually raises when a proxy
is forced onto the session. Full explorer + triplet_store suite: 572
passed.
Four findings from PR #916's automated review, all addressed:
- CodeQL (HIGH): the test HTTPS server's SSLContext allowed TLSv1/TLSv1.1
by not setting a minimum version. Added
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2.
- github-code-quality: unused `cryptography` local in
_make_self_signed_cert — importorskip's return value was never used.
- Qodo (reliability): _validate_fetch_url() only returned the first
validated IP, and _make_pinned_session() pinned to just that one
address, so a fetch would fail outright if the first-returned A/AAAA
record happened to be unreachable even though a later one would work.
_validate_fetch_url() now returns every validated IP (deduplicated,
in resolution order); _make_pinned_session() takes the full list and
falls back through each one via a custom Connection._new_conn
override, matching the fallback behavior a normal DNS-resolving
connection would already get for free. Verified with a real test:
pin to an unreachable loopback address followed by a real one, confirm
the fetch still succeeds by falling back; and a real test confirming
it still raises (rather than silently re-resolving the hostname) when
every pinned address is unreachable.
- Qodo (security): when an HTTP(S) proxy applies, the adapter falls back
to the unpinned path rather than pinning. This is a real, but
architecturally unavoidable, limitation from the client side: for a
forward proxy, the *proxy* performs its own DNS resolution of the
target host on the application's behalf, a resolution this process
has no visibility into or control over — there's no client-side pin
that closes that race. _validate_fetch_url's destination
classification still fully applies either way; only the secondary
DNS-pinning hardening doesn't extend through a proxy. Added an info
log when this fallback path is taken so it's observable rather than
silent, and expanded the code comment to make the reasoning explicit
for the next reader/reviewer rather than looking like an oversight.
Tests: 3 new tests in test_ontology_dns_pinning.py (multi-IP fallback
success, all-unreachable failure, deduplicated multi-record resolution).
Full explorer + triplet_store suite: 569 passed.
Two follow-up hardening items flagged as secondary/deferred during
GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236's fixes:
1. DNS check-then-use (TOCTOU) window in the ontology URL fetcher.
_validate_fetch_url() resolved and validated a hostname once, but
_fetch_url_sync() then let requests resolve the same hostname again
independently at connect time — a low-TTL or rebinding DNS answer
could differ between the two lookups, reopening the SSRF window the
validation exists to close.
_validate_fetch_url() now returns the validated IP, and a new
_make_pinned_session() builds a per-hop requests.Session whose
connection pool is pinned directly to that IP (bypassing DNS
resolution for the connection entirely), while explicitly restoring
the real hostname as the outgoing HTTP Host header and, for HTTPS,
the TLS SNI server_hostname/assert_hostname — so the connection
reaches the validated IP but still presents (and is verified
against) the real hostname's identity, keeping virtual hosting and
certificate validation correct.
Note: an earlier version of this fix set `_dns_host` post-construction
assuming it was decoupled from `host`, matching some other urllib3
releases; in the installed version (2.7.0), `host` is a property
that reads/writes `_dns_host` directly, so that approach silently
changed the Host header too. Verified with a real (non-mocked) local
HTTP server, a real local HTTPS server with a self-signed cert
(proving SNI/cert-hostname verification checks the real hostname,
not the pinned IP), and a negative control confirming a hostname/cert
mismatch is still correctly rejected — not silently bypassed.
2. Pre-wrapped object IRIs skipped full validation in
_format_object_for_sparql/_format_object_for_ntriples (Blazegraph,
RDF4J). A triplet object already wrapped in `<...>` only had its
inner content checked for a literal space or `>`, not run through
sparql_escaping.validate_uri() like the unwrapped-object branch —
flagged by automated review during GHSA-8vgg-8mr4-r236's fix. Both
branches now validate identically.
Tests: tests/explorer/test_ontology_dns_pinning.py (6 tests, including
2 real local-server end-to-end checks and 2 real-TLS checks with a
generated self-signed cert, gracefully skipped if `cryptography` isn't
installed); updated tests/explorer/test_ontology_ssrf.py for the new
per-hop session construction; 4 new tests in
tests/triplet_store/test_sparql_injection.py for the object-IRI fix.
Full explorer + triplet_store suite: 566 passed.
Two follow-up fixes to the initial ReDoS patch (CodeQL py/polynomial-redos
#1897), raised during code review:
--- Fix 1: _PREFIX_DECL regression — inline prologues and CRLF (#review-1) ---
The first ReDoS fix replaced the ambiguous trailing \s* with [ \t]*(?:\n|$),
but that introduced a behavioral regression:
* Inline prologues — PREFIX ex: <...> SELECT ... on a single line were no
longer stripped because the mandatory (?:\n|$) anchor never matched when
non-whitespace content followed the IRI on the same line.
* CRLF line endings — PREFIX ex: <...>\r\n failed because \r is not in
[ \t]* and the anchor expected a bare \n.
Root cause: the end-of-line anchor was unnecessary; the only thing needed
to eliminate backtracking ambiguity is ensuring the IRI body character class
and the trailing whitespace quantifier are disjoint.
Fix: change the IRI body from <[^>]*> to <[^>\r\n]*>, which:
- excludes CR and LF from the IRI match (semantically correct — SPARQL
IRIs cannot span line boundaries)
- makes [^>\r\n]* and the trailing [ \t]* have zero character overlap,
eliminating all backtracking ambiguity without any end-of-line anchor
No anchor is used, so both inline prologues and CRLF/LF endings work
naturally. ReDoS payloads (base< + !< x 10,000) still complete in <1 ms.
--- Fix 2: oversized-query length guard obscured error (#review-2) ---
The initial patch placed the _SPARQL_MAX_QUERY_LEN guard inside
_is_read_only_query(), which caused execute_sparql() to return the same
generic 'Only SELECT' error for both genuinely disallowed query types and
oversized inputs. Clients could not distinguish the two rejection reasons.
Fix: move the length check out of _is_read_only_query() and into
execute_sparql() as an explicit early gate, alongside the other resource
limits (_SPARQL_MAX_ROWS, _SPARQL_MAX_GRAPH_NODES). Oversized queries now
return a specific message naming the limit, the received length, and the
remediation step. _is_read_only_query() is documented to be length-agnostic.
_SPARQL_MAX_QUERY_LEN is relocated to the resource-limits block with the
other constants.
--- Tests added ---
tests/test_security_regression.py:
- test_inline_prefix_before_select_allowed (Fix 1 regression)
- test_crlf_line_endings_with_prefix (Fix 1 regression)
- test_crlf_multiple_prefixes_then_select (Fix 1 regression)
- test_inline_prefix_before_insert_still_blocked (Fix 1 security check)
- test_long_valid_query_not_rejected_by_is_read_only (Fix 2 separation)
tests/explorer/test_sparql_route.py:
- test_oversized_query_returns_distinct_length_error (Fix 2 error message)
- test_oversized_query_never_touches_the_graph (Fix 2 short-circuit)
- test_query_exactly_at_length_limit_is_accepted (Fix 2 boundary)
All 82 tests pass.
The _PREFIX_DECL pattern used \s* as a trailing quantifier after
<[^>]*>. On inputs that start with ase< but contain no closing >
(e.g. ase<!<<!<<!<...), the regex engine explores exponentially many
ways to split the match between [^>]* and \s*, causing polynomial
backtracking against user-controlled SPARQL query input.
Fix:
- Replace ^\s* / \s+ / \s* with ^[ \t]* / [ \t]+ / [ \t]*
so the leading/internal whitespace quantifiers only match horizontal
whitespace (no overlap with the <[^>]*> IRI part).
- Replace the ambiguous trailing \s* with [ \t]*(?:\n|$), which
matches only horizontal whitespace followed by a hard line boundary.
[^>]* and [ \t]* have disjoint character sets, eliminating the
backtracking ambiguity entirely.
- Add _SPARQL_MAX_QUERY_LEN = 10_000 guard at the top of
_is_read_only_query as defence-in-depth: rejects oversized input
before any regex work, bounding worst-case cost even if a future
pattern change reintroduces ambiguity.
Verified: ReDoS payload ase< + !< x 5000 completes in <1 ms.
Normal PREFIX/BASE stripping and read-only query detection unchanged.
Fixes: CodeQL py/polynomial-redos alert #1897
CWE: CWE-1333, CWE-730, CWE-400
* security: sanitize Cypher labels/relationship types/property keys (GHSA-482h-hw99-h62p)
Node labels and property keys passed to create_node/create_relationship
were interpolated directly into Cypher strings in the Neptune, Neo4j, and
FalkorDB graph stores. Property values are parameterized, but labels and
keys can't be bound as parameters, and nothing validated them, so a
document-derived entity type or property name could close the current
Cypher token early and append arbitrary statements (e.g. DETACH DELETE),
running with the application's database credentials.
- New shared semantica/graph_store/query_sanitize.py: sanitize_identifier()
generalizes age_store.py's existing _sanitize_label/_sanitize_rel_type
(the only backend that already validated this) into a helper the other
backends can import without an import cycle with graph_store.py/methods.py.
- Applied at every label/relationship-type/property-key interpolation site
in amazon_neptune.py, neo4j_store.py, falkordb_store.py, graph_store.py
(degree_centrality's own query builder), and methods.py
(update_relationship's own query builder) — create_node, create_nodes,
create_relationship, get_nodes, get_relationships, get_neighbors,
shortest_path, update_node, create_index, and all relationship-type
filters.
- depth/max_depth path-length parameters are also cast to int before
interpolation as defense-in-depth (they're already typed int, but
Python doesn't enforce that at runtime).
Added tests/graph_store/test_cypher_injection.py (12 tests covering the
sanitizer directly and reproducing the advisory's injection payload
against Neptune/Neo4j/FalkorDB create_node/create_relationship — asserts
the malicious query is never built or sent), plus regression tests for
graph_store.py's degree_centrality and methods.py's update_relationship.
Full graph_store test suite (224 tests) passes with no regressions.
* fix(graph-store): prevent depth-based Cypher injection
* test(graph-store): tighten injection regression assertions
* docs(changelog): add PR #910 (GHSA-482h Cypher injection) entry
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Two issues in the last round of commits:
1. explorer/auth.py added a new, opt-in APIKeyAuthMiddleware
(EXPLORER_API_KEY) and wired it into create_app(), but in doing so
removed the Depends(require_auth) dependency from every router and
deleted the /ws/graph-updates handshake check entirely. The new
middleware also fails OPEN (allows all requests) when its key is
unset, the opposite of require_auth's fail-closed design. Since
GHSA-j4mq-hprp-987v (the unauthenticated-Explorer-API advisory) is
already merged into main via require_auth, this would have reverted
a merged Critical fix the moment this branch merges. Removed
explorer/auth.py, restored the per-router dependencies and the
WebSocket auth check. Kept auth.py's one genuine improvement (adding
X-API-Key to the CORS allow_headers list) by folding it into the
existing CORS middleware config.
2. sparql.py's new _is_read_only_query() hardening (comment/PREFIX
stripping + forbidden-keyword scan) used `#[^\n]*` to strip SPARQL
comments, but a bare '#' also appears inside standard RDF namespace
IRIs (e.g. ".../1999/02/22-rdf-syntax-ns#") — the regex struck
everything after that '#' as a "comment", corrupting the query and
rejecting any legitimate SELECT using rdf:/rdfs:-style PREFIX
declarations. Confirmed by the fact the new hardening's own inlined
test copy failed against two of its own cases. Fixed by only
treating '#' as a comment-start at line-start or after whitespace,
which distinguishes ".../ns#" (preceded by a word character) from an
actual comment (preceded by whitespace/newline in every realistic
case, including the attacker's own comment-hiding PoC). Also fixed
the companion PREFIX/BASE regex, which required a prefix-name token
between the keyword and the IRI even for bare `BASE <...>`
declarations (which have none).
tests/test_security_regression.py's SPARQL section now imports the real
_is_read_only_query instead of maintaining a parallel inlined copy that
had silently drifted from — and shared the same bug as — the real
implementation; removed its TestAPIKeyAuth class (tested the now-deleted
auth.py) since equivalent, more thorough coverage already exists in
tests/explorer/test_explorer_auth.py. Updated tests/explorer/test_sparql_route.py's
multi-statement-injection test to reflect that the keyword scan now
catches "SELECT ... ; DROP ALL" itself rather than relying on rdflib's
parser, and added a new test confirming the parser still catches
multi-statement syntax that doesn't contain any forbidden keyword.
Full explorer/vector_store/security-regression/age_store suite: 543
passed (the only failures are 6 pre-existing, unrelated Pinecone-client
mocking issues).
The previous rework of the redirect loop closed the response on each
redirect hop but dropped the try/finally around the success path, so the
terminal response (the one actually read and returned) was left
unclosed, leaking the connection back to the pool unclosed under load.
Triplet.subject and Triplet.predicate (and, in some builders, .object)
were interpolated directly into SPARQL update/query strings in the
Blazegraph and RDF4J stores, and into a SELECT filter in the Jena store.
A subject containing '>' closes the '<...>' IRI token early, so the rest
of the value is parsed as more SPARQL. Entity names are document text in
the normal ingest pipeline, so anyone whose content gets processed could
append operations like CLEAR ALL, running with the application's store
credentials.
Applied the existing sparql_escaping.validate_uri (already used by
anzo_store.py, the one backend that was already hardened) at every
subject/predicate/object interpolation site:
- blazegraph_store.py: _build_insert_data, _triplets_to_rdf (unreachable
dead code today but same fix applied for consistency/future-proofing),
bulk_load's graph option, get_triplets's filter, delete_triplet.
- rdf4j_store.py: _triplets_to_ntriples, get_triplets's filter,
delete_triplet. (add_triplets's graph option was already validated.)
- jena_store.py: get_triplets's filter — the only vulnerable site;
add_triplets/delete_triplet already use rdflib's native Python API
(Graph.add/.remove with URIRef) rather than building query strings, so
they were never exploitable this way.
Added tests/triplet_store/test_sparql_injection.py (12 tests) reproducing
the advisory's own injection payload against all three backends' write
and read paths, asserting the malicious query is never built or sent.
Full triplet_store suite (330 tests) passes with no regressions.
Note: while adding read-path test coverage, found that jena_store.py's
get_triplets() WHERE-clause filter syntax is malformed SPARQL (missing a
FILTER()/separator before the equality conditions) — a pre-existing
correctness bug unrelated to this fix, worth a separate follow-up.
* security: require API-key auth on all Explorer API routes (GHSA-j4mq-hprp-987v)
Every Explorer route (bulk import/export, delete, LLM-backed ontology
generation, SPARQL, etc.) was mounted with no authentication, and both
server entrypoints bind 0.0.0.0 by default. Anyone reaching the port got
full read/write/delete on the graph.
- Add require_auth dependency (explorer/dependencies.py): checks
X-API-Key against SEMANTICA_API_KEY, fails closed with 503 if
unconfigured (not silently anonymous), 401 on wrong/missing key.
SEMANTICA_ALLOW_ANONYMOUS=true opts out explicitly for local dev.
- Wire dependencies=[Depends(require_auth)] into all 11 API routers in
both explorer/app.py and server.py. /health, /api/info, static assets,
and the SPA catch-all stay public.
- /ws/graph-updates handshake now checks the same key via header or
?api_key= query param (browsers can't set custom WS headers) before
accepting the connection.
- Default bind changed from 0.0.0.0 to 127.0.0.1 in server.py's main()
and cli.py's `server start`; the CLI warns if a non-loopback host is
passed explicitly without a key configured.
- Startup logging reports the resolved auth mode in both app factories.
- Document/generate SEMANTICA_API_KEY in the deploy recipes that expose
a public endpoint by default: docker-compose, Railway, Fly, Render.
Added tests/explorer/test_explorer_auth.py covering fail-closed default,
wrong/missing/correct key, anonymous opt-in, public-route exemptions, and
the WS handshake. Added tests/explorer/conftest.py defaulting the
pre-existing ~200 explorer tests to SEMANTICA_ALLOW_ANONYMOUS=true so
they keep exercising route logic without needing a key.
* fix CORS
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): add SSRF protection for WebIngestor and RESTIngestor
Block non-http(s) schemes and private/loopback/link-local targets before
outbound requests, with allow_private_ips opt-in for trusted deployments.
* fix(ingest): fail closed on SSRF DNS resolution errors
* fix(ingest): validate SSRF targets on every HTTP redirect hop
* fix(ingest): avoid blocking on SSRF DNS executor shutdown
* fix(ingest): parse allow_private_ips without truthy-string pitfalls
* docs(ingest): clarify robots.txt SSRF/validation comment
---------
Co-authored-by: Pravit Ampapathini
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
before re-validating (previously any relative redirect was rejected
outright), and close every response instead of leaking the connection
across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
error handling so the graph-size cap returns a clean SparqlResponse
error instead of an unhandled 500.
- add regression tests for all three.
* test(deduplication): cover ClusterBuilder, MergeStrategyManager, and batch paths
Add focused coverage for union-find clustering, property merge rules,
merge_duplicates, embedding similarity, and incremental detection (#866).
* test(deduplication): assert unrelated clusters without conditional skip
Make cluster-separation coverage fail closed by using mocked pairs and
unconditional assertions for distinct Apple vs Microsoft cluster IDs.
* test(deduplication): tighten update_clusters attachment assertions
Require the incremental path to place the new near-duplicate in the
same rebuilt cluster instead of accepting a vacuous cluster-count check.
* test(deduplication): strengthen incremental detect_duplicates wrapper checks
Assert real DuplicateCandidate matches, score threshold, and new×existing
routing instead of only checking that the wrapper returns a list.
* test(deduplication): verify metadata provenance behavior
Assert that preserve_provenance writes metadata.provenance fields and add a disabled-path test so regressions do not pass through merge_entities metadata alone.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@users.noreply.github.com>
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.
Also adds regression tests for scores >= 1 and a CHANGELOG entry.
Adds similarity_unavailable marker and warning logs to build_decision_context and explain_decision when a persistent backend (like FAISS) fails to reconstruct a vector. Updates docstrings to explicitly state this degraded-path behavior and guarantees schema stability. Adds regression tests to test vector retrieval failure behavior via caplog and context assertions.
- build_decision_context() and explain_decision(include_paths=True) both
accessed self.vectors directly, which is only initialized for the
inmemory backend, crashing with AttributeError on any persistent
backend (FAISS, Qdrant, Pinecone, etc.). Replaced with self.get_vector()
(#843's backend-agnostic accessor) + an is-not-None check — a verified
1:1 behavioral equivalent for the old 'decision_id in self.vectors'
guard on the inmemory path.
- Found a third, undocumented instance of the same bug during
verification: _filter_by_metadata() also accessed self.metadata/
self.vectors directly. Initial fix silently returned [] for persistent
backends, which was itself a new silent-failure bug (indistinguishable
from a genuine zero-match result). Reconciled to raise
NotImplementedError instead, matching the established precedent from
get_vector()/get_metadata() (#843) for 'backend exists but doesn't
support this operation' — confirmed via full grep of all 7 backend
wrapper classes that none currently implement filter_by_metadata,
so this path was previously dead-code-masked-as-working.
Tests: 14 new tests across two rounds — inmemory behavioral equivalence,
real (non-mocked) FAISS backend regression tests for all three methods,
and explicit coverage proving the NotImplementedError fires with a clear
message rather than the old silent-[] behavior. Full suite: 53 passed,
0 failed, 0 regressions across the 39 pre-existing tests.
self.indexer is only set for backend="inmemory", so save()/load() still
raised AttributeError for persistent backends (faiss, qdrant, etc.) even
after this PR's getattr() guards on self.vectors/self.metadata, since the
unguarded `self.indexer` access happened first. Guard it the same way and
delegate to the backend store's native save_index/load_index (currently
only FAISSStore implements these) so persistent-backend saves actually
persist instead of silently no-oping.
- Removed total=False from SearchResult TypedDict so all fields are strictly required
- Ensured distance: None is returned from backends that don't natively expose distance (Qdrant, Pinecone, SQLite, pgvector, in-memory)
- Standardized search result score to a consistent 0.0 - 1.0 similarity metric scale across all backend adapters
- Relaxed SearchResult id type to Union[str, int] to accommodate native integer IDs from Milvus and Qdrant without casting
- Updated schema verification tests
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
- Added insert_vectors alias to add_vectors for backward compatibility.
- Sanitized vector_id in get_vector and get_metadata to prevent query injection.
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
- Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
- Added metadata storage using JSON.
- Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
- Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
- Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
- VectorStore.get_vector() and get_metadata() were hardcoded to access
self.vectors and self.metadata dicts, which are only initialized for
the inmemory backend, causing AttributeError on all persistent backends
(FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).
Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
self.backend == 'inmemory' (zero behavior change) and delegate to
self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
- FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
NotImplementedError (FAISS has no metadata storage natively).
- QdrantStore: uses client.retrieve() with with_vectors/with_payload.
- PineconeStore: wraps existing fetch_vectors() call.
- MilvusStore: raises NotImplementedError (auto_id=True schema discards
string IDs at insert time, making by-ID lookup impossible in this
wrapper's current schema).
- WeaviateStore: uses collection.query.fetch_object_by_id().
- PgVectorStore: wraps existing get_vectors() SQL method.
- SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
FAISS backends with real (non-mocked) assertions.
All 28 tests pass.
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
Adds an Unreleased/Fixed entry for #841 (closes#840) — QdrantStore
search results were keyed "payload" instead of "metadata", breaking
HybridSearch.filter_by_metadata() for Qdrant results.
Adds an Unreleased/Added entry for #838 (closes#834), including the
follow-up fix that preserves ImportError for a missing pyoxigraph
install instead of masking it as a generic ProcessingError.
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
- Wire the Explorer frontend's node --test suites (test:graph-store,
test:graph-workspace, and the new test:plugin-registry regression
test) into CI. Previously only `npm run build` ran, so none of the
frontend tests -- including this fix's own regression coverage --
executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
transition doesn't leave the dev diagnostics panel stale.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
* fix(vector_store): stop dropping metadata for add_vectors-only backends
VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.
Fixes#832
* fix(vector_store): guard signature introspection in store_vectors
inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.
* docs(changelog): document VectorStore metadata-drop fix (#832, #835)
* test(vector_store): add regression coverage for metadata forwarding
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
- pluginRegistryPredicates.ts: consolidate 8-line JSDoc to 5 lines,
removing redundant detail that restated implementation mechanics
already obvious from the code.
- GraphWorkspace.tsx: shorten the lastScrubberMsRef comment from 5 lines
to 2; trim the handleDiagnosticsChange block comment by removing the
'rather than bailing out' implementation-alternative sentence; tighten
the distanceVisual inline comment.
- pluginRegistry.temporal.test.mjs: replace 17-line file-level JSDoc
with 9 lines focused on the invariant rather than the root-cause
narrative (already covered in pluginRegistryPredicates.ts); remove
two tsx loader implementation-detail comments; tighten two test-level
inline comments.
No logic, types, or test assertions changed. All 42 tests pass.
- Legacy top_k kwarg was read but not removed from options, so it got
forwarded via **options into VectorStore.search_vectors(), colliding
with backends that call search(..., top_k=k, **options) (e.g. sqlite,
pgvector) and raising "got multiple values for keyword argument
'top_k'". Now popped instead of just read.
- VectorStore.search_vectors()'s dispatch only recognized backend
methods named search/search_similar, so HybridSearch's delegation
still hit NotImplementedError for qdrant/milvus/pinecone, which name
their method search_vectors() with a differently-named count
parameter (limit vs k). Added a third dispatch branch that binds the
count positionally so it works regardless of the backend's parameter
name.
- Backend-delegated results defaulted a missing "distance" to the raw
score, silently reusing the local path's cosine-similarity convention
(distance = 1 - score) even for backends using unrelated metrics
(L2, inner product). A missing distance is now left as None instead
of a fabricated, metric-inconsistent value.
Two issues addressed:
1. Plugin-loading useEffect unnecessarily depended on temporalState.
After the #830 fix, no shouldLoad predicate reads temporalState, but
the effect's dep array still included it, causing extra re-runs on
every scrubber update. Removed temporalState from the dep array and
the shouldLoad call site. Made temporalState optional in the
LazyPluginRegistryEntry shouldLoad context type to match.
2. Regression test imported a local copy of shouldLoad instead of the
production predicate. Extracted all three shouldLoad predicates into
pluginRegistryPredicates.ts (pure module, no React/DOM dependencies),
wired GraphWorkspace.tsx to use the imported functions, and updated
the test to import and exercise the real production code via tsx.
Verified: introducing the old broken condition causes the test to fail;
the correct implementation passes all 7 assertions.
HybridSearch.search() directly accessed self.vector_store.vectors, a dict
that VectorStore only creates for backend="inmemory". Every other backend
(faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised
AttributeError. It now delegates to VectorStore.search_vectors() for
non-inmemory backends, applying metadata_filter as a post-filter and
normalizing results to a consistent {id, score, distance, metadata} shape.
Also fixes two related bugs surfaced while testing the backend-delegated
path end to end:
- vector_ids could remain None when explicit vectors/metadata were passed
without vector_ids, crashing downstream indexing.
- query_vector passed as a plain list crashed backend stores (e.g.
FAISSStore.search_similar) that call .ndim on it; now normalized to a
numpy array up front.
And in vector_store.py: VectorStore.store_vectors() silently dropped
metadata for FAISS (and any add_vectors-only backend) because it called
add_vectors(vectors, **options) without forwarding metadata, even though
FAISSStore.add_vectors() accepts it. This blocked HybridSearch's metadata
filtering from working at all against FAISS.
Fixes#833
temporalDiffState.ts belongs to feat/793-temporal-diff-ui and should not
appear in the #830 diff. Remove it from this branch's tracked files.
Add the pluginRegistry.temporal.test.mjs regression test that covers the
shouldLoad fix committed in the main #830 commit (it was never committed).
Add test:plugin-registry script to package.json so the regression test
can be run via npm run test:plugin-registry.
Two independent render loops were causing the Temporal panel to remain
stuck on 'Loading temporal...' in npm run dev:
Loop 1 — diagnostics state churn (GraphWorkspace.tsx):
handleDiagnosticsChange unconditionally called setGraphDiagnosticsState
with a new object on every invocation. buildEffectAvailability (called
inside GraphCanvas's diagnostics useEffect) always returns a new object,
so setGraphDiagnosticsState was called on every effect run, creating a
cycle: setGraphDiagnosticsState graphDiagnosticsState new
diagnosticsSnapshot new pluginContext new handleInteractionStateChange
new GraphCanvas re-renders diagnostics effect fires again.
Fix: before calling setGraphDiagnosticsState, compare the incoming
diagnostics field-by-field against the last accepted snapshot via a ref
(lastDiagnosticsRef). All effectAvailability entries, edgeClasses.updatedAt,
structureLayer.cacheKey/lastDrawAt/enabled, and distanceVisual identity
must differ for a state update to proceed. The ref approach avoids
scheduling a re-render at all, rather than bailing out inside a functional
updater after the render has already been committed.
Loop 2 — scrubberTime churn (GraphWorkspace.tsx + GraphWorkspaceShell.tsx):
TimelinePanel.tsx calls onTimeChange(defaultTime) whenever its useEffect
re-runs. React 18 concurrent mode re-runs effects with structurally-new
Date objects for the same timestamp when speculative renders discard
useMemo caches, causing setScrubberTime to be called repeatedly with a
new Date that has the same millisecond value — triggering temporalState
churn, the diagnostics effect, and eventually the same loop.
Fix: wrap setScrubberTime in an onTimeChange useCallback that compares the
incoming time's millisecond value against the last sent value (via
lastScrubberMsRef). Redundant calls with the same timestamp are dropped
before reaching setScrubberTime. Stable useCallback identity also prevents
TimelinePanel's useEffect from re-firing solely due to prop identity churn.
Both fixes applied to GraphWorkspace.tsx and identically to
GraphWorkspaceShell.tsx which has the same pattern.
Verified:
- npm run dev: 0 'Maximum update depth exceeded' errors
- Temporal panel renders with real data in dev mode
- Effects and Neighbors panels unaffected
- npm run build + preview: identical behavior, 0 errors
- All 42 frontend tests pass (34 graph-workspace, 1 graph-store, 7 plugin-registry)
fixed qodo review
applyDiffHighlight/clearDiffHighlight were writing baseColor only to
graphStore.graph (the store singleton), but Sigma is constructed with
displayGraphRef.current and the nodeReducer reads attributes from that
instance. When the display graph is a derived copy (aggregated,
focused, or grouped view), the store write has no effect on the
currently-rendered frame -- sigma.scheduleRefresh() flushes the
reducer over the display graph, which did not receive the mutation.
Fix: introduce writeBaseColor(context, nodeId, color) which writes to
BOTH the store graph (so the color propagates into the next display
graph rebuild via aggregateDisplayGraph's shallow attribute copy) AND
context.displayGraph (the live Graph instance currently bound to
Sigma, so the change is visible in the current frame immediately).
The dg !== graph guard skips the display-graph write when they happen
to be the same object (non-aggregated full view), avoiding a redundant
double-write in that case.
Original baseColor is still captured from the store graph (the
authoritative source, since aggregateDisplayGraph copies from there),
so restore remains correct across all view modes.
Adds a Compare section to the existing Temporal Context panel
(temporalOverlayPlugin.tsx) that lets a user pick two ISO timestamps
and diff the graph's node set between them via the existing, previously
UI-less GET /api/temporal/diff backend route.
- New temporalDiffState.ts: typed fetch wrapper (fetchTemporalDiff)
matching the route's added_nodes/removed_nodes response shape.
- Diff results recolor affected nodes via baseColor (not
ringColor/haloColor -- traced and confirmed those are only read by
the sigma reducer for hovered/selected/path-state nodes and are
silently discarded for default-state nodes).
- Validates both timestamps are present, parseable, and from < to
before firing a request.
- Distinct idle/loading/error/empty/success states -- an empty diff
(no changes) is rendered as its own state, not as an error.
- Cancels any in-flight request via AbortController on re-submission
and on unmount; restores each highlighted node's original baseColor
(captured before overwrite, not cleared to a fallback default) on
both paths.
- Reuses existing theme tokens (GRAPH_THEME.palette.semantic[2],
ui.control.dangerText) and existing button/input/loading/error
visual patterns already established in this same plugins directory
and in GraphInspectorPanel.tsx, rather than introducing new styling.
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
older database with the new code would break on the first insert/select
since the row width and _row_to_entry's fixed indices grew past the old
schema. Added test_migrates_pre_existing_old_schema_database.
- verify_chain() now also checks that sequence_id is exactly the
predecessor's plus one (no gap, no duplicate), in addition to the existing
previous_checksum comparison. Hardens against the narrow case where
compute_checksum()'s deliberate exclusion of entity_id could let two
distinct rows coincidentally share a checksum, which alone would let a
checksum-only comparison miss a gap. Added
test_verify_chain_detects_tampered_sequence_gap.
- Explorer provenance route: edge ids now include direction
(f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
already included it. The same (src, target) pair can legitimately appear
in both the upstream and downstream chains (cycles/overlap), and without
this the two edges collided on the same id. Added
test_add_chain_edges_ids_distinguish_direction.
- Removed an unused `Any` import in parse_provenance.py.
Part A - high-stakes trust blockers:
- Invalidation tombstones via ProvenanceManager.invalidate() (archive-then-append,
never mutates or deletes) instead of hard delete
- Hash-chained integrity: sequence_id/previous_checksum chain every entry to its
predecessor; new verify_chain() detects wholesale row deletion that a lone
per-row checksum cannot
- Typed Agent (AgentRecord: agent_type/is_automated) and Activity (ActivityRecord:
start/end timing), wired through all 18 *_provenance.py wrappers
- Split parent_entity_id into previous_version_id (correction) vs derived_from_id
(cross-source derivation), additive alongside the legacy combined field
- Downstream/descendant lineage traversal (get_descendants/trace_descendants,
reverse BFS) closing the dead direction="downstream" code path in the
Explorer's provenance route
- Qualified Association+hadRole and Invalidation in export_prov()
- New CLI: provenance invalidate|verify-chain|descendants
Part B - general PROV-O spec completeness:
- Qualified Generation/Usage/Derivation in export_prov()
- wasAssociatedWith, actedOnBehalfOf, wasInformedBy relations
- Bitemporal fields (valid_from/valid_until/revision_type/supersedes) plus
revision_history()/query_recorded_between(), closing the deprecated
kg.ProvenanceTracker's "no direct equivalent yet" migration gaps
- prov:Bundle/hadMember membership via bundle_id
- Configurable base_uri (--base-uri CLI flag), shared by RDFExporter's
NamespaceManager and OWLExporter's default ontology_uri so KG/OWL/PROV
exports co-resolve under one namespace instead of three hardcoded ones
Bugs fixed along the way:
- agent_id was a dead field: no track_* method read it from kwargs
- track_entities_batch silently absorbed typed kwargs into the metadata blob
- compute_checksum() had to exclude entity_id itself: hashing it made
track_entity's versioning-archive relabel permanently orphan any entry
already chained from the pre-relabel checksum, a false-positive "broken
chain" for a legitimate rename
- InMemoryStorage.get_chain_head() ignored the committed head whenever the
current transaction had staged entries, corrupting the next chain link
- several new ProvenanceEntry fields were wired into the dataclass and
export_prov() but not into SQLiteStorage's DDL/INSERT/row-mapping;
InMemoryStorage masked the gap. Added a permanent round-trip regression
test to catch this class of bug for future field additions
Flagged, not fixed (separate pre-existing issues, out of scope for #825):
- pipeline/pipeline_provenance.py imports a nonexistent module and wraps a
Pipeline dataclass with no run() method
- most *_provenance.py wrappers' backing classes are themselves missing or
incomplete (context_manager, deduplicator, normalizer, etc.)
- kg_provenance.py passes entity_type inside metadata={} instead of as a
top-level track_entity() kwarg across most of its call sites
* security: SHA-pin all Actions, harden release pipeline, add pin verification
Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.
- Pin every third-party GitHub Action across all workflows to a full commit
SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
check that confirms via the GitHub API that each pin still matches its
tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
contents: read); add a concurrency group so simultaneous tag pushes can't
race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
pull-requests: write and silently failing; add bounded artifact retention
for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
adopters, including what's enforced and what a fork needs to reconfigure
for itself (environment/branch protection, Trusted Publishing trust).
Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).
* fix: harden verify-action-pins per PR #824 bot review
Addresses real findings from the automated review on #824:
- The script previously only matched uses: lines that already contained a
40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
would never be scanned at all and the check would pass silently. It now
matches every uses: line and hard-fails on any ref that isn't a full
commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
previously only logged a warning and continued; that's now a hard
failure too, since an unverifiable pin is exactly the failure mode this
check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
so an edit to the verifier script itself wouldn't run the check that
verifies it. Added the script path to both trigger filters.
The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.
Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).
* fix: repair broken Safety scan and PR comment formatting
The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:
- Every line in the JS comment builder used \n (escaped backslash-n)
inside template literals, which JS renders as the literal two-character
string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
check_id - hence "undefined: <path>" for every entry.
Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).
While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:
- `safety check --json --output safety-report.json` is invalid in
Safety 3.x: --output now selects a console format (json/text/screen),
not a file path. The command errored on every run (swallowed by
`|| true`), so safety-report.json was never created and the PR comment
always fell back to a generic "scan completed" message. Switched to
`--save-json`, which is the correct flag for writing a JSON report to
disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
package_name.
- The job never installed Semantica's own dependencies before scanning,
so `safety check` (which defaults to scanning the environment) was
auditing the scanner tools' own dependencies, not Semantica's. Added
`pip install -e ".[llm-litellm]"` so the project's actual dependency
tree - including the LiteLLM extra this whole hardening effort is
about - is what gets scanned.
Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.
* fix: remove unused pypdf2 dependency (CVE-2023-36464)
Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.
PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.
* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use
Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").
Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").
Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.
Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).
* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening
Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).
* fix: close two remaining gaps missed by upstream bot-review fixes
verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
by the existing regex, so a SHA-pinned action written with quotes would
silently skip verification. Updated the main ERE to accept an optional
leading/trailing single or double quote around the owner/action@ref
value, and excluded quote chars from the inner character classes so the
ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
guard so the command doesn't fail when no *.yaml files exist.
security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
five scanner-failure modes (file missing, empty file, malformed JSON,
valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
genuine zero-vuln and real-vuln cases still behave correctly.
* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI
The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:
syntax error in conditional expression: unexpected token ')'
at line 27 on every CI run.
Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.
Verified in real bash 5.2.21 (Git for Windows):
No syntax error on the real 40-pin workflow tree (Checked 40)
Unquoted SHA pin: MATCH, correct repo+ref extracted
Double-quoted SHA pin: MATCH, correct repo+ref extracted
Single-quoted SHA pin: MATCH, correct repo+ref extracted
.yaml extension file: MATCH, correct repo+ref extracted
./local-action: NO MATCH (correct)
docker://: NO MATCH (correct)
* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md
The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:
- GitHub secret scanning and push protection (repo settings, not copied
on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
whether the upload-sarif step in codeql.yml does anything)
Added as items 5, 6, 7 matching the existing numbered bullet style.
* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)
verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:
old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
new: d1ba80a13dd99fba24a470575428917156a28b43
Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* fix(agno): surface unevaluable policy rules (#778)
* fix(agno): fixed qodo reviews
check_policy previously let unevaluable policy rules silently return
compliant=True with no signal (issue #778): a rule referencing a field
missing from decision_data, or a rule string not matching the expected
<field> <op> <value> format, both fell through _eval_rule's `return
True` and were treated as passed.
Both now raise ValueError, which routes through check_policy's existing
exception handler and surfaces as a `warnings` entry instead. compliant/
violations semantics are unchanged for every case that previously worked
correctly; an unevaluable rule is not counted as a violation since it's
genuinely unknown whether it would have passed.
Follow-up fixes from code review:
- policy_rules decoded via json.loads without checking it was a list;
a JSON-encoded bare string decoded to a Python str, so iterating it
evaluated one "rule" per character, amplifying a single input-shape
mistake into a wall of per-character warnings. A decoded string is
now treated as a single rule; any other non-list shape or non-string
list element produces exactly one warning instead.
- _eval_rule used `data.get(field) is None` to detect a missing field,
which can't distinguish an absent key from a key present with JSON
null - both produced the same "undefined field" warning. Field
presence is now checked with `field not in data` first, and a
present-but-null value gets its own distinct message.
Added regression tests for all of the above in
tests/integrations/agno/test_decision_kit.py (38 tests in the file,
128 passing across tests/integrations/agno/).
* fix(agno): reject non-object decision_data in check_policy
check_policy only validated that decision_data was well-formed JSON,
not that it decoded to an object. When it decoded to a list, `field
not in data` in _eval_rule silently became list-membership testing
of values instead of a dict key check - e.g. "confidence" not in
["confidence", 0.95] evaluates to False - so a matching rule fell
through to data["confidence"], raising a raw internal TypeError
("list indices must be integers or slices, not str") instead of any
meaningful diagnostic. Numbers, strings, and bools produced similarly
opaque TypeErrors deep inside _eval_rule.
check_policy now checks isinstance(data, dict) right after decoding
and rejects any other shape with a single clear violations entry,
the same way it already rejects malformed JSON.
Added 5 regression tests in tests/integrations/agno/test_decision_kit.py
covering list/number/string/bool/null decision_data shapes (43 tests
in the file, 133 passing across tests/integrations/agno/).
Addresses Copilot PR review comment on the #778 fix branch.
* docs(changelog): reference PR #822 in the check_policy changelog entry
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(provenance): log tracking failures and return None on storage error (closes#783)
* docs(provenance): document Optional return types and failure behavior (#783)
* fixed qodo reviews
- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)
- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True
- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable
- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()
- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Removed unused `validate_skos_hierarchy` import from
test_ontology_subissue3.py (flake8 F401); the test uses a `wraps=` spy
on the real add_nodes_and_edges instead of calling the helper directly.
- refresh_ontology tests now percent-encode the ontology URI with
urllib.parse.quote before interpolating it into the {ontology_uri:path}
request path, matching the already-encoded unknown-uri refresh test in
the same file instead of embedding a raw http://... URI with slashes.
- Reworded the cyclic-SKOS refresh test's comment and section header:
GraphSession.add_nodes_and_edges() documents pre-write validation and
lock-based mutual exclusion, not transactional rollback, so "atomic"
was replaced with "single combined add_nodes_and_edges() call" to avoid
implying rollback guarantees that don't exist.
Verified: tests/explorer/test_ontology_subissue3.py (34 passed) and
tests/explorer/ (204 passed), no regressions.
- validate_skos_hierarchy() re-walked every existing hierarchy edge in
the graph on each write, so one pre-existing cycle anywhere would
block all unrelated future SKOS writes. It now only traverses
concepts touched by the edges actually being written, while still
checking those against existing edges for cross-boundary cycles.
- In /api/ontology/load, `except HTTPException: raise` sat after a
broader `except Exception` clause that already matched HTTPException,
so a 422 raised after a successful OntologyIngestor parse was
silently swallowed and retried via the fallback RDF parser instead of
reaching the caller. Reordered the except clauses.
Co-authored-by: mikemikimike <13286568797@163.com>
- Added exc_info=True to both store failed and record_decision failed warning logs in _AgentScopedStore.upsert_memory() to preserve full traceback context for debugging
- Updated CHANGELOG.md entry to document traceback preservation
- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)
- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True
- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable
- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()
- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False
Signature introspection and the resulting call were sharing one
try/except, so a genuine bug inside a backend's get_causal_chain
(raising an unrelated TypeError) was misread as a signature mismatch,
causing an identical retry call before the real error surfaced.
Split introspection from the call so a successfully-introspected call
happens exactly once; the trial-and-error cascade now only runs when
inspect.signature itself fails. Also adds the CHANGELOG entry for
#781/#817, which was missing.
- Add safe input validation and bounds clamping on max_depth (1..100) to prevent DoS/memory exhaustion
- Use inspect.signature for accurate keyword dispatch with precise TypeError fallback
- Prevent masking of genuine internal TypeError exceptions inside graph backends
- Add security and input hardening regression tests
- Support legacy (depth kwarg) and positional-only get_causal_chain backend signatures in fallback path
- Add regression tests for signature compatibility
- Return explicit error dictionary when graph lacks get_causal_chain instead of silent empty list
- Forward direction and max_depth in fallback graph.get_causal_chain call
- Add regression tests for error signaling and parameter forwarding
* feat(triplet_store): add Altair Anzo triplet store backend
Adds AnzoStore as a fourth peer to BlazegraphStore/RDF4JStore/JenaStore,
speaking plain SPARQL 1.1 over HTTP (no new dependency needed). The one
structural difference from the existing backends is that Anzo addresses
data by a dataset/graphmart URI rather than a short namespace/repository
name, so the endpoint path percent-encodes it. Reuses the shared
sparql_escaping.py helpers and wires "anzo" into TripletStore's backend
dispatch and config env vars.
Closes#813
* fix(triplet_store): correct AnzoStore SPARQL syntax and validate IRIs
Addresses review findings from Qodo and Codex on PR #814:
- get_triplets(): constraints are now expressed via FILTER(...) instead of
bare equality expressions appended inside the WHERE group graph pattern
(e.g. "?s ?p ?o ?s = <...>"), which is not valid SPARQL and was rejected
by standards-compliant endpoints.
- bulk_load(): named-graph inserts now nest the GRAPH block inside the
INSERT DATA braces (INSERT DATA { GRAPH <g> { ... } }) per the SPARQL 1.1
Update grammar, instead of "INSERT DATA GRAPH <g> { ... }".
- bulk_load()/_build_insert_data()/delete_triplet()/get_triplets() now
validate subject/predicate/graph URIs via sparql_escaping.validate_uri
before interpolating them into SPARQL Update/Query strings, closing an
injection path where a value containing ">" or "}" could break out of
the intended <...> token.
- Corrected the store_type docstring/usage example: Anzo's linked-data-set
store type is "lds", not "dataset".
Extended tests/triplet_store/test_anzo_store.py with coverage for the
corrected query shapes and the new validation/injection-rejection paths
(38 tests total, up from 32). Full tests/triplet_store/ suite: 299/299
passing.
* test(triplet_store): expand AnzoStore regression coverage
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(provenance): centralize duplicated store-and-swallow logic into _save_entry() (closes#784)
- Added ProvenanceManager._save_entry(entry) as the single shared
checksum-compute + storage.store() + graceful-failure-swallow
pipeline, previously duplicated identically across track_entity,
track_relationship, track_chunk, and track_property_source.
- track_entities_batch/track_chunks_batch already delegate to
track_entity/track_chunk in a loop, so they inherit the fix for
free — left untouched, confirmed no direct duplication there.
- Byte-for-byte preserves today's swallow-and-continue behavior and
comment text; this is an architecture-only refactor. The silent-
failure behavior itself is unchanged and out of scope here — a fix
to it now only needs to happen in one place instead of four.
- Added 4 new regression tests (previously 0 of the 4 single-item
methods had failure-path coverage) proving storage.store() raising
is still caught and each method still returns its ProvenanceEntry.
Tests: tests/provenance/ 228 passed (+4 new), tests/explorer/test_provenance_manager_wiring.py 8 passed. 236/236, 0 failed.
* fix(provenance): drop out-of-transaction store attempt in track_entity fallback
The _save_entry refactor changed track_entity's pre-build exception
fallback (entry is None branch) to call _save_entry(), which makes a
real self.storage.store(entry) call. The original code only computed
a checksum here and never attempted storage again, since this branch
fires when something already failed before the entry was built inside
the atomic transaction. Storing outside that transaction bypasses the
BEGIN IMMEDIATE serialization #807 added, risking the same race it
fixed. Restored checksum-only behavior and added a regression test
asserting storage.store is not called on this path.
Also removed an untested hasattr(_store_with_conn) defensive branch
added during the refactor that wasn't in the original code, and added
a changelog entry.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Two review findings on #807/#812:
- retrieve() and trace_lineage() were routed through transaction()'s
BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
serialized behind every other read/write, defeating the WAL
concurrency this PR was meant to add. They now use a dedicated
_read_connection() (configured, no explicit BEGIN).
- track_entity()/track_chunk() swallowed all internal storage
exceptions unconditionally, so a single item's failure inside
track_entities_batch()/track_chunks_batch()'s shared transaction
never reached the batch loop's per-item except, inflating
tracked_count for entries that were never persisted. Both now
re-raise when called with a shared _conn (batch context) while
still degrading gracefully on standalone calls.
Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
- Reuse lineage['integrity_verified'] in _build_provenance in O(1) time when available, eliminating redundant SHA-256 verification loops across lineage chains.
- Improve compute_checksum dictionary handling in semantica/provenance/integrity.py so None values fall back cleanly to ProvenanceEntry defaults.
- Move json and verify_checksum imports to module top-level in semantica/provenance/manager.py to avoid function-local import overhead during get_lineage calls.
- Extend ProvenanceNode in semantica/explorer/schemas.py with audit evidence fields: source_document, source_location, source_quote, confidence, and checksum.
- Update _transform_audit_lineage in semantica/explorer/routes/provenance.py to populate these evidence fields for each lineage node from ProvenanceEntry records, while keeping default None values for orphan nodes.
- Include source_document, confidence, and checksum in markdown report rendering (_render_markdown) so exported markdown reports surface attribution and integrity evidence.
- Add unit test test_provenance_audit_evidence_fields_preserved in test_provenance_manager_wiring.py verifying that evidence fields are present across /api/provenance JSON responses and exported JSON/markdown reports.
- Update verify_checksum and compute_checksum in semantica/provenance/integrity.py to support both ProvenanceEntry objects and serialized dictionary entries.
- Add integrity_verified flag computed via verify_checksum to the dictionary returned by ProvenanceManager.get_lineage().
- Update _build_provenance in semantica/explorer/routes/provenance.py to verify every returned lineage entry before labeling the result as source=audit. If verification fails due to missing checksums or corrupted records, log a warning and fall back cleanly to graph traversal.
- Add unit test test_provenance_manager_wiring_checksum_failure_falls_back in test_provenance_manager_wiring.py verifying that tampered lineage entries trigger fallback to source=graph_traversal.
- Fix _transform_audit_lineage to classify all non-downstream ancestor derivation edges as 'upstream' instead of 'lateral', correcting multi-hop lineage direction in JSON and markdown reports.
- Add GraphSession.set_provenance_storage_path() to explicitly reject conflicting preconfigured storage paths or path mutations after provenance_manager initialization.
- Update create_app() to call active_session.set_provenance_storage_path(prov_path), preventing silent retention of conflicting paths or un-redirectable cached managers.
- Remove unused logging import in app.py.
- Add comprehensive unit tests in test_provenance_manager_wiring.py for upstream edge classification, markdown report grouping, conflicting path rejection, and manager initialization lockouts.
- Explorer's /api/provenance now queries the audit-grade ProvenanceManager
(SQLite-backed, checksummed) first, falling back to the naive 2-hop
graph traversal when no audit records exist for a node.
- Fixed a process-global mutable-state risk in the initial approach:
provenance storage path is threaded per-session via GraphSession,
not via ProvenanceManager's global set_default_storage_path classmethod.
- Added source: 'audit' | 'graph_traversal' to the response so callers
can distinguish which path served the data.
- Documented a known limitation: ProvenanceManager currently only
traces upstream/ancestor lineage, not descendants — the naive
fallback remains the only source for downstream relationships until
ProvenanceManager gains a reverse lookup (tracked separately).
- Warns (rather than silently no-ops) if a provided session's
provenance_manager was already constructed before create_app()
applied a provenance_storage_path.
- Never lets a provenance-manager failure crash the route; degrades
to the naive path with a logged warning instead.
Tests: 5 new tests in test_provenance_manager_wiring.py covering the
audit path, empty-record fallback, storage-failure degradation, app
startup wiring, and cross-session storage isolation. Full
tests/explorer/ + tests/provenance/ suite passing, order-invariant.
- README: get_table_lineage() takes table_name first, then catalog/schema
keyword args — the example had them in the wrong order, which would have
queried lineage for the wrong fully-qualified table when copy-pasted.
- modules.md: the ingest example used DatabricksIngestor without importing
it, causing a NameError if copy-pasted as-is.
- guides/ingest.md: corrected the claim that Databricks/Snowflake ingestors
return "the same shape as DBIngestor" — DBIngestor.execute_query() returns
a raw List[Dict] with no wrapper, unlike DatabricksData/SnowflakeData.
Makes enterprise lakehouse/warehouse ingestion (Databricks Unity Catalog +
Delta Lake, Snowflake) a first-class, prominently documented capability
across the README and guides, and adds matching runnable examples to
docs/guides/ingest.md. Also fixes several pre-existing inaccuracies caught
while auditing the ingest module docs against the actual source:
WebIngestor has no ingest_urls() (only singular ingest_url()), XMLIngestor's
XSD option is schema_path (not validate_xsd) and belongs on ingest() not the
constructor, and the "Available ingestors" list was missing DatabricksIngestor
while listing several classes not actually exported from semantica.ingest.
Extracts the row-cap-and-truncate loop (duplicated between the
CONSTRUCT/DESCRIBE and SELECT branches) into a shared _cap_rows()
helper, and adds a test for the previously-uncovered CONSTRUCT/DESCRIBE
truncation path. Addresses review nits on PR #805.
- Revert create_ontology silently falling back to a near-empty ontology on
generation failure; restores the HTTPException(500) behavior from #770/#787
that this PR had accidentally undone (and re-enables TestOntologyCreateFailures)
- Fold sh:Warning/sh:Info severity pySHACL results into the /shacl/validate
response's violations array instead of silently dropping them, so a
non-conforming report is never returned with an empty violations list
- Share a single nodes/edges fetch between _generated_shacl_for_uri and
_data_graph_turtle_for_uri via new _fetch_analysis_graph(), so /health
no longer re-queries and re-truncation-checks the same ontology twice
* fix(security): restrict Neptune cookbook SG, add VPC flow logs, harden IaC scan suppressions
Addresses open GHAS code scanning alerts:
- Neptune cookbook stack (neptune-setup.yaml) no longer opens the Bolt/OpenCypher
port to 0.0.0.0/0; a required ClientCidr parameter must be supplied instead.
Updated 21_Amazon_Neptune_Store.ipynb deploy instructions to match.
- Added VPC Flow Logs (CloudWatch Logs + IAM role) to the same stack.
- Documented why an account-wide IAM password policy resource does not belong
in a disposable per-learner CFN stack, with a justified ts:skip.
- Added inline `checkov:skip` / `ts:skip` comments to the knowledge-explorer
Helm templates (deployment/service/configmap) as a second suppression path
for the CKV_K8S_21/AC_K8S_0086/AC_K8S_0080 false positives, since the prior
annotation-only suppression was not being honored by the scanner.
* docs(changelog): document the Neptune and Helm chart security scan fixes
* fix(security): correct flow-log IAM scope and ClientCidr regex from review
- FlowLogRole granted logs:CreateLogStream/PutLogEvents on the bare log
group ARN, but those actions apply to log streams, not the group itself;
scoped them to "${FlowLogGroup.Arn}:log-stream:*" instead and moved the
Describe* actions (which don't support group/stream-level resource
restriction) to Resource: "*", matching AWS's documented flow-log IAM
policy shape. Without this, flow log delivery could silently fail.
- ClientCidr's AllowedPattern only checked digit count (1-3 digits per
octet), so malformed values like 999.999.999.999/32 passed parameter
validation and would only fail later when CloudFormation tried to
create the security group rule. Tightened the regex to enforce valid
IPv4 octet ranges (0-255) and prefix lengths (0-32).
* fix(security): harden IAM policy in neptune-setup and standardize Helm chart scan suppressions
- neptune-setup.yaml: split FlowLogRole policy into account-level statement (CreateLogGroup, DescribeLogGroups, DescribeLogStreams with Resource: '*') and log-group-scoped statement (CreateLogStream, PutLogEvents with !GetAtt FlowLogGroup.Arn) per AWS VPC Flow Logs least-privilege documentation.
- deployment.yaml: remove unreliable file-header skip comments (# checkov:skip / # ts:skip) and replace with resource-level metadata.annotations (checkov.io/skip and runterrascan.io/skip). Update seccomp rule ID from CKV_K8S_28 to checkov's actual seccomp rule CKV_K8S_31 on both Deployment and pod-template metadata.
- configmap.yaml / service.yaml: remove stale # ts:skip=AC_K8S_0086 file-header comments and add runterrascan.io/skip resource-level metadata annotations for consistency across all chart templates.
- .checkov.yaml: update documentation to explain resource-level metadata.annotations and reference CKV_K8S_31.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
sparql.py handles direct SPARQL query execution against the live graph with no test coverage anywhere in the repo. Adds coverage for the read-only allowlist (the actual security boundary here), row/timeout limits, error handling, and RDF projection fidelity.
- track_entity() no longer aliases a caller-supplied used_entities list
(it stored the reference directly and later mutated it via .append())
- Remove dead fallback branches in orchestrator.py/manager.py that
duplicated what Config.get()'s dotted-path resolution already does
- Add local --dry-run to `provenance audit` for parity with
`provenance export`
- `provenance check --strict` now warns instead of printing a success
checkmark before raising on a failed check
Add an Unreleased/Added entry for #786 covering the new export/import
Markdown format, idempotency and rollback guarantees, and the
Explorer/ContextGraph scoping decision from #765.
- Add entries alias in get_lineage() return dictionary so CLI and programmatic callers can access lineage entries via either key
- Update lineage() wrapper method to fallback to lineage_chain when entries is missing
- Add assertions in test_cli_lineage confirming lineage and entries lists are non-empty
- Add _default_storage_path, set_default_storage_path(), and test-isolation context manager default_storage_path() in ProvenanceManager
- Accept config kwarg in ProvenanceManager.__init__ to fix CLI initialization bug
- Implement audit_log(), lineage(), export_prov(), and check() on ProvenanceManager matching cli.py expectations
- Wire provenance.storage_path in Semantica.__init__ before pipeline stages execute
- Add comprehensive unit tests in tests/provenance/test_manager.py for CLI methods and test isolation
The set-state-in-effect refactor inlined each initial-fetch effect as a
standalone `fetchInitial`, duplicating the logic of the existing
reload/fetchOverview/fetchRegistry/loadVersions callbacks instead of
reusing them (required, since eslint-plugin-react-hooks v7 flags calling
an outside setState-touching function directly from an effect body, even
through an async gap - verified via a local lint probe). The duplicates
dropped the setError/flashMsg calls the originals had, so a failed
initial page load in AlignmentsTab, KGOverviewTab, OntologyManager, and
VersionsTab now failed silently instead of showing an error - a
regression of the exact bug #767/#790 fixed for these same files.
Also fixes LineageDiagram only clearing nodes/edges when the new
activeId was falsy, leaving the previous lineage view's stale diagram
on screen while switching directly between two ids.
The single-line checkov:skip=CKV_K8S_21 comment added in ed44260 was 286
characters, exceeding the repo's yamllint line-length rule (max 120,
.pre-commit-config.yaml). Split into three short comment lines: the skip
directive itself, then the rationale, in service.yaml, deployment.yaml,
and configmap.yaml.
Checkov's helm framework renders the chart without a namespace override,
so metadata.namespace (set to .Release.Namespace, bound only at install
time) always resolves to "default" and trips CKV_K8S_21 on service.yaml,
deployment.yaml, and configmap.yaml even though the chart is
namespace-agnostic by design.
Suppressed via per-file checkov:skip comments, following the same
convention already used for the Cloud Run false positives in
deploy/gcp/cloudrun-service.yaml.
* fix(#768): add ErrorBoundary to workspace Suspense blocks
* fix(#768): ensure ErrorBoundary retryCount only resets on recovery transition
* fix(#768): remove componentDidUpdate auto-reset to avoid premature reset on Suspense fallback
* fix(#768): reset ErrorBoundary retryCount only after a retry settles
Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.
KGOverviewTab dropped the nodes-fetch 207 warning whenever stats also
returned 207; HealthTab and AlignmentsTab still had the exact
silent-swallow pattern this PR set out to fix elsewhere in the same
folder. Also documents all of #790's fixes in the changelog.
* Fix#788: pin httpx<0.28.0 globally to fix TestClient breakage
Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.
Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.
Closes#788
* Add CHANGELOG entry for #788 httpx pin fix
Documents the httpx<0.28.0 global pin from this PR under Unreleased/Fixed.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Ensures network errors, API failures, and 207 Partial Success statuses are surfaced correctly to the user instead of failing silently in the frontend UI.
207 alone is indistinguishable from 200 to callers that only check
response.ok, so /api/analytics now raises 500 when every requested
metric fails and reserves 207 for genuine partial failure. Adds
regression tests for the temporal, analytics, and ontology-create
failure paths introduced in this PR, and logs the fix in the
changelog's Unreleased section.
Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.
Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.
Closes#788
- routes/temporal.py: temporal_patterns raises HTTPException(500) instead of
silently returning an empty-but-valid TemporalPatternResponse on exception
- routes/analytics.py: preserves existing partial-success body shape
(frontend already parses this), but sets response.status_code = 207 when
any individual metric computation fails, so callers get a real signal
instead of an indistinguishable 200
- routes/ontology.py: POST /create now raises HTTPException(500) on
generation failure instead of silently falling back to a partial/minimal
ontology and returning 200 with a misleading nodes_added count
Verified via git stash comparison that pre-existing test suite failures
(58 errors, Starlette TestClient/httpx version mismatch) are unrelated to
this change - identical failure count on modified and unmodified code.
Closes#770
Promotes the Unreleased changelog section (Databricks connector, SQLite
vector store, SPARQL CONSTRUCT templates, JenaStore named-graph support)
to 0.6.0 and syncs version references across pyproject.toml, __init__.py,
and docs.
- TemporalGraphQuery.query_time_range() and RDFExporter.export() both
expect {entities/relationships} (or {relationships} with source_id/
target_id keys), not ContextGraph.to_dict()'s {nodes, edges} shape.
Map the output before passing it in, and add an actual temporally-
bounded edge to the Temporal Intelligence example so the query has
something to find.
- add_causal_relationship() only accepts relationship_type values of
CAUSED, INFLUENCED, or PRECEDENT_FOR; replace the invented "triggers"/
"enables" values used in Decision Intelligence and the audit-trail
recipe, which would otherwise raise ValueError immediately.
PipelineBuilder.add_step() returns the created PipelineStep, not the
builder, so chaining .add_step().add_step() raised AttributeError.
Only connect_steps() and set_parallelism() return the builder and can
be chained.
Consolidates the platform reference into a single, premium README with
collapsible module/recipe sections so the docs and the deep-dive reference
no longer live in two places. Every code example was checked against the
actual semantica/ source and corrected where the API had drifted:
resolve_conflicts, register_source, ValidationResult.valid, clean_data,
execute_pipeline, ParquetExporter/LPGExporter/ReportGenerator calls,
graph.to_dict(), TemporalGraphQuery/TemporalNormalizer usage, the
Reasoner/ExplanationGenerator API, and the REST endpoint paths. Also
removed duplicated titles, snippets, and repeated example scenarios that
had crept in during the merge.
The CodeQL Analyze Python job failed on the #757 merge commit with
ECONNRESET while streaming the CodeQL bundle download in
codeql-action/init's "Setup CodeQL tools" step. This is unrelated to
the merged code — it's a known, currently-unaddressed gap in
codeql-action: the download error is retryable but the action doesn't
retry it internally (confirmed via codeql-action's issue tracker and
changelog).
Since a `uses:` step can't be wrapped by a shell-level retry action,
Initialize CodeQL now runs up to 3 times, cascading to the next
attempt only if the previous one failed, so the common case (success
on attempt 1) costs nothing extra.
Dataset.remove() on a bare 3-tuple resolves context=None internally,
which the underlying store treats as a wildcard and deletes the
matching triple from every graph, not just the default graph the
docstring promises. Pass self.graph.default_graph explicitly as the
context so delete_triplet stays scoped to the default graph, matching
the isolation guarantee default_union=False is meant to provide.
Also corrects a misleading comment: SPARQLStore is graph_aware=True
too, so graph-awareness isn't what requires SPARQLUpdateStore here —
it's SPARQLStore being read-only (.add()/.remove() raise TypeError).
Adds regression tests and a CHANGELOG entry for PR #757.
Updates the tagline, adds Ontology Management/SKOS to the feature pills, swaps
the yellow-highlight subhead for a cleaner italic style, and surfaces a
regulated-domains teaser linking to the existing "Built for High-Stakes
Domains" section.
Trims the README from a full module/API dump into a scannable pitch (hero,
why-Semantica, quick start, architecture, decision intelligence, one flagship
audit-trail recipe) and moves the exhaustive per-module reference, extra
recipes, and full integrations matrix into a new PLATFORM_REFERENCE.md.
1. Endpoint derivation regression: Detect if self.endpoint already contains
a Fuseki service suffix (/query, /update, /sparql) to prevent double-appending
(e.g., /ds/query/query). If it does, derive the base and construct both
paths properly.
2. Misleading serialize warning: Limit the named-graph data loss warning
to single-graph serializer formats (turtle, xml, n3, etc.). Multi-graph
formats (trig, nquads, nt) will correctly serialize all graphs without warning.
3. Zero-added error misdiagnosis: Track malformed triples accurately in
add_triplets(). If every triplet fails the local validation (ValueError/
AttributeError), raise a formatting-oriented ProcessingError instead of
assuming a store connectivity issue.
Includes comprehensive regression tests for all three cases.
Migrates JenaStore from rdflib.Graph to rdflib.Dataset with default_union=False
explicitly set, per maintainer-confirmed architecture for issue #756.
Changes:
- _initialize_graph: construct self.graph as Dataset(default_union=False) for
the in-memory path, and Dataset(store=SPARQLUpdateStore(...), default_union=False)
for the remote path. SPARQLUpdateStore.graph_aware=True satisfies Dataset's
hard requirement. Both paths verified against rdflib source.
- add_triplets: accept and honor graph= option. When supplied, Dataset.graph(uri)
creates/retrieves the named-graph context and the triple is written via a
4-tuple (which SPARQLUpdateStore maps to INSERT DATA { GRAPH <uri> { ... } }).
When graph= is omitted, the 3-tuple path routes to Dataset's default graph,
preserving pre-migration semantics exactly.
- serialize: add WARNING log when named-graph content would be silently dropped
by a single-graph serializer (turtle/xml/n3). Log includes triple count and
recommends trig/nquads formats. No warning when only the default graph is used.
- create_model: document that triplet_count now counts triples across all graphs
(default + named) as a consequence of this migration. Semantics shift made
visible, not silent.
- delete_triplet: document that graph= parity is a known gap, deferred to a
future follow-up per maintainer's stated scope (add_triplets only).
Decisions applied:
1. triplet_count semantics shift: documented in create_model docstring
2. delete_triplet graph= parity: explicitly out of scope, noted in docstring
3. Existing store.graph=Graph() tests: left unchanged; new tests added
to cover the real _initialize_graph path
Tests added (TestJenaStoreDatasetMigration):
- test_initialize_graph_produces_dataset_not_graph
- test_initialize_graph_dataset_has_default_union_false
- test_add_triplets_with_graph_option_writes_to_named_graph
- test_add_triplets_without_graph_option_writes_to_default_graph
- test_add_triplets_named_graph_isolated_from_default_query
- test_serialize_logs_warning_when_named_graph_content_present
- test_serialize_no_warning_when_only_default_graph_used
Also updated test_add_triplets_remote_endpoint_fires_insert_data_via_update_store
to patch Dataset instead of Graph (the remote path now creates Dataset(store=...)).
Full suite: 269 passed, 0 failed (tests/triplet_store/ + tests/pipeline/)
The exception-propagation comment and Raises docstring in
execute_construct_template stated that add_triplets signals failure
exclusively via a returned dict. This became stale after the JenaStore fix
(previous commit) which introduced ProcessingError propagation for complete
batch failures.
Updated to document both paths:
- dict-based failure: success=False in returned dict (BlazegraphStore, RDF4J, etc.)
- raised ProcessingError: JenaStore full-batch failure now raises directly
No logic changed. 262 tests pass.
The remote-endpoint path in _initialize_graph was instantiating the read-only
rdflib SPARQLStore, causing every add_triplets() call against a remote Fuseki
endpoint to silently fail: SPARQLStore.add() raises TypeError which was swallowed
by the broad except Exception per-triplet handler and returned as success=True/added=0.
Changes:
- Import SPARQLUpdateStore alongside SPARQLStore
- _initialize_graph: use SPARQLUpdateStore(query_endpoint=<base>/query,
update_endpoint=<base>/update) per standard Fuseki REST API conventions
- Fix constructor: self.endpoint=config.get('endpoint') always returned None
because the named positional 'endpoint' param captures the kwarg before **config;
now uses endpoint or config.get('endpoint')
- Narrow per-triplet except to (ValueError, AttributeError); add ProcessingError
when entire batch fails to prevent misleading success=True/added=0 return
Tests added (TestJenaStoreRemoteEndpointUsesUpdateStore): 4 new test cases
Full suite: 262 passed (tests/triplet_store/ + tests/pipeline/)
_format_object_for_ntriples decided IRI vs. literal purely from the
presence of datatype/lang metadata, defaulting anything without it to
<obj>. Any plain literal object (e.g. typical NER/extraction output
like "Alice", or an untyped Turtle literal round-tripped through the
new CONSTRUCT path) was wrapped as an invalid IRI instead of a quoted
literal, diverging from BlazegraphStore's _is_uri_value-first check.
Port _is_uri_value from BlazegraphStore so RDF4JStore checks whether
the object is actually URI-shaped before falling back to literal
handling, with a plain-quoted-literal fallback instead of <obj>.
Extends CONSTRUCT support to JenaStore, which uses rdflib.Graph natively rather
than an HTTP protocol - CONSTRUCT results come as native 3-tuples with no
Accept-header/parsing dance needed, unlike Blazegraph/RDF4J.
- CONSTRUCT-aware execute_sparql: reuses shared sparql_escaping.CONSTRUCT_QUERY_RE,
extracts datatype/language from rdflib Literal objects into the same 4-tuple
metadata contract used by Blazegraph/RDF4J
- Non-CONSTRUCT path (SELECT/ASK) confirmed byte-for-byte unchanged (Property 9)
- execute_construct_template confirmed backend-agnostic against JenaStore, zero
changes needed
- Named-graph support explicitly out of scope - JenaStore wraps a single
rdflib.Graph with no named-graph concept; add_triplets continues to silently
ignore graph= exactly as before. Tracked separately as a follow-up issue
requiring a Graph -> ConjunctiveGraph/Dataset migration.
Extends the Blazegraph-only CONSTRUCT support from #322 (commit 4f2c6c82's
approved pattern) to RDF4JStore:
- CONSTRUCT-aware execute_sparql: Accept: text/turtle, rdflib Turtle parsing,
4-tuple (s, p, o, metadata) contract with datatype/language preservation
- Named-graph writes via RDF4J's REST context parameter, N-Triples-encoded
(angle-bracket-wrapped IRI), confirmed against RDF4J's Protocol.java source
- graph=None preserves existing behavior exactly (no context param sent,
not context=null - verified as a distinct, deliberate choice)
- _CONSTRUCT_QUERY_RE moved to sparql_escaping.py as a shared, backend-agnostic
constant; Blazegraph now delegates to it, zero behavioral change confirmed
- execute_construct_template (construct_templates.py) required zero changes -
confirmed backend-agnostic via end-to-end integration tests against RDF4JStore
29 new tests, full suite 245/245 passing. Jena support remains out of scope
for this PR - tracked separately in #754's remaining scope.
Replace remaining Hawksight-AI GitHub org links and the old
semantica-dev noreply email with the current semantica-agi org
and kaif@getsemantica.ai contact, so security/support contacts
match pyproject.toml.
Implements #322: ConstructTemplate/ParameterDescriptor/ConstructTemplateRegistry
with injection-safe {{param}} rendering, Blazegraph CONSTRUCT-aware execute_sparql
extension, execute_construct_template (render->execute->parse->persist), and a
construct_template pipeline step. RDF4J/Jena support deferred to a follow-up issue.
Closes#322
Documents the 9 pre-existing test failures caused by never-implemented
kg.ProvenanceTracker compatibility methods, the deprecation fix, and
the follow-up migration guide addition in this PR.
Every deprecation warning added in this PR (and the class docstring)
points to docs/migration/kg-provenance-tracker.md, but the file was
never added, so the reference was dead. Adds the guide with a
method-mapping table to semantica.provenance.ProvenanceManager.
Add graph-apache-age extra (psycopg2-binary) which was previously
undeclared despite age_store.py depending on it, wire it into
graph-all, and document install commands for FalkorDB/AGE/Neptune
alongside Neo4j. Note that RDF triple stores need no extra since they
talk SPARQL over HTTP via the core `requests` dependency. Also align
README's "Triplet Stores" table label to "Triple Stores (RDF)" to
match the standard term used elsewhere in the docs, while keeping the
TripletStore interface name in backticks.
Semantica already ships both an RDF triplet-store stack (Blazegraph,
Apache Jena, Eclipse RDF4J via a unified TripletStore/SPARQL interface)
and an LPG graph-store stack (Neo4j, FalkorDB, Apache AGE, AWS Neptune
via Cypher), but the README only surfaced the LPG side. Add a hero
highlight line, a "What Semantica gives you" bullet, and split the
Features-at-a-Glance table row so both formats and all backends are
named explicitly.
- get_table_lineage() gains include_column_lineage=True, resolving
per-column upstream/downstream references via Unity Catalog's
column-lineage API (one request per column, opt-in)
- DatabricksConnector.connect() now reuses an already-open connection
instead of opening a second one; ingest_table()/ingest_query() only
close the connection they opened themselves, so using the ingestor
as a context manager no longer leaks the connection opened by
__enter__
- get_table_schema()/get_table_lineage()/list_tables() now validate
both catalog and schema are resolved before calling Unity Catalog,
matching list_tables()'s existing catalog check
- 8 new regression tests (35 total)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
Review follow-up: only append archived_history_id to used_entities when
explicit_parent_supplied is True. Previously it was appended unconditionally,
so the no-explicit-parent re-track path ended up with the same history id in
both parent_entity_id and used_entities, duplicating the reference in
get_lineage() output.
- get_lineage() aggregated metadata by iterating trace_lineage()'s BFS
order and calling dict.update() on each entry, so ancestor metadata
(now reachable via derived_from chains) could overwrite the queried
entity's own metadata on conflicting keys. Reverse the iteration so
the queried entity (always lineage_entries[0]) is applied last and
wins, matching the documented "most recent entry's metadata takes
precedence" intent.
- track_entity()'s derived_from guard only accepted a concrete dict,
silently ignoring other collections.abc.Mapping implementations
(e.g. types.MappingProxyType). Switch the isinstance check to
Mapping so any mapping-like metadata is honored.
Addresses Qodo review findings on PR #741.
track_entity() only auto-linked a parent by looking up `source` as an
existing entity_id, so two entities sharing a real source URL (e.g. a
document and a decision derived from it) never got connected, and
metadata["derived_from"] was stored but never consulted by any linking
or traversal code.
track_entity() now treats metadata["derived_from"] as an explicit
parent link (unless parent_entity_id was already passed directly), so
the existing BFS in trace_lineage() picks it up for free.
Closes#735
Rule is a mutable dataclass, so an already-registered rule's priority
could change after being added; the dedup early-return skipped the
priority re-sort, so re-adding a rule after mutating its priority
left self.rules stale relative to that change. The duplicate branch
now re-sorts before returning, matching the append path.
- add_rule()'s duplicate-skip path now logs at warning level instead
of debug, so a skipped duplicate is visible by default rather than
silent in typical logging configs
- The duplicate-rule log message now stringifies conditions via
map(str, ...) before joining, since Rule.conditions is List[Any]
and non-string entries would otherwise raise TypeError
add_rule() unconditionally appended to self.rules, so re-running the
same setup code on an existing Reasoner instance (e.g. re-executing a
Jupyter cell) duplicated every rule; forward_chain() would then match
the duplicated rules but silently return no new results since the
conclusions were already in self.facts, with no error or warning.
add_rule() now compares an incoming rule's rule_type, conditions, and
conclusion against existing rules and returns the existing Rule
instead of appending a duplicate, keeping repeated add_rule() calls
with the same definition idempotent.
* Add shacl extra to pyproject.toml (fixes#736)
* docs(changelog): add entry for shacl extra fix (#736)
by @Sameer6305
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
'By default, initializing ... with storage_path=...' read as if passing
storage_path were the default, contradicting the very next sentence
about the no-argument in-memory default. Rephrased so the in-memory
default isn't undercut by the first sentence.
* docs: improve SHACL validation guide onboarding and workflow guidance
* docs: fix SHACL validation implementation mismatches
* docs: fix stale violation URIs and drop unused imports in SHACL guide
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
Condense the CLI section to the essential install/usage snippet and
command groups, pointing to docs.getsemantica.ai for the full
reference instead of maintaining static terminal mockups in the README.
Consolidate the hero around a single category claim (Context and
Accountability Layer for AI agents), drop the named-competitor
comparison table (LangChain/LlamaIndex/Mem0/Zep/Palantir Foundry),
remove GitHub alert-box tips/notes, cut redundant module/changelog
sections, strip vanity feature counts, and trim em dashes for a
cleaner, more premium read.
- Add SQLITE_VEC_AVAILABLE flag via importlib.util.find_spec so the test
suite's skipif actually reflects whether sqlite-vec is installed; it was
previously undefined, causing all sqlite vector store tests to be
silently skipped regardless of installation state.
- Actually apply PRAGMA synchronous=NORMAL alongside journal_mode=WAL when
use_wal=True, matching the documented behavior; document use_wal as an
opt-in kwarg in the docstring and usage guide.
- Correct _is_safe_identifier error messages (regex never allowed hyphens).
- Batch get() and update() with IN(...)/executemany instead of per-id
round trips, consistent with add()/delete().
- Fix flaky test_update_vectors assertion that relied on list.index()
over dicts containing numpy arrays.
- Reorder sqlite_vec_store import alphabetically in vector_store/__init__.py.
Co-Authored-By: Luffy2208 <209925020+Luffy2208@users.noreply.github.com>
Revert badge rows to flat-square (for-the-badge rendered as
mismatched oversized blocks), convert the subtitle to a native
blockquote for GitHub's built-in muted-grey text styling, and
trim "The" from the tagline.
Update the hero tagline, subtitle, and comparison table to frame
Semantica as Palantir-grade knowledge/decision intelligence that is
open source, self-hostable, and priced for startups through
Fortune 500, not just enterprise budgets.
* docs: improve conflict resolution guide onboarding and workflow guidance
* docs: fix conflict resolution implementation mismatches
* docs: correct credibility-weighted example output values
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
The merging example told readers to use merge_entity_group() for
already-confirmed duplicate groups, but the code right below it still
called merge_duplicates() on group.entities, which re-runs duplicate
detection redundantly. Update the call to match the stated guidance.
- Correct the unsupported-rule-key pitfall: absent keys fail compliance,
present keys (any value) pass — the previous wording had this backwards.
- Remove required_ltv/pd/lgd/dsti/credit_score: True from the mortgage
example. required_* checks equality against the given value, so True
against a real numeric field silently marks compliant decisions as
non-compliant (verified: a fully passing decision still returned False).
The min_/max_ rules already enforce presence of ltv/dsti/credit_score.
* docs: improve export guide onboarding and workflow guidance
* docs: fix export guide implementation mismatches
* docs: revert .content to .text in export guide examples
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
* docs: improve agent memory guide onboarding and usage guidance
* docs: align Agent Memory guide with persistence implementation
* docs: fix misleading index_path persistence claim across guides
VectorStore's index_path kwarg is silently absorbed into FAISSStore's
**config and never read anywhere in faiss_store.py, so it does not make
the FAISS index persist across restarts as several docs implied. Real
persistence requires an explicit VectorStore.save()/.load() call, or
AgentContext.save()/.load() which cascades to it.
- docs/reference/context.md: rewrite the "Persist your vector store"
tip to explain the actual save()/load() mechanism instead of the
dead index_path kwarg.
- docs/guides/graphrag.md, decision-intelligence.md, ingest.md,
semantic-extraction.md: drop the dead index_path=... kwarg from
VectorStore(backend="faiss", ...) constructor calls.
Follow-up to #691, which fixed the same false claim in
docs/guides/agent-memory.md but missed these other files.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* feat: implement Apache Arrow and Feather file ingestion support (#235)
* fix(arrow): eliminate double full-scan and clean up reader wrapper
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
* docs(changelog): add [Unreleased] entries for Arrow ingestion (#705)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
- Remove duplicate Data Quality Info block; content moved into Common Pitfalls as a dedicated pitfall entry, keeping the critical advanced_analytics=True warning as the sole callout
- Align node count threshold: Common Pitfalls now consistently references 100+ nodes (was '< 50 nodes'), matching the When To Use recommendation
- Fix CVE in SUNBURST example: CVE-2024-3400 → CVE-2020-10148, matching context-graphs.md
- Correct load_from_graph fact format: predicates/args are lowercased (threatactor(apt29), not ThreatActor(APT29)); scoped to DatalogReasoner only; removed incorrect metadata-to-predicate claim
- Move Common Pitfalls section after </Tabs> so it renders outside the tab component and is visible to all readers
- Replace non-existent shortest_path() with get_neighbors() + path_to_anchor
- Remove non-existent extract_subgraph() calls from all domain tab examples
- Clarify automated extraction requires knowledge_graph= constructor arg and list input
- Distinguish save_to_file() (graph only) from AgentContext.save() (graph + FAISS + memory)
- Add resolve_links() step to serialization section for cross-graph link restoration
- Link duplicate entities pitfall to the deduplication guide and its API
- Replace ctx.store() + graph.to_dict() with direct entity/relationship
dict to avoid key mismatch (to_dict() returns nodes/edges; generator
reads entities/relationships)
- Fix prop type filter: 'datatype' → 'data' (value set by PropertyGenerator)
- Fix domain/range printing: both are stored as lists, not scalars
- Clarify Reasoning bullet: OWL inference requires an external reasoner,
Semantica only exports the ontology
- Remove duplicate LLM-vs-graph-generator pitfall already covered by the
Info callout in the LLMOntologyGenerator section
* docs: improve pipeline guide onboarding and workflows
* fix(docs): correct broken pipeline guide examples from review
- Remove Option 1 (register_step_handler + string name): ExecutionEngine
never resolves string handler names via step_registry, so it raised
TypeError at runtime; replace with the single working pattern
- Add missing step_type positional arg to all new add_step() calls
- Use connect_steps() for checkpoint dependency instead of the
dependencies= kwarg, consistent with every other example in the file
- Move extract_entities definition above its call site to fix NameError
- Replace docstring on save_checkpoint with inline comment to match
the no-docstring convention used by all other handlers in the file
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Expand intro to cover Git (dict/code_files) and stream (StreamMessage/.content) return shapes, which the previous two-class split omitted
- Add missing imports and AgentContext setup to the Source 1 internal-docs snippet (NameError on copy-paste)
- Add advanced_analytics=True to ContextGraph in both Business Examples (required for extract_entities=True to populate graph analytics)
- Replace bare `pass` credential with YOUR_DB_PASSWORD placeholder to match the YOUR_*_KEY convention used elsewhere
- Guard nullable description/resolution columns in ticket_texts with `(r[...] or '')` to prevent TypeError on NULL rows
- Replace misleading time.sleep() rate-limit advice with accurate description of RESTIngestor's built-in 429 retry/backoff and how to tune it
Without an explicit namespace in metadata, checkov (CKV_K8S_21) flags
every resource as using the default namespace. Using .Release.Namespace
lets helm install --namespace semantica --create-namespace correctly
scope all resources to the target namespace.
An empty/comment-only YAML file is parsed as NoneType by PyYAML.
Checkov requires a dict; adding skip-check: [] satisfies the parser
without globally suppressing any checks.
- gcp/cloudrun-service.yaml: add comment + README sed one-liner so PROJECT_ID
is substituted before gcloud run services replace (was a literal placeholder
that caused image-pull failure on the declarative deploy path)
- azure/main.parameters.json: replace wildcard allowedOrigins "*" with a
REPLACE_ME placeholder; add README note to set the real URL after first deploy
- kubernetes/networkpolicy.yaml + helm networkpolicy template: add from: selector
(ingress-nginx namespace + same-namespace pods) so ingress is no longer
allow-all; restrict egress to FalkorDB port 6379 and DNS port 53 instead of
the allow-all egress: - {} wildcard
- helm/values.yaml: expose networkPolicy.ingressNamespace and falkordbPort values
- kubernetes/deployment.yaml: add secretRef for knowledge-explorer-secrets so
FALKORDB_PASSWORD is actually injected into the container
- app.py: add _mutation_bridge_installed guard to prevent closure stacking when
the same GraphSession is passed to create_app() more than once; remove
duplicate app.state.allowed_origins assignment (single source of truth is
app.state.explorer_settings); add comment on falkordb_host/port dead config
- tests: update allowed_origins assertions to use explorer_settings dict
- .checkov.yaml: remove global CKV_K8S_21/28/30 suppressions; rely on per-file
inline checkov:skip comments in cloudrun-service.yaml so future real K8s
manifests are not silently exempted
checkov scans deploy/gcp/cloudrun-service.yaml as a Kubernetes resource
because it has apiVersion: serving.knative.dev/v1. It flags CKV_K8S_21
('default namespace should not be used') because Cloud Run services have
no metadata.namespace field — they are project/region scoped, not
namespace scoped. Add CKV_K8S_21 to .checkov.yaml skip-check and to the
inline skip comment in cloudrun-service.yaml.
Root cause of 6 consecutive CI failures:
MSDO 0.215.0's guardian.cmd wrapper breaks the build whenever checkov exits
with code 1. Checkov exits 1 on ANY violation, including MEDIUM/LOW findings
that are all 'below minimum severity'. This makes Active results = 0 and
'Found no breaking results', yet Guardian still raises BreakException because
it treats the tool's exit code as a first-class breaking signal. The
.checkov.yaml soft-fail setting was never read because the MSDO runner
bypasses repository config files.
Fix:
- Remove checkov from the MSDO tools list (stops the guardian.cmd crash)
- Add a dedicated 'checkov' job on ubuntu-latest using the official
bridgecrewio/checkov-action@v12, which runs a current checkov release,
runs on Linux, and correctly reads .checkov.yaml and respects soft_fail
- Set soft_fail: true in the action so low/medium findings appear in the
Security tab without ever blocking the build
- MSDO continues to run eslint, templateanalyzer (Bicep/ARM), and terrascan;
these tools all have well-behaved exit codes and produce no active results
after the security fixes applied earlier in this PR
.checkov.yaml:
- Replace soft-fail: true (was a failed workaround for MSDO) with
skip-check: [CKV_K8S_28, CKV_K8S_30] — correct suppression for the
Knative false-positives (Cloud Run enforces seccomp + AppArmor at
platform level without requiring K8s annotations)
checkov scans deploy/gcp/cloudrun-service.yaml as a Kubernetes resource
(it has apiVersion: serving.knative.dev/v1) and raises CKV_K8S_28 /
CKV_K8S_30. Adding those annotations to spec.template.metadata.annotations
caused checkov to crash (exit 1 with no SARIF output) — likely a bug in
checkov's AppArmor check when it tries to match the annotation container
name against containers in a Knative RevisionSpec. Fix:
- Remove the AppArmor / seccomp annotations from the template metadata
- Add checkov:skip comments at the file top so the false-positive checks
are suppressed cleanly (Cloud Run enforces these at platform level)
Also drop the legacy seccomp.security.alpha.kubernetes.io/pod annotation
from deploy/helm/knowledge-explorer/values.yaml: run #186 confirmed that
the modern podSecurityContext.seccompProfile.type: RuntimeDefault field
already satisfies CKV_K8S_28 for the Helm chart without the annotation.
Adding the annotation alongside the modern field was causing the same
crash in checkov's Helm-rendered output.
checkov crashes (exit 1) on two constructs introduced in earlier commits:
1. deploy/gcp/cloudrun-service.yaml: pod-level spec.template.spec.securityContext
is not part of Knative RevisionSpec. checkov's Knative parser panics on
this unknown field. Remove it — CKV_K8S_28 (seccomp) and CKV_K8S_30
(AppArmor) are already satisfied by the legacy annotations in
spec.template.metadata.annotations; the container-level securityContext
that IS valid in Cloud Run Gen 2 is kept.
2. deploy/azure/main.bicep: 'vnetInternal ? { ... } : null' compiles to
ARM null() which crashes checkov's Bicep/ARM parser. Replace the inline
null ternary with two concrete variable objects (vnetConfigInternal and
vnetConfigExternal) so both branches are well-typed objects.
Active results are 0 and 'Found no breaking results' but MSDO still fails
because checkov exits with code 1 whenever it finds any violation
(including MEDIUM/LOW below the minimum severity threshold). MSDO v1.12.0
treats a non-zero tool exit code as a breaking result even when Guardian
reports no active findings.
soft-fail: true makes checkov exit 0 in all cases. MSDO Guardian still
reads the full SARIF output and would surface any HIGH/CRITICAL findings
as active results that break the build, so the security posture is
unchanged.
AZR-000363 (Azure.ContainerApp.PublicAccess) — line 29 managedEnvironment:
- Add vnetConfiguration.internal: true (default) so the environment uses
an internal load balancer instead of a public IP
- Parameterize with vnetInternal (bool, default true) and
infrastructureSubnetId so operators can provide their subnet on deploy
AZR-000361 (Azure.ContainerApp.ManagedIdentity) — line 40 containerApp:
- Add identity.type = SystemAssigned so the Container App can
authenticate to Azure services without storing credentials
Also update main.parameters.json and README with the new parameters.
The 2 active checkov HIGH results (CKV_K8S_28 + CKV_K8S_30) were coming
from deploy/gcp/cloudrun-service.yaml — checkov scans it as a Kubernetes
resource (apiVersion: serving.knative.dev/v1) and flagged missing AppArmor
and seccomp on that file, regardless of the fixes made to the k8s/ and
helm/ manifests.
deploy/gcp/cloudrun-service.yaml:
- Add container name (explorer) so AppArmor annotation key matches
- Add AppArmor annotation to pod template metadata (CKV_K8S_30)
- Add legacy seccomp annotation (AC_K8S_0080 / CKV_K8S_28)
- Add pod-level seccompProfile: RuntimeDefault (CKV_K8S_28)
- Add container securityContext (runAsNonRoot, allowPrivilegeEscalation)
Cloud Run Gen 2 supports all of these fields
deploy/kubernetes/deployment.yaml:
- Pin image tag from ':latest' to ':0.5.0' (AC_K8S_0068 / AC_K8S_0069)
- Add legacy seccomp pod annotation alongside existing seccompProfile field
deploy/helm/knowledge-explorer/values.yaml:
- Add legacy seccomp annotation to podAnnotations so it renders into
the Helm-generated pod template alongside the modern seccompProfile
checkov HIGH (2 breaking results, CKV_K8S_30):
- Add AppArmor annotation to k8s deployment pod template
(container.apparmor.security.beta.kubernetes.io/explorer: runtime/default)
- Add AppArmor annotation via Helm values.yaml podAnnotations so it
renders into the Helm-generated pod template
Terrascan warnings (AC_K8S_0087 / AC_K8S_0080 / AC_K8S_0073):
- Add runAsNonRoot: true and seccompProfile: RuntimeDefault at container
securityContext level in both k8s deployment and Helm values (these
were only at pod spec level before)
Terrascan AC_K8S_0002 (noHttps):
- Add nginx ssl-redirect annotation to k8s ingress so HTTPS enforcement
is explicit at the ingress controller layer
Terrascan AC_K8S_0013 (noOwnerLabel):
- Add owner label to k8s namespace.yaml
Terrascan AC_K8S_0068 (imageWithLatestTag):
- Change Helm values.yaml image.tag from 'latest' to '' (falls back to
.Chart.AppVersion at render time)
- Pin values.prod.yaml to explicit release tag 0.5.0
- GCP: remove --allow-unauthenticated, restrict ingress to
internal-and-cloud-load-balancing, replace wildcard ALLOWED_ORIGINS=*
with a substitution variable (_ALLOWED_ORIGINS) so operators supply a
real URL at deploy time; same fix in cloudrun-service.yaml
- Fly.io: replace hardcoded FALKORDB_HOST=localhost with the correct
.internal private-network hostname pattern; update README accordingly
- docker-compose.dev.yml: add missing top-level networks: block so the
frontend service can join the semantica network without --file layering
- K8s/Helm: add readOnlyRootFilesystem: true + runAsUser: 1000 to
container securityContext; mount an emptyDir /tmp so uvicorn can write
temp files
- app.py: fix _read_explorer_settings() or-chain, use in os.environ
checks so an explicit ALLOWED_ORIGINS="" produces an empty allow-list
instead of silently falling through to localhost defaults; remove dead
app.state.falkordb_host/port attributes
- docs: update four locations that still documented {"status":"healthy"}
to reflect the new {"status":"ok"} health response
- tests: update test assertion to read falkordb settings from
app.state.explorer_settings instead of removed top-level attributes
* feat(export): implement Neo4j Bulk CSV Exporter and update registry docs (#261)
* fix(export): address review bugs in Neo4j CSV exporter
- _write_csv: filter **options to known csv.writer dialect params only,
preventing TypeError when callers pass kwargs like delimiter= or encoding=
that would reach csv.writer twice or as unknown arguments
- export_neo4j_csv: split kwargs into constructor-level init_params vs
per-call call_kwargs before forwarding, eliminating the double-pass that
caused dialect params to collide inside _write_csv
- _prepare_export: remove dead node_id_lookup dict that was built but never
consumed by any caller
- export_knowledge_graph dispatch: drop the ambiguous "neo4j" format alias
(kept "neo4j_csv" and "neo4j-csv"); "neo4j" conflicts with the codebase's
established meaning of the live Bolt/Cypher store backend; add inline
comment clarifying that file_path is treated as an output directory for
this format
- export_usage.md: fix all three wrong API examples — constructor params
node_label_sep/strict_validation corrected to label_separator/strict,
non-existent nodes_path/rels_path kwargs removed, convenience-method
example updated to show the correct positional output_dir argument
Co-Authored-By: KaifAhmad1 <kaif2208@gmail.com>
* docs(changelog): add Neo4j Bulk CSV Export entry for PR #665
Documents the new Neo4jCSVExporter feature contributed by @Luffy2208
and the five follow-up bug fixes (TypeError on dialect kwargs,
double-pass kwargs split, dead node_id_lookup removal, ambiguous
format="neo4j" alias removal, and wrong API examples in docs).
Co-Authored-By: KaifAhmad1 <kaif2208@gmail.com>
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: KaifAhmad1 <kaif2208@gmail.com>
Replace uniform cursor-bar hover effects with a differentiated,
light animation layer per element type. Adds global polish:
smooth scroll, custom scrollbar, brand-colored text selection,
page fade-in entrance, emerald focus rings, H1 gradient underline
accent, styled blockquotes, gradient HR dividers, uppercase table
headers, and CTA button glow — all tuned to the #080C10 dark
background and #10B981 emerald brand color.
* docs: add Changelog tab and clean up overview page
- Remove v0.5.0 release banner and stats grid from docs/index.md
- Add Changelog navigation tab to docs/docs.json after FAQ
- Create docs/changelog.md sourced from CHANGELOG.md with full Mintlify
formatting: one accordion per release (Unreleased → v0.0.1), icons,
pip install snippets, Added/Fixed/Security sub-sections, and a change
type legend
* docs: fix broken index#whats-new link in quickstart — point to changelog
* docs: fix onboarding examples for GraphBuilder and temporal queries
* docs: fix provenance import, hollow example, and query comment (#656 follow-up)
- Fix wrong import: ProvenanceTracker lives in semantica.kg, not semantica.provenance
- Replace hollow provenance accordion with actual track_entity/get_all_sources example
- Annotate query="" in TemporalGraphQuery.query_at_time as reserved for future use
* docs: use ProvenanceManager from semantica.provenance in W3C PROV-O example
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* docs: add choose-your-module onboarding guide
* fix(docs): correct export code examples against actual API signatures
- export_to_rdf() returns a string; use export() for file output
- format="json-ld" is invalid; correct value is "jsonld"
- ParquetExporter/LPGExporter/ArangoAQLExporter take file_path as a
required positional arg, not output= / output_dir= kwargs
- ArangoAQLExporter().export(graph) was missing file_path entirely,
which would raise TypeError at runtime
- Remove misleading 'with provenance embedded' comment (no such param)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Fix What's new → link in Info banner (now a proper <a> tag, always clickable)
- Replace 4-stat CardGroup on index with inline premium stats row
- Convert every <CardGroup>/<Card> block site-wide to markdown bullet lists:
content sections → bold-title bullets with sub-bullets, nav cards → [Title](href) — description
- Add cursor-animated list item hover effects to custom.css:
green inset left border, subtle background tint, marker color change on hover
- Affects index, getting-started, quickstart, concepts, modules, faq, architecture,
installation, cookbook, glossary, learning-more, explorer-setup, cli-setup,
community, contributing-guide, governance, citation, project-license,
all integrations pages, and all 20+ reference module pages
Removes all notebooks, data files, and exports under cookbook/use_cases/
(advanced_rag, biomedical, blockchain, capability_gap_defense, cybersecurity,
finance, intelligence, renewable_energy, supply_chain) and the corresponding
docs/use-cases.md page.
Cleans up all references in docs/cookbook.md, docs/docs.json,
docs/concepts.md, docs/modules.md, and docs/learning-more.md.
- Move Discord, GitHub, PyPI, and Follow on X links from sidebar anchors to top-right navbar
- Lock dark mode as default via appearance.strict and hide theme toggle
- Add custom.css with hover highlighting for tables, code blocks, cards, callouts, and inline code
- Move all Tips and Common Pitfalls sections inline next to their relevant content across all 25 reference docs
- Polish context.md: remove duplicates, condense callouts, upgrade Cookbooks to CardGroup
- Convert Troubleshooting and Performance Optimization sections in installation.md, cli-setup.md, explorer-setup.md, learning-more.md, and faq.md from plain headers to AccordionGroup
- Change navigation-hint Tip callouts to Info in concepts.md, faq.md, glossary.md, and modules.md
* fix: replace invalid Mintlify theme 'venus' with 'mint'
* docs: replace em dashes with colons across all docs files
* fix: strip UTF-8 BOM from all docs files (broke frontmatter detection)
- llms.md: use showcase models (llama-3.3-70b-versatile, gpt-4o) in
provider examples and use-case tables; clarify defaults vs recommended
in Defaults and Reproducibility section
- split.md: document that chunk_size is in characters with migration note
- ingest.md: add Note that glob patterns are not supported by ingest()
- explorer-setup.md: remove hardcoded "1.5 seconds" timing claim
- cli-setup.md: expand semantica-worker description with concrete usage
- mcp_server.md: clarify turtle/ttl are aliases for the same RDF format
- semantic_extract.md: remove emoji from code comments
Adds an explicit top-level `permissions` block to the GitHub Actions CI workflow.
This change sets the `GITHUB_TOKEN` permission scope to the minimum required level (`contents: read`), following the principle of least privilege and addressing the CodeQL alert `actions/missing-workflow-permissions`.
The workflow only requires read access to repository contents for checkout and CI tasks, so no additional permissions are needed.
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(explorer): resolve blank dashboard UI and ship frontend bundle in wheel
Fixes#631 — the Explorer server started successfully but the browser showed
a blank page because semantica/static/ was gitignored and never present after
a fresh install or clone.
Changes:
- ci.yml / release.yml: add Node 20 setup + npm ci && npm run build before
python -m build so every wheel contains a CI-built frontend bundle
- pyproject.toml: add package-data patterns (static/*, static/assets/*) so
setuptools includes the bundle in the wheel; add MANIFEST.in for sdist coverage
- app.py: replace silent empty-HTML fallback with a 200 page that clearly
explains the missing bundle and links to /docs; fix CORS allow_credentials
to default false, gated behind EXPLORER_CORS_CREDENTIALS env var to prevent
credentialed cross-origin requests on unauthenticated endpoints
- __init__.py: warn at startup when --host is non-loopback (unauthenticated
network exposure)
- explorer/README.md: full rewrite covering pip-install mode (primary path,
no Node required) and dev-server mode (contributors), CLI flags, env vars,
workspace table, troubleshooting for the blank-page symptom
- README.md: update Knowledge Explorer section with correct command and link
to the new setup guide
* fix(explorer): set build.target esnext to fix esbuild CI failure
esbuild >=0.28 (forced via npm overrides) conflicts with Vite 6 defaults on
Linux CI — it tries to lower destructuring syntax for the implicit browser
target list but errors out. Explicit target: 'esnext' tells esbuild to emit
native syntax unchanged, bypassing the transpilation error entirely. Safe for
a developer tool that runs in modern browsers.
* test(explorer): verify packaged frontend bundle
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Consolidate dual import (module alias + from-import) to a single
`import ... as progress_module` alias and qualify all references.
Replace bare `BaseException` catch with `Exception` in the thread
runner helper.
Co-Authored-By: Zohaib Hassnain <zohaib179949@gmail.com>
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
Updates CHANGELOG [Unreleased] to record that hardcoded GROQ_API_KEY
fallback values were stripped from advanced_rag/01, advanced_rag/02,
blockchain/01_DeFi_Protocol_Intelligence, and biomedical/01 — covering
secret scanning alerts #1–#6 (gsk_SLLE0, gsk_S4dBVJ, gsk_SLOv6,
gsk_lR6Qcj, gsk_ToJis6, gsk_LmbQBr). Keys already removed from HEAD
in a5da533; all 6 keys must be revoked in the Groq console.
* security: fix 9 Dependabot/CodeQL alerts — DOMPurify, vite, uuid, workflow permissions
- Add explicit permissions block to defender-for-devops.yml (CodeQL #25)
- Upgrade vite 5.4.x → 6.4.3; bundled esbuild 0.21.5 → 0.25.12 (Dependabot #2, #7)
- Force dompurify ^3.4.0 via npm overrides; resolves 6 DOMPurify XSS alerts (#4–#6, #8–#11)
- Force uuid ^13.0.1 via npm overrides; fixes buffer bounds check (Dependabot #12)
* fix(ci): exclude bandit from MSDO scan on windows-latest
bandit_runner.exe builds a per-file command line; on a large Python repo
the total command string exceeds the Windows CreateProcess limit and the
process fails to start (Win32 ERROR_FILENAME_EXCED_RANGE 206).
Exclude bandit via the tools param and retain checkov, eslint,
templateanalyzer, terrascan, and binskim.
* fix(ci): drop binskim (no binaries), enable Neptune audit logging
- Remove binskim from MSDO tools: repo has no compiled binaries so
BinSkim raises AnalyzeArgumentNoValuesException and breaks the run
- Add EnableCloudwatchLogsExports: [audit] to NeptuneCluster to fix
Checkov CKV_AWS_101 (the one error-level result breaking the build)
Remove all horizontal rule dividers for a cleaner premium look.
Replace all Hawksight-AI references with semantica-agi org URLs and update footer attribution from Hawksight AI to Semantica.
Deep-copy **options in ingest_examples and batch_public_apis so that
mutable values (e.g. params dicts) are not shared across iterations.
Add rate_limit_delay to the config_only_key strip list in ingest_public_api
so it is not forwarded twice when passed via kwargs.
Replaces the bare GIF with a structured "See Semantica in Action"
section featuring a clickable YouTube thumbnail for the Knowledge
Explorer Tour (https://youtu.be/QfnNZg4-dZA) above the original GIF,
with named subsections and a feature-list subtitle.
* test(benchmarks): add git-lfs infrastructure validation checks
* fix(benchmarks): add assertions and skip markers to LFS validation tests
Three tests had no assertions and always passed vacuously. Replace with
real assertions gated by pytest.mark.skip so logic is reviewed now and
enforcement is enabled later by removing the decorator. Also fix fragile
CWD-relative paths to use Path(__file__)-anchored roots, drop unused os
import and dead expected_patterns list, replace os.walk with Path.rglob,
and add missing newline at EOF.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Replaces the bare GIF with a structured "See Semantica in Action"
section featuring a clickable YouTube thumbnail for the Knowledge
Explorer Tour (https://youtu.be/QfnNZg4-dZA) above the original GIF,
with named subsections and a feature-list subtitle.
* test(benchmarks): add git-lfs infrastructure validation checks
* fix(benchmarks): add assertions and skip markers to LFS validation tests
Three tests had no assertions and always passed vacuously. Replace with
real assertions gated by pytest.mark.skip so logic is reviewed now and
enforcement is enabled later by removing the decorator. Also fix fragile
CWD-relative paths to use Path(__file__)-anchored roots, drop unused os
import and dead expected_patterns list, replace os.walk with Path.rglob,
and add missing newline at EOF.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Extract _discover_modules() to scan benchmarks/ at runtime so the
--module choices list stays accurate as directories are added or
removed; eliminates the stale context_graph_effectiveness entry and
the missing infrastructure entry from the original implementation
- Add an existence guard before passing the resolved path to pytest so
a valid-looking choice that maps to a missing directory fails fast
with a clear error instead of silently collecting 0 tests and exiting 0
- Print the active module filter to the console so users can confirm
the filtered scope in runner output
CLI section:
- Intro updated to mention startup dashboard and Rich polish
- Data In: added semantica watch examples; removed --watch flag from ingest
(watch is now its own command)
- Developer Tools: new subsection covering init, doctor, changelog, shell,
info with representative examples
What's New in v0.5.0:
- Added Modern CLI Experience subsection listing all 11 improvements:
startup dashboard, grouped help, doctor, init, watch, changelog, shell,
progress bars, elapsed timing, error cards, Windows UTF-8 fix
In TOML, declaring [project.urls] inside the [project] block causes all
subsequent key-value pairs (including dependencies = [...]) to be parsed
as project.urls.* keys, producing:
ValueError: invalid pyproject.toml config: project.urls.dependencies
must be string
Fix: move [project.urls] to after the dependencies array closes and before
[project.optional-dependencies], which is the correct TOML position for a
sub-table of [project].
Elapsed timing
- CLIContext._start records time.perf_counter() at context creation
- _ok() appends elapsed seconds to every success message automatically
Structured error cards
- _show_error_card() renders a red-bordered Rich Panel with title, detail,
and an actionable hint line
- _ERROR_HINTS maps common exception types to fix suggestions
- _run_with_error_handling() now routes all errors through the card renderer
instead of raising plain click.ClickException
Rich progress bars
- kg build: per-source Progress bar (SpinnerColumn + BarColumn +
MofNCompleteColumn + TimeElapsedColumn) when multiple --source flags given;
single-source path keeps the spinner
- ingest: spinner added (was missing entirely); shows filename and recursive flag
semantica changelog
- Hits GitHub releases API via stdlib urllib; compares latest tag against
__version__; renders release notes in a rounded Panel; --json supported
semantica doctor
- Checks: Python version, semantica/rich versions, graph store reachability,
vector store importability, LLM provider env vars, config file, log dir
- Rich table with ✓/⚠/✗ per check; summary error/warning count at bottom
semantica init
- Interactive wizard: graph backend, vector backend, optional LLM key
- Writes ~/.semantica/config.yaml via yaml.dump; --force to overwrite
semantica watch
- Wraps watchdog Observer; matches configurable glob patterns; auto-ingests
on created/modified events; graceful Ctrl+C shutdown
- Guards ImportError with pip install semantica[watch] hint
_HELP_SECTIONS updated to surface init, doctor, changelog, watch
- description: rewritten to lead with the accountability/provenance
angle and name concrete capabilities; drops emoji which render
inconsistently across PyPI clients
- keywords: expanded from 8 to 23 terms covering modern search queries
(ai-agents, llm, graph-rag, decision-intelligence, provenance, etc.)
- classifiers: added Information Analysis, Text Processing::Linguistic,
Database Engines/Servers, Information Technology audience
- [project.urls]: new section with Homepage, Documentation, Repository,
Changelog, Bug Tracker, Discord — shown prominently on the PyPI page
and drive clicks to GitHub/docs
- optional-dependencies: added watch = [watchdog>=3.0.0]; bundled into all
2026-06-04 17:37:11 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Prevents UnicodeEncodeError on the default cp1252 code page when Rich
renders box-drawing characters and emoji in the startup banner and panels.
Placed before all other imports so Click and Rich capture the already-
reconfigured streams. Uses reconfigure() (Python 3.7+) which modifies the
existing TextIOWrapper in-place rather than replacing sys.stdout.
- _BANNER: ASCII art shown when `semantica` is run with no subcommand
- _show_startup: dashboard panel with Graph Store / Vector Store / Profile
status cards; suppressed under --quiet and --json
- RichGroup: click.Group subclass that renders --help with grouped sections
(Data Ingestion, Intelligence, Knowledge Graph, Analytics, Export & Viz,
Infrastructure, Services, Tools) plus a Quick Start block
- main decorator: cls=RichGroup + invoke_without_command=True to wire both
- `semantica shell`: interactive REPL that dispatches subcommands while
sharing the parent CLIContext; supports readline on Unix for line editing
- Guard parse_cmd spinner with `fmt == "json"` (default format) to prevent
Rich status output from polluting machine-readable stdout in piped usage
- Remove unused `Rule` import from cli.py
- Remove unused `_orig_print` variable in verify_rich_cli.py
- Unify semantica.cli import style in verify_rich_cli.py; use cli_mod.main
The docs validation workflow runs python docs_check.py with no pip
install step, so rich is not available. Wrap the rich import in a
try/except ModuleNotFoundError and fall back to plain print() calls
so the script works in both environments:
- With rich installed: coloured pass/FAIL output
- Without rich (CI): plain text pass/FAIL output, same exit codes
## Summary
Overhaul the CLI and all library modules to produce polished, modern
terminal output comparable to tools like uv, gh, and cargo. Rich was
already a declared dependency but barely used — this commit wires it
throughout every layer.
## Changes by layer
### semantica/cli.py — visual overhaul
- Add imports: `box`, `Panel`, `Rule`, `Syntax`, `Text` from Rich
- Add 7 style constants (`_BRAND`, `_KEY`, `_VAL`, `_DIM`, `_SUCCESS`,
`_WARN_STY`, `_TABLE_BOX`) for a consistent colour palette
- `_ok()` now prefixes output with a green ✓ checkmark
- New `_info()` helper (neutral · bullet, respects --quiet)
- New `_warn()` helper (yellow ⚠ prefix, never suppressed)
- New `_pprint()` helper: renders dicts/lists as syntax-highlighted JSON
(Rich Syntax, monokai theme) instead of raw Python repr; strings
pass through unchanged; respects --quiet
- `info` command: banner replaced with a rounded Rich Panel showing
version + tagline; component table uses SIMPLE_HEAD box
- All 7 table sites updated: `box=SIMPLE_HEAD`, `show_edge=False`,
consistent `_KEY`/`_VAL` column styles (KG Stats, Reasoning Engines,
Recent Decisions, Configured Backends, Backup Info, MCP Tools)
- `_run_build()`: `console.status(spinner="dots")` wraps the blocking
build call; skipped under --quiet / --json
- `parse`, `extract`, `embed generate`, `reason run`, `reason explain`,
`deduplicate`: each wraps its long-running operation in a status
spinner, guarded by --quiet / --json
- All 30+ `console.print(result)` calls replaced with `_pprint()`
- All raw `[yellow]Warning:[/yellow]` and "not running" patterns
replaced with the new `_warn()` / `_WARN_STY` style
### semantica/explorer/__init__.py
- Error messages use `Console(stderr=True)` with `[bold red]Error:[/bold red]`
- Graph loading wrapped in `console.status()` spinner
- Startup info replaced with a cyan-bordered Rich Panel showing URL,
API docs, and health endpoint
### Library internals — replace print() with structured logger calls
All modules below had active `print()` calls that bypassed the logging
framework, corrupted spinners, and polluted stdout in piped/programmatic
use. All replaced with appropriate `self.logger.*` calls:
- `semantica/kg/graph_builder.py` — 23 calls: entity resolution
progress, graph structure steps, GraphStore persistence timing, and
the two `='*60` completion banners → `self.logger.info/debug()`
- `semantica/semantic_extract/methods.py` — 4 verbose-mode debug
prints → `logger.debug()`
- `semantica/semantic_extract/relation_extractor.py` — progress +
error prints → `self.logger.debug/warning()` with `exc_info`
- `semantica/semantic_extract/triplet_extractor.py` — same pattern
- `semantica/semantic_extract/semantic_network_extractor.py` — batch
error prints → `self.logger.warning/error()`
- `semantica/semantic_extract/coreference_resolver.py` — error print
→ `self.logger.error()`
- `semantica/semantic_extract/providers.py` — debug print →
`self.logger.debug()`
### Tooling
- `benchmarks/benchmarks_runner.py`: Rule banner, ✓/✗/⚠ status lines,
Rule separators around regression alert
- `benchmarks/infrastructure/compare.py`: removed manual ANSI escape
codes; comparison output is now a Rich Table with SIMPLE_HEAD;
summary uses coloured Rule + styled SUCCESS/FAILURE messages
- `cookbook/advanced/snowflake_ingestion_examples.py`: `_section()`
helper using Rule; tabular data rendered as Rich Table; result lines
use ✓/✗/⚠ prefixes; logger.error already present, retained
- `docs_check.py`: `pass`/`FAIL` lines use `[bold green]` /
`[bold red]`; summary uses styled output
## Tests
- `tests/test_cli_commands.py`: fix 3 pre-existing mock mismatches
- `test_kg_stats_json_with_mock`: mock now uses `compute_metrics()`
(the method the code actually calls) instead of `get_statistics()`
- `test_dry_run_not_needed_extract_is_read_only` and
`test_stdin_input`: mock now provides `NERExtractor`,
`RelationExtractor`, `TripletExtractor`, `EventDetector`
(the classes the code imports) instead of `SemanticAnalyzer`
Result: 230/230 tests pass (was 227/230)
- `tests/verify_rich_cli.py`: new verification script; exercises all
14 command groups (92 --help checks, table rendering, dry-run
formatting, --json mode, _pprint helper); 111 pass, 0 fail
- Wire --confidence, --model, --temporal flags to extractors via a flat
extractor_config dict (min_confidence, llm_model, include_temporal)
instead of the unused kwargs dict and sectioned to_dict() spread
- Pass confidence_threshold=confidence directly to RelationExtractor
which exposes it as a named parameter alongside **config
- Remove dead SemanticAnalyzer import and unreachable else branch from
extract; unsupported modes now consistently raise ClickException
- Add _serialize_extract_result() to convert dataclass/list results to
plain dicts so JSON and YAML output is machine-readable, not str()
- Fix kg_stats: remove graph={} arg from compute_metrics() so it uses
the analyzer's loaded graph instead of always computing on empty data
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Covers all 22 command groups introduced in issue #568:
global flags, data in, processing, KG, intelligence (reason/decision/temporal),
provenance, validation, ontology, export, visualize, orchestration
(pipeline/store/backup), services (server/explorer/mcp), and shell completion.
Each section shows real invocation examples rather than flag tables.
P1 — runtime-breaking API mismatches:
- decision record/list/query/trace/similar/impact/check: all six decision
commands now call decision_methods / decision_query using a GraphStore
from _get_graph_store(cli_ctx) instead of passing config= kwargs that
don't exist on the underlying API signatures.
- embed index: load vectors from the Parquet/JSON file into List[np.ndarray]
before calling create_index(), which expects vectors not a file path string.
P2 — stub implementations replaced with real logic:
- backup sync: now collects local data sources via _collect_backup_sources
and performs an incremental copy (skips files whose dst mtime >= src mtime).
- backup restore: detects .enc / tar.gz / .tar / directory, decrypts SEM1
format when --enc, extracts tar archives with leading prefix stripped, or
copies directory trees back to cwd.
P3 — correctness bugs:
- backup create: archive now includes actual config/ontology/store data files
via _collect_backup_sources; manifest records the file list.
- extract: --output now works for all formats (table/rdf/yaml), not only JSON.
- backup create: empty keyfile now raises a clear error instead of silently
producing an unencrypted archive.
- normalize: use Path.is_file() instead of Path.exists() to avoid accidentally
reading a directory that matches the input text.
- visualize: without --output, emit to stdout; do not silently write kg.html.
Minor:
- _setup_cli_logging: replace opaque _ = (quiet, json_output, exc) tuple
with del to suppress unused-variable lint.
- reason list: try to source engines from the reasoning module registry;
fall back to the hardcoded list.
- deduplicate --action report: use method="pairwise" to produce individual
pair objects with similarity scores, distinct from --action detect.
- tests: remove mixed import (from semantica.cli import main) — all 192
runner.invoke calls now use cli_module.main as CodeQL flagged.
- tests: add two focused embed-index regression tests that verify vectors
are loaded from the file before create_index is called.
- embed search: embed query text before calling search_vectors (was passing
raw string to query_vector positional arg, causing TypeError on every call)
- ontology version: import OntologyVersionManager not OntologyVersioning
(symbol never existed; command always failed even with package installed)
- ingest --watch: forward watch flag into _ingest() kwargs (was accepted
but silently dropped, so --watch had no effect)
- store migrate: replace fake success stub with honest ClickException pointing
to the export+embed-index workaround (no bulk-dump API exists in vector store layer)
Fix the export runtime mismatch where get_export_method expected the existing (task, name) registry contract but the CLI passed only the format argument.
Expands semantica/cli.py from a 2-command stub into a complete terminal
interface covering every capability described in issue #568, and ships
253 tests covering all new commands, flags, and error paths.
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
- Remove incorrect # pragma: no cover from _run_with_error_handling
generic Exception branch (test_runtime_errors_are_click_safe already
covers it via the monkeypatched RuntimeError path)
- Add _require_ctx() guard: converts None ctx.obj into a clean
ClickException instead of an AttributeError (protects standalone_mode=False
/ library-use callers); apply to info, kg_build, build_alias commands
- Rename serve group -> services to avoid collision with the future
`semantica server` flat command specified in issue #568; update docstring
to document planned subcommand layout
- Fix command-level config logging: re-call setup_logging() with the
command-level config logging section when -c is used (setup_logging
clears handlers before adding, so no accumulation risk)
- Fix missing log_level_override in command_ctx: global --log-level was
silently dropped when a per-command -c config was present, breaking
the override chain for any nested _build_runtime_config calls
- Add return-shape docstring on _run_build documenting the expected
build_knowledge_base() return dict structure
- Add type annotation to runner fixture (-> CliRunner) so Pylance
correctly types runner.invoke() -> Result across all test functions
- Expand test suite: 25 -> 32 tests
* test_info_command_shows_framework_components
* test_info_command_shows_config_path_when_supplied
* test_log_level_global_override_stores_in_context
* test_command_config_preserves_global_log_level_override
* test_build_result_with_stats_shows_source_count
* test_build_result_without_stats_shows_generic_success
* test_build_result_none_shows_generic_success
* test_require_ctx_raises_click_exception_on_none
* test_require_ctx_returns_ctx_unchanged
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- keep command-level config from overriding logging unless --log-level is set
- validate YAML/JSON config roots and surface parse failures as Click errors
- tighten CLI tests around isolation and cleanup
- add CLI runtime context, global config/log-level handling, and click-safe error wrapping
- implement kg build as a thin wrapper over existing orchestrator build flow
- keep hidden legacy build alias and route both build handlers through shared internal path
- add focused CLI tests for help UX, config flag compatibility, alias parity, and clean error output
- keep tests lightweight by mocking heavy build execution paths
- Rename logo PNG to semantica-logo.png (lowercase, hyphenated)
- Update docs.json logo (light/dark) and favicon to reference new PNG
- Replace legacy purple favicon with new teal brain neural network icon
- llms.md: replace non-exported Anthropic/Ollama imports with LiteLLM provider-prefix pattern; replace ReasoningEngine with Reasoner; replace create_provider with LiteLLM in YAML config example and tip
- concepts.md: replace ReasoningEngine with Reasoner/ReteEngine/GraphReasoner; fix DatalogReasoner.reason() to evaluate()/query(); replace TemporalKnowledgeGraph with TemporalGraphQuery; replace DistanceCalculator with SimilarityCalculator; replace EntityDeduplicator with DuplicateDetector/EntityMerger
- kg.md: replace non-exported build_knowledge_graph with method_registry.execute()
- semantic_extract.md: replace Anthropic import with LiteLLM
- index.md: replace Anthropic/Ollama imports with LiteLLM
- modules.md: fix TemporalKnowledgeGraph, DistanceCalculator, OntologyManager, ReasoningEngine, DatalogEngine, start_explorer, create_provider across code examples and module index table
- triplet_store.md: replace non-exported NamespacePrefixManager with semantica.ontology.NamespaceManager
Replace plain markdown in every docs/reference/ file and docs/concepts.md with
rich Mintlify JSX components — CardGroup, Steps, Tabs, AccordionGroup, Tip,
Warning, Note, and CodeGroup — for a consistent, navigable, production-grade
developer experience.
Frontmatter, intro, heading, Tabs, and Module Map all said "three-layer"
while the architecture-overview.svg and its alt text showed four layers.
Adds Layer 3 (Intelligence: KG, vector store, ontology, triplet store,
embeddings) and renumbers the former Layer 3 Application to Layer 4.
Diagrams (docs/assets/img/diagrams/):
- architecture-overview.svg: 4-column layered architecture
- pipeline-flow.svg: 8-step numbered pipeline flow
- kg-structure.svg: entity/relation graph with typed nodes and labeled edges
- graphrag-flow.svg: dual-path retrieval (vector + graph) to LLM to grounded answer
- extraction-pipeline.svg: NER/Relation/Coreference fan-out to Triplet Generator
- agent-context-flow.svg: AgentContext hub with VectorStore and ContextGraph
- reasoning-chain.svg: forward-chaining inference with explanation path
Wordmark logo (light + dark SVG variants):
- Green rounded-square S icon + Semantica text in green
- docs.json updated to use wordmark SVGs for light and dark modes
Pages updated with diagrams:
- index.md, architecture.md, quickstart.md, concepts.md
- reference/kg.md, reference/pipeline.md, reference/semantic_extract.md
- reference/context.md, reference/reasoning.md
- installation.md: revert card title back to "Getting Started" to match
the Tip text that already links to it by that name
- getting-started.md: restore pip install semantica[all] code block that
was removed in the original PR; users need the copy-paste snippet even
when the Installation guide is the canonical reference; also standardize
link text to "Installation" (was "Installation guide")
- index.md: add Installation card as first entry in "Start Here" CardGroup
so the prose ("install first, then open Quickstart") is backed by an
actual card to click
- quickstart.md: standardize link text to "Installation" (was "Installation guide")
mint export fails with 'file does not exist' for pages named 'contributing'
and 'license' — these are reserved by Mintlify's GitHub integration layer.
Renamed to contributing-guide.md and project-license.md and updated all
nav entries and cross-links throughout the docs.
Also adds .gitattributes LF rules to prevent CRLF issues from Windows devs.
- Validate docs structure with docs_check.py (Python)
- Validate Mintlify build with mint validate (Node 20 LTS)
- Export static site with mint export, deploy to GitHub Pages
- Deploy job skipped on PRs (validate-only for branches)
- docs.yml: replace mkdocs build/deploy with python docs_check.py;
Mintlify deployment is handled by its own GitHub App
- ci.yml: remove dead paths-ignore refs to deleted mkdocs.yml and
requirements-docs.txt
* Add XML file ingestion support
* fix(xml-ingestor): add ingest_string test and document ingest() return keys
- Add test_xml_ingestor_ingests_string to cover the public ingest_string()
method which had no test coverage
- Document all source_type return keys in the ingest() docstring so callers
know to use result["xml"] rather than result["data"] for XML sources
* docs(changelog): add unreleased entry for XML ingestion support (#560)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* docs(readme): redesign for better traction and narrative clarity
- Reorder sections: Problem → Solution → Quick Start → What's New → Integrations
- Add website and docs badges to the top badge strip
- Improve hero tagline and narrative blockquote
- Restore v0.4.0 (Temporal, SKOS, SHACL) and v0.3.0 release sections
- Remove duplicate modules list; consolidate into single table
- Fix broken emoji characters in Enterprise section
- Add blank lines around all headings and list blocks
* docs(readme): concise rewrite with accurate v0.5.0 features and compact layout
- OntologyManager: remove red error banner on HTTP 500; always fall back
to empty state silently (error banners reserved for user actions only)
- AlignmentsTab: remove offline-backend warning when both registry and
alignments requests fail; show empty form silently
- ShaclStudio: fix Monarch tokenizer crash — [@] character class prevents
Monaco from misinterpreting @prefix/@base as language-property refs;
wrap beforeMount in try/catch so any Monaco setup failure cannot crash
the React tree
Decision workspace:
- Add AbortController per loadChain() call; abort previous request when a
new decision is selected, preventing stale out-of-order chain responses
- Guard all setState calls with signal.aborted so unmounted component
state updates are skipped; cancel in-flight request on unmount via a
dedicated cleanup effect
SPARQL workspace:
- Guard results table on both result.rows && result.columns to prevent
runtime crash when backend omits columns field
- Use (result.columns ?? []) inside rows.map() to satisfy TypeScript
narrowing inside the closure
- Add .catch() to clipboard.writeText() — silently swallows permission
errors (query remains visible in the editor as fallback)
- Fix CSV export anchor: append to body before click, remove after, to
ensure cross-browser compatibility
Import/Export workspace:
- Fix download anchor: append to document.body before a.click() and
remove afterwards, matching the standard compatible pattern
Lineage workspace:
- Replace 🔗 emoji empty-state icon with lucide-react Link2 for
consistent theming and sizing
Diff & Merge workspace:
- Add "Sample preview" banner above the mock diff table so users know
the displayed fields are illustrative until the backend is connected
OntologyManager:
- Restore non-blocking warning (flash message) when HTTP response is
non-OK and not a 404; network errors (backend down) stay silent
AlignmentsTab:
- When both registry and alignments promises reject, surface a soft
error banner so users know data is missing rather than just empty
Four issues raised in code review:
- Mode.JSON retry now strips response_format from create_kwargs before
calling json_client.chat.completions.create, preventing incompatible
kwargs from being forwarded to a client configured for a different mode.
- Add exc_info=True to the generate_structured fallback warning in the
manual repair loop so the gateway rejection traceback is visible in
production logs, consistent with the other warnings added in this PR.
- Remove the duplicate is_available definition in GroqProvider. Python
silently kept only the second definition; the first (with diagnostic
branching) was dead code and could cause confusion on future edits.
- Validate base_url scheme in OpenAIProvider._init_client. Non-HTTP(S)
schemes (file://, ftp://, javascript:, etc.) are now rejected with a
ValueError at init time, preventing SSRF if base_url originates from
configuration rather than hardcoded values.
Add 3 new tests: SSRF scheme rejection, valid-URL acceptance, and
exc_info presence on the generate_structured fallback warning (20/20 pass).
Update CHANGELOG.md with full description of all fixes under [Unreleased].
Three bugs caused NERExtractor to silently return pattern-based entities
even when method="llm" was configured:
1. exc_info=True missing on method-failure warning in NERExtractor —
the root exception was swallowed, making the gateway error invisible
in logs even with DEBUG enabled.
2. OpenAIProvider.generate_structured always sent response_format=json_object
to the API. Custom/enterprise gateways (Qwen, LLaMA proxies, internal
gateways) often reject this parameter, causing both the instructor path
and the manual repair loop to fail with the same error on every retry.
3. generate_typed manual repair loop had no fallback when generate_structured
itself raised — it retried the same failing call up to max_retries times,
then propagated the error, triggering _extract_fallback (pattern extraction).
Fixes:
- Add exc_info=True to the method-failure warning so the full traceback
appears in logs and users can diagnose the root cause.
- Skip response_format=json_object in OpenAIProvider.generate_structured
when base_url is set (custom endpoint), since standard OpenAI gateways
don't require it and third-party ones reject it.
- In the generate_typed manual repair loop, catch generate_structured
failures and immediately retry via plain generate() + _parse_json,
breaking the retry-the-same-failing-call loop for custom gateways.
Also adds 17 targeted regression tests covering all three bug paths,
including the exact gateway configuration reported in the issue.
- Add multilingual README links section (30 languages via readme-i18n.com)
- Pin version badge to 0.5.0 with correct release tag link
- Add "What's New in v0.5.0" section covering Distance Intelligence,
Ontology Hub Suite, Parquet ingestion, indexed search, and security fixes
* Added Parquet ingest support (#234)
* docs: Add Parquet ingestion support to CHANGELOG
- Add comprehensive changelog entry for PR #548
- Document ParquetIngestor class and key features
- Include author credit (@Luffy2208) and PR reference
- Follow existing changelog format and structure
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Convert mcp_server.py to package structure (semantica/mcp_server/)
- Add __init__.py and __main__.py for python -m support
- Add semantica-mcp console script entry point in pyproject.toml
- Fix API method calls (extract -> extract_entities/relations/triplets)
- Remove non-existent _result_cache imports
- Update documentation with both usage methods
Resolves pipx installation issue where semantica.mcp_server was not available.
Provides two ways to run: 'semantica-mcp' command or 'python -m semantica.mcp_server'.
- bug_001: top_k_per_entity now uses OR semantics — keep a candidate if
EITHER entity is under quota, preventing high-quality candidates being
silently dropped when a popular counterpart saturates its quota
- bug_002: validate max_results and top_k_per_entity at construction;
negative or non-int values raise ValueError instead of silent empty output
- bug_003: validate min_similarity in [0.0, 1.0] at construction;
out-of-range values raise ValueError
- bug_004: harden ConflictDetector method='relationship' normalization —
always produces List[Dict] before calling detect_relationship_conflicts
- quality_001: update detect_duplicates + incremental_detect docstrings to
reflect configurable sort_by field (not hardcoded 'confidence')
- quality_002: add _normalize_entity_id helper (always str) used in both
_apply_result_limits and _build_duplicate_groups for consistent ID handling
Backward compatible: callers not using new params see no behavior change.
58 tests pass (0 failures)
Fixes#534
- New __init__ params: max_results, top_k_per_entity, min_similarity, sort_by
- _apply_result_limits: drop below min_similarity, sort by sort_by field,
enforce top_k_per_entity per entity, cap at max_results globally
- Wired into detect_duplicates() and incremental_detect()
- 30 new tests in TestResultLimiting; full suite 42/42 passed
Fixes#533
- Removes duplicate `detect_conflicts` definition that was silently overridden,
causing AttributeError for callers passing `method=` or `property_name=` kwargs
- Merges dispatcher logic into the surviving method with `method="all"` default
supporting: "all", "value", "property", "type", "relationship", "temporal",
"logical", "entity"
- Fixes `method="relationship"` incorrectly defaulting `relationships` to the
entities list; now defaults to `[]` with dict normalization
- Removes unreachable dead code block after try/except raise in
`detect_entity_conflicts`
* fix(deps): remove gpu extra from [all] to fix Windows installation failure
faiss-gpu has no Windows builds, so semantica[all] failed with
'No matching distribution found for faiss-gpu>=1.7.0' on Windows.
Removed gpu from both [all] lines — semantica[gpu] remains available
as an explicit opt-in for Linux GPU environments.
Closes#532
* docs(changelog): record faiss-gpu Windows installation failure fix (#532)
Closes#531
- Replace 5 direct sys.stdout.write() calls in ConsoleProgressDisplay.update()
with self._safe_write() so emoji/block characters are encoded safely on
Windows cp1252 consoles
- Add TestProgressTrackerEncoding regression tests (3 cases) covering
_safe_write, pipeline header, and auto emoji-disable on cp1252
test_retry_logic.py injected sys.modules["openai"] = MagicMock() at module
level so providers.py could be imported without the real openai package.
Those mocks were never restored, leaving openai (and spacy, instructor etc.)
as MagicMock objects for the entire test session. This caused
test_pr482_deepseek_openai tests to receive a MagicMock when importing
openai.OpenAI, making MagicMock(spec=OpenAI) raise InvalidSpecError.
Fix: save original sys.modules entries before injection and restore them
immediately after the semantica imports that needed the mocks complete.
The mock objects remain bound inside the already-imported provider module,
so test_retry_logic tests are unaffected; other test modules now see the
real packages again.
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
subprocess.CompletedProcess[str] as a return annotation is not subscriptable
at runtime on Python 3.8, causing test collection to abort before any tests
run. Adding PEP 563 deferred evaluation makes all annotations strings at
import time, restoring 3.8 compatibility without changing behaviour on 3.9+.
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-05 13:40:11 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* fix(ingest): lazy-load optional ingestion backends
* fix(ingest): address qodo review — use ModuleNotFoundError and guard ConfigurationError
Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.
Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
* docs(changelog): record lazy ingest backends fix and qodo review fixes (#535)
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
* fix(tests): set exc.name on blocker's ModuleNotFoundError to match Python import machinery
OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.
Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.
Co-authored-by: Zohaib Hassan (@ZohaibHassan16) <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Zohaib Hassan (@ZohaibHassan16) <zohaib179949@gmail.com>
OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.
Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.
Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Backend (semantica/explorer/routes/ontology.py):
- suggest-alignments: add TF-IDF character-ngram embeddings via sklearn
(SimilarityCalculator-compatible cosine scoring) so embedding_similarity
is populated in results; combined score = 0.4*label + 0.6*embedding when
available, falling back to label-only when sklearn is absent
- suggest-alignments: add token-overlap prefilter before SequenceMatcher so
zero-Jaccard pairs are skipped without computing full similarity; add
_MAX_ENTITIES_PER_SIDE=500 per-ontology cap on top of the existing
_MAX_ANALYSIS_NODES global cap
- suggest-alignments: remove dead try/except OntologyEngine.create_alignment
block that always failed silently (no TripletStore configured); replace
with a comment explaining the intentional ephemeral-only storage model
- health: replace O(alignments x entities) any() scans for alignment coverage
with O(1) set membership checks via assessed_ids
- shacl/validate: run rdflib.Graph().parse(format='turtle') syntax check on
the submitted Turtle before returning; invalid syntax now raises 422 instead
of returning a misleading unavailable/success response
Frontend:
- AlignmentsTab: add pairwise alignment matrix section that groups recorded
alignments by (source_ontology, target_ontology) pair; each cell shows
color-coded relation badges per RELATION_COLORS; clicking a badge populates
the create/edit form for quick editing; matrix is shown when at least two
ontologies are loaded
- ShaclStudio: add selectedShapeId state and fullShacl ref; each shape row in
the library is now a clickable button that extracts its Turtle block from
the full SHACL and pre-populates the Monaco editor; a "View all" toggle
restores the full SHACL; selected shape ID is shown in the editor header
- GraphWorkspace: fix viewMode race in external focus effect — call
setSelectedNodeId directly instead of going through focusNode(), which
captured a stale viewMode in its closure; remove focusNode from the
dependency array since it is no longer called
Tests (14 passing, was 11):
- Add test_suggest_alignments_returns_embedding_similarity: asserts
embedding_similarity is non-null when sklearn is available
- Add test_shacl_validate_rejects_invalid_turtle_syntax: asserts 422 on
syntactically invalid Turtle
- Add test_health_alignment_coverage_uses_set_lookup: asserts alignment
dimension score is non-zero after recording an alignment, verifying the
O(1) set lookup path works correctly end-to-end
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Backend:
- Replace false conforms=True SHACL stub with status=unavailable always;
live validation cannot be wired until OntologyEngine.validate_graph is
connected to a data graph — a stub that returns conforms=True misleads
users editing shapes
- Cap node/edge fetches in health, suggest-alignments, and SHACL generation
at _MAX_ANALYSIS_NODES (5 000) with a logger.warning when the graph
exceeds the limit; unbounded limit=999_999 fetches cause OOM on large graphs
- Set SHACL health dimension score to 0.0 (was 70.0) when status=unavailable;
exclude unavailable dimensions from the total_score average so they neither
inflate nor deflate the result
- Allow alignments to reference external/unloaded URIs (e.g. schema.org)
without raising 404; label falls back to URI fragment or caller-supplied
source_label/target_label fields added to OntologyAlignmentRequest
- Fix _alignment_id to use uuid.NAMESPACE_OID instead of NAMESPACE_URL;
the composite key is not a URL
- Fix _summarize_shapes to normalise \r\n before splitting on .\n so shape
parsing works correctly on Windows line endings
Frontend:
- Wrap handleSave/handleSuggest/handleRemove/handleAcceptSuggestion in
useCallback in AlignmentsTab for consistency with sibling components
- Add ephemeral-storage banner in AlignmentsTab warning that alignments are
session-memory-only and not persisted across restarts
- Fix exportReport in HealthTab to append/remove anchor from document before
clicking and defer URL.revokeObjectURL to avoid Blob URL leak in some browsers
- Derive health dimension grid column count from health.dimensions.length
instead of the hardcoded repeat(5, ...) that breaks if the backend adds
or removes a dimension
- Add minimal Monarch tokenizer for the Monaco turtle language registration
in ShaclStudio so prefix declarations, IRIs, SHACL properties, comments,
and string literals are syntax-highlighted; previously the editor rendered
as plain text despite theme rules being defined
Tests (11 passing, was 5):
- Rename test_shacl_validate_has_stable_contract to
test_shacl_validate_returns_unavailable and assert status == unavailable
- Add test_shacl_validate_rejects_empty_turtle (expects 422)
- Add test_health_returns_404_for_unknown_ontology
- Add test_health_shacl_dimension_is_zero_when_unavailable with total_score check
- Add test_delete_unknown_alignment_returns_404
- Add test_alignment_upsert_is_idempotent (verifies ID stability and created_at
preservation across updates)
- Add test_alignment_accepts_external_uri (verifies no 404 for schema.org URIs)
- Relax test_alignment_suggestions_are_ranked label assertions to substring
checks so the test survives similarity algorithm changes
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
- Fix domain_uri/range_uri always being truthy strings
- Only create rdfs:domain/rdfs:range edges when domain/range are non-empty strings
- Add proper validation with .strip() to handle whitespace-only values
- Apply fix to both 'data' and 'text' mode ontology creation
- Prevents pollution of graph with invalid edges to namespace root
Fixes issue where empty domain/range values like '' or None would still create
edges pointing to namespace root (e.g., 'https://ex/#/') instead of being
properly omitted.
The pattern `<[^>]+>\s+<[^>]+>` in _detect_format() was flagged by CodeQL
(py/polynomial-redos, CWE-1333/730/400) as a polynomial regular expression
on uncontrolled user data.
The `<...>` branch was already unreachable — strings starting with '<' return
'xml' two lines above — but CodeQL does not track that control flow path.
Fix: replace the entire re.match() call with plain startswith / 'in' checks:
- N-Triples with URI subjects are already handled by the XML branch.
- Only blank-node-subject N-Triples (_:word <uri> ...) need detection here,
which is correctly expressed as startswith('_:') and ' <' in stripped.
- Removed the now-unused `import re`.
Closes security advisory #23.
2026-05-01 15:26:17 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Bug 4 — Upload format misdetected:
- Added xml→'xml' and json→'json-ld' to the extension→format map so
.xml and .json files are no longer misidentified as turtle.
- Changed the fallback from '|| "turtle"' to '?? ""' (empty string for
unknown extensions) so the backend _detect_format() runs instead of
blindly assuming turtle for any unrecognised extension.
- Omit the format key entirely from the load request body when no format
was detected, letting the backend auto-detect from content heuristics.
- Added .n3 to the file picker accept list and dropzone hint text.
Bug 1 — Broken registry filters:
fetchRegistry no longer sends format/kind values (owl/skos/internal/external)
as the status query param; those filters are applied client-side via
filteredEntries which already had the correct logic. Only the text search
param q is delegated to the backend.
Bug 2 — Toggle/refresh URI corruption:
Removed removesuffix('/toggle') and removesuffix('/refresh') from
toggle_ontology and refresh_ontology. Starlette's route regex already
strips the literal suffix from the captured path param; the removesuffix
call was a no-op for normal URIs but corrupted any ontology URI that
legitimately ends with /toggle or /refresh.
Bug 3 — SSRF in URL fetch:
Added _validate_fetch_url() which rejects non-http/https schemes and
resolves the hostname to block private, loopback, link-local, reserved,
and multicast addresses before requests.get() is called. Applied to all
three fetch sites: preview, load, and refresh.
Bug 5 — Inconsistent XML hardening:
_parse_rdf_sync now calls _safe_parse_rdf() from
semantica/explorer/utils/rdf_parser.py instead of g.parse() directly,
applying the existing defusedxml-based XXE protection for RDF/XML inputs.
Bug 6 — Search scans whole graph:
search_entities now calls session.search(q, limit*6) which hits the
GraphSearchIndex instead of fetching up to 999,999 nodes and doing a
linear Python substring scan. Results are post-filtered by _SEARCHABLE_TYPES
and entity_type before being returned up to the requested limit.
- Replace invalid inset-left with inset: 0 0 0 72px on ::before at <=680px
- Add matching mobile inset fix to ::after (was still at 88px)
- Merge duplicate .landing-capability-band CSS rule blocks into one
- Fix non-standard font-weight: 850 -> 800 on .landing-launcher-item-title
- Remove unused eyebrow field from LandingAction type and all data entries
- Extract static 42-dot SVG preview array to module-level PREVIEW_DOTS constant
Co-Authored-By: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
- Align _coerce_embedding_vector inner dict-probe key list with
_extract_node_embeddings outer key list (add 'embeddings', reorder to
generic-first) so nested embedding dicts resolve consistently.
- Add TODO comment on _extract_node_embeddings to cache per-session
graph revision and avoid O(N) re-scan on every semantic request.
- Add deprecation docstrings to legacy path-segment routes
(/node/{id}/path and /node/{id}/semantic-neighborhood) documenting
the known slash-in-ID limitation and pointing to the query-param
alternatives.
- Extract _FakeSimilarity to module level so it is shared without
duplication across test classes.
- Rewrite test_legacy_semantic_neighborhood_still_works_for_simple_ids
as a fully isolated TestClient session instead of mutating the
shared module-scoped 'client' fixture, preventing cross-test
state pollution.
- Extract _make_slash_node_session helper to reduce boilerplate in the
slash-safe route tests.
Co-Authored-By: ZohaibHassan16 <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <98801504+KaifAhmad1@users.noreply.github.com>
- Fix dead `if (anchorNodeId)` conditional in buildHeatmapRenderSnapshot
(anchor is always truthy past the early-return guard on line 263)
- Replace O(n) array .includes() with WeakMap-cached Set.has() in
resolveDistanceNodeStyle heatmap path — prevents per-node O(n) scan
during every Sigma reducer pass on large graphs
- Rename GraphDistanceBucketCounts.threeHop → threeHopPlus across
types.ts, graphSceneState.ts, and GraphWorkspace.tsx so the field
name reflects that it accumulates distance ≥ 3, not exactly 3;
update status-strip labels to "3+ hop" accordingly
- Restore hasMetrics guard in PathDistanceIntelPanel to suppress the
empty metric grid <div> when a path result carries no optional metrics
Co-Authored-By: ZohaibHassan16 <zohaib@hawksight.ai>
Co-Authored-By: KaifAhmad1 <mohammadk78600@gmail.com>
- Extract ENTITY_SHAPE_ALIASES and classifyEntityShape into a shared
graphEntityShape.ts utility — resolveEntityShape was duplicated with
divergent signatures in useLoadGraph.ts and graphSceneState.ts; both
now import from one place so aliases can never drift
- graphSceneState.resolveEntityShape falls back to classifyEntityShape
for nodes created programmatically that bypass useLoadGraph
- Fix graphTheme.ts indentation around fullGraphStructure,
fullGraphStructureLayer, and interaction — closing braces were at
wrong indent levels making the nesting visually misleading
- Add comment on fullGraphStructureLayer.mode explaining it is
intentionally "off" as a staged-rollout gate (flip to "auto" to enable
cross-community canvas curve rendering)
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Merge origin/main (Distance Intelligence #502) into feat/explorer-visual-refresh.
Conflict was in the viewModeItems useMemo: the PR's new cluster-based toolbar
structure diverged from main's coreToolbarGroups additions.
Resolution:
- Keep PR's viewModeItems as a clean 3-item segmented control (Full/Grouped/Focused)
- Port Distance Intelligence controls (ego mode, heatmap, structural/semantic overlay)
into a new distanceToolbarItems useMemo that slots into the cluster toolbar as a
"Distance" cluster, visible only when a node is selected
- Wire distanceToolbarItems into toolbarClusters between "local-structure" and
"analysis" clusters
- All other Distance Intelligence additions (state vars, BFS helpers, useEffects,
ego depth slider, GraphInspectorPanel onFocusNode prop) merged cleanly
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Merge blockers (ZohaibHassan16):
- fix: distance-matrix raises HTTP 503 when metric=semantic but no
similarity backend is available, instead of silently returning hop
distances labeled as semantic
- fix: distance-enriched export now requires node_subset (HTTP 422 if
omitted), preventing unbounded all-pairs O(n^2) export over full graph
- fix: DistanceExportRequest default include corrected from ["hops",
"distance_band"] to ["source_id", "target_id", "hop_count",
"distance_band"] so default exports are unambiguous and use the correct
column name
- fix: confidence decay edge weight index now reads graph_dict.get("edges")
or graph_dict.get("relationships") to handle both graph dict shapes,
fixing always-1.0 decay when session returns relationships key
Bot findings (github-code-quality / chatgpt-codex):
- fix: remove unused Iterable import from distance_exporter.py
- fix: remove unused Response import from graph.py
- fix: move logger init before optional KG import; replace empty
except ImportError: pass with logger.debug in distance_exporter.py
- fix: replace two bare except Exception: pass in temporal.distance_history
with logger.warning including source, target, metric, and timestamp context
- fix: remove mixed import style in test_qual003 — use only module import
and reference CausalChainAnalyzer through it
- Fix `import.meta.env.DEV` crash in graphSceneState.ts that broke the
entire test:graph-workspace suite (module load fails in Node.js/tsx)
- Export `resolveGroupedDisplayNodeId` from graphSceneState.ts and
remove the identical copy in GraphWorkspace.tsx
- Add `checkGroupedViewAvailability` helper (Louvain only, no centrality)
so grouped view availability can be checked cheaply on every graph change
- Gate full community graph build (`groupedDisplayCandidate`) on
`viewMode === 'grouped'` to avoid running Louvain + centrality on every
graph version tick when the user is not in grouped view
- Remove dead ternary in `focusNode` where both branches returned `nodeId`
- Add 7 new tests covering resolveGroupedDisplayNodeId,
resolveGroupedDisplayStateSnapshot, and checkGroupedViewAvailability
Co-Authored-By: ZohaibHassan16 <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <mohammadk78600@gmail.com>
Merge conflict resolution:
- Kept fix/graph-motion's conditional layout-stop (only in focused mode)
to preserve live layout motion for derived graphs — the core intent of
this PR.
Must-fix items resolved:
1. plugin.json — removed "hooks": "./hooks/hooks.json" (re-added by this
branch, already removed in PR #489 on main as it is auto-loaded).
Kept "agents": "./agents".
2. Double Louvain per render — added groupedViewAvailable useMemo in
GraphWorkspace (deps: [graphVersion]) that runs community detection
once. Passed result into resolveDisplayGraph and resolveDisplayStateSnapshot
via new groupedViewAvailable option; both functions skip their internal
computeGraphAnalyticsBase call when the value is pre-supplied.
3. graphVersion in displayState deps — removed graphVersion from the
displayState memo dep array. displayState now depends on the stable
boolean groupedViewAvailable, not on every ADD_NODE/ADD_EDGE tick,
so Louvain no longer re-fires on every WebSocket update.
4. hideLabelsOnMove / hideEdgesOnMove flipped to true — intentional:
suppressing labels and edges during pan reduces visual noise and is
part of the flicker-reduction fix described in the PR.
Co-authored-by: ZohaibHassan16 <zohaibhassan1696@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
1. Prevent active-but-disabled Focused button by only disabling when
viewMode is not already "focused" (viewMode !== "focused" && !canActivateFocusedMode).
2. Generalize inspector fallback copy — stale/invalid node IDs are not
necessarily grouped items, so remove the misleading "Activate Focused
mode" hint.
3. Move pluginRuntimeRef.current read out of render by converting
canActivateFocusedMode from useMemo to useState + useEffect, resolving
two ESLint "cannot access refs during render" errors and the missing
toolbar-memo dependency warning.
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Claude Code auto-loads hooks/hooks.json. Declaring it explicitly in
manifest.hooks causes: 'Duplicate hooks file detected ... already-loaded'.
Same pattern as agents: manifest should only reference *additional* hook
files beyond the default.
Two separate schema issues blocked `/plugin marketplace add ./plugins`
followed by `/plugin install semantica@semantica-local`:
1. `marketplace.json` was missing the required top-level `owner` object.
Claude Code rejects with: `owner: Invalid input: expected object,
received undefined`.
2. `plugin.json` declared `"agents": "./agents"` (string), but Claude
Code's manifest schema rejects non-array `agents` with:
`Validation errors: agents: Invalid input`. Auto-discovery from
the default `agents/` directory works when the field is omitted,
provided agents are flat `<name>.md` files with frontmatter (Claude
Code's subagent convention) rather than `<name>/AGENT.md`
subdirectories.
Changes:
- add `owner` object to `marketplace.json`
- drop `agents` field from `plugin.json` (falls back to auto-discovery)
- rename `agents/<name>/AGENT.md` -> `agents/<name>.md` (frontmatter
content is unchanged, just the path)
After this, the documented local-install flow succeeds end-to-end.
- Replace deepseek.Client with openai.OpenAI(base_url="https://api.deepseek.com/v1")
in DeepSeekProvider._init_client(); the deepseek PyPI package has no Client class
- Add self.base_url = "https://api.deepseek.com/v1" to DeepSeekProvider.__init__()
(missing from original PR; caused AttributeError on every instantiation)
- Fix verbose_mode NameError in BaseProvider.generate_typed() instructor path
- Update pyproject.toml: llm-deepseek extra now declares openai>=1.0.0
- Update _init_client warning message to reference openai library
- Add 19 tests in tests/semantic_extract/test_pr482_deepseek_openai.py
- Update CHANGELOG.md
Co-authored-by: liling <liling@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
- Replace list.sort() on every upsert with bisect.insort() — O(log n) per
insert instead of O(n log n); bulk rebuild still sorts once at the end
- Replace list.remove() in remove() with bisect.bisect_left + pop() — O(log n)
find instead of O(n) scan
- Wrap handle_graph_mutation() index mutations in self._lock — mutation bridge
fires from a background thread and was racing concurrent search/rebuild calls
- Drop source/target upserts in add_edge() — edges don't change node text so
the index documents are identical; removes unnecessary cache invalidation
- Sort tag values in _cache_key() — ["a","b"] and ["b","a"] now share a cache
entry since _passes_filters() uses set intersection (order-independent)
- Restore @app.get("/") root handler missing from this branch vs main
Co-authored-by: ZohaibHassan16 <zohaibhassan16@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
- Resolve all merge conflict markers in provenance.py, app.py, .gitignore
- Revert broken session.get_nodes()/get_edges() to session.graph.nodes/edges
- Keep undirected=True ego_graph fix for upstream ancestor traversal
- Add direction field to ProvenanceEdge (upstream/downstream/lateral)
- Group lineage edges in _render_markdown by direction section
- Move ProvenanceNode/ProvenanceEdge/ProvenanceResponse to schemas.py
- Restore complete router import set in app.py (sparql, vocabulary, etc.)
Co-authored-by: Sameer6305 <sameer6305@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
- Add _ttl_block() helper to accumulate all predicate-object pairs before
writing, producing a single valid Turtle subject block terminated by one
period — eliminates the bug where rdfs:subClassOf / domain / range were
appended after a closed '.' block
- Add missing data_properties loop to _export_owl_turtle so
owl:DatatypeProperty declarations are no longer silently dropped
- Add _escape_ttl_str() to escape quotes, backslashes, newlines, carriage
returns, and tabs inside Turtle string literals (rdfs:label, rdfs:comment,
owl:versionInfo)
- Unify optional-field null checks to consistent x = prop.get(); if x: pattern
- Add 43 tests in tests/export/test_owl_exporter.py covering syntax validity,
data properties, string escaping, null handling, and header output
- Update CHANGELOG.md with [Unreleased] entry
Closes#478
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend PathResponse with hop_count (len(path)-1) and distance_band
("direct"|"near"|"mid-range"|"distant") as first-class API fields
- Add classify_path_distance() to semantica/utils/helpers.py as the
single source of truth for hop-count thresholds; both the route and
the visualizer import from it, eliminating duplicate threshold logic
- Populate hop_count and distance_band in find_path route via
classify_path_distance(); remove local _classify_distance() copy
- Add highlight_path: list[str] param to KGVisualizer.visualize_network;
path edges rendered as a distance-aware orange trace (opacity and
stroke width scale from direct→distant: 1.0/4px to 0.35/1.5px)
- Fix bidirectional edge lookup: only forward pairs (A→B) along the
path are added to path_edge_set; reverse back-edges in directed
graphs are no longer incorrectly highlighted
- Add logger.warning when highlight_path contains node IDs absent from
the layout position map, surfacing silent no-op mismatches
- Extend frontend PathResponse type in GraphInspectorPanel.tsx and
GraphWorkspaceShell.tsx with hop_count: number and distance_band
literal union to match the updated API contract
- Add 10 new tests: 2 API-level and 8 unit tests covering all four
band boundaries (0, 1, 2, 3, 4, 6, 7, 20 hops); 104 explorer tests
pass, 0 failures introduced
- Update CHANGELOG.md
- PathFinder.bfs_shortest_path() and dijkstra_shortest_path() gain a
directed: bool = True parameter. When False, a temporary undirected
view (graph.to_undirected()) is used for traversal only; the original
directed edges are preserved and returned in the response.
- _make_undirected_view() helper added to PathFinder; falls back safely
for non-NetworkX graph types.
- GET /api/graph/node/{id}/path exposes ?directed=false query param.
- PathResponse gains a directed: bool field echoing the mode used.
- Route now returns 404 on empty path (previously returned 200 with
path: []).
- 21 new tests: 12 unit (TestBidirectionalPathFinding) + 9 API-level
(TestBidirectionalPathRoute). All 120 tests pass.
- CHANGELOG updated under [Unreleased].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add semantica/kg/knowledge_graph.py with KnowledgeGraph dataclass
(entities, relationships, metadata) plus __len__ and __bool__ helpers
- Export KnowledgeGraph from semantica/kg/__init__.py
- Add KGVisualizer._convert_knowledge_graph() for explicit, non-mutating
conversion from KnowledgeGraph to internal dict format
- Route isinstance(graph, KnowledgeGraph) through _convert_knowledge_graph
inside _normalize_graph so all five visualize_* entry points accept
KnowledgeGraph directly without any manual conversion
- Add TestFormalKnowledgeGraphType (15 tests)
Closes#471
- routes/graph.py: raise HTTPException(404) for missing nodes; wrap path_finder call in try/except → HTTPException(404)
- routes/decisions.py: raise HTTPException(404) on chain, compliance, precedents sub-routes
- routes/annotations.py: raise HTTPException(404) on create (missing node) and delete (missing annotation)
- routes/enrich.py: raise HTTPException(404/503/422) for missing nodes, unavailable services, and extraction errors
- routes/temporal.py: fix TemporalPatternDetector method name detect_patterns → detect_temporal_patterns
- explorer/app.py: root / now serves built index.html when available, falls back to minimal HTML shell; add HTMLResponse import
- tests/explorer/test_explorer_api.py: test_extract accepts 503 (spacy/transformers not installed is a valid service state)
All 45 explorer API integration tests pass.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Clarify plugin README install and usage steps
* feat(explorer): add welcome message to root endpoint and bump version to 0.4.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(plugins): update all plugin READMEs for v0.4.0 with full platform list
- Rewrite main community guide with platform table (8 plugins), skills/agents
inventory, Knowledge Explorer section, and per-platform install steps
- Add v0.4.0 badge and Knowledge Explorer section to VS Code, Cline,
Continue, Windsurf, and OpenClaw READMEs
- Fix inconsistent tool count (12 → 17) across all READMEs
- Bump Python requirement from 3.8+ to 3.10+ across all plugins
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs: add PR description for utils → main
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: remove PR_DESCRIPTION.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove duplicate integrations table from bottom of README
- Move Agentic Frameworks section to top alongside AI tools table
- Show only Agno as supported; list LangChain, LangGraph, CrewAI, LlamaIndex, AutoGen, OpenAI Agents SDK, Google ADK as coming soon
- Add logo icons for all agentic frameworks matching existing plugin style
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add integrations/openclaw/ with OpenClawKGTool (REST) and
OpenClawMCPConfig (mcporter.json generator)
- Add plugins/.openclaw-plugin/ bundle (plugin.json, marketplace.json,
README) with MCP + native tool support
- Add OpenClaw badge to README header
- Reorganize "Works With Every AI Tool" table into labeled groups:
Native Plugin Bundle, MCP Server + Plugin, MCP Server, REST API
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
KGVisualizer.visualize_network() (and sibling methods) only accepted a raw
dict. Passing a KnowledgeGraph object — the natural output of
GraphBuilder.build() — silently returned without rendering.
Added _normalize_graph() which duck-types the input: dicts pass through
unchanged; any object exposing .entities / .relationships attributes is
converted to the canonical dict form; anything else raises a clear
ProcessingError naming the offending type.
_normalize_graph() is called as the first statement in visualize_network(),
visualize_communities(), visualize_centrality(), visualize_entity_types(),
and visualize_relationship_matrix().
Also adds 21 tests in tests/visualization/test_kg_visualizer_normalize_graph.py
covering the helper directly, the end-to-end regression for #458, and
a guard that every public method routes through _normalize_graph.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
End-to-end example using DatalogReasoner, GraphBuilder, ContextGraph,
GraphAnalyzer, ExplanationGenerator, DatalogFact, and DatalogRule.
Covers ancestor query, KG dependency analysis, RBAC policy, and org hierarchy.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 21:43:04 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Tools grid:
- Claude Code/Cursor/Codex: 'Native plugin' (plugins/ dirs exist in repo)
- All other tools: 'REST API' (no MCP server impl in codebase — Semantica
has an MCP CLIENT for ingesting from MCP servers, not an MCP server)
- Codex CLI added back (has real plugin bundle at plugins/.codex-plugin/)
Plugin Bundles section:
- Full table of all 17 skills with descriptions matching SKILL.md files
- Full table of all 3 agents (kg-assistant, decision-advisor, explainability)
- Hooks entry referencing plugins/hooks/hooks.json
MCP Client section:
- Correct framing: MCPClient in semantica/ingest/mcp_client.py pulls
data FROM MCP servers into KG (not an MCP server itself)
- Code snippet + supported schemes
REST API Server section:
- Lists all 10 route modules from semantica/explorer/routes/ with paths
- WebSocket /ws endpoint
- Health check
Agno integration section:
- Expanded to table showing all 5 actual files in integrations/agno/
with class names and descriptions matching source code
AI Coding Tools table:
- Corrected connection types and setup notes to match actual code
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add 'AI Coding Tools & IDEs' table under Integrations listing every
tool from the visual grid with connection type and setup note:
Claude Code, Cursor, Windsurf, Claude Desktop, VS Code, GitHub
Copilot, Cline, Roo Code, Continue, Goose, Kilo Code, Aider,
Amazon Q, Zed, Claude SDK, REST API (109 endpoints)
- Add Neo4j to Graph Databases list (was in modules but missing here)
- Add Email and Repository ingestors to Data Sources
- Expand LLM Providers: add Groq, HuggingFace, Ollama entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AI tools grid (removed Gemini CLI, Codex CLI; added VS Code, GitHub
Copilot, Continue, Amazon Q, Zed — all confirmed MCP-supporting tools
with significant user bases in 2026):
Row 1: Claude Code, Cursor, Windsurf, Claude Desktop, VS Code,
GitHub Copilot, Cline, Roo Code
Row 2: Continue, Goose, Kilo Code, Aider, Amazon Q, Zed,
Claude SDK, Any agent REST API
Agentic frameworks grid (added LangGraph and OpenAI Agents SDK, expanded
to 8 entries): Agno, LangChain, LangGraph, LlamaIndex, AutoGen, CrewAI,
OpenAI Agents SDK, Google ADK
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New '🖥️ Semantica Knowledge Explorer' section placed after Plugins,
with a workspace-tab table (Graph, Timeline, Decisions, Registry,
Entity Resolution, KG Overview, Ontology), a 4-line quick-start
snippet, requirements line, and a pointer to explorer/README.md
- Added explorer/ row to the detailed Modules table with a link
- Added explorer/ bullet to the condensed Modules list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- GraphWorkspace: set isRunningPredictions=true before link-prediction fetch
and false in finally block; pass isRunningPredictions prop to
LazyGraphInspectorPanel so the inspector button disables and shows a
spinner during the request (was declared but never wired — broke
noUnusedLocals TypeScript build)
- DecisionWorkspace: add AbortController to the /api/decisions useEffect
so the fetch is cancelled on unmount; add per-call AbortController to
handleSelectDecision for /api/decisions/:id/chain; add res.ok guards
before .json() on both fetches; encodeURIComponent on decision_id to
prevent path-injection edge cases
- index.css: add missing @keyframes skeleton-pulse rule (0%/100% opacity
0.45, 50% opacity 0.85) — KGOverviewTab skeletonBarStyle referenced
this animation but it was never defined, leaving skeleton bars static
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents all 12 vulnerability fixes (CRITICAL→LOW), 4 post-review bug
fixes, and CodeQL infrastructure changes under [Unreleased] following
the existing Keep a Changelog format.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(agent_memory): implement MemoryItem.to_dict() / from_dict() for safe JSON
persistence — timestamps serialised via isoformat(), embeddings dropped (not
JSON-safe, regenerated on demand); save() and load() now round-trip correctly
without TypeError or AttributeError (Bug #1)
fix(sparql): add asyncio.Semaphore(_SPARQL_MAX_CONCURRENT=4) around graph.query
so timed-out threads cannot exhaust the default ThreadPoolExecutor; add
`truncated: bool` field to SparqlResponse so callers know when the 5 000-row
cap was hit (Bug #2)
fix(export_import): trim _ALLOWED_IMPORT_EXTENSIONS to {.json, .csv} — the only
formats the handler actually parses; removes .graphml/.gexf/.ttl/.rdf that
passed the allowlist check but hit a hard 422 inside the handler (Bug #3)
fix(codeql): remove blanket rule-ID auto-dismiss job; replace with a commented
template for pinning specific alert numbers — prevents future real alerts of
the same rule being silently suppressed (Bug #4)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-12 14:35:30 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
semantica/static/ is already in .gitignore but the 19 newly-hashed
build artifacts introduced by the main merge were still tracked.
Runs git rm --cached to complete the untracking so future frontend
builds do not create dirty working-tree diffs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bug 1 — non-string IDs crash store():
_resolve_iri() called .startswith() directly on local, causing AttributeError
when upstream graph builders emit integer entity/relationship IDs. Fixed by
coercing local to str() at entry; None/empty returns a safe urn: sentinel.
Bug 2 — prefixed W3C terms mis-resolved under base_uri:
Values like 'owl:Thing' and 'xsd:date' were not recognised as absolute IRIs
and with base_uri set were rewritten to e.g. https://example.com/owl:Thing,
corrupting standard OWL/XSD IRIs in stored triples. Fixed by adding a known-
prefix expansion table (xsd/rdf/rdfs/owl/skos/semantica) that is checked before
base_uri is applied, matching the same prefix map already used in blazegraph_store.
Added 5 regression tests covering both bugs: integer IDs with/without base_uri,
owl:Thing domain/range, xsd:date range, and rdfs:/skos: parent class expansion.
store() was minting urn:entity:, urn:class:, and urn:property: URIs for every
bare local name, even when the ontology carried a namespace.base_uri. This made
instance data and ontology class data irreconcilable in SPARQL joins.
- Extract base_uri from ontology.namespace.base_uri (or ontology.uri as fallback)
- Introduce _resolve_iri(local, kind) closure that appends the local name to
base_uri when present, keeping urn: fallback only when no base URI is known
- Apply _resolve_iri consistently for entity URIs, entity types, relationship
predicates, ontology class URIs, parent class URIs, property URIs, and
property domain/range URIs
- Explicit entity.uri values are never overridden
- Added 9 regression tests in TestTripletStoreOntologyNamespace covering all
IRI expansion paths, urn: fallback, explicit URI passthrough, top-level uri
key fallback, and trailing-slash safety
- Added _resolve_datatype_iri() to expand known prefixes (xsd/rdf/rdfs/owl/skos)
to full IRIs instead of blindly wrapping in <...>, fixing invalid SPARQL like
<xsd:integer>
- Validated language tags against RFC 5646 regex to prevent SPARQL injection
via metadata["lang"] values containing whitespace or punctuation
- Validated datatype IRIs for whitespace/special characters before interpolation
- Extended test suite from 7 to 15 cases covering prefix expansion, injection
rejection, and all accepted input forms
This commit transforms the raw 150k-element graph into a high-performance, exploratory UI:
- Implemented Universal Sizing (logarithmic scale based on node degree) and a Procedural Color Mapper (string hashing) to automatically size and colorize categorical data.
- Built the 'Focus Mode' engine using Sigma reducers. Hovering or clicking a node instantly isolates it and its 1-hop neighbors while muting the canvas, eliminating visual noise.
- Applied an enterprise-grade visual style, featuring deep radial background gradients, structural grid overlays, and a sliding glassmorphism metadata HUD.
- Shifted from DOM-bound state mutations to direct WebGL render pipelines to maintain visual performance.
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — wait for the issue to be assigned to you before opening a PR, to avoid duplicate work.
## Description
<!-- Provide a clear description of your changes -->
# Local composite actions (./x) and Docker image refs (docker://...) use a
# different pinning mechanism and aren't in scope here.
[["$content"=~ uses:\ +\./ ]]&&continue
[["$content"=~ uses:\ +docker:// ]]&&continue
if[["$content"=~ $USES_PATTERN]];then
repo="${BASH_REMATCH[1]}"
ref="${BASH_REMATCH[3]}"
checked=$((checked +1))
if[[ ! "$ref"=~ ^[0-9a-fA-F]{40}$ ]];then
echo"::error file=$file,line=$lineno::$repo is pinned to '$ref', not a full commit SHA. Mutable tags/branches can be silently re-pointed (see the LiteLLM/Trivy 2026 incident) - pin to a commit SHA instead."
echo"::warning file=$file,line=$lineno::$repo@$sha has no trailing '# vX' comment recording which tag it corresponds to - add one for auditability."
continue
fi
resolved=$(gh api "repos/$repo/commits/$tag" --jq '.sha' 2>/dev/null)
if[[ -z "$resolved"]];then
echo"::error file=$file,line=$lineno::Could not resolve '$repo@$tag' via the GitHub API (rate limit, deleted tag, or typo). Treating as unverifiable = failure."
fail=1
continue
fi
if[["$resolved" !="$sha"]];then
echo"::error file=$file,line=$lineno::$repo is pinned to $sha but tag '$tag' now resolves to $resolved. Update the pin or the comment."
# Guard 1: fail loudly if Safety exited before writing a report at all
# (network error, API auth failure, tool crash). Without this check a
# missing or empty file causes jq to fall back to "0", making a broken
# scanner indistinguishable from a clean scan.
if [ ! -s safety-report.json ]; then
echo "::error::Safety scan produced no report (safety-report.json is missing or empty). Treating as failure — check for network errors, API auth failures, or Safety crashes in the logs above."
# Guard 2: ensure VULNS is a non-negative integer before the -gt
# comparison. "null" (missing/null key) or "" (jq parse failure) would
# cause bash's -gt to throw an arithmetic error and fall through to the
# success branch — the same silent-pass bug as a missing file.
if ! [[ "$VULNS" =~ ^[0-9]+$ ]]; then
echo "::error::Safety report exists but 'vulnerabilities' is missing or non-numeric (got: '${VULNS}'). The report may be malformed or Safety may have written an error-only JSON. Treating as failure."
exit 1
fi
if [ "$VULNS" -gt 0 ]; then
echo "❌ Security vulnerabilities found: $VULNS"
echo "CI will fail to prevent merging of vulnerable dependencies"
lines.push('<details>', '<summary>Show all findings</summary>', '');
lines.push(...items);
lines.push('', '</details>');
} else {
semgrepResults = '## No Security Patterns Found\\n';
lines.push(...shown);
}
} catch (e) {
semgrepResults = '## Semgrep scan completed\\n';
return lines.join('\n');
}
// Create summary comment
const comment = `# 🔒 Security Scan Results\\n\\n${safetyResults}\\n\\n${banditResults}\\n\\n${semgrepResults}\\n\\n---\\n\\n*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*\\n\\n📊 **Security Policy**: CI fails on vulnerabilities and HIGH severity issues.`;
.map((issue) => `- \`${issue.test_name}\` in \`${issue.filename}:${issue.line_number}\``)
);
const semgrepSection = renderSection(
'Semgrep — static analysis patterns',
'semgrep-report.json',
(data) => (data.results || []).map(
(issue) => `- \`${issue.check_id}\` in \`${issue.path}:${issue.start?.line ?? '?'}\``
)
);
const comment = [
'# 🔒 Security Scan Results',
'',
safetySection,
'',
banditSection,
'',
semgrepSection,
'',
'---',
'',
'*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*',
'',
'📊 **Security Policy**: CI fails on Safety vulnerabilities and Bandit HIGH-severity findings. Semgrep findings above are informational and do not block merge.',
> Note: `Docs` and `Cookbook` are external resources maintained outside this file and may change over time. If a link is unavailable, refer to the repository `README.md` and in-repo documentation as canonical fallbacks.
Thank you for your interest in contributing! Every contribution, no matter how small, is valuable. 🎉
⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/sV34vps5hH)**
⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/semantica-agi/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/sV34vps5hH)**
> **New to contributing?** Start with a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/sV34vps5hH) community.
> **New to contributing?** Start with a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/sV34vps5hH) community.
---
## 🚀 Quick Start
1. Find a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue)
2. [Fork Semantica](https://github.com/Hawksight-AI/semantica/fork) & clone the repository
1. Find a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue)
2. [Fork Semantica](https://github.com/semantica-agi/semantica/fork) & clone the repository
3. Make your changes
4. Submit a pull request!
**Need help?** Join [Discord](https://discord.gg/sV34vps5hH) or [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
**Need help?** Join [Discord](https://discord.gg/sV34vps5hH) or [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions)
---
## 🗂️ Working on an Existing Issue
If you want to work on an open GitHub issue, please follow these steps to keep things coordinated and avoid duplicate effort:
1.**Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome.
2.**Comment if you'd like the issue reserved.** Leaving a comment like *"I'd like to take this on"* is the fastest way to get assigned, but it isn't required — maintainers can also assign an issue directly to a contributor (e.g., based on recent activity in the repo) without waiting for a comment first.
3.**Wait for assignment.** A maintainer will assign the issue when appropriate, whether or not a comment was left. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
4.**Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work.
```bash
git checkout -b fix/short-description # or feature/short-description
```
5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue.
> **Why this matters:** Assignment (with or without a comment) helps maintainers track who is working on what and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
Not sure where to start? Try a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH).
---
## 🔀 Duplicate PRs & Issue Priority
When more than one pull request targets the same issue, maintainers triage using this order of priority. These rules decide between PRs that are otherwise following the [assignment workflow above](#-working-on-an-existing-issue) — opening a PR before being assigned doesn't grant priority on its own, and an unassigned PR can still be closed as a duplicate once someone else is assigned to the issue.
1. **Contributor-raised issue with an existing PR.** If the person who opened the issue has also opened a PR for it, that PR is prioritized (they still need to be assigned before it's merged).
2. **Maintainer-raised issue with a claim comment.** If we opened the issue and someone has commented asking to work on it, we assign it to them and check their PR before picking up any other PR for the same issue.
3. **No prior assignment or comment.** If multiple PRs exist and no one was assigned or claimed the issue first, priority goes to whichever contributor has the most consistent activity in the repo over the last 60 days (e.g., merged PRs, substantive reviews, or issue triage participation) — not just PR volume.
4. **Late duplicate PRs.** If a PR is opened after another contributor has already been assigned to the issue, we close the duplicate early rather than let it sit open, and point the author to another open issue (or ask them to check `main` for newly opened ones). This avoids contributors spending time updating a PR that won't be merged.
5. **Overlapping scope.** If a PR covers multiple issues, or there's genuine overlap between competing PRs, maintainers discuss it on [Discord](https://discord.gg/sV34vps5hH) before deciding rather than resolving it unilaterally.
**Why this matters:** it keeps triage predictable, avoids wasted contributor effort on PRs that won't merge, and helps retain active contributors.
---
@@ -78,7 +116,7 @@ Thank you for your interest in contributing! Every contribution, no matter how s
**What:** Report bugs you find
**How:** Use the [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
**How:** Use the [bug report template](https://github.com/semantica-agi/semantica/issues/new?template=bug_report.md)
**Include:** Description, steps to reproduce, expected vs actual behavior, environment details
@@ -88,7 +126,7 @@ Thank you for your interest in contributing! Every contribution, no matter how s
**What:** Suggest new features or improvements
**How:** Use the [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
**How:** Use the [feature request template](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md)
**Include:** Problem statement, proposed solution, use cases
@@ -108,7 +146,7 @@ Thank you for your interest in contributing! Every contribution, no matter how s
## 🎉 Major Release: Distance Intelligence & Ontology Hub Complete
> First stable, full public release of Semantica. Covers everything shipped across three release stages: 0.3.0-alpha (2026-02-19), 0.3.0-beta (2026-03-07), and 0.3.0 stable (2026-03-10).
- **12 Critical Vulnerabilities Fixed**: Eval injection, XXE, SQL injection, and more
- **SSRF Protection**: Comprehensive URL validation and hostname resolution
- **Input Validation**: Enhanced file upload restrictions and format detection
- **CORS & Headers**: Proper security headers and WebSocket protection
---
## v0.3.0 — Stable (2026-03-10)
## 📊 **BY THE NUMBERS**
### Context Graph Feature Completeness
**Temporal Validity Windows** (by @KaifAhmad1)
Nodes and edges now carry first-class `valid_from` / `valid_until` ISO datetime fields. These are stored directly on `ContextNode` and `ContextEdge` dataclasses — not in metadata — and survive full serialisation round-trips through `save_to_file()` / `load_from_file()` and `to_dict()` / `from_dict()`.
- `ContextNode.is_active(at_time=None)` and `ContextEdge.is_active(at_time=None)` — returns `True` if the node/edge is live at the given time (defaults to now). Handles both tz-aware and tz-naive datetime inputs correctly.
- `ContextGraph.find_active_nodes(node_type=None, at_time=None)` — filters the entire graph and returns only nodes within their validity window.
- `add_node(valid_from=..., valid_until=...)` and `add_edge(valid_from=..., valid_until=...)` — pass validity fields directly in the call signature.
- Bug fix: `is_active()` previously crashed with `TypeError` when passed a tz-aware `datetime` (e.g. `datetime.now(timezone.utc)`). Fixed by normalising all inputs to tz-naive UTC via a new `_parse_iso_dt()` helper.
- Bug fix: validity fields were silently lost in `add_nodes()`, `add_edges()`, `to_dict()`, and `from_dict()`. All four paths now correctly preserve and restore them.
**Weighted Multi-Hop BFS** (by @KaifAhmad1)
`ContextGraph.get_neighbors(node_id, hops=1, relationship_types=None, min_weight=0.0)` now accepts a `min_weight` threshold. Any edge with weight below the threshold is skipped during BFS traversal, allowing callers to confine multi-hop queries to high-confidence causal links. Default `0.0` is fully backward-compatible.
**Cross-Graph Navigation** (by @KaifAhmad1)
Separate `ContextGraph` instances can now be linked and navigated between — hierarchically, like separate knowledge domains that reference each other.
- `link_graph(other_graph, source_node_id, target_node_id, link_type="CROSS_GRAPH") -> str` — creates a navigable bridge and returns a `link_id`. Records a dedicated `"cross_graph_link"` typed marker node internally (not a phantom `"entity"`) and a marker edge.
- `navigate_to(link_id) -> (other_graph, target_node_id)` — jumps to the target graph and entry node for a given link.
- `graph_id` field — each `ContextGraph` now carries a stable UUID so instances can identify each other across save/load.
- `save_to_file()` — now writes a `links` section alongside nodes and edges, containing `link_id`, `source_node_id`, `target_node_id`, and `other_graph_id` for every cross-graph link.
- `load_from_file()` — restores `graph_id` and populates `_unresolved_links` from the `links` section.
- `resolve_links(registry: Dict[str, ContextGraph]) -> int` — reconnects unresolved links post-load. Pass `{graph_id: graph_instance}` for each linked graph; returns the count of successfully resolved links. `navigate_to()` raises a clear `KeyError` with a `resolve_links()` hint if called before resolution.
- Bug fix: the previous implementation auto-created the synthetic marker target as an `"entity"` node (phantom pollution). Fixed by explicitly pre-creating a `"cross_graph_link"` typed `ContextNode` before the marker edge.
- 14 new tests in `tests/context/test_cross_graph_navigation.py` covering all scenarios including full save/load round-trips with partial registry resolution.
**Other Fixes** (by @KaifAhmad1)
- `PipelineBuilder.add_step()` return type annotation corrected from `"PipelineBuilder"` to `"PipelineStep"` — the implementation was already correct; only the annotation and docstring were stale.
- `test_hybrid_search_performance` timing computation fixed — now accumulates a true `search_times` list instead of reusing the last loop iteration's `start_time`; threshold relaxed to `< 5.0s` for real `sentence-transformers` (384-dim) latency on development machines.
- `_parse_relation_result` in `methods.py` — unmatched subjects/objects now produce a synthetic `UNKNOWN` entity instead of silently dropping the relation. All co-founders returned by the LLM are preserved in the output.
- Duplicate relation fix — an orphaned legacy block that appended every relation twice has been removed.
- `extraction_method` parameter added — typed extraction paths now correctly record `"llm_typed"` in relation metadata instead of `"llm"`.
- `_match_pattern` in `reasoner.py` rewritten — splits patterns on `?var` placeholders first, then escapes only literal segments. Pre-bound variables resolve to exact literals, repeated variables use backreferences, non-greedy `.+?`prevents over-consumption of separators.
- Added `tests/reasoning/test_reasoner.py` (4 tests) and `tests/semantic_extract/test_relation_extractor.py` (6 tests).
- `RDFExporter` now accepts `"ttl"`, `"nt"`, `"xml"`, `"rdf"`, and `"json-ld"` as format aliases in `export_to_rdf()`. Aliases resolve before format validation — zero public API changes.
@@ -24,7 +24,7 @@ Security vulnerabilities should be reported privately to prevent potential explo
### 2. Report Security Issue
Create a [GitHub Security Advisory](https://github.com/Hawksight-AI/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix.
Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us via the security email listed in `SUPPORT.md`.
Include the following information:
@@ -37,7 +37,7 @@ Include the following information:
### 3. Response Timeline
- **Initial Response**: Within 48 hours
- **Initial Response**: Within 24 hours for critical issues; within 48 hours for non-critical issues
- **Status Update**: Within 7 days
- **Resolution**: Depends on severity and complexity
@@ -112,6 +112,101 @@ We regularly update dependencies to address security vulnerabilities. However, y
- Be cautious with external API calls
- Implement proper authentication and authorization
## CI/CD Supply-Chain Security
Semantica's build and release pipeline is explicitly hardened against
CI/CD supply-chain attacks — the class of attack behind the March 2026
LiteLLM/Trivy incident, where a compromised third-party Action with a
**mutable tag** was used to steal a long-lived publishing token, after which
malicious packages were pushed straight to PyPI without ever touching the
source repository. Every control below maps directly to closing one step of
that attack chain.
### Immutable build inputs
- **Risk**: a tag (`@v4`, `@release/v1`) is re-pointed by a compromised upstream maintainer or account, silently changing what every consumer's CI runs.
**Control**: every third-party GitHub Action in every workflow is pinned to a full 40-character commit SHA, with the human-readable tag kept only as a trailing comment (e.g. `actions/checkout@3d3c42e... # v7`).
- **Risk**: a SHA pin drifts out of sync with its own comment over time, or is mistyped.
**Control**: `verify-action-pins.yml` fails closed on any `uses:` reference that isn't a full commit SHA (catching a newly added mutable tag, not just auditing existing pins), resolves every pinned tag via the GitHub API on each workflow change, on every push to `main`, and weekly, and fails if the SHA no longer matches the tag it claims to be — an API lookup that can't be resolved is treated as a failure, not a silent skip.
- **Risk**: manually re-pinning ~15 actions across 8 workflow files on every upstream release is error-prone.
**Control**: Dependabot (`github-actions` ecosystem) opens a grouped PR that bumps the SHA *and* the tag comment together whenever an action releases — pins never require hand-editing.
### Publishing pipeline (highest-privilege path)
- **Risk**: a long-lived `PYPI_TOKEN` sitting in repo/org secrets is exfiltrated by any compromised step.
**Control**: PyPI publishing uses Trusted Publishing (OIDC) (`id-token: write`) — there is no long-lived PyPI credential anywhere in this repository to steal.
- **Risk**: a compromised CI run publishes to PyPI with no human in the loop.
**Control**: the publish job runs only inside a protected `pypi` GitHub Environment with a required human reviewer — every release needs manual approval in the Actions UI before it runs.
- **Risk**: the release job could be triggered from an arbitrary branch/ref.
**Control**: the `pypi` environment's deployment-branch policy is restricted to `v*` tags only.
- **Risk**: a scanner or unrelated job inherits publish-level credentials.
**Control**: `release.yml` sets `permissions: contents: read` at the workflow level; `contents: write` / `id-token: write` / `attestations: write` are granted only to the release job, never workflow-wide.
- **Risk**: two tag pushes race through the publish pipeline simultaneously.
**Control**: `concurrency: group: release-${{ github.ref }}` serializes releases per tag.
- **Risk**: a consumer can't verify a wheel on PyPI actually came from this repo's CI.
**Control**: SLSA build provenance is attested for every release via `actions/attest-build-provenance`, producing a signed, verifiable record of the exact commit and workflow run that produced the artifact (checkable with `gh attestation verify`).
### Repository controls
- **Risk**: unreviewed or force-pushed changes land on `main`.
**Control**: `main` requires 1 approving PR review (stale approvals dismissed on new pushes), resolved conversations, and blocks force-pushes and branch deletion.
- **Risk**: a PR merges without its security/CI checks passing.
**Control**: merges require the `build`, `Analyze Python` (CodeQL), and `security-scan` checks to pass, in strict mode (checks must be re-run against the latest `main`).
- **Risk**: a compromised scanner job reaches secrets or write access.
**Control**: scanning jobs (`CodeQL`, `security-scan.yml`, `security.yml`, `defender-for-devops.yml`) run with read-only, least-privilege permissions (typically `contents: read` + `security-events: write` only) and never share a job, environment, or secret scope with the publish job.
- **Risk**: secrets are committed accidentally.
**Control**: GitHub secret scanning and push protection are both enabled at the repository level, rejecting pushes that contain recognizable credential patterns before they land in history.
## Automated Security Scanning
Every scan below runs continuously in CI, not just at release time:
- **CodeQL** (`security-and-quality` query pack) — Python source: injection, unsafe deserialization, and other code-level vulnerability classes. Runs in `codeql.yml` on every push/PR to `main` and weekly.
- **Bandit** — Python-specific security anti-patterns (hardcoded secrets, unsafe `eval`/`pickle`, weak crypto, etc.); CI fails on any HIGH-severity finding. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **Semgrep** (`p/security` ruleset) — cross-language static-analysis security patterns. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **Safety** — known CVEs in Semantica's own installed dependencies, including optional LLM-provider extras such as LiteLLM; CI fails on any match. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **pip-audit** — independent, PyPA-maintained vulnerability database cross-check against installed dependencies (Safety and pip-audit use different advisory sources, so both run). Runs in `security.yml` weekly.
- **Microsoft Defender for DevOps** (`eslint`, `templateanalyzer`, `terrascan`) — JavaScript/TypeScript lint-security rules and infrastructure-as-code misconfigurations. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
- **Checkov** — Kubernetes, Helm, Dockerfile, GitHub Actions, and secrets-pattern IaC scanning; results upload to the same Security tab as CodeQL. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
- **GitGuardian** — secret-detection check on every pull request, installed as a GitHub App integration (not a repo-local workflow). Runs on every PR.
- **GitHub secret scanning + push protection** — blocks known credential patterns before they're pushed, and continuously scans existing history. Platform-level, continuous.
- **Dependabot** — version/security PRs for Python, Docker, and GitHub Actions dependencies, grouped where relevant to reduce review noise. Configured in `.github/dependabot.yml`, runs weekly for security-relevant packages and monthly for docs dependencies.
- **`verify-action-pins.yml`** — enforces that every Action reference is a full commit SHA (failing on a newly introduced mutable tag) and confirms each SHA still matches the tag it claims to be. Runs on every workflow change, every push to `main`, and weekly.
All SARIF-producing scanners (CodeQL, Checkov, Microsoft Defender) publish
findings to the repository's **Security → Code scanning alerts** tab, giving
a single audit trail across tools rather than scattered per-tool reports.
### Adopting this posture in a fork or downstream deployment
Teams standing up their own instance of Semantica, or forking it for an
internal/regulated deployment, can reuse this posture directly:
1. Keep Dependabot's `github-actions` ecosystem entry — it is what keeps
SHA pins current without manual maintenance.
2. Re-run `verify-action-pins.yml` after re-pointing the repository's Actions
at your own mirrors, if you do so.
3. If you publish your own PyPI package from a fork, configure your own
Trusted Publishing trust relationship on PyPI (Trusted Publishing is
scoped to a specific `owner/repo` + workflow filename) and your own
protected environment with your own required reviewers — these are not
transferable from this repository.
4. Branch protection, environment protection, and repository secret
scanning are repository *settings*, not workflow files — cloning or
forking the repo does **not** copy them. They must be re-applied via
the GitHub UI or API on the new repository.
5. GitHub secret scanning and push protection are repository settings that
don't carry over to a fork either — re-enable both under the new
repository's Security settings, not just Dependabot.
6. GitGuardian runs as a GitHub App installation scoped to this specific
repository, not a workflow file — a fork gets no secret-detection
coverage from it until the app is installed separately on the new repo.
7. CodeQL's `upload-sarif` step in `codeql.yml` only runs meaningfully if
Default Setup is *not* already enabled for the repository (it's designed
to skip gracefully otherwise) — check whether Default Setup or Advanced
Setup is active on the new repository and adjust expectations for where
CodeQL findings show up accordingly.
## Dependency Security Policy
### Regular Updates
@@ -156,8 +251,8 @@ We appreciate responsible disclosure. Security researchers who help us improve t
For security-related questions or concerns:
- **GitHub Issues**: [Create an issue](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix
This document outlines the architecture, directory structure, and usage of the performance benchmarking suite for the Semantica Agentic RAG framework.
## Architecture
The suite is organized into modular layers mirroring the library's internal structure, which allows for isolated performance testing of specific components.
### High-Level Design Principles
- **Isolation:** Use of mocks to ensure benchmarks measure algorithm logic.
- **Virtualization:** A custom `conftest.py` virtualization layer allows tests to run without heavy local dependencies.
- **Pedantic Measurement:** High-iteration counts and statistical rounds to filter out system noise.
## Directory Structure
Based on the current production environment, the suite is organized as follows:
| core_processing/ | Throughput tests for NER, extraction, and graph building. |
| export/ | Serialization benchmarks for JSON, CSV, RDF, and GraphML. |
| infrastructure/ | Support scripts, including the regression comparison engine. |
| input_layer/ | Ingestion, parsing, and splitting performance. |
| normalize/ | Text cleaning, encoding handling, and date normalization. |
| ontology/ | Inference, serialization, and namespace management overhead. |
| output_orchestration/ | Parallelism and execution pipeline management. |
| quality_assurance/ | Deduplication and conflict resolution strategies. |
| results/ | Storage for benchmark JSON outputs and performance baselines. |
| storage/ | Latency tests for Vector stores (FAISS) and Triplet stores (Jena). |
| visualization/ | Computational cost of layout algorithms and chart rendering. |
## Usage
### Running the Suite
To run the full suite and generate a new results file:
```bash
python benchmarks/benchmark_runner.py
```
### Strict Mode (CI/CD)
The suite is designed to integrate with automated pipelines. Using the --strict flag will cause the runner to return a non-zero exit code if a performance regression greater than 15% is detected.
```bash
python benchmarks/benchmark_runner.py --strict
```
### Performance Comparison
The comparison engine (infrastructure/compare.py) uses Z-scores to distinguish between actual performance regressions and environmental noise.
- Regression: Change > 15% AND Z-score > 2.0.
- Noise: Change > 15% but Z-score < 2.0.
### Updating Baseline
When a performance change is intentional (e.g., a more complex but necessary algorithm is added), update the "gold standard" baseline:
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
"\n",
"# Datalog-Style Reasoning\n",
"\n",
"End-to-end guide to Semantica's **`DatalogReasoner`** — a native bottom-up semi-naive fixpoint engine — wired together with `GraphBuilder`, `ContextGraph`, `GraphAnalyzer`, `ExplanationGenerator`, and the supporting data-classes (`DatalogFact`, `DatalogRule`, `InferenceResult`, `Rule`).\n",
"# Everything that transitively depends on the database\n",
"db_deps = sorted(r[\"X\"] for r in dr.query(\"transitive_dep(?X, database)\"))\n",
"print(\"Components that transitively depend on Database:\")\n",
"for c in db_deps:\n",
" print(\" \", c)\n",
"\n",
"# What does pythonsdk transitively depend on?\n",
"sdk_chain = sorted(r[\"Y\"] for r in dr.query(\"transitive_dep(pythonsdk, ?Y)\"))\n",
"print(f\"\\nPython SDK full dependency chain: {sdk_chain}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Part 3 — ContextGraph + `load_from_graph()`\n",
"\n",
"`DatalogReasoner.load_from_graph(graph)` accepts any `ContextGraph` directly: it calls `graph.find_edges()` and `graph.find_nodes()` and converts each result into EDB facts automatically."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ── Build an in-memory ContextGraph ───────────────────────────────────────\n",
"# ContextGraph.add_node / add_edge are the canonical way to build in-memory KGs\n",
"cg = ContextGraph()\n",
"\n",
"# Nodes\n",
"for person in [\"alice\", \"bob\", \"carol\", \"dave\", \"eve\"]:\n",
"## Part 6 — Engine Introspection: DatalogFact & DatalogRule\n",
"\n",
"After reasoning, the engine's internal state is fully accessible via `DatalogFact` and `DatalogRule` data-classes. Use this for auditing, debugging, or downstream export."
"This notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n",
"\n",
"### Key Features\n",
"\n",
"- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n",
"- **OpenCypher Support**: Query using standard OpenCypher syntax\n",
"- VPC with public subnets and Internet Gateway\n",
"- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
"- IAM user with least-privilege access for OpenCypher queries\n",
"- Security group allowing Bolt protocol (port 8182) access\n",
"\n",
"> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
"\n",
"**Outputs:**\n",
"- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
"- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
"- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
"- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
"- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
"| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
"| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
"\n",
"> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
"\n",
"---"
]
"source": "# Amazon Neptune Graph Store\n\n## Overview\n\nThis notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n\n### Key Features\n\n- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n- **OpenCypher Support**: Query using standard OpenCypher syntax\n- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n- **Native ~id Support**: Leverages Neptune's native element ID handling\n- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n\n### Prerequisites\n\n- An Amazon Neptune Database cluster\n- AWS credentials configured (boto3, environment variables, or IAM role)\n- Network access to your Neptune cluster (VPC, security groups)\n- Your public IP address or VPN/office CIDR (run `curl ifconfig.me` to find your public IP), used below to restrict database access\n\n#### Quick Setup with CloudFormation\n\nIf you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n\n```bash\n# Deploy the Neptune stack (takes ~15-20 minutes)\n# Replace 203.0.113.25/32 with your own public IP (run `curl ifconfig.me` to find it)\n# or your office/VPN CIDR. This restricts who can reach the database on the\n# network level - never widen it to 0.0.0.0/0 outside of a short-lived local experiment.\naws cloudformation create-stack \\\n --stack-name semantica-neptune \\\n --template-body file://neptune-setup.yaml \\\n --parameters ParameterKey=ClientCidr,ParameterValue=203.0.113.25/32 \\\n --capabilities CAPABILITY_NAMED_IAM\n\n# Wait for stack creation to complete\naws cloudformation wait stack-create-complete --stack-name semantica-neptune\n\n# Get the outputs (endpoint, port, credentials)\naws cloudformation describe-stacks --stack-name semantica-neptune \\\n --query 'Stacks[0].Outputs' --output table\n```\n\nThe template creates:\n- VPC with public subnets, Internet Gateway, and VPC Flow Logs (to CloudWatch Logs)\n- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n- IAM user with least-privilege access for OpenCypher queries\n- Security group allowing Bolt protocol (port 8182) access only from the `ClientCidr` you specify\n\n> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager. The `ClientCidr` parameter is required (no default) precisely so the database is never silently exposed to the whole internet.\n\n**Outputs:**\n- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n\n**Cleanup:**\n```bash\naws cloudformation delete-stack --stack-name semantica-neptune\n```\n\n**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n\n| Resource | Cost (USD) |\n| --- | --- |\n| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n| Storage (10 GB) | ~1/month |\n| I/O requests | ~1-5/month |\n| Public IPv4 address | ~3.60/month (0.005/hr) |\n| VPC Flow Logs (CloudWatch Logs) | ~1-2/month depending on traffic |\n| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n\n> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n\n---"
Graph Retrieval-Augmented Generation (GraphRAG): A New Era for Intelligent Search
GraphRAG is an advanced technique that combines the retrieval capabilities of vector databases with the structural reasoning of knowledge graphs. Unlike traditional RAG, which relies solely on vector similarity, GraphRAG leverages the relationships between entities to provide more contextually accurate and comprehensive answers.
Key Components:
1. Knowledge Graph: A structured representation of data where nodes represent entities and edges represent relationships.
2. Vector Search: Finds semantically similar text chunks.
3. Graph Traversal: Navigates the knowledge graph to find related entities that might not be semantically similar but are structurally relevant.
Benefits:
- Improved Context: By following relationships, the system can understand the broader context of a query.
- Multi-hop Reasoning: Can answer complex questions that require connecting multiple pieces of information.
- Reduced Hallucinations: Grounding answers in a verified knowledge structure reduces the likelihood of generating false information.
{"entities":[{"id":"python_org","name":"Python Software Foundation","type":"Organization"},{"id":"guido_van_rossum","name":"Guido van Rossum","type":"Person"}],"relationships":[{"source":"guido_van_rossum","target":"python_org","type":"FOUNDED"}]}
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)\n",
"\n",
"# Drug Discovery Pipeline - Vector Similarity Search\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates a **complete drug discovery pipeline** using Semantica's modular architecture. We'll use individual modules directly to build a comprehensive system for drug-target interaction prediction using vector similarity search and knowledge graphs.\n",
"# Map to biomedical categories based on context\n",
"drugs = [e for e in all_entities if e.label == \"PRODUCT\" or (e.label == \"ORG\" and any(kw in e.text.lower() for kw in [\"drug\", \"pharma\", \"medication\"]))]\n",
"proteins = [e for e in all_entities if e.label == \"ORG\" or (e.label == \"PRODUCT\" and any(kw in e.text.lower() for kw in [\"protein\", \"enzyme\", \"receptor\", \"kinase\", \"target\"]))]\n",
"\n",
"print(f\"Extracted {len(drugs)} drugs and {len(proteins)} proteins\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Extracting Drug-Target Relationships\n",
"\n",
"Extract relationships between drugs and proteins to understand drug-target interactions.\n"
"print(\"Exported knowledge graph to JSON and GraphML formats\")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
KaifAhmad1
