mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
explainability-scope-clarification
2343
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 KaifAhmad1
|
50927f99b5 |
docs: surface explainability scope note near the top of the README
Moves a concise version of the system-level vs. foundation-model explainability clarification up next to the opening pitch, so it's visible before readers scroll to the high-stakes-domains section. |
||
|
KaifAhmad1
|
476237952d |
docs: clarify explainability is system-level, not foundation-model internal
Adds a consistent scope note to README and docs (concepts, FAQ, index) stating Semantica does not expose or reconstruct an LLM's internal reasoning/chain-of-thought. It explains and audits the AI system around the model: context, provenance, policies, decisions, and execution history. |
||
|
hariandZohaib Hassnain
|
70aa9d01bf |
fix(normalize): validate symbol currencies (#940)
* fix(normalize): validate symbol currencies Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com> * fix(normalize): match currency codes by token boundaries Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com> --------- Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com> Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> |
||
|
Mohd Kaif
|
c53ca4e84b |
docs: formalize issue assignment and duplicate-PR triage workflow (#1030)
* docs(contributing): formalize issue assignment and duplicate-PR triage workflow Comments are no longer required before an issue can be assigned - maintainers may assign directly based on recent activity. Also documents the duplicate-PR priority order for triage (contributor PR, claimed issue, activity tiebreak, late duplicates, overlapping scope). * docs(contributing): clarify assignment precedence and define activity tiebreak Addresses Qodo review feedback on PR #1030: the duplicate-PR priority list now states these rules apply on top of the assignment workflow (opening a PR pre-assignment doesn't grant priority), and the "most active" tiebreak now specifies a concrete 60-day window and signals instead of being subjective. |
||
|
pravit-ampandPravit Ampapathini
|
15171fd31a |
fix(parse): import get_progress_tracker in ExcelParser (#1016)
ExcelParser.__init__ called get_progress_tracker() without importing it, so every instantiation raised NameError and the class was unusable. The existing test imported ExcelParser but never constructed it, so nothing caught it. Same defect as #530 in SimilarityCalculator, which was fixed without sweeping the rest of the codebase. Add construction coverage for every parser exported from semantica.parse, driven off __all__ so later additions are covered automatically. These live outside test_parse_comprehensive.py, whose setUp patches get_progress_tracker into each parse module and would mock away the interaction under test. Closes #1014 Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> |
||
|
Guofang.Tang
|
8177d88753 |
fix(kg): preserve isolated nodes in graph analytics (#1011)
* fix(kg): preserve isolated nodes in graph analytics * fix(kg): support node fallbacks and community payloads --------- |
||
|
Shinde vinayak rao patil
|
d94d8f6ab8 |
Feat/crewai integration (#988)
* feat(crewai): add first-class CrewAI integration (#962) Add native CrewAI support so Crew agents can share a ContextGraph and AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching the existing agno integration pattern. - SemanticaKGTool: 5 KG actions (extract_entities, extract_relations, add_to_graph, query_graph, find_related) with sync run()/async arun() - SemanticaDecisionTool: 5 decision-intelligence actions (record_decision, find_precedents, trace_causal_chain, analyze_impact, check_policy) over AgentContext - SemanticaKnowledgeSource: serializes a ContextGraph into crew knowledge storage; bridges legacy load_content() and current validate_content()/aadd() contracts for crewai>=0.80.0 - All classes degrade gracefully when crewai is absent - New pip extra crewai=... included in the all bundle - 70 new tests (stub-based present-case + subprocess degradation path) - Docs: integrations/crewai.md, docs.json nav, README matrix updates * fix(crewai): harden tools against real Semantica dataclass shapes (#962) Bugs found during live testing with crewai 1.15.16: - SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses ('str' object has no attribute 'end_char'): string names were passed to extract_relations(entities=...), which requires Entity objects, and the tool read .name/.source/.target instead of Entity's .text/.label and Relation's .subject/.object. Add shape-agnostic field helpers. - SemanticaDecisionTool() created an AgentContext without a knowledge_graph, so _decision_backend was never set and record_decision raised 'Decision tracking is not enabled'. Wire in a ContextGraph. - record_decision hard-failed when the agent omitted optional fields; fall back to category='general', reasoning='agent decision', outcome='recorded'. Add tests covering real Entity/Relation dataclass shapes and the live auto-created AgentContext path (now 77 crewai tests, 212 total). * fix(crewai): make find_related traverse edges undirected (#962) ContextGraph.get_neighbors only follows outgoing edges, so a node whose only edge is incoming (A -> B) reported no related concepts. Rebuild a bidirectional adjacency from find_edges() in SemanticaKGTool._find_related so 'related' honors both directions. * fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962) - Exclude live graph/context/extractor state from JSON serialization (model_dump(mode="json")) so CrewAI checkpointing no longer raises PydanticSerializationError; model_post_init self-heals defaults on restore - query_graph now searches node content via graph.query() plus id/type - trace_causal_chain returns an explicit error when causal tracing is unavailable instead of substituting similarity precedents; call trace_decision_causality(..., max_depth=...) with the correct kwarg name - find_precedents propagates max_precedents/limit to the backend instead of being silently capped at 10 - Serialize add_to_graph batches under a module lock to prevent concurrent double-counting; skip nameless entities instead of creating repr()-junk nodes - aadd() runs CPU-bound serialization in a thread executor - Mirror crewai args_schema serialize/restore in the conftest stub and add serialization regression tests (crewai: 92 tests) * fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962) - _eval_rule now coerces rule values type-aware: bool("false") was truthy, so 'enabled == false' reported a violation for enabled=false, and string datums like "0.90" were compared lexicographically instead of numerically - _trace_causal_chain no longer raises AttributeError (which escaped _run) when the decision context lacks knowledge_graph; returns honest error JSON - SemanticaKnowledgeSource storage failures log an actionable ERROR; without a configured crew embedder agents previously retrieved nothing silently - add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a process-global one: independent graphs no longer serialize each other and re-entrant extractor callbacks cannot deadlock - entity/relation confidence=None normalizes to 1.0 instead of failing the whole extraction with float(None) - add subprocess integration test against real crewai covering Crew-level serialization round-trip and checkpoint restore (stub tests cannot see it) - docs: embedder requirement for SemanticaKnowledgeSource; resume contract note * fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962) Re-verification against real crewai showed the embedder-missing failure raises ValueError even though storage IS wired, so the old except-ValueError branch mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure. Distinguish by storage presence instead of exception type: storage is None -> DEBUG keep-in-memory (legitimate standalone use); storage wired but save() raises -> actionable ERROR. Add regression test mirroring real crewai's ValueError-on-missing-embedder behavior. * fix(crewai): expose run()/arun() entry points in degraded mode (#962) The public crewai contract is run()/arun(); without crewai installed they were missing (only the private _run existed), so the documented 'usable without crewai' path raised AttributeError at the entry point. Define them in degraded mode only, leaving crewai's BaseTool implementations untouched when present. Extend the degradation subprocess test to exercise run() and arun(). * fix(crewai): standardize query shape, field-name rules, and restore-state flag - _query_graph: id/type matches now return the same schema as content matches (id/type/label/content/score) instead of a bare list - _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys (e.g. "risk-score >= 0.9") are addressable in policy rules - add had_live_state/reconstructed_state so checkpoint-restored tools and knowledge sources signal that their live graph/context was lost and an empty one reconstructed; knowledge source no longer hides the loss by eagerly rebuilding its graph inside __init__ (pydantic calls __init__ during model_validate) * fix(crewai): address Qodo review — confidence errors, string trim, holistic availability - record_decision: stop calling float() in _run, so malformed confidence values surface as JSON errors (via _record_decision's handling) instead of crashing the tool - _coerce_value: return the stripped string for non-numeric literals so whitespace-padded decision_data fields match policy rules - centralize crewai availability in _availability.py so the exported CREWAI_AVAILABLE flag is holistic across tools and knowledge source (previously each module probed crewai independently and the package flag came from decision_tool only) * ci: regenerate requirements-ci.txt for the crewai extra The crewai extra in pyproject.toml brings in crewai, crewai-tools and transitive deps (chromadb, lancedb, ...). Recompile with uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes. * ci: keep crewai out of the locked CI dependency set crewai (all versions) hard-requires chromadb~=1.1.0, which carries a pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c) with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in the 'all' extra failed pip-audit and the safety check on requirements-ci.txt. - drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is unchanged and still installs crewai) - stop listing crewai-tools in the extra: the integration only uses crewai core (BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps - regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0 vulnerabilities, staleness check matches * docs(crewai): document crewai extra scope and chromadb CVE-2026-45829 - CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not part of the 'all' bundle, with the chromadb CVE-2026-45829 reason - integrations/crewai/README.md: add a security warning that installing the extra pulls chromadb~=1.1.0, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 --------- |
||
|
|
5579851208 |
fix(export): harden YAML export input handling (#958)
* refactor(export): centralize graph-payload key normalization
Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.
Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.
Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.
Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.
CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.
* fix(export): reject non-mapping input to the YAML exporters
export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.
Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.
Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.
* fix(export): address YAML exporter review findings
- semantica/export/yaml_exporter.py — import Sequence from typing
instead of collections.abc. `Sequence[str]` in _require_mapping's
annotation is evaluated at function-definition time; collections.abc.Sequence
only became subscriptable in Python 3.9, so on the 3.8 this project
declares support for, importing this module raised TypeError.
typing.Sequence has supported subscripting since 3.5.3. Mapping stays
imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
and read exported YAML through a context manager instead of an
unclosed yaml.safe_load(open(...)).
* fix(export): reject YAML export payloads with no recognized key
Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.
- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
collections through normalize_graph_payload(), which raises rather than
returning empty collections for an unrecognized mapping. Adopting the
shared resolver rather than repeating the check locally also brings the
'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
from this library's own graph type to YAML, used in
examples/capability_gap_context_graphs_example.py — exports its records
instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
defaulted lookups and had the same defect; it goes through the resolver
too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
its own key set. Schemas are a separate vocabulary with no aliasing, so
_require_recognized_keys lives in this module rather than in the shared
graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
An export_json envelope carries one, so accepting it would readmit the
case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
output directory, so a rejected export leaves nothing behind.
The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.
Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.
Closes #953.
* fix(export): reject payloads whose records resolve to nothing
Addresses the Qodo findings on #958.
Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.
- utils/helpers.py — split the check in two. _require_recognized_keys keeps
the presence rule; _require_nothing_dropped runs after resolution and
refuses a payload that resolved to nothing while an unread key still holds
records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
always carries a populated 'statistics' dict, and an empty graph must stay
exportable, so 'metadata', 'statistics' and 'count' are named as context
rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
both checks through the shared helpers rather than its own copy, so the
two vocabularies cannot drift apart in what counts as a silent-empty
export.
Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.
- export/yaml_exporter.py — the serialization span now says it serialized,
not that it exported, and export() opens its own span around the
filesystem work that stops as failed on error. Nothing reports a completed
export until the bytes are on disk.
Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.
Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.
* fix(export): validate collection values and make Neo4j mappings strict
Two gaps at the boundary the shared normalizer is supposed to own.
_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.
Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.
Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.
* fix(export): close YAML schema and record validation gaps
Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived. Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.
Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects. Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary. Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.
Tests: 101 tests pass across
tests/utils/test_normalize_graph_payload.py
tests/export/test_yaml_exporter_key_recognition.py
tests/export/test_yaml_exporter_input_validation.py
tests/export/test_neo4j_csv_exporter.py
* fix(export): close exception-type and record-shape gaps in normalize_graph_payload
LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).
Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).
normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.
Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.
* docs(changelog): add entry for #958 YAML export input hardening
Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes #956, #952, #953.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
Lakshay Saini
|
115e7965cd |
fix(explorer): gate temporal requests on graph load (#1003)
Explorer was firing temporal requests before the graph even loaded. When the backend is down, /api/graph/nodes fails but the temporal bounds and snapshot effects didn't care , they fired anyway, off in their own corner, ignoring whether the graph actually came up. Every page load with no backend meant three failed requests instead of one, and a scrubber that had nothing to scrub. Added two small predicate functions and gated the temporal effects on them. Basically: don't ask for time-based data until you know the graph itself loaded. An empty graph still counts as loaded, so that case isn't broken. Confirmed with the backend down, before and after: three failing requests down to one. Fixes #982. |
||
|
yzxcj797
|
8639cb9f16 |
fix(seed): pass connection string to DBIngestor and stop mislabeling OSError in load_from_database (#995)
Fix DBIngestor calls in load_from_database , it was never actually reaching the db. execute_query/export_table need the connection string as their first arg, but we were only passing it to the constructor's config dict, which those methods don't read. Every call blew up with a TypeError before connecting. Also split the ImportError/OSError handling , they were caught together so a real connection failure got reported as "module not available", which sent people looking in the wrong place. OSError now surfaces as an actual failure with the original exception chained via `from e`. Fixes #973. |
||
|
|
f1e7e64ad1 |
feat(context): add retraction and purge to ContextGraph (#957)
* feat(context): add retraction and purge to ContextGraph ContextGraph had 56 public methods and none that removed anything: the only option was clear(), which discards the whole graph. Removing one entity meant exporting to a dict, filtering by hand and rebuilding, losing provenance. Add two operations with deliberately different contracts. retract_node/retract_edge close the entity's validity window. The entity stops being active going forward, but state_at() before the retraction still returns it, so decisions recorded against it remain explainable. This reuses the valid_from/valid_until machinery already present rather than adding a new subsystem. purge_node/purge_edge remove the entity outright, from history as well as from the active view, leaving a tombstone that records that a purge happened and why but never the purged content. Scope is this graph only; copies in AgentMemory or a bound vector store are not reached, so it is one step of an erasure workflow rather than the whole of it. Both record themselves through the existing mutation_callback path. MutationRecord already documented REMOVE_NODE/REMOVE_EDGE in its operation vocabulary, so retraction emits UPDATE_NODE and purge emits REMOVE_NODE with no changes required to change_management. Incident-edge lookup scans self.edges rather than _adjacency, which is keyed by source only and would otherwise leave inbound edges pointing at a removed node. Purge updates edges, edge_type_index and _adjacency together so the indexes cannot drift, and clear() now resets the retraction and tombstone records. * fix(context): address review findings on retraction and purge * fix(context): close every duplicate when retracting/purging by edge_id edge_id is content-derived and not yet guaranteed unique (#922, fix pending in #926): two identical add_edge() calls produce two edge objects sharing one id. retract_edge()/purge_edge() resolved "the edge" via the first matching object only, so a duplicate was silently left untouched (still live, still active) while the call returned True and recorded a tombstone/retraction claiming it was fully handled. Repeat purge_edge() calls also silently overwrote the tombstone's reason/purged_at on each partial attempt instead of no-op'ing once nothing remained to purge. retract_node()'s cascade had the same root cause from the other direction: it checked the live _retractions dict mid-loop, so the first duplicate's just-written record made the second look already handled and it was skipped outright, left permanently active. retract_edge()/purge_edge() now act on every edge matching the id under a single record; the cascade's dedup check is snapshotted before the loop starts so within-call duplicates are still closed rather than skipped. Adds TestDuplicateEdgeId (5 tests) reproducing all three paths. * docs(changelog): document retraction/purge feature Adds an Unreleased/Added entry for #955/#957 covering retract_node, retract_edge, purge_node, purge_edge and the get/list accessors, plus the duplicate-edge_id fix caught and applied during review. --------- Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
pravit-amp
|
6df97cf0a0 |
fix(triplet_store): stop CONSTRUCT detection matching inside a leading comment (#951)
CONSTRUCT_QUERY_RE skipped comments with a bare \#[^\n]*, whose trailing * backtracks. For '# CONSTRUCT ...\nSELECT ...' the engine gave back everything after the '#', so the CONSTRUCT inside the comment satisfied the query-form keyword and a SELECT/ASK was reported as a CONSTRUCT. All four SPARQL backends delegate to this regex, so such a query took the CONSTRUCT branch of execute_sparql, which sends Accept: text/turtle and parses the body as Turtle — failing with a misleading 'Failed to parse CONSTRUCT response as Turtle'. Require a comment to reach a line terminator. Both LF and CR are accepted because the SPARQL grammar ends a comment at either; matching only LF would regress CR-terminated comments into false negatives. Add regression tests covering both directions across all four backends.> |
||
|
|
84ce3c5155 |
fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#926)
* fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#922) * docs(changelog): document add_edge dedupe fix Adds an Unreleased/Fixed entry for #922/#926 so the ContextGraph edge-dedupe bug and its fix are recorded per Keep a Changelog format. --------- Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Guofang.Tang
|
b8175ea801 |
fix(kg): make k-shortest path search side-effect free (#1000)
* fix(kg): make k-shortest path search side-effect free * fix(kg): respect traversal direction for edge exclusion |
||
|
|
557e29ee14 |
fix(explorer): repair /api/enrich/extract (always 503) and the /api/decisions routes (always 500) (#886)
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes Two Explorer API endpoints fail on every install. /api/enrich/extract imported extract_entities and extract_relations from semantic_extract.methods, where neither name is defined — that module ships only the per-strategy variants (extract_entities_ml, extract_relations_regex, ...), and nothing re-exports a plain facade. The resulting ImportError was caught and reported as "semantic_extract module not available. Ensure spacy and transformers are installed.", so a wiring bug looked like a missing dependency. The route now calls NamedEntityRecognizer and RelationExtractor directly, the classes the README documents, and feeds the extracted entities into relation extraction rather than re-deriving them. The 503 branch stays for a genuinely absent module. Every /api/decisions* route returned 500 once the graph held a decision: record_decision() stores timestamp as datetime.now().timestamp(), a float, while DecisionResponse types the field as str, so pydantic rejected the value the library itself wrote. A before-mode field validator on DecisionResponse normalizes float, int and datetime inputs to ISO-8601, covering every route that builds the model instead of only the list endpoint. The existing tests missed both: test_extract accepted 503 as a pass, and the decision fixtures are hand-built nodes carrying no timestamp at all. Both are tightened, and a TestRecordedDecisions class exercises the routes against decisions created through record_decision(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(semantic_extract): cache spaCy models instead of loading one per call extract_entities_ml(), extract_relations_similarity() and extract_relations_dependency() called spacy.load() on every invocation, so the model was re-read from disk and re-initialized per call. On a short sentence that is ~120 ms of loading around ~2 ms of work, and successive calls never got cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the pipeline ner_extract step and POST /api/enrich/extract, and process_batch() multiplies it by the number of documents. The module already had a cached loader for one code path — get_nlp_model() and its _nlp_cache global — but the extraction functions bypassed it. Adds load_spacy_model(), a process-level cache keyed by model name behind a lock so concurrent callers do not each start a load, and routes the five call sites through it. Errors are left uncached and propagate unchanged, so the existing OSError fallbacks to pattern extraction still fire. get_nlp_model() keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for similarity work, so its model is not interchangeable with the NER one. Cache entries record the spacy module object they came from. Several tests patch methods.spacy with a mock and assert on load calls; without that guard a name-keyed cache would hand a previous test's mock to a later one. Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm: extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(explorer): harden extraction and timestamp handling * fix(explorer): catch OverflowError/OSError in decision timestamp validator DecisionResponse._normalize_timestamp only guarded against NaN/inf via math.isfinite(), but datetime.fromtimestamp() raises OverflowError or OSError for finite epoch values outside the platform's representable range (e.g. milliseconds stored where seconds were expected). Those exceptions escaped the pydantic validator unhandled, reintroducing an unhandled 500 on /api/decisions* for exactly the bug class this PR closes. Also exclude bool from the numeric branch, since bool is an int subclass and was being silently coerced to epoch 0/1. * docs: add changelog entry for PR #886 (explorer extract/decisions fixes) Documents the extraction 503, decisions timestamp 500, and folded-in spaCy caching fixes, plus the review-round hardening from Sameer6305 and the timestamp overflow/bool fix from this follow-up commit. --------- Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
yzxcj797
|
c1be6dd7dc | docs: fix dead allcontributors emoji-key link (#987) | ||
|
Zohaib Hassnain
|
42afc06003 |
ci: refresh github/codeql-action pin to current v4 (#986)
The pin was 5595ccaf..., but upstream has since moved the v4 tag to ff2f1c62.... The Verify Action Pins workflow flags this drift on every PR that touches any workflow file, regardless of whether that PR changed codeql.yml or defender-for-devops.yml. Verified the new SHA against the GitHub API directly (not just the CI error text) and confirmed .github/scripts/verify-action-pins.sh passes clean locally (40/40 action references OK, exit 0). |
||
|
Yunare MaiaandZohaib Hassnain
|
4513b61e40 |
ci: pin Python dependencies in requirements-ci.txt for reproducible CI (#945)
* ci: pin Python dependencies in requirements-ci.txt for reproducible CI Adds a committed lockfile pinning all transitive dependencies at exact versions (uv pip compile, Python 3.11, all extras — 1581 lines), the Python equivalent of explorer/package-lock.json + npm ci. - CI installs from requirements-ci.txt before building the wheel - CI verifies the lockfile is byte-identical to a fresh compile (fails on staleness after pyproject.toml changes) - CONTRIBUTING documents the regeneration command Closes #938 Signed-off-by: Yunare Maia <yunare@gmail.com> * ci: address Qodo review — security scans use pinned deps, exclude gpu extras - security-scan.yml installs from requirements-ci.txt instead of "./[llm-litellm]" so Safety scans the exact CI/release dependency tree - security.yml runs pip-audit -r requirements-ci.txt for the same parity - lockfile regenerated with --extra all (the cross-platform set) instead of --all-extras, which pulled faiss-gpu/cupy from the Linux-only gpu extra and co-installed faiss-cpu + faiss-gpu in CI - uv pinned to 0.12.1 (the version that generated the lockfile) in CI and CONTRIBUTING so regeneration is deterministic Signed-off-by: Yunare Maia <yunare@gmail.com> * ci: make lockfile staleness check immune to upstream releases The previous check re-resolved pyproject.toml without constraints, so any upstream package release (e.g. boto3 1.43.69 -> 1.43.70) failed CI even when nothing in the repo changed — exactly the time-dependent drift Qodo flagged. The check now re-resolves with requirements-ci.txt as a constraint and compares only version lines, so it detects intentional pyproject.toml changes but ignores upstream releases. CONTRIBUTING updated to match. Signed-off-by: Yunare Maia <yunare@gmail.com> * ci: fix security workflows — install pip-audit; order tooling after pinned deps Security workflow: the pip-audit install step was lost in the rebase conflict merge — pip-audit was invoked but never installed (exit 127). Security-scan workflow: installing safety first let the pinned requirements-ci.txt overwrite its transitive deps (rich), breaking the safety CLI at runtime (RuntimeError: Type not yet supported). Tooling is now installed AFTER the pinned set. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ci): address review — hashes, build isolation, release builds, docs (4/4) ZohaibHassan16's review flagged 4 supply-chain gaps; all addressed: 1. **Release builds now use the lockfile**: release.yml installs requirements-ci.txt and runs `python -m build --no-isolation` so the sdist/wheel is built against the exact tested dependency set. 2. **Build isolation pinned**: [build-system].requires is now setuptools==84.0.0 + wheel==0.48.0 (exact pins, no ranges). 3. **Hashes**: requirements-ci.txt regenerated with --generate-hashes (5,708 sha256 hashes, verified against PyPI). Staleness check updated to strip the `\` line continuations hashes introduce. 4. **CONTRIBUTING.md documents the separate environment**: hashes, never-install-into-dev note, build-system pins, --no-isolation release builds. Validated: stale-check diff clean, hash spot-check matches PyPI. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ci): apply --no-isolation to CI build + align benchmark to Python 3.11 Follow-up to ZohaibHassan16's second review round: 1. ci.yml was still running `python -m build` with build isolation (unpinned setuptools/wheel from PyPI) — now `python -m build --no-isolation` against the pinned deps, matching release.yml. 2. benchmark.yml was on Python 3.12 while the lockfile is compiled for 3.11 — aligned to 3.11 so every workflow runs the same environment. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ci): install pinned wheel before --no-isolation build python -m build --no-isolation failed with 'Missing dependencies: wheel==0.48.0' because wheel is build-time only — uv's lockfile excludes it, so installing requirements-ci.txt alone left the build env without it. Both ci.yml and release.yml now install wheel==0.48.0 (the same pin [build-system] declares) before building. Validated locally: wheel builds clean with --no-isolation. Signed-off-by: Yunare Maia <yunare@gmail.com> --------- Signed-off-by: Yunare Maia <yunare@gmail.com> Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> |
||
|
hsd2514andZohaib Hassnain
|
8a4ebafb9a |
fix(context): honor explicit causal edges in decision tracing (#983)
* fix(context): honor explicit causal edges in decision tracing trace_decision_causality() inferred causes purely from shared NER entities plus timestamp ordering, so relationships recorded through add_causal_relationship() never affected the trace. When entity extraction returned nothing, trace_decision_chain() came back empty even though an explicit CAUSED edge was stored in the graph. Traverse the explicit CAUSED/INFLUENCED/PRECEDENT_FOR edges first, since they are the ground truth the caller recorded, and keep the entity and timestamp inference as an additive fallback for pairs with no explicit link. Edges whose source has no decision record (for example a graph restored via from_dict) are skipped so a stale edge cannot abort the trace. analyze_decision_influence() now reports explicitly linked decisions as direct influence rather than surfacing them only as indirect, and no longer lists the same decision under both direct and indirect. Closes #975 * fix(context): address review feedback on causal edge tracing Follow-up to the explicit causal edge fix, covering the issues raised in review. A stored edge weight of 0.0 was coerced to the 1.0 default by a truthiness check, inflating confidence_decay in the causal chain report. add_edge() is public and can create causal edges with any weight, so use an explicit None check instead. Explicit causes were collected into a dict keyed by source_id, so multiple causal edges between the same pair of decisions overwrote each other and only the last was traced. Collect every edge instead, keeping a separate set of source ids for the entity fallback exclusion. Cycle detection used a single traversal-wide visited set, so a decision reached through one branch became unreachable through another and branching graphs silently lost valid chains. Detect cycles per path instead; max_depth still bounds the traversal. Build a reverse index of causal edges once per call rather than scanning the edge list at every visited node, and use edge_type_index in the influence analysis. The three causal edge types are now a shared constant. Adds regression tests for zero weights, parallel edges, branching graphs and cycle termination. * fix(context): bound causal trace and report truncation Per-path cycle detection keeps branching graphs correct but makes the traversal combinatorial in max_depth: on a densely connected graph the number of distinct causal paths grows by roughly the branching factor per level, so a raised max_depth could return hundreds of thousands of chain reports and take seconds of CPU. Add a max_chains bound, defaulting to 10000. Rather than dropping chains silently, which is the exact failure this fix set out to eliminate, the traversal stops at the bound and appends a {"truncated": True, ...} marker so callers can always tell the trace is incomplete. A warning is logged with the same detail. Pass max_chains=None for the previous unbounded behaviour. Graphs that fit within the bound are unaffected. --------- Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> |
||
|
manjunath bhaskar
|
80b9bea0d5 |
fix(ingest): lock the repo host DNS resolve cache against concurrent mutation (#979)
* fix(ingest): lock the repo host DNS resolve cache against concurrent mutation
_REPO_HOST_RESOLVE_CACHE is a module level OrderedDict shared by every
RepoIngestor instance and thread. _resolve_repo_host_ips and
_prune_repo_host_resolve_cache read, wrote, and iterated it with no lock,
so concurrent ingest_repository() calls (e.g. from a thread pool) could
mutate the dict while another thread was iterating it during pruning.
This reliably raised RuntimeError: OrderedDict mutated during iteration
under ordinary concurrent usage, not just adversarial input.
Reproduced with 32 threads hammering _resolve_repo_host_ips with a low
TTL and small cache cap so pruning and eviction happen on nearly every
call; the crash showed up within the first few hundred iterations on
every run before the fix and did not reproduce at all after it.
Fix adds a threading.Lock guarding every read, write, and prune of the
cache. The blocking socket.getaddrinfo call stays outside the lock so a
slow DNS lookup for one host cannot stall cache access for other hosts.
Added a regression test, TestRepoHostResolveCacheThreadSafety, that
drives 32 threads through _resolve_repo_host_ips with a short TTL and
small cache cap and asserts no exception is raised.
Full test suite: 4088 passed, 332 failed, 140 errors both before and
after this change (same counts on main), all from missing optional
dependencies in this local environment (snowflake, sqlite-vec, spaCy
models, faiss/torch version mismatches), not from this fix. The ingest
and SSRF focused test files pass cleanly: 106 passed, 0 failed.
* test(ingest): fail fast on the first hung thread in the resolve-cache race test
join(timeout=30) alone doesn't fail the test if a worker hangs -- it
just returns after the timeout with the thread still running, and the
test falls through to the errors check, which trivially passes since
a hung thread never got far enough to append one. A future deadlock
could slip past this test looking green.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s instead. Worker threads are daemon=True so a genuine hang can't
also block the test process from exiting.
Verified the assertion is load-bearing, not cosmetic: temporarily
injected an artificial 9999s sleep into the first worker in a
throwaway copy of the test and confirmed the test now fails in ~31s
with a clear message, instead of the ~16 minutes a mass hang would
otherwise cost. That copy was never committed.
Addresses the review comment on #979 from ZohaibHassan16 and Qodo's
automated review.
* test(ingest): fail fast on the first hung thread, for real this time
The previous commit (
|
||
|
Lakshay Saini
|
5bc09a5f5a |
refactor(explorer): remove dead graph workspace shell (#984)
* refactor(explorer): remove dead graph workspace shell * refactor(explorer): remove unused graph runtime stage |
||
|
|
75f88b1c40 |
Fix/explorer backend failure states (#980)
* fix(explorer): show a retryable error when the graph fails to load The dependency-pre-bundle overlay had no failure path: on a fetch error it kept rendering the last progress frame forever with no retry. Route isError/error out of the load query, surface a real error card with the underlying message, and let retry re-fetch without a full page reload. * fix(explorer): reflect real backend connectivity on the landing page The status dot and 'System Online' text were static, so a dead backend still looked healthy. Track checking/online/offline explicitly and drive both off the same state so they can't disagree. * feat(explorer): let search results be dismissed, round relevance scores The results strip had no close affordance and stayed pinned until the next search. Add a header row with a dismiss button, and round scores to whole numbers instead of showing three decimals of a raw relevance value nobody can act on. * feat(explorer): add typeahead suggestions to graph search Typing in the search box now debounces a query against the existing search endpoint and shows a combobox dropdown, with arrow-key navigation, Enter/click to jump straight to a node, and Escape to dismiss. Previously nothing happened until the full form was submitted. * fix(explorer): abort stale typeahead requests and clear suggestions on error Clearing the search box while a suggestion fetch was in flight never aborted it, so a late response could reopen the dropdown with results for a query that was no longer typed. A non-OK response also left whatever suggestions were already on screen untouched instead of clearing them. Abort on every effect cleanup (not just unmount) and clear suggestions on any non-abort failure. * docs(changelog): add entry for Explorer backend failure states fix Documents the (#980, closes #977) fix in the Unreleased/Fixed section. --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Shubham SrivastavaandMohd Kaif
|
80b1cca07b |
test(semantic_extract): guard openai-dependent tests and assert on the logger, not stdout (#935)
* test(semantic_extract): skip openai-dependent tests when the SDK is absent, assert logs not stdout * test(semantic_extract): pass logger name to assertLogs to match suite convention All 11 existing assertLogs call sites in the suite pass a logger name string rather than a Logger instance; tests/reasoning/test_reasoner.py uses this exact .logger.name form. Behaviour is unchanged. --------- Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
|
1c0cebb1c3 |
security(context): harden Markdown import against TOCTOU symlink races (#932)
* security(context): harden Markdown import against TOCTOU symlink races Closes #856 * fix(context): harden markdown import security tests * docs(changelog): add entry for Markdown import TOCTOU symlink hardening Documents the (#932, closes #856) fix in the Unreleased/Fixed section. --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Guofang.Tang
|
94d0c3dc07 |
fix(kg): remap relationship endpoints after entity resolution (#978)
* fix(kg): remap relationship endpoints after entity resolution * fix(kg): harden relationship endpoint remapping |
||
|
Ikko Eltociear Ashimine
|
c0a051903f |
docs: update CONTRIBUTING.md (#976)
fix GiHub link. |
||
|
sushuaiyu
|
09c4b1b570 |
test(context): skip symlink test without Windows privilege (#908)
* test(context): skip symlink test without Windows privilege * test(context): name Windows privilege error code --------- |
||
|
Yunare Maia
|
c5d13a45db |
feat(seed): allow_private_ips opt-in for trusted internal API sources (#959)
* feat(seed): add allow_private_ips opt-in for trusted internal API sources (Closes #943) SeedDataManager.load_from_api now delegates to the shared SSRF guard (semantica/ingest/ssrf.py, added in #906) instead of raw requests.get, gaining redirect validation and bounded DNS resolution for free. New config option allow_private_ips (parsed via the shared parse_bool helper) lets trusted internal deployments load from private APIs while the secure default (block private/loopback/link-local) is unchanged. Tests updated to mock request_with_ssrf_guard; new tests cover the block-by-default behavior and the opt-in flag reaching the guard. 19/19 green in test_seed_manager.py, 25/25 across both seed suites. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ssrf): strip sensitive headers on cross-host redirects (Qodo finding) request_with_ssrf_guard reused the caller's headers on every redirect hop, so an Authorization bearer token from load_from_api could leak to a different redirect target host. Now strips Authorization and Proxy-Authorization when the redirect origin (netloc) changes, while keeping them for same-host hops (matching requests semantics). 2 new tests: cross-host redirect drops the credential; same-host keeps it. 37/37 green in test_ssrf_protection.py. load_from_api docstring now also documents cloud-metadata blocking and per-hop redirect validation. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ssrf): strip credentials on https->http downgrade redirects (review feedback) _should_strip_auth now mirrors requests' should_strip_auth semantics: strip on hostname change, port change, or scheme downgrade; keep the credential only for the safe http->https upgrade on default ports. Previously only netloc was compared, so an https->http redirect on the same host replayed the Authorization header in cleartext. --------- Signed-off-by: Yunare Maia <yunare@gmail.com> |
||
|
|
611874e63e |
security: apply SSRF guard to feed ingestion requests (#928)
* security: apply SSRF guard to feed ingestion requests FeedIngestor and FeedMonitor fetched feed and website URLs with plain requests.get/head calls, bypassing the SSRF validation already used by web_ingestor.py and api_ingestor.py. This allowed feed URLs pointing at loopback, link-local, or other private network addresses to be fetched directly. Route all outbound requests in feed_ingestor.py through request_with_ssrf_guard, gated by the same allow_private_ips config option the other ingestors expose. * test: mock the correct request boundary in test_discover_feeds_empty The test still patched requests.get after discover_feeds() moved to request_with_ssrf_guard(), which calls requests.request and performs real DNS resolution. That left the test hitting live network/DNS. * docs(changelog): document FeedIngestor SSRF guard fix (#928, closes #927) Records the SSRF guard applied to all 5 feed-ingestion request sites, the Qodo-flagged test-mock fix, independent PoC verification, and the carried-over exception-swallowing behavior in discover_feeds(). --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
43bac6170c |
fix(vector_store): make VectorManager methods work on persistent backends (#855) (#914)
* fix(vector_store): make VectorManager methods work on persistent backends (#855) maintain_store() and collect_statistics() reached into VectorStore internals (.vectors/.metadata), which only exist for the inmemory backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus, ...) crashed with AttributeError. Add a public backend-agnostic VectorStore.count() accessor following the get_vector()/get_metadata() precedent (#843) and the NotImplementedError-on-unsupported-capability precedent of _filter_by_metadata() (#848): inmemory counts its dict, persistent backends delegate to count() when available, and raise NotImplementedError otherwise. VectorManager methods now go through count(); maintain_store() keeps the exact inmemory semantics (separate vector/metadata dict counts) and reports a 1:1 count for persistent backends, where metadata is stored alongside each vector. Tests: 10 hermetic unit tests covering inmemory, delegation and the NotImplementedError path. Core vector_store suite: 40 passed. * fix(vector_store): raise NotImplementedError when count() unavailable Address Qodo review findings on #914: - Persistent backend with no wrapped store no longer silently returns 0 (which masked a missing initialization as an empty, healthy store); it now raises NotImplementedError like get_vector()/get_metadata(). - A mis-shaped adapter exposing a non-callable 'count' attribute now surfaces a clean NotImplementedError instead of a TypeError, via a getattr + callable() capability check. Adds regression tests for both cases. * fix(vector_store): implement count() on FAISS/SQLite/PgVector backends (#914) - FAISSStore.count(): returns len(index.vector_ids); 0 when no index exists yet - SQLiteVecStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*)) - PgVectorStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*)) - VectorStore.count(): fix misleading NotImplementedError message; now describes how to add count() support to a backend adapter rather than claiming only the inmemory backend can ever support counting - VectorManager.maintain_store(): split inmemory and persistent paths: * inmemory: independently reads len(vectors) and len(metadata) and compares them as an integrity check (original semantics preserved) * persistent: calls store.count(); returns metadata_count=None because metadata is co-located with vectors in the backend and cannot be counted independently; never manufactures metadata_count=vector_count as a vacuous tautology (#914 Qodo review) - Tests: rewrite test_vector_manager_persistent.py with 31 tests covering dispatch logic, inmemory divergence detection, persistent metadata_count=None invariant, FAISSStore/PgVectorStore via mocks, and SQLiteVecStore via real in-memory SQLite (skipped when sqlite-vec absent) * docs(changelog): document VectorManager persistent-backend count fix (#914, closes #855) Records the VectorStore.count() accessor, the FAISS/SQLite/PgVector implementations added during review, and the maintain_store() metadata_count fix (no longer fabricates equality for persistent backends). --------- Co-authored-by: Sameer6305 <sskadam6305@gmail.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
|
91d02a0f29 |
fix(ingest): harden RepoIngestor GitPython clone surface (#868) (#905)
* fix(ingest): harden RepoIngestor against GitPython URL and option injection
Bump GitPython to >=3.1.58, allowlist clone kwargs, and validate repo URLs
before clone_from to close env-var exfiltration and option-injection paths.
* fix(ingest): accept scp-like SSH remotes in RepoIngestor URL validation
* fix(ingest): resolve repo hostnames to block SSRF via private IPs
* fix(ingest): map malformed repo URL parse errors to ValidationError
* fix(ingest): bound and prune repo host resolve cache
Cap the repository host DNS cache, prune expired entries on access, and evict the oldest entries so long-running processes cannot accumulate unbounded host lookups from user-supplied repo URLs.
* fix(ingest): cap host resolve cache and tighten env-var token checks
Bound the repo host DNS cache with pruning and oldest-entry eviction, and narrow URL env-var blocking to actual $VAR/${VAR} tokens so literal dollar signs are not rejected.
* fix(ingest): preserve repo path compatibility and NAT64 support
* docs(changelog): document RepoIngestor GitPython hardening (#905, closes #868)
Records the clone-surface hardening (GitPython floor, clone-option
allowlist, URL/SSRF validation), the two fixes made during review
(NAT64 false-positive, local-path regression), and a known residual
gap: the SSRF host check doesn't classify RFC 6598 CGNAT space
(100.64.0.0/10) as blocked since ipaddress.is_private doesn't cover it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@wifi-10-43-175-99.wifi.berkeley.edu>
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
|
7c3372c062 |
fix(explorer): align dev esbuild target (#966)
Co-authored-by: le-czs <243511553+le-czs@users.noreply.github.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
|
2cfb5de43d |
feat(export): add opt-in metric_errors column to DistanceExporter (#960)
* feat(export): add opt-in metric_errors column to DistanceExporter
Add a 'metric_errors' field to compute_pairs() output that lets
downstream consumers programmatically distinguish legitimate 'no path'
(None) from computation failures (None + error name).
Usage:
rows = exporter.compute_pairs(include=[..., 'metric_errors'])
# row['metric_errors'] == '' → all metrics succeeded
# row['metric_errors'] == 'hop_count,weighted_distance' → those failed
Design decisions:
- Opt-in: column only appears when explicitly requested via include=
- Default export schema unchanged (backward compatible)
- Comma-separated metric names (not exception messages) — stable for
programmatic filtering without exposing internal error details
- Helpers now return (value, error_name | None) tuples internally
Follow-up to #879, as discussed in its review thread.
* fix: address Qodo findings — track betweenness errors and remove unused constant
1. _betweenness() now returns (dict, error) tuple like the other helpers,
so betweenness computation failures appear in metric_errors.
2. Removed unused _ERROR_COLUMNS constant (dead code).
All 77 tests in tests/export/ pass.
* docs(changelog): add entry for opt-in metric_errors column (#960)
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
|
0fa3483b96 |
fix(context): clarify get_node_property not-found contract (#877) (#882)
* fix(context): clarify get_node_property not-found contract (#877) Add default= param to get_node_property and get_node_attributes so callers can distinguish node-missing from property-missing using a sentinel. Fix add_node_attribute calling mutation_callback outside the lock. Tests added for all cases. * fix(context): address Qodo review findings (#877) * fix(context): wrap add_node_attribute mutation_callback in try/except (#877) The PR claimed to move the callback back inside `with self._lock`, but the diff only dropped a stray blank line -- the call stayed outside the lock, unchanged. That's actually correct: self._lock is an RLock, and _add_internal_node/_add_internal_edge deliberately release the lock before invoking the callback too, so a slow/misbehaving callback never holds up other threads. The real gap was that, unlike those two siblings, this call site didn't catch exceptions from the callback. Wrapped it the same way, with a regression test. --------- Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
18f1d55d77 |
test(normalize): make optional tests deterministic (#881)
* test(normalize): make optional tests deterministic Signed-off-by: aoright <102943475+aoright@users.noreply.github.com> * docs(changelog): add entry for #881 / #860 normalize test determinism fixes --------- Signed-off-by: aoright <102943475+aoright@users.noreply.github.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
b0080c3602 |
fix(export): log DistanceExporter metric computation failures instead of swallowing them (#879)
* fix(export): log DistanceExporter metric computation failures instead of swallowing them
The four private metric helpers in DistanceExporter (_betweenness,
_hop_distance, _weighted_distance, _semantic_similarity) each catch a bare
Exception and return None/{} with no signal. That makes an exported None
indistinguishable from a legitimate "no path exists" result, corrupting
downstream CSV/JSONL/DataFrame exports with no way to tell a real gap from a
swallowed error.
Log each caught exception at warning level with the offending source/target
before returning the existing sentinel. The exported row shape and values are
unchanged; only the observability of the failure changes.
Fixes #874
* fix(export): route DistanceExporter warnings through the semantica logger tree
get_logger(__name__) doubled the semantica. prefix (__name__ is already
semantica.export.distance_exporter), so the warnings this PR adds landed on
semantica.semantica.export.distance_exporter, a branch setup_logging() never
configures and does not reach the app's log handler. Also reworded the three
except-Exception log messages: they said "recording as no path", which
overclaims what a generic exception means.
Addresses review feedback from @KaifAhmad1 on #879.
* docs(changelog): add DistanceExporter logging fix entry
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
Shubham Srivastava
|
1ee3f2f214 |
fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract (#941)
* fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract
_extract_from_text() defaulted ner_method, relation_method and
triplet_method to "llm" and ran relation extraction unconditionally,
contradicting the build() docstring ("ml"/"pattern"/False) and the
standalone extractor defaults. Any raw-text build() therefore required a
provider, an API key, and network access without saying so.
Defaults are now ml/pattern/pattern with extract_relations=False. LLM
extraction is unchanged and now opt-in via explicit kwargs.
Also documents relation_method and extract_triplets, which the docstring
never listed, and drops the stale "Default to LLM methods as per
requirement" comment.
Closes #930
* perf(kg): reuse extractors across texts instead of rebuilding per source
Addresses review feedback on #941. NERExtractor.__init__ loads its spaCy
model eagerly when the method includes "ml", so switching the default
from "llm" to "ml" made _extract_from_text() reload the model once per
source in a multi-document build.
Extractors are now cached per (kind, method) on the builder. Adds tests
asserting single construction across repeated texts, that distinct
methods still get distinct extractors, and that the default path runs
end to end without any provider call.
* fix(kg): keep fallback method lists working with the extractor cache
The extractor cache keyed directly on `method`, but all three extractors
accept a list for fallback ordering (e.g. ner_method=["pattern", "ml"]),
so a list argument raised TypeError: unhashable type: 'list' before
extraction started. Lists are now converted to tuples for the cache key
only; the extractor still receives the original value.
Also seeds _extraction_stats in __init__. It was previously created only
in build(), so calling _extract_from_text() directly — as the report's
repro does — raised an AttributeError that the broad except swallowed and
logged as "Entity extraction failed".
Adds coverage for list methods on all three extractors, cache reuse for
equal lists, and distinct entries for different orderings.
* fix(kg): forward extracted relations into triplet extraction
_extract_from_text() passed only entities= to extract_triplets(), so
TripletExtractor re-derived relations itself whenever relations is None,
using a method taken from triplet_method rather than relation_method.
That duplicated work and could yield triplets inconsistent with the
relations already extracted.
relations is now initialized to None, holds the extracted list when
extract_relations=True succeeds, and is forwarded to extract_triplets().
When extraction is disabled or fails, None is passed and
TripletExtractor's existing self-derivation is unchanged.
Folded in at maintainer request rather than tracked as #944.
* docs(changelog): note that #878 documented the LLM defaults before this landed
#878 merged while this was in review and resolved the same code/docstring
mismatch in the opposite direction. Records that #930's decision makes
the code the side that changes, and that #878's docstring formatting is
retained.
|
||
|
|
1a3dd5038a |
docs(kg): document GraphBuilder public methods (#878)
* docs(kg): document GraphBuilder public methods * test(kg): skip module-level doctest to fix suite run * docs(kg): restore GraphBuilder option documentation * docs(kg): document default values for build() extraction options extract_relations, extract_triplets, ner_method, relation_method, and triplet_method all have concrete defaults in _extract_from_text(), but the build() docstring only stated a default for extract, inconsistent with CONTRIBUTING.md's docstring convention of noting parameter defaults. * docs: add changelog entry for GraphBuilder docstrings (#878, #876) --------- Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
c1154b6ed6 |
fix(security): header injection, link-prediction DoS, import ID sanitization (#912)
* fix(security): sanitize node_id in Content-Disposition to prevent header injection (CWE-113) * fix(security): cap link prediction at 10k nodes with semaphore to prevent OOM DoS (CWE-770) * fix(security): sanitize imported node IDs to prevent stored header injection chain (CWE-20) * test(security): add self-contained PoC runner with real measured output * test(security): add regression tests for header injection, DoS cap, import sanitization * fix(security): comprehensive fix for header injection, DoS, and import ID sanitization * fix: move semaphore to wrap entire data-load+scoring region, use node-specific edge queries (Qodo #2, #3) * fix: sanitize edge source/target IDs to match sanitized node IDs (Qodo #4) * fix: scope 999_999 check to predict_links function via AST (Qodo #1) * fix: add explicit None guard to _sanitize_import_node_id * fix(security): close import-sanitizer bypass, enforce link-prediction cap before the expensive scan Follow-up to the fixes in this PR, found in review: - export_import.py's "properties" in raw_node fast path stored the id verbatim, completely skipping _sanitize_import_node_id() -- a node payload of {"id": "<crlf>", "properties": {}} (the shape this app's own /api/export produces) bypassed the VULN-3 fix entirely. That branch now sanitizes id before storing. - The link-prediction 10k-node cap checked `total` only after calling session.get_nodes()/get_edges(), which normalize the graph's entire matching set before applying `limit` -- so the DoS guard ran after the expensive work it exists to prevent had already happened, on every request regardless of graph size. Added GraphSession.get_raw_counts(), an O(1) check against the raw len(graph.nodes)/len(graph.edges), and moved the size check ahead of the normalizing calls (also added an edge-count cap). - 5 of the existing regression tests asserted that literal words like "Set-Cookie"/"Content-Type" disappear from the sanitized value -- the sanitizer strips \r\n\x00"\ , not letters, so those assertions failed against this PR's own fix as submitted. Corrected to assert on the actual security property (no \r/\n survives), and added end-to-end tests that exercise the real /api/import -> /api/provenance/report route chain so the properties-key bypass has regression coverage. Full explorer suite: 241 passed. tests/test_security_regression_pr2.py: 30 passed. --------- Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
687a180721 |
fix(context): take the lock in ContextGraph.to_dict() (#929)
* fix(context): take the lock in ContextGraph.to_dict()
to_dict() iterated self.nodes.values() and self.edges without holding
self._lock, so a concurrent writer raised "RuntimeError: dictionary changed
size during iteration". It was the only reader on the class that did not take
the lock -- stats(), density(), find_nodes(), find_edges(), get_neighbors(),
get_nodes_by_label(), state_at() and save_to_file() all hold it.
Commit
|
||
|
|
bc63e962c9 |
test(seed): use a real file for CSV loading (#873)
Co-authored-by: nightcityblade <nightcityblade@gmail.com> Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <kaifahmad087@gmail.com> |
||
|
|
9ec7959899 |
Bump fastapi minimum version to fix PYSEC-2024-38 (starlette DoS) (#871)
* security(deps): bump fastapi floor to >=0.109.1 (PYSEC-2024-38) The [explorer] extra declared fastapi>=0.100.0, which allows the vulnerable 0.109.0 (PYSEC-2024-38, HTTP response splitting). Raise the floor to 0.109.1, the patched release. One-line change, no functional impact -- the 0.109.x API is stable and backward-compatible. Fixes #869 * fix(deps): bump fastapi to >=0.109.2 and python-multipart to >=0.0.7 for PYSEC-2024-38 PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p) is a ReDoS in python-multipart < 0.0.7: an attacker sends a crafted Content-Type header that causes catastrophic backtracking in the multipart regex, stalling the event loop and causing a DoS on any endpoint that parses form data. The original PR bumped fastapi to >=0.109.1, but that version pins starlette<0.36.0,>=0.35.0 and cannot install starlette 0.36.2+ (which contains the fix via python-multipart>=0.0.7). FastAPI 0.109.2 is the first version that pins starlette>=0.36.3 (verified against PyPI metadata). Two changes are necessary: 1. fastapi>=0.109.1 -> fastapi>=0.109.2: ensures starlette>=0.36.3 is installed as a transitive dependency, which in turn pulls the fixed python-multipart>=0.0.7. 2. python-multipart>=0.0.6 -> python-multipart>=0.0.7: closes the direct dependency path. python-multipart is listed explicitly in the explorer extra, so without this floor a resolver could still install 0.0.6 and leave the vulnerability present even with the fastapi bump. The fix targets only the 'explorer' optional dependency group, which is the only code surface where FastAPI and form-data parsing are used. No functional API changes between 0.109.1 and 0.109.2; 239 Explorer tests pass without modification. * ci(security): gate pip-audit on explorer-extra dependency PRs, add changelog entry for PYSEC-2024-38 The Security workflow's pip-audit job ran weekly against a bare Python env with none of Semantica's optional extras installed, and always continue-on-error'd -- it would never have flagged the vulnerable fastapi/python-multipart floors this PR fixes, or the first attempt at the fix that left python-multipart>=0.0.6 in place. security-scan.yml's Safety check has the same blind spot (only installs [llm-litellm]). pip-audit now also runs on pull_request when pyproject.toml changes, installs semantica[all] so it can actually see extras like [explorer], and fails the build on findings for that trigger. Scheduled/dispatch runs stay non-blocking pending a full pass over the [all] tree. Also documents the fix (#871, closes #869) in CHANGELOG.md, including the correction made during review after the original fastapi-only bump turned out not to close the vulnerability. * fix(deps): raise setuptools floor to >=83.0.0 (CVE-2026-59890), harden audit env The new pull_request pip-audit gate (previous commit) caught this on its first run: pip install -e ".[all]" resolved setuptools==79.0.1, vulnerable to CVE-2026-59890 / GHSA-h35f-9h28-mq5c / PYSEC-2026-3447 (Unicode normalization lets a MANIFEST.in exclude/prune pattern be bypassed on macOS APFS/HFS+, leaking excluded files into a built sdist). Fixed in setuptools 83.0.0. [build-system] requires had the same too-permissive floor this whole PR is about (setuptools>=61.0). Raised to >=83.0.0. Also upgrade pip/ setuptools explicitly in the Security workflow before running pip-audit, since [build-system] requires only governs isolated build environments, not the ambient one actions/setup-python provisions and pip-audit scans. --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Mohd Kaif
|
22bf581094 |
Merge pull request #870 from oiahoon/fix/mcp-server-version
fix(mcp): report package version |
||
|
KaifAhmad1
|
6328bfe52d | docs(changelog): add entry for MCP server version fix (#870, closes #863) | ||
|
KaifAhmad1
|
81bb5f2ed8 |
fix(mcp): report package version in standalone mcp/ server too
semantica/mcp_server/__init__.py was fixed to stop hardcoding 0.4.0, but the separate top-level mcp/ package (run via `python -m mcp.server`, documented in mcp/__init__.py as a supported way to configure Claude Desktop/Windsurf/etc. from a source checkout) still hardcoded 0.4.0 in three places: mcp/__init__.py, mcp/server.py, and mcp/resources/registry.py. Reuses semantica.__version__ directly, matching the pattern just adopted in semantica/mcp_server/__init__.py, so both implementations stay in sync with the package version going forward. |
||
|
Sameer6305
|
b8e8b2f227 |
fix(mcp): use semantica.__version__ as authoritative MCP version source
The previous implementation used importlib.metadata.version('semantica') as
the primary version source with a PackageNotFoundError fallback to
semantica.__version__. This caused two of the three new regression tests to
fail in editable/development installs, where dist-info (egg-info) is written
at install time and is not automatically updated on subsequent version bumps.
In this repo, pyproject.toml declares version as a static field (not dynamic),
and semantica/__init__.py maintains __version__ in sync with it by convention.
semantica.__version__ is therefore the authoritative source of truth and is
always present whenever semantica.mcp_server is importable -- the importlib
.metadata indirection adds no value and can return a stale value.
Changes:
- semantica/mcp_server/__init__.py: replace the importlib.metadata try/except
block with a direct 'from semantica import __version__ as _SEMANTICA_VERSION'
- tests/test_mcp_server_version.py: rewrite tests to assert both MCP version
surfaces (SERVER_INFO['version'] and semantica://schema/info) against
semantica.__version__ as the single ground truth; add 0.4.0 regression
canaries and a cross-surface consistency assertion; remove the mirrored
importlib.metadata resolution that masked the staleness problem
The root-level mcp/ directory (a separate unpublished companion implementation
not included in the built package) is intentionally left unchanged -- it is
outside the scope of issue #863 which targets the semantica-mcp entry point.
|
||
|
Sameer Kadam
|
f821fa7e2e | Merge branch 'main' into fix/mcp-server-version | ||
|
Mohd Kaif
|
229cb69c50 |
Merge pull request #857 from TaherTadpatri/fix/AttributError_in_filter_by_metadata_on_persistent_backend
Added custom _filter_by_metadata for each memory backend |
||
|
Sameer Kadam
|
8ef7c9f760 | Merge branch 'main' into fix/mcp-server-version | ||
|
KaifAhmad1
|
cab995dc97 |
fix: address code review findings in backend metadata filtering
- pinecone_store: call self.index.describe_index_stats() instead of the nonexistent self.describe_index_stats(), and use a unit query vector instead of an all-zero vector so filter_by_metadata() works on cosine-metric indexes (the library's own default) - pgvector_store: apply the existing lowercase true/false bool handling to the list-filter branch too, and use the jsonb ?| operator so list-valued metadata fields match on intersection instead of being compared as a single JSON-text blob - sqlite_vec_store: use json_each() with a json_type guard so list-valued metadata fields match on intersection, mirroring the in-memory backend's set-intersection semantics - faiss_store: filter_by_metadata(limit=0) now returns [] instead of one result - milvus_store: reject NaN/Infinity filter values up front with a clear ValidationError instead of building an invalid expression that gets silently swallowed - update the #848 FAISS NotImplementedError test to reflect that FAISS now implements real filter_by_metadata() (this PR's whole point) - add regression tests for each fix; sqlite tests run against the real sqlite-vec extension |