Fix DBIngestor calls in load_from_database , it was never actually reaching the db.
execute_query/export_table need the connection string as their first arg,
but we were only passing it to the constructor's config dict, which
those methods don't read. Every call blew up with a TypeError before
connecting.
Also split the ImportError/OSError handling , they were caught together
so a real connection failure got reported as "module not available",
which sent people looking in the wrong place. OSError now surfaces as
an actual failure with the original exception chained via `from e`.
Fixes#973.
* feat(context): add retraction and purge to ContextGraph
ContextGraph had 56 public methods and none that removed anything: the only
option was clear(), which discards the whole graph. Removing one entity meant
exporting to a dict, filtering by hand and rebuilding, losing provenance.
Add two operations with deliberately different contracts.
retract_node/retract_edge close the entity's validity window. The entity stops
being active going forward, but state_at() before the retraction still returns
it, so decisions recorded against it remain explainable. This reuses the
valid_from/valid_until machinery already present rather than adding a new
subsystem.
purge_node/purge_edge remove the entity outright, from history as well as from
the active view, leaving a tombstone that records that a purge happened and why
but never the purged content. Scope is this graph only; copies in AgentMemory
or a bound vector store are not reached, so it is one step of an erasure
workflow rather than the whole of it.
Both record themselves through the existing mutation_callback path.
MutationRecord already documented REMOVE_NODE/REMOVE_EDGE in its operation
vocabulary, so retraction emits UPDATE_NODE and purge emits REMOVE_NODE with no
changes required to change_management.
Incident-edge lookup scans self.edges rather than _adjacency, which is keyed by
source only and would otherwise leave inbound edges pointing at a removed node.
Purge updates edges, edge_type_index and _adjacency together so the indexes
cannot drift, and clear() now resets the retraction and tombstone records.
* fix(context): address review findings on retraction and purge
* fix(context): close every duplicate when retracting/purging by edge_id
edge_id is content-derived and not yet guaranteed unique (#922, fix
pending in #926): two identical add_edge() calls produce two edge
objects sharing one id. retract_edge()/purge_edge() resolved "the
edge" via the first matching object only, so a duplicate was silently
left untouched (still live, still active) while the call returned
True and recorded a tombstone/retraction claiming it was fully
handled. Repeat purge_edge() calls also silently overwrote the
tombstone's reason/purged_at on each partial attempt instead of
no-op'ing once nothing remained to purge.
retract_node()'s cascade had the same root cause from the other
direction: it checked the live _retractions dict mid-loop, so the
first duplicate's just-written record made the second look already
handled and it was skipped outright, left permanently active.
retract_edge()/purge_edge() now act on every edge matching the id
under a single record; the cascade's dedup check is snapshotted
before the loop starts so within-call duplicates are still closed
rather than skipped.
Adds TestDuplicateEdgeId (5 tests) reproducing all three paths.
* docs(changelog): document retraction/purge feature
Adds an Unreleased/Added entry for #955/#957 covering retract_node,
retract_edge, purge_node, purge_edge and the get/list accessors, plus
the duplicate-edge_id fix caught and applied during review.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
CONSTRUCT_QUERY_RE skipped comments with a bare \#[^\n]*, whose trailing *
backtracks. For '# CONSTRUCT ...\nSELECT ...' the engine gave back everything
after the '#', so the CONSTRUCT inside the comment satisfied the query-form
keyword and a SELECT/ASK was reported as a CONSTRUCT.
All four SPARQL backends delegate to this regex, so such a query took the
CONSTRUCT branch of execute_sparql, which sends Accept: text/turtle and parses
the body as Turtle — failing with a misleading 'Failed to parse CONSTRUCT
response as Turtle'.
Require a comment to reach a line terminator. Both LF and CR are accepted
because the SPARQL grammar ends a comment at either; matching only LF would
regress CR-terminated comments into false negatives.
Add regression tests covering both directions across all four backends.>
* fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#922)
* docs(changelog): document add_edge dedupe fix
Adds an Unreleased/Fixed entry for #922/#926 so the ContextGraph
edge-dedupe bug and its fix are recorded per Keep a Changelog format.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Qodo finding: embed generate wrote scalar dim_* columns, but embed index only
detects embeddings when a column's values are list/np.ndarray. This broke the
generate→index pipeline with 'No vector column found'.
Fix: write a single 'embedding' column where each value is a list[float],
matching what embed index's isinstance(df[c].iloc[0], (list, np.ndarray))
check expects. Row indices serve as ids (embed index will pass ids=None
to create_index, which is acceptable — vectors index correctly regardless).
Also addressed from Qodo review:
- .parquet suffix check is now case-insensitive (.lower())
- pandas already a core dependency (bot was wrong)
- pyarrow dependency remains added
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes
Two Explorer API endpoints fail on every install.
/api/enrich/extract imported extract_entities and extract_relations from
semantic_extract.methods, where neither name is defined — that module ships
only the per-strategy variants (extract_entities_ml, extract_relations_regex,
...), and nothing re-exports a plain facade. The resulting ImportError was
caught and reported as "semantic_extract module not available. Ensure spacy
and transformers are installed.", so a wiring bug looked like a missing
dependency. The route now calls NamedEntityRecognizer and RelationExtractor
directly, the classes the README documents, and feeds the extracted entities
into relation extraction rather than re-deriving them. The 503 branch stays
for a genuinely absent module.
Every /api/decisions* route returned 500 once the graph held a decision:
record_decision() stores timestamp as datetime.now().timestamp(), a float,
while DecisionResponse types the field as str, so pydantic rejected the value
the library itself wrote. A before-mode field validator on DecisionResponse
normalizes float, int and datetime inputs to ISO-8601, covering every route
that builds the model instead of only the list endpoint.
The existing tests missed both: test_extract accepted 503 as a pass, and the
decision fixtures are hand-built nodes carrying no timestamp at all. Both are
tightened, and a TestRecordedDecisions class exercises the routes against
decisions created through record_decision().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(semantic_extract): cache spaCy models instead of loading one per call
extract_entities_ml(), extract_relations_similarity() and
extract_relations_dependency() called spacy.load() on every invocation, so the
model was re-read from disk and re-initialized per call. On a short sentence
that is ~120 ms of loading around ~2 ms of work, and successive calls never got
cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the
pipeline ner_extract step and POST /api/enrich/extract, and process_batch()
multiplies it by the number of documents.
The module already had a cached loader for one code path — get_nlp_model() and
its _nlp_cache global — but the extraction functions bypassed it.
Adds load_spacy_model(), a process-level cache keyed by model name behind a
lock so concurrent callers do not each start a load, and routes the five call
sites through it. Errors are left uncached and propagate unchanged, so the
existing OSError fallbacks to pattern extraction still fire. get_nlp_model()
keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for
similarity work, so its model is not interchangeable with the NER one.
Cache entries record the spacy module object they came from. Several tests
patch methods.spacy with a mock and assert on load calls; without that guard a
name-keyed cache would hand a previous test's mock to a later one.
Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm:
extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(explorer): harden extraction and timestamp handling
* fix(explorer): catch OverflowError/OSError in decision timestamp validator
DecisionResponse._normalize_timestamp only guarded against NaN/inf via
math.isfinite(), but datetime.fromtimestamp() raises OverflowError or
OSError for finite epoch values outside the platform's representable
range (e.g. milliseconds stored where seconds were expected). Those
exceptions escaped the pydantic validator unhandled, reintroducing an
unhandled 500 on /api/decisions* for exactly the bug class this PR
closes. Also exclude bool from the numeric branch, since bool is an
int subclass and was being silently coerced to epoch 0/1.
* docs: add changelog entry for PR #886 (explorer extract/decisions fixes)
Documents the extraction 503, decisions timestamp 500, and folded-in
spaCy caching fixes, plus the review-round hardening from Sameer6305
and the timestamp overflow/bool fix from this follow-up commit.
---------
Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Fixes#994:
1. Prevent self-recursion in methods.py: generate_embeddings, embed_text,
calculate_similarity, and pool_embeddings all registered themselves as
custom methods, causing infinite self-calls when dispatch invoked them
without explicitly passing method parameter.
Fix: check custom_method is not the function itself before recursing.
2. Fix embed generate --output corrupt output: the CLI wrote
json.dumps(result, default=str) which produced plaintext repr of numpy
arrays (e.g. '[1.49e-01 4.85e-02 ...]') instead of proper Parquet.
Fix: detect .parquet extension (case-insensitive), convert numpy array
to pandas DataFrame with dim_* columns and id index, use to_parquet().
Non-parquet extensions fall back to JSON with clear ImportError message.
3. Add pyarrow>=14.0.0 to core dependencies (previously only in
ingest-parquet/ingest-arrow optional extras). The documented quick-start
flow of embed generate --output ... requires pyarrow out of the box.
(Note: pandas>=1.3.0 is already a core dependency; pyarrow is the
missing piece.)
Note: .github/workflows/* files are excluded from this PR as they require
a token with workflow scope. Upstream workflows are unchanged.
The pin was 5595ccaf..., but upstream has since moved the v4 tag to
ff2f1c62.... The Verify Action Pins workflow flags this drift on every
PR that touches any workflow file, regardless of whether that PR
changed codeql.yml or defender-for-devops.yml.
Verified the new SHA against the GitHub API directly (not just the CI
error text) and confirmed .github/scripts/verify-action-pins.sh passes
clean locally (40/40 action references OK, exit 0).
* ci: pin Python dependencies in requirements-ci.txt for reproducible CI
Adds a committed lockfile pinning all transitive dependencies at exact
versions (uv pip compile, Python 3.11, all extras — 1581 lines), the
Python equivalent of explorer/package-lock.json + npm ci.
- CI installs from requirements-ci.txt before building the wheel
- CI verifies the lockfile is byte-identical to a fresh compile (fails
on staleness after pyproject.toml changes)
- CONTRIBUTING documents the regeneration command
Closes#938
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: address Qodo review — security scans use pinned deps, exclude gpu extras
- security-scan.yml installs from requirements-ci.txt instead of
"./[llm-litellm]" so Safety scans the exact CI/release dependency tree
- security.yml runs pip-audit -r requirements-ci.txt for the same parity
- lockfile regenerated with --extra all (the cross-platform set) instead
of --all-extras, which pulled faiss-gpu/cupy from the Linux-only gpu
extra and co-installed faiss-cpu + faiss-gpu in CI
- uv pinned to 0.12.1 (the version that generated the lockfile) in CI and
CONTRIBUTING so regeneration is deterministic
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: make lockfile staleness check immune to upstream releases
The previous check re-resolved pyproject.toml without constraints, so any
upstream package release (e.g. boto3 1.43.69 -> 1.43.70) failed CI even
when nothing in the repo changed — exactly the time-dependent drift Qodo
flagged. The check now re-resolves with requirements-ci.txt as a
constraint and compares only version lines, so it detects intentional
pyproject.toml changes but ignores upstream releases. CONTRIBUTING
updated to match.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: fix security workflows — install pip-audit; order tooling after pinned deps
Security workflow: the pip-audit install step was lost in the rebase
conflict merge — pip-audit was invoked but never installed (exit 127).
Security-scan workflow: installing safety first let the pinned
requirements-ci.txt overwrite its transitive deps (rich), breaking the
safety CLI at runtime (RuntimeError: Type not yet supported). Tooling is
now installed AFTER the pinned set.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): address review — hashes, build isolation, release builds, docs (4/4)
ZohaibHassan16's review flagged 4 supply-chain gaps; all addressed:
1. **Release builds now use the lockfile**: release.yml installs
requirements-ci.txt and runs `python -m build --no-isolation` so the
sdist/wheel is built against the exact tested dependency set.
2. **Build isolation pinned**: [build-system].requires is now
setuptools==84.0.0 + wheel==0.48.0 (exact pins, no ranges).
3. **Hashes**: requirements-ci.txt regenerated with --generate-hashes
(5,708 sha256 hashes, verified against PyPI). Staleness check updated
to strip the `\` line continuations hashes introduce.
4. **CONTRIBUTING.md documents the separate environment**: hashes,
never-install-into-dev note, build-system pins, --no-isolation release
builds.
Validated: stale-check diff clean, hash spot-check matches PyPI.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): apply --no-isolation to CI build + align benchmark to Python 3.11
Follow-up to ZohaibHassan16's second review round:
1. ci.yml was still running `python -m build` with build isolation
(unpinned setuptools/wheel from PyPI) — now `python -m build
--no-isolation` against the pinned deps, matching release.yml.
2. benchmark.yml was on Python 3.12 while the lockfile is compiled for
3.11 — aligned to 3.11 so every workflow runs the same environment.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): install pinned wheel before --no-isolation build
python -m build --no-isolation failed with 'Missing dependencies:
wheel==0.48.0' because wheel is build-time only — uv's lockfile
excludes it, so installing requirements-ci.txt alone left the build
env without it. Both ci.yml and release.yml now install wheel==0.48.0
(the same pin [build-system] declares) before building. Validated
locally: wheel builds clean with --no-isolation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(context): honor explicit causal edges in decision tracing
trace_decision_causality() inferred causes purely from shared NER entities
plus timestamp ordering, so relationships recorded through
add_causal_relationship() never affected the trace. When entity extraction
returned nothing, trace_decision_chain() came back empty even though an
explicit CAUSED edge was stored in the graph.
Traverse the explicit CAUSED/INFLUENCED/PRECEDENT_FOR edges first, since
they are the ground truth the caller recorded, and keep the entity and
timestamp inference as an additive fallback for pairs with no explicit
link. Edges whose source has no decision record (for example a graph
restored via from_dict) are skipped so a stale edge cannot abort the trace.
analyze_decision_influence() now reports explicitly linked decisions as
direct influence rather than surfacing them only as indirect, and no
longer lists the same decision under both direct and indirect.
Closes#975
* fix(context): address review feedback on causal edge tracing
Follow-up to the explicit causal edge fix, covering the issues raised in
review.
A stored edge weight of 0.0 was coerced to the 1.0 default by a truthiness
check, inflating confidence_decay in the causal chain report. add_edge() is
public and can create causal edges with any weight, so use an explicit None
check instead.
Explicit causes were collected into a dict keyed by source_id, so multiple
causal edges between the same pair of decisions overwrote each other and
only the last was traced. Collect every edge instead, keeping a separate set
of source ids for the entity fallback exclusion.
Cycle detection used a single traversal-wide visited set, so a decision
reached through one branch became unreachable through another and branching
graphs silently lost valid chains. Detect cycles per path instead; max_depth
still bounds the traversal.
Build a reverse index of causal edges once per call rather than scanning the
edge list at every visited node, and use edge_type_index in the influence
analysis. The three causal edge types are now a shared constant.
Adds regression tests for zero weights, parallel edges, branching graphs and
cycle termination.
* fix(context): bound causal trace and report truncation
Per-path cycle detection keeps branching graphs correct but makes the
traversal combinatorial in max_depth: on a densely connected graph the
number of distinct causal paths grows by roughly the branching factor per
level, so a raised max_depth could return hundreds of thousands of chain
reports and take seconds of CPU.
Add a max_chains bound, defaulting to 10000. Rather than dropping chains
silently, which is the exact failure this fix set out to eliminate, the
traversal stops at the bound and appends a {"truncated": True, ...} marker
so callers can always tell the trace is incomplete. A warning is logged with
the same detail. Pass max_chains=None for the previous unbounded behaviour.
Graphs that fit within the bound are unaffected.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): lock the repo host DNS resolve cache against concurrent mutation
_REPO_HOST_RESOLVE_CACHE is a module level OrderedDict shared by every
RepoIngestor instance and thread. _resolve_repo_host_ips and
_prune_repo_host_resolve_cache read, wrote, and iterated it with no lock,
so concurrent ingest_repository() calls (e.g. from a thread pool) could
mutate the dict while another thread was iterating it during pruning.
This reliably raised RuntimeError: OrderedDict mutated during iteration
under ordinary concurrent usage, not just adversarial input.
Reproduced with 32 threads hammering _resolve_repo_host_ips with a low
TTL and small cache cap so pruning and eviction happen on nearly every
call; the crash showed up within the first few hundred iterations on
every run before the fix and did not reproduce at all after it.
Fix adds a threading.Lock guarding every read, write, and prune of the
cache. The blocking socket.getaddrinfo call stays outside the lock so a
slow DNS lookup for one host cannot stall cache access for other hosts.
Added a regression test, TestRepoHostResolveCacheThreadSafety, that
drives 32 threads through _resolve_repo_host_ips with a short TTL and
small cache cap and asserts no exception is raised.
Full test suite: 4088 passed, 332 failed, 140 errors both before and
after this change (same counts on main), all from missing optional
dependencies in this local environment (snowflake, sqlite-vec, spaCy
models, faiss/torch version mismatches), not from this fix. The ingest
and SSRF focused test files pass cleanly: 106 passed, 0 failed.
* test(ingest): fail fast on the first hung thread in the resolve-cache race test
join(timeout=30) alone doesn't fail the test if a worker hangs -- it
just returns after the timeout with the thread still running, and the
test falls through to the errors check, which trivially passes since
a hung thread never got far enough to append one. A future deadlock
could slip past this test looking green.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s instead. Worker threads are daemon=True so a genuine hang can't
also block the test process from exiting.
Verified the assertion is load-bearing, not cosmetic: temporarily
injected an artificial 9999s sleep into the first worker in a
throwaway copy of the test and confirmed the test now fails in ~31s
with a clear message, instead of the ~16 minutes a mass hang would
otherwise cost. That copy was never committed.
Addresses the review comment on #979 from ZohaibHassan16 and Qodo's
automated review.
* test(ingest): fail fast on the first hung thread, for real this time
The previous commit (f94e3b38) claimed to check is_alive() right after
each individual join, but a git staging mistake meant it actually
committed the old batched version instead (checking all 32 threads
only after the whole join loop finished) -- ZohaibHassan16 caught this
by timing it directly, 5 hanging threads took ~5x longer than 1
hanging thread, which the per-thread version would not do.
This commit was built by resetting to the current branch tip, verifying
byte-for-byte against a separately saved copy of the intended fix, and
confirming the actual committed git object (not just `git diff`) has
the inline check before pushing anything.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s regardless of how many threads hang.
---------
* fix(explorer): show a retryable error when the graph fails to load
The dependency-pre-bundle overlay had no failure path: on a fetch
error it kept rendering the last progress frame forever with no
retry. Route isError/error out of the load query, surface a real
error card with the underlying message, and let retry re-fetch
without a full page reload.
* fix(explorer): reflect real backend connectivity on the landing page
The status dot and 'System Online' text were static, so a dead
backend still looked healthy. Track checking/online/offline explicitly
and drive both off the same state so they can't disagree.
* feat(explorer): let search results be dismissed, round relevance scores
The results strip had no close affordance and stayed pinned until the
next search. Add a header row with a dismiss button, and round scores
to whole numbers instead of showing three decimals of a raw relevance
value nobody can act on.
* feat(explorer): add typeahead suggestions to graph search
Typing in the search box now debounces a query against the existing
search endpoint and shows a combobox dropdown, with arrow-key
navigation, Enter/click to jump straight to a node, and Escape to
dismiss. Previously nothing happened until the full form was
submitted.
* fix(explorer): abort stale typeahead requests and clear suggestions on error
Clearing the search box while a suggestion fetch was in flight never
aborted it, so a late response could reopen the dropdown with results
for a query that was no longer typed. A non-OK response also left
whatever suggestions were already on screen untouched instead of
clearing them. Abort on every effect cleanup (not just unmount) and
clear suggestions on any non-abort failure.
* docs(changelog): add entry for Explorer backend failure states fix
Documents the (#980, closes#977) fix in the Unreleased/Fixed section.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* test(semantic_extract): skip openai-dependent tests when the SDK is absent, assert logs not stdout
* test(semantic_extract): pass logger name to assertLogs to match suite convention
All 11 existing assertLogs call sites in the suite pass a logger name
string rather than a Logger instance; tests/reasoning/test_reasoner.py
uses this exact .logger.name form. Behaviour is unchanged.
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
* feat(seed): add allow_private_ips opt-in for trusted internal API sources (Closes#943)
SeedDataManager.load_from_api now delegates to the shared SSRF guard
(semantica/ingest/ssrf.py, added in #906) instead of raw requests.get,
gaining redirect validation and bounded DNS resolution for free.
New config option allow_private_ips (parsed via the shared parse_bool
helper) lets trusted internal deployments load from private APIs while
the secure default (block private/loopback/link-local) is unchanged.
Tests updated to mock request_with_ssrf_guard; new tests cover the
block-by-default behavior and the opt-in flag reaching the guard.
19/19 green in test_seed_manager.py, 25/25 across both seed suites.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ssrf): strip sensitive headers on cross-host redirects (Qodo finding)
request_with_ssrf_guard reused the caller's headers on every redirect hop,
so an Authorization bearer token from load_from_api could leak to a
different redirect target host. Now strips Authorization and
Proxy-Authorization when the redirect origin (netloc) changes, while
keeping them for same-host hops (matching requests semantics).
2 new tests: cross-host redirect drops the credential; same-host keeps it.
37/37 green in test_ssrf_protection.py. load_from_api docstring now also
documents cloud-metadata blocking and per-hop redirect validation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ssrf): strip credentials on https->http downgrade redirects (review feedback)
_should_strip_auth now mirrors requests' should_strip_auth semantics:
strip on hostname change, port change, or scheme downgrade; keep the
credential only for the safe http->https upgrade on default ports.
Previously only netloc was compared, so an https->http redirect on the
same host replayed the Authorization header in cleartext.
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
* security: apply SSRF guard to feed ingestion requests
FeedIngestor and FeedMonitor fetched feed and website URLs with plain
requests.get/head calls, bypassing the SSRF validation already used by
web_ingestor.py and api_ingestor.py. This allowed feed URLs pointing at
loopback, link-local, or other private network addresses to be fetched
directly.
Route all outbound requests in feed_ingestor.py through
request_with_ssrf_guard, gated by the same allow_private_ips config
option the other ingestors expose.
* test: mock the correct request boundary in test_discover_feeds_empty
The test still patched requests.get after discover_feeds() moved to
request_with_ssrf_guard(), which calls requests.request and performs
real DNS resolution. That left the test hitting live network/DNS.
* docs(changelog): document FeedIngestor SSRF guard fix (#928, closes#927)
Records the SSRF guard applied to all 5 feed-ingestion request sites,
the Qodo-flagged test-mock fix, independent PoC verification, and the
carried-over exception-swallowing behavior in discover_feeds().
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(vector_store): make VectorManager methods work on persistent backends (#855)
maintain_store() and collect_statistics() reached into VectorStore
internals (.vectors/.metadata), which only exist for the inmemory
backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus,
...) crashed with AttributeError.
Add a public backend-agnostic VectorStore.count() accessor following
the get_vector()/get_metadata() precedent (#843) and the
NotImplementedError-on-unsupported-capability precedent of
_filter_by_metadata() (#848): inmemory counts its dict, persistent
backends delegate to count() when available, and raise
NotImplementedError otherwise. VectorManager methods now go through
count(); maintain_store() keeps the exact inmemory semantics (separate
vector/metadata dict counts) and reports a 1:1 count for persistent
backends, where metadata is stored alongside each vector.
Tests: 10 hermetic unit tests covering inmemory, delegation and the
NotImplementedError path. Core vector_store suite: 40 passed.
* fix(vector_store): raise NotImplementedError when count() unavailable
Address Qodo review findings on #914:
- Persistent backend with no wrapped store no longer silently returns 0
(which masked a missing initialization as an empty, healthy store);
it now raises NotImplementedError like get_vector()/get_metadata().
- A mis-shaped adapter exposing a non-callable 'count' attribute now
surfaces a clean NotImplementedError instead of a TypeError, via a
getattr + callable() capability check.
Adds regression tests for both cases.
* fix(vector_store): implement count() on FAISS/SQLite/PgVector backends (#914)
- FAISSStore.count(): returns len(index.vector_ids); 0 when no index exists yet
- SQLiteVecStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- PgVectorStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- VectorStore.count(): fix misleading NotImplementedError message; now describes
how to add count() support to a backend adapter rather than claiming only the
inmemory backend can ever support counting
- VectorManager.maintain_store(): split inmemory and persistent paths:
* inmemory: independently reads len(vectors) and len(metadata) and compares
them as an integrity check (original semantics preserved)
* persistent: calls store.count(); returns metadata_count=None because
metadata is co-located with vectors in the backend and cannot be counted
independently; never manufactures metadata_count=vector_count as a vacuous
tautology (#914 Qodo review)
- Tests: rewrite test_vector_manager_persistent.py with 31 tests covering
dispatch logic, inmemory divergence detection, persistent metadata_count=None
invariant, FAISSStore/PgVectorStore via mocks, and SQLiteVecStore via real
in-memory SQLite (skipped when sqlite-vec absent)
* docs(changelog): document VectorManager persistent-backend count fix (#914, closes#855)
Records the VectorStore.count() accessor, the FAISS/SQLite/PgVector
implementations added during review, and the maintain_store()
metadata_count fix (no longer fabricates equality for persistent backends).
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
* fix(ingest): harden RepoIngestor against GitPython URL and option injection
Bump GitPython to >=3.1.58, allowlist clone kwargs, and validate repo URLs
before clone_from to close env-var exfiltration and option-injection paths.
* fix(ingest): accept scp-like SSH remotes in RepoIngestor URL validation
* fix(ingest): resolve repo hostnames to block SSRF via private IPs
* fix(ingest): map malformed repo URL parse errors to ValidationError
* fix(ingest): bound and prune repo host resolve cache
Cap the repository host DNS cache, prune expired entries on access, and evict the oldest entries so long-running processes cannot accumulate unbounded host lookups from user-supplied repo URLs.
* fix(ingest): cap host resolve cache and tighten env-var token checks
Bound the repo host DNS cache with pruning and oldest-entry eviction, and narrow URL env-var blocking to actual $VAR/${VAR} tokens so literal dollar signs are not rejected.
* fix(ingest): preserve repo path compatibility and NAT64 support
* docs(changelog): document RepoIngestor GitPython hardening (#905, closes#868)
Records the clone-surface hardening (GitPython floor, clone-option
allowlist, URL/SSRF validation), the two fixes made during review
(NAT64 false-positive, local-path regression), and a known residual
gap: the SSRF host check doesn't classify RFC 6598 CGNAT space
(100.64.0.0/10) as blocked since ipaddress.is_private doesn't cover it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@wifi-10-43-175-99.wifi.berkeley.edu>
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Bug fixes:
- _get_graph: call load_from_file (graph.load does not exist; SEMANTICA_KG_PATH was silently ignored and the graph started empty).
- query_decisions: read category from metadata.category (top-level category was always empty, so category filtering returned nothing).
- find_precedents / query: lower default similarity threshold to 0.05 so short CJK queries can match.
- extract_entities/extract_relations: return the entity text field (previously returned the spaCy type label as 'label' and dropped the actual text); expose model/language/method params so non-English (e.g. zh_core_web_sm) NER works.
New tools:
- query_graph: node detail / bidirectional neighbours (up to 5 hops, in-edges included) / keyword search.
- update_node: update node properties (e.g. action status todo/doing/done) and persist to SEMANTICA_KG_PATH.
- delete_node: soft-archive a node (status=archived) and persist.
Co-Authored-By: Claude <noreply@anthropic.com>
DecisionResponse.timestamp is typed str, but decision nodes store a float epoch. Coerce non-str timestamps so GET /api/decisions stops returning 422 Unprocessable Content.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a character-bigram overlap-coefficient fallback to _calculate_decision_content_similarity so CJK scenarios (no whitespace tokenization) can match recorded decisions; the previous whitespace Jaccard was always 0 for CJK.
Rebuild _decisions/_decision_index/_entity_index/_temporal_index from persisted decision nodes at the end of load_from_file, otherwise find_precedents_by_scenario and decision_count break after a reload since save_to_file does not serialize the internal decision indexes.
Co-Authored-By: Claude <noreply@anthropic.com>
gensim (core dependency) has no prebuilt cp314 wheel, and the slim base image lacks gcc to build from source, so 'pip install .[explorer]' fails on python:3.14-slim. Pin to python:3.13-slim (still satisfies requires-python>=3.8) until gensim ships a cp314 wheel.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(context): clarify get_node_property not-found contract (#877)
Add default= param to get_node_property and get_node_attributes so
callers can distinguish node-missing from property-missing using a
sentinel. Fix add_node_attribute calling mutation_callback outside
the lock. Tests added for all cases.
* fix(context): address Qodo review findings (#877)
* fix(context): wrap add_node_attribute mutation_callback in try/except (#877)
The PR claimed to move the callback back inside `with self._lock`, but
the diff only dropped a stray blank line -- the call stayed outside the
lock, unchanged. That's actually correct: self._lock is an RLock, and
_add_internal_node/_add_internal_edge deliberately release the lock
before invoking the callback too, so a slow/misbehaving callback never
holds up other threads. The real gap was that, unlike those two
siblings, this call site didn't catch exceptions from the callback.
Wrapped it the same way, with a regression test.
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(export): log DistanceExporter metric computation failures instead of swallowing them
The four private metric helpers in DistanceExporter (_betweenness,
_hop_distance, _weighted_distance, _semantic_similarity) each catch a bare
Exception and return None/{} with no signal. That makes an exported None
indistinguishable from a legitimate "no path exists" result, corrupting
downstream CSV/JSONL/DataFrame exports with no way to tell a real gap from a
swallowed error.
Log each caught exception at warning level with the offending source/target
before returning the existing sentinel. The exported row shape and values are
unchanged; only the observability of the failure changes.
Fixes#874
* fix(export): route DistanceExporter warnings through the semantica logger tree
get_logger(__name__) doubled the semantica. prefix (__name__ is already
semantica.export.distance_exporter), so the warnings this PR adds landed on
semantica.semantica.export.distance_exporter, a branch setup_logging() never
configures and does not reach the app's log handler. Also reworded the three
except-Exception log messages: they said "recording as no path", which
overclaims what a generic exception means.
Addresses review feedback from @KaifAhmad1 on #879.
* docs(changelog): add DistanceExporter logging fix entry
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract
_extract_from_text() defaulted ner_method, relation_method and
triplet_method to "llm" and ran relation extraction unconditionally,
contradicting the build() docstring ("ml"/"pattern"/False) and the
standalone extractor defaults. Any raw-text build() therefore required a
provider, an API key, and network access without saying so.
Defaults are now ml/pattern/pattern with extract_relations=False. LLM
extraction is unchanged and now opt-in via explicit kwargs.
Also documents relation_method and extract_triplets, which the docstring
never listed, and drops the stale "Default to LLM methods as per
requirement" comment.
Closes#930
* perf(kg): reuse extractors across texts instead of rebuilding per source
Addresses review feedback on #941. NERExtractor.__init__ loads its spaCy
model eagerly when the method includes "ml", so switching the default
from "llm" to "ml" made _extract_from_text() reload the model once per
source in a multi-document build.
Extractors are now cached per (kind, method) on the builder. Adds tests
asserting single construction across repeated texts, that distinct
methods still get distinct extractors, and that the default path runs
end to end without any provider call.
* fix(kg): keep fallback method lists working with the extractor cache
The extractor cache keyed directly on `method`, but all three extractors
accept a list for fallback ordering (e.g. ner_method=["pattern", "ml"]),
so a list argument raised TypeError: unhashable type: 'list' before
extraction started. Lists are now converted to tuples for the cache key
only; the extractor still receives the original value.
Also seeds _extraction_stats in __init__. It was previously created only
in build(), so calling _extract_from_text() directly — as the report's
repro does — raised an AttributeError that the broad except swallowed and
logged as "Entity extraction failed".
Adds coverage for list methods on all three extractors, cache reuse for
equal lists, and distinct entries for different orderings.
* fix(kg): forward extracted relations into triplet extraction
_extract_from_text() passed only entities= to extract_triplets(), so
TripletExtractor re-derived relations itself whenever relations is None,
using a method taken from triplet_method rather than relation_method.
That duplicated work and could yield triplets inconsistent with the
relations already extracted.
relations is now initialized to None, holds the extracted list when
extract_relations=True succeeds, and is forwarded to extract_triplets().
When extraction is disabled or fails, None is passed and
TripletExtractor's existing self-derivation is unchanged.
Folded in at maintainer request rather than tracked as #944.
* docs(changelog): note that #878 documented the LLM defaults before this landed
#878 merged while this was in review and resolved the same code/docstring
mismatch in the opposite direction. Records that #930's decision makes
the code the side that changes, and that #878's docstring formatting is
retained.
* docs(kg): document GraphBuilder public methods
* test(kg): skip module-level doctest to fix suite run
* docs(kg): restore GraphBuilder option documentation
* docs(kg): document default values for build() extraction options
extract_relations, extract_triplets, ner_method, relation_method, and
triplet_method all have concrete defaults in _extract_from_text(), but
the build() docstring only stated a default for extract, inconsistent
with CONTRIBUTING.md's docstring convention of noting parameter
defaults.
* docs: add changelog entry for GraphBuilder docstrings (#878, #876)
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(security): sanitize node_id in Content-Disposition to prevent header injection (CWE-113)
* fix(security): cap link prediction at 10k nodes with semaphore to prevent OOM DoS (CWE-770)
* fix(security): sanitize imported node IDs to prevent stored header injection chain (CWE-20)
* test(security): add self-contained PoC runner with real measured output
* test(security): add regression tests for header injection, DoS cap, import sanitization
* fix(security): comprehensive fix for header injection, DoS, and import ID sanitization
* fix: move semaphore to wrap entire data-load+scoring region, use node-specific edge queries (Qodo #2, #3)
* fix: sanitize edge source/target IDs to match sanitized node IDs (Qodo #4)
* fix: scope 999_999 check to predict_links function via AST (Qodo #1)
* fix: add explicit None guard to _sanitize_import_node_id
* fix(security): close import-sanitizer bypass, enforce link-prediction cap before the expensive scan
Follow-up to the fixes in this PR, found in review:
- export_import.py's "properties" in raw_node fast path stored the id
verbatim, completely skipping _sanitize_import_node_id() -- a node
payload of {"id": "<crlf>", "properties": {}} (the shape this app's
own /api/export produces) bypassed the VULN-3 fix entirely. That
branch now sanitizes id before storing.
- The link-prediction 10k-node cap checked `total` only after calling
session.get_nodes()/get_edges(), which normalize the graph's entire
matching set before applying `limit` -- so the DoS guard ran after
the expensive work it exists to prevent had already happened, on
every request regardless of graph size. Added
GraphSession.get_raw_counts(), an O(1) check against the raw
len(graph.nodes)/len(graph.edges), and moved the size check ahead of
the normalizing calls (also added an edge-count cap).
- 5 of the existing regression tests asserted that literal words like
"Set-Cookie"/"Content-Type" disappear from the sanitized value -- the
sanitizer strips \r\n\x00"\ , not letters, so those assertions failed
against this PR's own fix as submitted. Corrected to assert on the
actual security property (no \r/\n survives), and added end-to-end
tests that exercise the real /api/import -> /api/provenance/report
route chain so the properties-key bypass has regression coverage.
Full explorer suite: 241 passed. tests/test_security_regression_pr2.py: 30 passed.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* fix(context): take the lock in ContextGraph.to_dict()
to_dict() iterated self.nodes.values() and self.edges without holding
self._lock, so a concurrent writer raised "RuntimeError: dictionary changed
size during iteration". It was the only reader on the class that did not take
the lock -- stats(), density(), find_nodes(), find_edges(), get_neighbors(),
get_nodes_by_label(), state_at() and save_to_file() all hold it.
Commit 1d1ae398 introduced the RLock and added 26 "with self._lock:" blocks;
to_dict already existed and was not among them. save_to_file is safe only
incidentally -- it holds the lock and builds its payload inline rather than
delegating to to_dict, so it never reaches the unguarded loops.
Beyond the RuntimeError, the unguarded body could also return a torn snapshot:
the statistics block reads len(self.nodes)/len(self.edges) after building the
node and edge lists, so a write landing in between yields counts that
contradict the payload they describe.
self._lock is an RLock, so this composes with the callers that already hold it
(build_from_conversation and build_from_documents both return self.to_dict()
from inside a locked block). Neither external caller -- agent_context's
_capture_checkpoint_state nor triplet_store's knowledge-graph conversion --
defines a lock of its own, so there is no ordering inversion.
Add tests/context/test_context_graph_thread_safety.py: a deterministic check
that to_dict() blocks while another thread holds _lock (no race window
needed), a reentrancy check, and three checks under concurrent writes covering
the RuntimeError, statistics/payload agreement, and duplicate node ids. Four
of the five fail against the unfixed method.
Closes#923
* test(context): make to_dict lock tests deterministic and hang-proof
Wait for the worker thread to actually start before asserting to_dict()
blocks on _lock, and run the reentrancy check in a joined worker so a
non-reentrant lock fails the test instead of hanging CI.
* test(context): assert worker threads actually stopped after timed joins
A join(timeout=...) on a daemon thread returns even if the thread is
still running, so a deadlock would leak a live thread into subsequent
tests instead of failing. Assert not is_alive() after each timed join.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* security(deps): bump fastapi floor to >=0.109.1 (PYSEC-2024-38)
The [explorer] extra declared fastapi>=0.100.0, which allows the
vulnerable 0.109.0 (PYSEC-2024-38, HTTP response splitting). Raise the
floor to 0.109.1, the patched release. One-line change, no functional
impact -- the 0.109.x API is stable and backward-compatible.
Fixes#869
* fix(deps): bump fastapi to >=0.109.2 and python-multipart to >=0.0.7 for PYSEC-2024-38
PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p) is a ReDoS in
python-multipart < 0.0.7: an attacker sends a crafted Content-Type header
that causes catastrophic backtracking in the multipart regex, stalling the
event loop and causing a DoS on any endpoint that parses form data.
The original PR bumped fastapi to >=0.109.1, but that version pins
starlette<0.36.0,>=0.35.0 and cannot install starlette 0.36.2+ (which
contains the fix via python-multipart>=0.0.7). FastAPI 0.109.2 is the
first version that pins starlette>=0.36.3 (verified against PyPI metadata).
Two changes are necessary:
1. fastapi>=0.109.1 -> fastapi>=0.109.2: ensures starlette>=0.36.3 is
installed as a transitive dependency, which in turn pulls the fixed
python-multipart>=0.0.7.
2. python-multipart>=0.0.6 -> python-multipart>=0.0.7: closes the direct
dependency path. python-multipart is listed explicitly in the explorer
extra, so without this floor a resolver could still install 0.0.6 and
leave the vulnerability present even with the fastapi bump.
The fix targets only the 'explorer' optional dependency group, which is
the only code surface where FastAPI and form-data parsing are used.
No functional API changes between 0.109.1 and 0.109.2; 239 Explorer tests
pass without modification.
* ci(security): gate pip-audit on explorer-extra dependency PRs, add changelog entry for PYSEC-2024-38
The Security workflow's pip-audit job ran weekly against a bare Python
env with none of Semantica's optional extras installed, and always
continue-on-error'd -- it would never have flagged the vulnerable
fastapi/python-multipart floors this PR fixes, or the first attempt at
the fix that left python-multipart>=0.0.6 in place. security-scan.yml's
Safety check has the same blind spot (only installs [llm-litellm]).
pip-audit now also runs on pull_request when pyproject.toml changes,
installs semantica[all] so it can actually see extras like [explorer],
and fails the build on findings for that trigger. Scheduled/dispatch
runs stay non-blocking pending a full pass over the [all] tree.
Also documents the fix (#871, closes#869) in CHANGELOG.md, including
the correction made during review after the original fastapi-only bump
turned out not to close the vulnerability.
* fix(deps): raise setuptools floor to >=83.0.0 (CVE-2026-59890), harden audit env
The new pull_request pip-audit gate (previous commit) caught this on its
first run: pip install -e ".[all]" resolved setuptools==79.0.1, vulnerable
to CVE-2026-59890 / GHSA-h35f-9h28-mq5c / PYSEC-2026-3447 (Unicode
normalization lets a MANIFEST.in exclude/prune pattern be bypassed on
macOS APFS/HFS+, leaking excluded files into a built sdist). Fixed in
setuptools 83.0.0.
[build-system] requires had the same too-permissive floor this whole PR
is about (setuptools>=61.0). Raised to >=83.0.0. Also upgrade pip/
setuptools explicitly in the Security workflow before running pip-audit,
since [build-system] requires only governs isolated build environments,
not the ambient one actions/setup-python provisions and pip-audit scans.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
semantica/mcp_server/__init__.py was fixed to stop hardcoding 0.4.0,
but the separate top-level mcp/ package (run via `python -m
mcp.server`, documented in mcp/__init__.py as a supported way to
configure Claude Desktop/Windsurf/etc. from a source checkout) still
hardcoded 0.4.0 in three places: mcp/__init__.py, mcp/server.py, and
mcp/resources/registry.py.
Reuses semantica.__version__ directly, matching the pattern just
adopted in semantica/mcp_server/__init__.py, so both implementations
stay in sync with the package version going forward.
The previous implementation used importlib.metadata.version('semantica') as
the primary version source with a PackageNotFoundError fallback to
semantica.__version__. This caused two of the three new regression tests to
fail in editable/development installs, where dist-info (egg-info) is written
at install time and is not automatically updated on subsequent version bumps.
In this repo, pyproject.toml declares version as a static field (not dynamic),
and semantica/__init__.py maintains __version__ in sync with it by convention.
semantica.__version__ is therefore the authoritative source of truth and is
always present whenever semantica.mcp_server is importable -- the importlib
.metadata indirection adds no value and can return a stale value.
Changes:
- semantica/mcp_server/__init__.py: replace the importlib.metadata try/except
block with a direct 'from semantica import __version__ as _SEMANTICA_VERSION'
- tests/test_mcp_server_version.py: rewrite tests to assert both MCP version
surfaces (SERVER_INFO['version'] and semantica://schema/info) against
semantica.__version__ as the single ground truth; add 0.4.0 regression
canaries and a cross-surface consistency assertion; remove the mirrored
importlib.metadata resolution that masked the staleness problem
The root-level mcp/ directory (a separate unpublished companion implementation
not included in the built package) is intentionally left unchanged -- it is
outside the scope of issue #863 which targets the semantica-mcp entry point.
- pinecone_store: call self.index.describe_index_stats() instead of the
nonexistent self.describe_index_stats(), and use a unit query vector
instead of an all-zero vector so filter_by_metadata() works on
cosine-metric indexes (the library's own default)
- pgvector_store: apply the existing lowercase true/false bool handling
to the list-filter branch too, and use the jsonb ?| operator so
list-valued metadata fields match on intersection instead of being
compared as a single JSON-text blob
- sqlite_vec_store: use json_each() with a json_type guard so list-valued
metadata fields match on intersection, mirroring the in-memory
backend's set-intersection semantics
- faiss_store: filter_by_metadata(limit=0) now returns [] instead of one
result
- milvus_store: reject NaN/Infinity filter values up front with a clear
ValidationError instead of building an invalid expression that gets
silently swallowed
- update the #848 FAISS NotImplementedError test to reflect that FAISS
now implements real filter_by_metadata() (this PR's whole point)
- add regression tests for each fix; sqlite tests run against the real
sqlite-vec extension
yzxcj797