mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
60eb595d6216588dda8d50d9bd0ff2ba821fb90d
482
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 FABIOTESS
|
60eb595d62 |
fix(export): keep JSON-LD payloads in the default graph
A JSON-LD document with a top-level @id and a top-level @graph is a named graph. Its members become quads named by that @id, and the default graph is left empty. rdflib.Graph.parse() keeps the default graph and discards the rest without reporting anything, so every consumer that loads an export the ordinary way saw the document header and none of the data. _convert_to_jsonld wrote the payload into @graph and then stamped a document @id beside it, which named every list export and every generic-dict export. export_knowledge_graph made it worse: it converted the graph to JSON-LD and handed the finished document back to export(), which converted it a second time. The converted document no longer carries entities/relationships keys, so the second pass treated it as opaque and buried the whole knowledge graph inside @graph, under a name that is a wall-clock timestamp. A two-entity, one-relationship graph exported to JSON-LD parsed as 2 triples with Graph() and 21 quads with Dataset(). The 19 missing triples were the entire knowledge graph. The document node now goes inside @graph when the payload lives there, and is the document itself otherwise, so no export names its own graph by accident. An already-converted document is merged rather than nested, which also stops the export carrying two document nodes and two @context blocks. Semantica's reader has the mirror of this bug (#1129), so these exports could not be read back by Semantica either. |
||
|
Mohd Kaif
|
48a05b00a6 | Merge branch 'main' into timezone-aware-timestamps | ||
|
FABIOTESSandClaude Opus 5
|
e03212cd66 |
fix(provenance): compare timestamp ranges by instant, not by spelling
Review finding on #1121, and correct: with new entries carrying +00:00 and entries written earlier carrying nothing, query_recorded_between() and audit_log() compared ISO strings directly, which orders by how a timestamp is spelled rather than when it happened. Two consequences, both introduced by the offset this PR adds: - An inclusive naive bound naming a stored offset-bearing timestamp sorts below it, because the stored value is the longer string, so the record it names is excluded from its own range. - A bound in another offset lands wherever its digits fall. "2026-08-19T19:45:00+05:30" is 14:15Z, before an entry at 14:19Z, but string comparison puts it after. Both paths now compare instants, through a new to_utc_datetime() helper that reads a missing offset as UTC. That is what the naive values actually were: provenance stamped with datetime.utcnow(), so reading them as UTC keeps a stored naive value and the same instant written with an offset comparing equal instead of ordering by representation. It is also the read side the remaining 147 call sites will need whenever the rest of the package is converted. A bound that cannot be read as a timestamp keeps the historical string comparison rather than raising on a call that used to work. Five new tests cover the inclusive naive bound, the other-offset bound, legacy and offset-bearing entries ordered together, audit_log's since filter, and the unreadable-bound fallback. The first two fail with manager.py reverted; the rest are guards. 569 provenance, export and ontology tests pass, and the full-suite failure set is unchanged at 329, all from optional dependencies missing locally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
FABIOTESSandClaude Opus 5
|
83c04a57d6 |
fix(export,provenance): write timestamps with an explicit UTC offset (#1114)
semantica/export/ stamped every value with datetime.now().isoformat(), which reads the machine's local clock. semantica/provenance/ stamped its own with datetime.utcnow().isoformat(), which reads UTC. Both return a naive datetime and both serialize identically, so once the value is out of the process nothing distinguishes them: the same string means two different instants depending on which module wrote it. In RDF the consequence is silent rather than loud. Under XSD 1.1 a value with no timezone compared against one with a timezone is indeterminate whenever the two fall inside the 14-hour window, SPARQL turns an indeterminate comparison into an error, and FILTER discards errors as non-matches. Loading a Semantica-stamped export into Oxigraph next to two correctly stamped ones and asking which were written before a given instant returns the other two and drops ours, with no error anywhere. prov:generatedAtTime, prov:startedAtTime, prov:endedAtTime and prov:atTime all carry values written this way, so an audit trail cannot be ordered against timestamps from any other system. Adds utc_now()/utc_now_iso() to semantica/utils/helpers.py, exported from semantica.utils, and uses them at all 29 call sites in export/ (json_exporter, yaml_exporter, report_generator, export_provenance) and provenance/ (manager, schemas, bridge_axiom). Values now read 2026-08-19T14:19:04.229937+00:00: one unambiguous instant, comparable against any correctly stamped value, and valid xsd:dateTimeStamp. sem:exportedAt's range in the vocabulary that landed with #1109 is tightened from xsd:dateTime to xsd:dateTimeStamp accordingly. Its comment had to explain why the weaker range was necessary; that reason is gone. datetime.utcnow() is also deprecated as of Python 3.12 and scheduled for removal. Constructing a ProvenanceEntry under -W error::DeprecationWarning on 3.13 raised; it no longer does. Two new test modules cover offset presence on every export and provenance path, PROV-O literals valid as xsd:dateTimeStamp, comparison against a timezone-aware instant without TypeError, the Oxigraph filter that dropped the naive value, the declared range matching what the exporter writes, and the document @id remaining a valid IRI with +00:00 in it. The filter test picks a bound inside the indeterminate window on purpose: a bound years away is determinate even for a naive value, and the test would pass without the fix. 13 of the 14 fail with this commit's semantica/export, semantica/provenance and vocabulary reverted. The remaining 147 naive call sites, in context/, vector_store/, seed/ and elsewhere, are deliberately untouched: those timestamps are compared against values parsed back from previously stored naive strings, so converting the write side alone would raise TypeError on existing data. That sweep needs a read-side migration and belongs in its own change. No new failures across the suite: 329 pre-existing failures before and after, all from optional dependencies missing in the local environment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
FABIOTESSandClaude Opus 5
|
75b026c6dd |
fix(export): mint JSON-LD @ids the same way the RDF serializers do (#1101)
The #1101 fix covered serialize_to_turtle, serialize_to_ntriples and serialize_to_rdfxml. Both JSON-LD writers were left interpolating the entity's own text into f"semantica:entity/{text}" and the endpoints into f"semantica:rel/{source}_{target}". Three consequences, all reproducible on 0.6.5 through the public API: - An entity whose text contains a space, which is most organisation and person names an extractor produces, mints an invalid IRI. A JSON-LD parser drops that node in full and says nothing, so the entity is simply missing from the export: rdflib reads 6 triples for {"text": "AcmeCorp"} and 2 for {"text": "Acme Corp"}. - serialize_to_jsonld resolved endpoints from source_id/target_id only, while the rest of the module accepts source/target too. Every relationship carrying the second form minted the identical "semantica:rel/_", so all of them collapsed onto one node and their types and endpoints merged into a graph nobody wrote. - The JSON-LD @id and the Turtle IRI for one entity disagreed (ns#entity/Acme Corp vs ns#entity_a73cb4563ee2e72c), so the two serializations of one knowledge graph were two different graphs. Both writers now use mint_entity_iri/mint_relationship_iri, resolving endpoints both ways and passing the list index the RDF paths pass, so one knowledge graph carries one node identity whichever serializer wrote it. JSONExporter.export_entities and export_relationships also declare the semantica prefix their @context was already writing "semantica:entities" against. Without the declaration a processor reads that as an IRI in the scheme semantica rather than the namespace expansion, which is the original #1101 defect on a third path: rdflib returns the predicate literally as semantica:entities. tests/export/test_jsonld_iri_minting.py parses each export with a real JSON-LD processor rather than asserting on the JSON text, and covers all seven claims above. Each test fails on the parent commit. 236 export and ontology tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
KaifAhmad1andfabio-rovai
|
2d75952476 |
fix: close remaining review gaps in vocabulary/deterministic-IRI PR
serialize_to_rdfxml still defaulted entity_type to the bare string "semantica:Entity" written into an rdf:resource attribute, which isn't namespace-expanded the way a Turtle angle-bracket or XML element name is - the same #1101 failure mode, just on the path the original tests didn't cover. Now uses the full-IRI DEFAULT_ENTITY_TYPE like the Turtle path. json_exporter.py emits semantica:format and @type: "semantica:KnowledgeGraph", neither of which was declared in the vocabulary or included in EMITTED_TERMS, so the "undeclared terms fail the build" guarantee didn't actually cover them. Both are now declared with rdfs:label/comment and added to the guard set. MANIFEST.in didn't mirror the pyproject.toml package-data addition, so a source-distribution install could ship without the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only PATH, breaking it on Windows and any host needing other inherited env vars; now overrides only PYTHONHASHSEED on top of the inherited environment. Also folds mint_entity_iri/mint_relationship_iri's hand-rolled hashlib.sha256(...).hexdigest() into the existing hash_data() helper this file already imports alongside. 229 export and ontology tests pass, including a new regression test for the RDF/XML default-type fix. Co-Authored-By: fabio-rovai <fabio@thetesseractacademy.com> |
||
|
FABIOTESSandClaude Opus 5
|
e55c03bd39 |
fix: resolve temporal endpoints both ways, and stop declaring a range the exporters contradict
Both from review on #1109. The temporal fallback minted from source_id only, while the main serializer accepts source_id or source. Relationships using the second form therefore hashed two empty strings, and once the IRI became deterministic that turned a latent problem into an active one: unrelated relationships at the same list index collided on the same IRI across exports, so their temporal data aliased when loaded together. Endpoints are now resolved the way serialize_to_turtle resolves them, before minting. The vocabulary declared sem:confidence with range xsd:decimal, which the N-Triples serializer contradicts by typing the same value xsd:float. Neither is safe to declare while the two serializers disagree, since the Turtle path writes the value bare and the Turtle grammar reads that as xsd:decimal. The range is dropped with the reasoning recorded on the term and a pointer to #1100, which tracks the disagreement itself. Extends the drift guard rather than only fixing the instance: a new test asserts that any range this vocabulary declares matches the datatype the serializers actually emit, so the class of contradiction that review caught fails the build next time. 228 export and ontology tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
FABIOTESSandClaude Opus 5
|
e1092ac507 |
feat(ontology): declare the Semantica vocabulary, and mint entity IRIs deterministically
Closes #1107, closes #1101. Every RDF export mints terms in https://semantica.dev/ns#, and nothing declared what those terms meant. The namespace returns 404 and no vocabulary shipped with the package, so a consumer receiving an export could not tell semantica:text from a typo of it: in the open world an undeclared IRI is unknown rather than wrong, and every RDF tool treats the two alike. Closed-world checking is what separates them, and it needs a document to check against. semantica/ontology/vocabulary/semantica-ns.ttl declares the fourteen terms the exporters actually emit, drawn from the emitting call sites rather than from what a vocabulary ought to contain. It ships inside the package so it loads without a network round trip, and is the same document intended to be served at the namespace IRI once hosting and content negotiation are sorted. tests/ontology/test_vocabulary.py ties the document to the code: every term the serializers can write must be declared, so adding a term to an exporter without declaring it fails the build rather than shipping an undeclared IRI. The vocabulary alone would not have made those IRIs resolve, because the fallback path minted them from Python's builtin hash(). That is randomised per process, so the same entity received a different IRI on every run and exports could not be diffed, deduplicated against an earlier load, or joined to a provenance record written by an earlier process. Minting now uses SHA-256 and writes a full IRI in the declared namespace rather than semantica:entity_N, which inside angle brackets is an IRI in the scheme "semantica" rather than the prefix expansion, and so never joined with anything written through the prefix. The same applies to the default entity and relationship types in the Turtle path. 134 export tests and 91 ontology tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Guofang.Tang
|
b77e3e8c3c |
fix(kg): preserve entity_id aliases during entity merging (#1086)
* fix(kg): preserve entity_id aliases during merge * fix(kg): unify entity ID extraction semantics |
||
|
江俊杰
|
08a6e7c053 |
fix(ontology): render real SHACL constraint values in explain_violations
explain_violations previously rendered hardcoded placeholders (min_count=1, max_count=1) and misused the violation message as the datatype/class value, so plain-English explanations were inaccurate. The root cause is that _run_pyshacl never read the real constraint parameters from sh:sourceShape when building each SHACLViolation. Changes: - SHACLViolation: add min_count/max_count/datatype/class_ fields and include them in to_dict() - _run_pyshacl: back-reference sh:sourceShape to extract the real sh:minCount/sh:maxCount/sh:datatype/sh:class values - explain_violations: render the real values, falling back to "?" or descriptive text when unknown Note: sh:qualifiedMinCount/qualifiedMaxCount are not handled and fall back to the "?" placeholder. Adds regression tests covering both the formatting path and the sh:sourceShape back-reference (skips when pyshacl/rdflib are absent). |
||
|
cxzg007and江俊杰
|
a1194a155d |
feat(context): add to_kg_dict() adapter for canonical KG shape (#1081)
* feat(context): add to_kg_dict() adapter for canonical KG shape
Convert ContextGraph internal nodes/edges/source representation into the canonical entities/relationships/source_id shape consumed by RDFExporter and TemporalGraphQuery. Add entities_only filtering that drops dangling relationships, plus README examples and unit tests.
* fix(context): harden to_kg_dict against null props and non-str node ids
- Guard properties/metadata with 'or {}' so nodes loaded from JSON null
no longer raise TypeError when copied (Qodo bug 1)
- Coerce entity id to str(n.node_id) so it matches ContextEdge's
str-coerced endpoints, preventing valid relationships from being
dropped during entities_only filtering (Qodo bug 3)
* fix(kg): accept source_id/target_id endpoints in validator and temporal query
to_kg_dict() emits canonical source_id/target_id keys, but GraphValidator
and TemporalGraphQuery only read the legacy source/target keys, so its
output failed validation and lost relationships (Qodo bug 2).
- GraphValidator: resolve endpoints from either key variant and treat a
resolvable source/target (plus type) as satisfying required fields
- TemporalGraphQuery.analyze_evolution/find_paths: read either variant
- tests: add regression coverage for null props/metadata (bug 1),
non-string node ids (bug 3), and KG-utility consumability (bug 2)
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
|
||
|
Guofang.Tang
|
17d878cbf3 |
fix(kg): honor exact entity resolution (#1026)
* fix(kg): honor exact entity resolution * fix(kg): preserve entities without identifiers * fix(kg): ignore blank exact entity names --------- |
||
|
KaifAhmad1
|
43b207c1c5 |
fix(security): address privately disclosed zip-slip, SQLi, SSRF, XSS, and SPARQLi findings
Fixes a set of runtime trust-boundary issues from a private security
disclosure (checkout
|
||
|
|
5c2901ae27 |
docs(context): fix unrunnable ContextGraph docstring example (#921)
* docs(context): fix unrunnable ContextGraph docstring example The module docstring's Example Usage block called add_node/add_edge with keyword arguments they do not accept. add_node(node_id, node_type, ...) takes node_type positionally and has no properties parameter, so the documented call raised TypeError; add_edge's parameter is edge_type, so type= fell through to **properties and polluted edge metadata while appearing to work. Two of the three broken forms failed silently rather than raising, storing a nested properties dict or a stray type key instead of erroring. Add regression tests that execute the documented calls and assert the docstring itself does not reintroduce the invalid kwargs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(context): close two blind spots in the docstring regression guards The guards added in the previous commit could pass while checking nothing. _example_block() terminated the capture at the first "\n\n". The Example Usage block already contains ">>> " spacer lines, so any reformatting that turned one into a bare blank line would truncate the capture -- potentially to empty -- and the guards would then scan a block that no longer held the add_node/add_edge calls they exist to police. Both guards also iterated over re.findall() without asserting a match. Zero matches meant zero assertions and a green test, so the two failure modes compounded: a truncated block produced no matches, and no matches produced a pass. Terminate the block at the next top-level section header (^\S) or end of docstring instead, so blank lines inside the example are harmless, and assert the captured block, the parsed statement list, and each guard's match list are all non-empty. Extract statements with doctest.DocTestParser rather than a line regex. This also catches a call reformatted across "..." continuation lines, which the ">>> graph.add_node(.*" pattern silently skipped, and lets test_documented_calls_execute exec the docstring's own statements instead of a retyped copy that could drift from it. Full doctest.testmod isn't usable here: add_node/add_edge return True and the docs carry no expected-output lines, so it reports 4 spurious failures. Narrow the kwarg check to (?<![\w])type\s*= so a legitimate node_type= or edge_type= in the docs no longer trips a guard aimed at bare type=. Verified by mutating the module docstring and re-running the guards: extra blank lines with a valid example still pass; regressed add_node/add_edge, a type= on a continuation line, deleted calls, and a deleted section all fail; a legitimate node_type= passes. 6 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(context): correct precedent lookup in docstring example --------- Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
Shubham Srivastava
|
baf8f01f85 |
test(export): guard Parquet tests on pyarrow itself, not the exporter import (#1056)
* test(export): guard Parquet tests on pyarrow itself, not the exporter import Closes #1054 * test(export): guard on PARQUET_AVAILABLE so the skip matches the runtime check find_spec only proves pyarrow is discoverable, not importable. Addresses review feedback on #1056. --------- |
||
|
Sameer KadamandKaifAhmad1
|
04602a0e0e |
fix(security): prevent Authorization header leakage across redirects (#947) (#1067)
* fix(security): prevent auth header leakage across redirects * fix(security): harden redirect credential handling Address Copilot and Qodo review findings for #947. - Remove unused variables, imports, and unnecessary pass statements from tests. - Harden cross-origin redirect handling for per-request auth credentials. - Strip session-level auth handlers before cross-origin redirect hops. - Prevent session.auth from regenerating Authorization headers. - Disable trust_env during cross-origin hops to prevent .netrc credential injection. - Restore session auth and trust_env state reliably with try/finally. - Add regression coverage for auth=, session.auth, trust_env, and multi-hop redirects. - Preserve existing security behavior and same-origin authentication semantics. Validated with 189/189 security and affected tests passing. * fix(security): scope allow_private_ips to same-host redirects, fix error handling gaps Follow-up to review findings on #1067: - MCPClient hardcoded allow_private_ips=True for every redirect hop, not just its operator-configured host, so a compromised/malicious MCP server could 302 into private address space (e.g. cloud metadata) unchecked. request_with_ssrf_guard() gains allow_private_ips_on_redirect: a redirect target inherits the original host's private-IP trust only when it matches that host; MCPClient now pins it to False. - detect_public_api() only caught requests.exceptions.RequestException, but the SSRF guard raises ValidationError for blocked hosts/redirects, unlike its sibling ingest_public_api(). Now catches and re-raises it the same way. - detect_public_api()/ingest_public_api() forwarded session/allow_private_ips through **options into request_with_ssrf_guard(), which already passes both explicitly -- a caller supplying either would hit a duplicate-kwarg TypeError. Both are now popped from request_options first. New regression coverage for all three in tests/ingest/, plus a CHANGELOG entry under Unreleased/Security. --------- Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Shahzaib AhmadandShahzaib Ahmad
|
eedf1425ca |
Fix flatten_dict key collisions (#1062)
* Fix flatten_dict key collisions * Fix flatten_dict formatting --------- Co-authored-by: Shahzaib Ahmad <malikshahzaib7145@example.com> |
||
|
KaifAhmad1
|
a8194dfc60 |
fix(split): catch broken-runtime spaCy failures in SemanticChunker
SemanticChunker.__init__ only caught OSError around load_spacy_model(), while NERExtractor's identical call (fixed earlier in this PR) also catches generic Exception for a model that is installed but fails at runtime. Bring SemanticChunker in line so a broken spaCy config degrades to fallback chunking instead of crashing __init__. Adds a regression test mirroring the existing NERExtractor case, and a CHANGELOG entry for #998/#1042. |
||
|
Sameer6305
|
c7415f2e92 | fix: complete spaCy model cache integration | ||
|
Sameer Kadam
|
3331df28ad | Merge branch 'main' into spacy-cache-split-chunking | ||
|
Accute9
|
de5e20dc55 | resolved merge conflict | ||
|
Accute9
|
0f252ab355 | Fixed max line length (88) issues and eager imports | ||
|
Aneesh MandapatiandCopilot Autofix powered by AI
|
0b77e5fe94 |
Refactor for flake8 max line length (88) issue
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Accute9
|
893b6db3c3 | regression tests added and tested for routing spaCy model loads through cache | ||
|
hariandZohaib Hassnain
|
70aa9d01bf |
fix(normalize): validate symbol currencies (#940)
* fix(normalize): validate symbol currencies Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com> * fix(normalize): match currency codes by token boundaries Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com> --------- Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com> Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> |
||
|
pravit-ampandPravit Ampapathini
|
15171fd31a |
fix(parse): import get_progress_tracker in ExcelParser (#1016)
ExcelParser.__init__ called get_progress_tracker() without importing it, so every instantiation raised NameError and the class was unusable. The existing test imported ExcelParser but never constructed it, so nothing caught it. Same defect as #530 in SimilarityCalculator, which was fixed without sweeping the rest of the codebase. Add construction coverage for every parser exported from semantica.parse, driven off __all__ so later additions are covered automatically. These live outside test_parse_comprehensive.py, whose setUp patches get_progress_tracker into each parse module and would mock away the interaction under test. Closes #1014 Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> |
||
|
Guofang.Tang
|
8177d88753 |
fix(kg): preserve isolated nodes in graph analytics (#1011)
* fix(kg): preserve isolated nodes in graph analytics * fix(kg): support node fallbacks and community payloads --------- |
||
|
Shinde vinayak rao patil
|
d94d8f6ab8 |
Feat/crewai integration (#988)
* feat(crewai): add first-class CrewAI integration (#962) Add native CrewAI support so Crew agents can share a ContextGraph and AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching the existing agno integration pattern. - SemanticaKGTool: 5 KG actions (extract_entities, extract_relations, add_to_graph, query_graph, find_related) with sync run()/async arun() - SemanticaDecisionTool: 5 decision-intelligence actions (record_decision, find_precedents, trace_causal_chain, analyze_impact, check_policy) over AgentContext - SemanticaKnowledgeSource: serializes a ContextGraph into crew knowledge storage; bridges legacy load_content() and current validate_content()/aadd() contracts for crewai>=0.80.0 - All classes degrade gracefully when crewai is absent - New pip extra crewai=... included in the all bundle - 70 new tests (stub-based present-case + subprocess degradation path) - Docs: integrations/crewai.md, docs.json nav, README matrix updates * fix(crewai): harden tools against real Semantica dataclass shapes (#962) Bugs found during live testing with crewai 1.15.16: - SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses ('str' object has no attribute 'end_char'): string names were passed to extract_relations(entities=...), which requires Entity objects, and the tool read .name/.source/.target instead of Entity's .text/.label and Relation's .subject/.object. Add shape-agnostic field helpers. - SemanticaDecisionTool() created an AgentContext without a knowledge_graph, so _decision_backend was never set and record_decision raised 'Decision tracking is not enabled'. Wire in a ContextGraph. - record_decision hard-failed when the agent omitted optional fields; fall back to category='general', reasoning='agent decision', outcome='recorded'. Add tests covering real Entity/Relation dataclass shapes and the live auto-created AgentContext path (now 77 crewai tests, 212 total). * fix(crewai): make find_related traverse edges undirected (#962) ContextGraph.get_neighbors only follows outgoing edges, so a node whose only edge is incoming (A -> B) reported no related concepts. Rebuild a bidirectional adjacency from find_edges() in SemanticaKGTool._find_related so 'related' honors both directions. * fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962) - Exclude live graph/context/extractor state from JSON serialization (model_dump(mode="json")) so CrewAI checkpointing no longer raises PydanticSerializationError; model_post_init self-heals defaults on restore - query_graph now searches node content via graph.query() plus id/type - trace_causal_chain returns an explicit error when causal tracing is unavailable instead of substituting similarity precedents; call trace_decision_causality(..., max_depth=...) with the correct kwarg name - find_precedents propagates max_precedents/limit to the backend instead of being silently capped at 10 - Serialize add_to_graph batches under a module lock to prevent concurrent double-counting; skip nameless entities instead of creating repr()-junk nodes - aadd() runs CPU-bound serialization in a thread executor - Mirror crewai args_schema serialize/restore in the conftest stub and add serialization regression tests (crewai: 92 tests) * fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962) - _eval_rule now coerces rule values type-aware: bool("false") was truthy, so 'enabled == false' reported a violation for enabled=false, and string datums like "0.90" were compared lexicographically instead of numerically - _trace_causal_chain no longer raises AttributeError (which escaped _run) when the decision context lacks knowledge_graph; returns honest error JSON - SemanticaKnowledgeSource storage failures log an actionable ERROR; without a configured crew embedder agents previously retrieved nothing silently - add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a process-global one: independent graphs no longer serialize each other and re-entrant extractor callbacks cannot deadlock - entity/relation confidence=None normalizes to 1.0 instead of failing the whole extraction with float(None) - add subprocess integration test against real crewai covering Crew-level serialization round-trip and checkpoint restore (stub tests cannot see it) - docs: embedder requirement for SemanticaKnowledgeSource; resume contract note * fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962) Re-verification against real crewai showed the embedder-missing failure raises ValueError even though storage IS wired, so the old except-ValueError branch mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure. Distinguish by storage presence instead of exception type: storage is None -> DEBUG keep-in-memory (legitimate standalone use); storage wired but save() raises -> actionable ERROR. Add regression test mirroring real crewai's ValueError-on-missing-embedder behavior. * fix(crewai): expose run()/arun() entry points in degraded mode (#962) The public crewai contract is run()/arun(); without crewai installed they were missing (only the private _run existed), so the documented 'usable without crewai' path raised AttributeError at the entry point. Define them in degraded mode only, leaving crewai's BaseTool implementations untouched when present. Extend the degradation subprocess test to exercise run() and arun(). * fix(crewai): standardize query shape, field-name rules, and restore-state flag - _query_graph: id/type matches now return the same schema as content matches (id/type/label/content/score) instead of a bare list - _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys (e.g. "risk-score >= 0.9") are addressable in policy rules - add had_live_state/reconstructed_state so checkpoint-restored tools and knowledge sources signal that their live graph/context was lost and an empty one reconstructed; knowledge source no longer hides the loss by eagerly rebuilding its graph inside __init__ (pydantic calls __init__ during model_validate) * fix(crewai): address Qodo review — confidence errors, string trim, holistic availability - record_decision: stop calling float() in _run, so malformed confidence values surface as JSON errors (via _record_decision's handling) instead of crashing the tool - _coerce_value: return the stripped string for non-numeric literals so whitespace-padded decision_data fields match policy rules - centralize crewai availability in _availability.py so the exported CREWAI_AVAILABLE flag is holistic across tools and knowledge source (previously each module probed crewai independently and the package flag came from decision_tool only) * ci: regenerate requirements-ci.txt for the crewai extra The crewai extra in pyproject.toml brings in crewai, crewai-tools and transitive deps (chromadb, lancedb, ...). Recompile with uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes. * ci: keep crewai out of the locked CI dependency set crewai (all versions) hard-requires chromadb~=1.1.0, which carries a pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c) with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in the 'all' extra failed pip-audit and the safety check on requirements-ci.txt. - drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is unchanged and still installs crewai) - stop listing crewai-tools in the extra: the integration only uses crewai core (BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps - regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0 vulnerabilities, staleness check matches * docs(crewai): document crewai extra scope and chromadb CVE-2026-45829 - CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not part of the 'all' bundle, with the chromadb CVE-2026-45829 reason - integrations/crewai/README.md: add a security warning that installing the extra pulls chromadb~=1.1.0, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 --------- |
||
|
|
5579851208 |
fix(export): harden YAML export input handling (#958)
* refactor(export): centralize graph-payload key normalization
Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.
Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.
Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.
Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.
CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.
* fix(export): reject non-mapping input to the YAML exporters
export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.
Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.
Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.
* fix(export): address YAML exporter review findings
- semantica/export/yaml_exporter.py — import Sequence from typing
instead of collections.abc. `Sequence[str]` in _require_mapping's
annotation is evaluated at function-definition time; collections.abc.Sequence
only became subscriptable in Python 3.9, so on the 3.8 this project
declares support for, importing this module raised TypeError.
typing.Sequence has supported subscripting since 3.5.3. Mapping stays
imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
and read exported YAML through a context manager instead of an
unclosed yaml.safe_load(open(...)).
* fix(export): reject YAML export payloads with no recognized key
Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.
- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
collections through normalize_graph_payload(), which raises rather than
returning empty collections for an unrecognized mapping. Adopting the
shared resolver rather than repeating the check locally also brings the
'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
from this library's own graph type to YAML, used in
examples/capability_gap_context_graphs_example.py — exports its records
instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
defaulted lookups and had the same defect; it goes through the resolver
too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
its own key set. Schemas are a separate vocabulary with no aliasing, so
_require_recognized_keys lives in this module rather than in the shared
graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
An export_json envelope carries one, so accepting it would readmit the
case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
output directory, so a rejected export leaves nothing behind.
The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.
Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.
Closes #953.
* fix(export): reject payloads whose records resolve to nothing
Addresses the Qodo findings on #958.
Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.
- utils/helpers.py — split the check in two. _require_recognized_keys keeps
the presence rule; _require_nothing_dropped runs after resolution and
refuses a payload that resolved to nothing while an unread key still holds
records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
always carries a populated 'statistics' dict, and an empty graph must stay
exportable, so 'metadata', 'statistics' and 'count' are named as context
rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
both checks through the shared helpers rather than its own copy, so the
two vocabularies cannot drift apart in what counts as a silent-empty
export.
Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.
- export/yaml_exporter.py — the serialization span now says it serialized,
not that it exported, and export() opens its own span around the
filesystem work that stops as failed on error. Nothing reports a completed
export until the bytes are on disk.
Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.
Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.
* fix(export): validate collection values and make Neo4j mappings strict
Two gaps at the boundary the shared normalizer is supposed to own.
_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.
Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.
Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.
* fix(export): close YAML schema and record validation gaps
Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived. Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.
Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects. Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary. Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.
Tests: 101 tests pass across
tests/utils/test_normalize_graph_payload.py
tests/export/test_yaml_exporter_key_recognition.py
tests/export/test_yaml_exporter_input_validation.py
tests/export/test_neo4j_csv_exporter.py
* fix(export): close exception-type and record-shape gaps in normalize_graph_payload
LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).
Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).
normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.
Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.
* docs(changelog): add entry for #958 YAML export input hardening
Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes #956, #952, #953.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
yzxcj797
|
8639cb9f16 |
fix(seed): pass connection string to DBIngestor and stop mislabeling OSError in load_from_database (#995)
Fix DBIngestor calls in load_from_database , it was never actually reaching the db. execute_query/export_table need the connection string as their first arg, but we were only passing it to the constructor's config dict, which those methods don't read. Every call blew up with a TypeError before connecting. Also split the ImportError/OSError handling , they were caught together so a real connection failure got reported as "module not available", which sent people looking in the wrong place. OSError now surfaces as an actual failure with the original exception chained via `from e`. Fixes #973. |
||
|
|
f1e7e64ad1 |
feat(context): add retraction and purge to ContextGraph (#957)
* feat(context): add retraction and purge to ContextGraph ContextGraph had 56 public methods and none that removed anything: the only option was clear(), which discards the whole graph. Removing one entity meant exporting to a dict, filtering by hand and rebuilding, losing provenance. Add two operations with deliberately different contracts. retract_node/retract_edge close the entity's validity window. The entity stops being active going forward, but state_at() before the retraction still returns it, so decisions recorded against it remain explainable. This reuses the valid_from/valid_until machinery already present rather than adding a new subsystem. purge_node/purge_edge remove the entity outright, from history as well as from the active view, leaving a tombstone that records that a purge happened and why but never the purged content. Scope is this graph only; copies in AgentMemory or a bound vector store are not reached, so it is one step of an erasure workflow rather than the whole of it. Both record themselves through the existing mutation_callback path. MutationRecord already documented REMOVE_NODE/REMOVE_EDGE in its operation vocabulary, so retraction emits UPDATE_NODE and purge emits REMOVE_NODE with no changes required to change_management. Incident-edge lookup scans self.edges rather than _adjacency, which is keyed by source only and would otherwise leave inbound edges pointing at a removed node. Purge updates edges, edge_type_index and _adjacency together so the indexes cannot drift, and clear() now resets the retraction and tombstone records. * fix(context): address review findings on retraction and purge * fix(context): close every duplicate when retracting/purging by edge_id edge_id is content-derived and not yet guaranteed unique (#922, fix pending in #926): two identical add_edge() calls produce two edge objects sharing one id. retract_edge()/purge_edge() resolved "the edge" via the first matching object only, so a duplicate was silently left untouched (still live, still active) while the call returned True and recorded a tombstone/retraction claiming it was fully handled. Repeat purge_edge() calls also silently overwrote the tombstone's reason/purged_at on each partial attempt instead of no-op'ing once nothing remained to purge. retract_node()'s cascade had the same root cause from the other direction: it checked the live _retractions dict mid-loop, so the first duplicate's just-written record made the second look already handled and it was skipped outright, left permanently active. retract_edge()/purge_edge() now act on every edge matching the id under a single record; the cascade's dedup check is snapshotted before the loop starts so within-call duplicates are still closed rather than skipped. Adds TestDuplicateEdgeId (5 tests) reproducing all three paths. * docs(changelog): document retraction/purge feature Adds an Unreleased/Added entry for #955/#957 covering retract_node, retract_edge, purge_node, purge_edge and the get/list accessors, plus the duplicate-edge_id fix caught and applied during review. --------- Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
pravit-amp
|
6df97cf0a0 |
fix(triplet_store): stop CONSTRUCT detection matching inside a leading comment (#951)
CONSTRUCT_QUERY_RE skipped comments with a bare \#[^\n]*, whose trailing * backtracks. For '# CONSTRUCT ...\nSELECT ...' the engine gave back everything after the '#', so the CONSTRUCT inside the comment satisfied the query-form keyword and a SELECT/ASK was reported as a CONSTRUCT. All four SPARQL backends delegate to this regex, so such a query took the CONSTRUCT branch of execute_sparql, which sends Accept: text/turtle and parses the body as Turtle — failing with a misleading 'Failed to parse CONSTRUCT response as Turtle'. Require a comment to reach a line terminator. Both LF and CR are accepted because the SPARQL grammar ends a comment at either; matching only LF would regress CR-terminated comments into false negatives. Add regression tests covering both directions across all four backends.> |
||
|
|
84ce3c5155 |
fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#926)
* fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#922) * docs(changelog): document add_edge dedupe fix Adds an Unreleased/Fixed entry for #922/#926 so the ContextGraph edge-dedupe bug and its fix are recorded per Keep a Changelog format. --------- Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Guofang.Tang
|
b8175ea801 |
fix(kg): make k-shortest path search side-effect free (#1000)
* fix(kg): make k-shortest path search side-effect free * fix(kg): respect traversal direction for edge exclusion |
||
|
|
557e29ee14 |
fix(explorer): repair /api/enrich/extract (always 503) and the /api/decisions routes (always 500) (#886)
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes Two Explorer API endpoints fail on every install. /api/enrich/extract imported extract_entities and extract_relations from semantic_extract.methods, where neither name is defined — that module ships only the per-strategy variants (extract_entities_ml, extract_relations_regex, ...), and nothing re-exports a plain facade. The resulting ImportError was caught and reported as "semantic_extract module not available. Ensure spacy and transformers are installed.", so a wiring bug looked like a missing dependency. The route now calls NamedEntityRecognizer and RelationExtractor directly, the classes the README documents, and feeds the extracted entities into relation extraction rather than re-deriving them. The 503 branch stays for a genuinely absent module. Every /api/decisions* route returned 500 once the graph held a decision: record_decision() stores timestamp as datetime.now().timestamp(), a float, while DecisionResponse types the field as str, so pydantic rejected the value the library itself wrote. A before-mode field validator on DecisionResponse normalizes float, int and datetime inputs to ISO-8601, covering every route that builds the model instead of only the list endpoint. The existing tests missed both: test_extract accepted 503 as a pass, and the decision fixtures are hand-built nodes carrying no timestamp at all. Both are tightened, and a TestRecordedDecisions class exercises the routes against decisions created through record_decision(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(semantic_extract): cache spaCy models instead of loading one per call extract_entities_ml(), extract_relations_similarity() and extract_relations_dependency() called spacy.load() on every invocation, so the model was re-read from disk and re-initialized per call. On a short sentence that is ~120 ms of loading around ~2 ms of work, and successive calls never got cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the pipeline ner_extract step and POST /api/enrich/extract, and process_batch() multiplies it by the number of documents. The module already had a cached loader for one code path — get_nlp_model() and its _nlp_cache global — but the extraction functions bypassed it. Adds load_spacy_model(), a process-level cache keyed by model name behind a lock so concurrent callers do not each start a load, and routes the five call sites through it. Errors are left uncached and propagate unchanged, so the existing OSError fallbacks to pattern extraction still fire. get_nlp_model() keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for similarity work, so its model is not interchangeable with the NER one. Cache entries record the spacy module object they came from. Several tests patch methods.spacy with a mock and assert on load calls; without that guard a name-keyed cache would hand a previous test's mock to a later one. Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm: extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(explorer): harden extraction and timestamp handling * fix(explorer): catch OverflowError/OSError in decision timestamp validator DecisionResponse._normalize_timestamp only guarded against NaN/inf via math.isfinite(), but datetime.fromtimestamp() raises OverflowError or OSError for finite epoch values outside the platform's representable range (e.g. milliseconds stored where seconds were expected). Those exceptions escaped the pydantic validator unhandled, reintroducing an unhandled 500 on /api/decisions* for exactly the bug class this PR closes. Also exclude bool from the numeric branch, since bool is an int subclass and was being silently coerced to epoch 0/1. * docs: add changelog entry for PR #886 (explorer extract/decisions fixes) Documents the extraction 503, decisions timestamp 500, and folded-in spaCy caching fixes, plus the review-round hardening from Sameer6305 and the timestamp overflow/bool fix from this follow-up commit. --------- Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
hsd2514andZohaib Hassnain
|
8a4ebafb9a |
fix(context): honor explicit causal edges in decision tracing (#983)
* fix(context): honor explicit causal edges in decision tracing trace_decision_causality() inferred causes purely from shared NER entities plus timestamp ordering, so relationships recorded through add_causal_relationship() never affected the trace. When entity extraction returned nothing, trace_decision_chain() came back empty even though an explicit CAUSED edge was stored in the graph. Traverse the explicit CAUSED/INFLUENCED/PRECEDENT_FOR edges first, since they are the ground truth the caller recorded, and keep the entity and timestamp inference as an additive fallback for pairs with no explicit link. Edges whose source has no decision record (for example a graph restored via from_dict) are skipped so a stale edge cannot abort the trace. analyze_decision_influence() now reports explicitly linked decisions as direct influence rather than surfacing them only as indirect, and no longer lists the same decision under both direct and indirect. Closes #975 * fix(context): address review feedback on causal edge tracing Follow-up to the explicit causal edge fix, covering the issues raised in review. A stored edge weight of 0.0 was coerced to the 1.0 default by a truthiness check, inflating confidence_decay in the causal chain report. add_edge() is public and can create causal edges with any weight, so use an explicit None check instead. Explicit causes were collected into a dict keyed by source_id, so multiple causal edges between the same pair of decisions overwrote each other and only the last was traced. Collect every edge instead, keeping a separate set of source ids for the entity fallback exclusion. Cycle detection used a single traversal-wide visited set, so a decision reached through one branch became unreachable through another and branching graphs silently lost valid chains. Detect cycles per path instead; max_depth still bounds the traversal. Build a reverse index of causal edges once per call rather than scanning the edge list at every visited node, and use edge_type_index in the influence analysis. The three causal edge types are now a shared constant. Adds regression tests for zero weights, parallel edges, branching graphs and cycle termination. * fix(context): bound causal trace and report truncation Per-path cycle detection keeps branching graphs correct but makes the traversal combinatorial in max_depth: on a densely connected graph the number of distinct causal paths grows by roughly the branching factor per level, so a raised max_depth could return hundreds of thousands of chain reports and take seconds of CPU. Add a max_chains bound, defaulting to 10000. Rather than dropping chains silently, which is the exact failure this fix set out to eliminate, the traversal stops at the bound and appends a {"truncated": True, ...} marker so callers can always tell the trace is incomplete. A warning is logged with the same detail. Pass max_chains=None for the previous unbounded behaviour. Graphs that fit within the bound are unaffected. --------- Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> |
||
|
manjunath bhaskar
|
80b9bea0d5 |
fix(ingest): lock the repo host DNS resolve cache against concurrent mutation (#979)
* fix(ingest): lock the repo host DNS resolve cache against concurrent mutation
_REPO_HOST_RESOLVE_CACHE is a module level OrderedDict shared by every
RepoIngestor instance and thread. _resolve_repo_host_ips and
_prune_repo_host_resolve_cache read, wrote, and iterated it with no lock,
so concurrent ingest_repository() calls (e.g. from a thread pool) could
mutate the dict while another thread was iterating it during pruning.
This reliably raised RuntimeError: OrderedDict mutated during iteration
under ordinary concurrent usage, not just adversarial input.
Reproduced with 32 threads hammering _resolve_repo_host_ips with a low
TTL and small cache cap so pruning and eviction happen on nearly every
call; the crash showed up within the first few hundred iterations on
every run before the fix and did not reproduce at all after it.
Fix adds a threading.Lock guarding every read, write, and prune of the
cache. The blocking socket.getaddrinfo call stays outside the lock so a
slow DNS lookup for one host cannot stall cache access for other hosts.
Added a regression test, TestRepoHostResolveCacheThreadSafety, that
drives 32 threads through _resolve_repo_host_ips with a short TTL and
small cache cap and asserts no exception is raised.
Full test suite: 4088 passed, 332 failed, 140 errors both before and
after this change (same counts on main), all from missing optional
dependencies in this local environment (snowflake, sqlite-vec, spaCy
models, faiss/torch version mismatches), not from this fix. The ingest
and SSRF focused test files pass cleanly: 106 passed, 0 failed.
* test(ingest): fail fast on the first hung thread in the resolve-cache race test
join(timeout=30) alone doesn't fail the test if a worker hangs -- it
just returns after the timeout with the thread still running, and the
test falls through to the errors check, which trivially passes since
a hung thread never got far enough to append one. A future deadlock
could slip past this test looking green.
Assert immediately after each individual join rather than after the
whole loop: checking only once every thread has been joined means a
mass hang costs up to 32*30s = 16 minutes before the test even reaches
the check. Failing on the first hung thread caps the worst case at
~30s instead. Worker threads are daemon=True so a genuine hang can't
also block the test process from exiting.
Verified the assertion is load-bearing, not cosmetic: temporarily
injected an artificial 9999s sleep into the first worker in a
throwaway copy of the test and confirmed the test now fails in ~31s
with a clear message, instead of the ~16 minutes a mass hang would
otherwise cost. That copy was never committed.
Addresses the review comment on #979 from ZohaibHassan16 and Qodo's
automated review.
* test(ingest): fail fast on the first hung thread, for real this time
The previous commit (
|
||
|
Shubham SrivastavaandMohd Kaif
|
80b1cca07b |
test(semantic_extract): guard openai-dependent tests and assert on the logger, not stdout (#935)
* test(semantic_extract): skip openai-dependent tests when the SDK is absent, assert logs not stdout * test(semantic_extract): pass logger name to assertLogs to match suite convention All 11 existing assertLogs call sites in the suite pass a logger name string rather than a Logger instance; tests/reasoning/test_reasoner.py uses this exact .logger.name form. Behaviour is unchanged. --------- Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
|
1c0cebb1c3 |
security(context): harden Markdown import against TOCTOU symlink races (#932)
* security(context): harden Markdown import against TOCTOU symlink races Closes #856 * fix(context): harden markdown import security tests * docs(changelog): add entry for Markdown import TOCTOU symlink hardening Documents the (#932, closes #856) fix in the Unreleased/Fixed section. --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
Guofang.Tang
|
94d0c3dc07 |
fix(kg): remap relationship endpoints after entity resolution (#978)
* fix(kg): remap relationship endpoints after entity resolution * fix(kg): harden relationship endpoint remapping |
||
|
sushuaiyu
|
09c4b1b570 |
test(context): skip symlink test without Windows privilege (#908)
* test(context): skip symlink test without Windows privilege * test(context): name Windows privilege error code --------- |
||
|
Yunare Maia
|
c5d13a45db |
feat(seed): allow_private_ips opt-in for trusted internal API sources (#959)
* feat(seed): add allow_private_ips opt-in for trusted internal API sources (Closes #943) SeedDataManager.load_from_api now delegates to the shared SSRF guard (semantica/ingest/ssrf.py, added in #906) instead of raw requests.get, gaining redirect validation and bounded DNS resolution for free. New config option allow_private_ips (parsed via the shared parse_bool helper) lets trusted internal deployments load from private APIs while the secure default (block private/loopback/link-local) is unchanged. Tests updated to mock request_with_ssrf_guard; new tests cover the block-by-default behavior and the opt-in flag reaching the guard. 19/19 green in test_seed_manager.py, 25/25 across both seed suites. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ssrf): strip sensitive headers on cross-host redirects (Qodo finding) request_with_ssrf_guard reused the caller's headers on every redirect hop, so an Authorization bearer token from load_from_api could leak to a different redirect target host. Now strips Authorization and Proxy-Authorization when the redirect origin (netloc) changes, while keeping them for same-host hops (matching requests semantics). 2 new tests: cross-host redirect drops the credential; same-host keeps it. 37/37 green in test_ssrf_protection.py. load_from_api docstring now also documents cloud-metadata blocking and per-hop redirect validation. Signed-off-by: Yunare Maia <yunare@gmail.com> * fix(ssrf): strip credentials on https->http downgrade redirects (review feedback) _should_strip_auth now mirrors requests' should_strip_auth semantics: strip on hostname change, port change, or scheme downgrade; keep the credential only for the safe http->https upgrade on default ports. Previously only netloc was compared, so an https->http redirect on the same host replayed the Authorization header in cleartext. --------- Signed-off-by: Yunare Maia <yunare@gmail.com> |
||
|
|
611874e63e |
security: apply SSRF guard to feed ingestion requests (#928)
* security: apply SSRF guard to feed ingestion requests FeedIngestor and FeedMonitor fetched feed and website URLs with plain requests.get/head calls, bypassing the SSRF validation already used by web_ingestor.py and api_ingestor.py. This allowed feed URLs pointing at loopback, link-local, or other private network addresses to be fetched directly. Route all outbound requests in feed_ingestor.py through request_with_ssrf_guard, gated by the same allow_private_ips config option the other ingestors expose. * test: mock the correct request boundary in test_discover_feeds_empty The test still patched requests.get after discover_feeds() moved to request_with_ssrf_guard(), which calls requests.request and performs real DNS resolution. That left the test hitting live network/DNS. * docs(changelog): document FeedIngestor SSRF guard fix (#928, closes #927) Records the SSRF guard applied to all 5 feed-ingestion request sites, the Qodo-flagged test-mock fix, independent PoC verification, and the carried-over exception-swallowing behavior in discover_feeds(). --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
43bac6170c |
fix(vector_store): make VectorManager methods work on persistent backends (#855) (#914)
* fix(vector_store): make VectorManager methods work on persistent backends (#855) maintain_store() and collect_statistics() reached into VectorStore internals (.vectors/.metadata), which only exist for the inmemory backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus, ...) crashed with AttributeError. Add a public backend-agnostic VectorStore.count() accessor following the get_vector()/get_metadata() precedent (#843) and the NotImplementedError-on-unsupported-capability precedent of _filter_by_metadata() (#848): inmemory counts its dict, persistent backends delegate to count() when available, and raise NotImplementedError otherwise. VectorManager methods now go through count(); maintain_store() keeps the exact inmemory semantics (separate vector/metadata dict counts) and reports a 1:1 count for persistent backends, where metadata is stored alongside each vector. Tests: 10 hermetic unit tests covering inmemory, delegation and the NotImplementedError path. Core vector_store suite: 40 passed. * fix(vector_store): raise NotImplementedError when count() unavailable Address Qodo review findings on #914: - Persistent backend with no wrapped store no longer silently returns 0 (which masked a missing initialization as an empty, healthy store); it now raises NotImplementedError like get_vector()/get_metadata(). - A mis-shaped adapter exposing a non-callable 'count' attribute now surfaces a clean NotImplementedError instead of a TypeError, via a getattr + callable() capability check. Adds regression tests for both cases. * fix(vector_store): implement count() on FAISS/SQLite/PgVector backends (#914) - FAISSStore.count(): returns len(index.vector_ids); 0 when no index exists yet - SQLiteVecStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*)) - PgVectorStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*)) - VectorStore.count(): fix misleading NotImplementedError message; now describes how to add count() support to a backend adapter rather than claiming only the inmemory backend can ever support counting - VectorManager.maintain_store(): split inmemory and persistent paths: * inmemory: independently reads len(vectors) and len(metadata) and compares them as an integrity check (original semantics preserved) * persistent: calls store.count(); returns metadata_count=None because metadata is co-located with vectors in the backend and cannot be counted independently; never manufactures metadata_count=vector_count as a vacuous tautology (#914 Qodo review) - Tests: rewrite test_vector_manager_persistent.py with 31 tests covering dispatch logic, inmemory divergence detection, persistent metadata_count=None invariant, FAISSStore/PgVectorStore via mocks, and SQLiteVecStore via real in-memory SQLite (skipped when sqlite-vec absent) * docs(changelog): document VectorManager persistent-backend count fix (#914, closes #855) Records the VectorStore.count() accessor, the FAISS/SQLite/PgVector implementations added during review, and the maintain_store() metadata_count fix (no longer fabricates equality for persistent backends). --------- Co-authored-by: Sameer6305 <sskadam6305@gmail.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> |
||
|
|
91d02a0f29 |
fix(ingest): harden RepoIngestor GitPython clone surface (#868) (#905)
* fix(ingest): harden RepoIngestor against GitPython URL and option injection
Bump GitPython to >=3.1.58, allowlist clone kwargs, and validate repo URLs
before clone_from to close env-var exfiltration and option-injection paths.
* fix(ingest): accept scp-like SSH remotes in RepoIngestor URL validation
* fix(ingest): resolve repo hostnames to block SSRF via private IPs
* fix(ingest): map malformed repo URL parse errors to ValidationError
* fix(ingest): bound and prune repo host resolve cache
Cap the repository host DNS cache, prune expired entries on access, and evict the oldest entries so long-running processes cannot accumulate unbounded host lookups from user-supplied repo URLs.
* fix(ingest): cap host resolve cache and tighten env-var token checks
Bound the repo host DNS cache with pruning and oldest-entry eviction, and narrow URL env-var blocking to actual $VAR/${VAR} tokens so literal dollar signs are not rejected.
* fix(ingest): preserve repo path compatibility and NAT64 support
* docs(changelog): document RepoIngestor GitPython hardening (#905, closes #868)
Records the clone-surface hardening (GitPython floor, clone-option
allowlist, URL/SSRF validation), the two fixes made during review
(NAT64 false-positive, local-path regression), and a known residual
gap: the SSRF host check doesn't classify RFC 6598 CGNAT space
(100.64.0.0/10) as blocked since ipaddress.is_private doesn't cover it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@wifi-10-43-175-99.wifi.berkeley.edu>
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
|
2cfb5de43d |
feat(export): add opt-in metric_errors column to DistanceExporter (#960)
* feat(export): add opt-in metric_errors column to DistanceExporter
Add a 'metric_errors' field to compute_pairs() output that lets
downstream consumers programmatically distinguish legitimate 'no path'
(None) from computation failures (None + error name).
Usage:
rows = exporter.compute_pairs(include=[..., 'metric_errors'])
# row['metric_errors'] == '' → all metrics succeeded
# row['metric_errors'] == 'hop_count,weighted_distance' → those failed
Design decisions:
- Opt-in: column only appears when explicitly requested via include=
- Default export schema unchanged (backward compatible)
- Comma-separated metric names (not exception messages) — stable for
programmatic filtering without exposing internal error details
- Helpers now return (value, error_name | None) tuples internally
Follow-up to #879, as discussed in its review thread.
* fix: address Qodo findings — track betweenness errors and remove unused constant
1. _betweenness() now returns (dict, error) tuple like the other helpers,
so betweenness computation failures appear in metric_errors.
2. Removed unused _ERROR_COLUMNS constant (dead code).
All 77 tests in tests/export/ pass.
* docs(changelog): add entry for opt-in metric_errors column (#960)
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
|
0fa3483b96 |
fix(context): clarify get_node_property not-found contract (#877) (#882)
* fix(context): clarify get_node_property not-found contract (#877) Add default= param to get_node_property and get_node_attributes so callers can distinguish node-missing from property-missing using a sentinel. Fix add_node_attribute calling mutation_callback outside the lock. Tests added for all cases. * fix(context): address Qodo review findings (#877) * fix(context): wrap add_node_attribute mutation_callback in try/except (#877) The PR claimed to move the callback back inside `with self._lock`, but the diff only dropped a stray blank line -- the call stayed outside the lock, unchanged. That's actually correct: self._lock is an RLock, and _add_internal_node/_add_internal_edge deliberately release the lock before invoking the callback too, so a slow/misbehaving callback never holds up other threads. The real gap was that, unlike those two siblings, this call site didn't catch exceptions from the callback. Wrapped it the same way, with a regression test. --------- Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
18f1d55d77 |
test(normalize): make optional tests deterministic (#881)
* test(normalize): make optional tests deterministic Signed-off-by: aoright <102943475+aoright@users.noreply.github.com> * docs(changelog): add entry for #881 / #860 normalize test determinism fixes --------- Signed-off-by: aoright <102943475+aoright@users.noreply.github.com> Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com> |
||
|
|
b0080c3602 |
fix(export): log DistanceExporter metric computation failures instead of swallowing them (#879)
* fix(export): log DistanceExporter metric computation failures instead of swallowing them
The four private metric helpers in DistanceExporter (_betweenness,
_hop_distance, _weighted_distance, _semantic_similarity) each catch a bare
Exception and return None/{} with no signal. That makes an exported None
indistinguishable from a legitimate "no path exists" result, corrupting
downstream CSV/JSONL/DataFrame exports with no way to tell a real gap from a
swallowed error.
Log each caught exception at warning level with the offending source/target
before returning the existing sentinel. The exported row shape and values are
unchanged; only the observability of the failure changes.
Fixes #874
* fix(export): route DistanceExporter warnings through the semantica logger tree
get_logger(__name__) doubled the semantica. prefix (__name__ is already
semantica.export.distance_exporter), so the warnings this PR adds landed on
semantica.semantica.export.distance_exporter, a branch setup_logging() never
configures and does not reach the app's log handler. Also reworded the three
except-Exception log messages: they said "recording as no path", which
overclaims what a generic exception means.
Addresses review feedback from @KaifAhmad1 on #879.
* docs(changelog): add DistanceExporter logging fix entry
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
|
||
|
Shubham Srivastava
|
1ee3f2f214 |
fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract (#941)
* fix(kg): align GraphBuilder raw-text extraction defaults with the documented contract
_extract_from_text() defaulted ner_method, relation_method and
triplet_method to "llm" and ran relation extraction unconditionally,
contradicting the build() docstring ("ml"/"pattern"/False) and the
standalone extractor defaults. Any raw-text build() therefore required a
provider, an API key, and network access without saying so.
Defaults are now ml/pattern/pattern with extract_relations=False. LLM
extraction is unchanged and now opt-in via explicit kwargs.
Also documents relation_method and extract_triplets, which the docstring
never listed, and drops the stale "Default to LLM methods as per
requirement" comment.
Closes #930
* perf(kg): reuse extractors across texts instead of rebuilding per source
Addresses review feedback on #941. NERExtractor.__init__ loads its spaCy
model eagerly when the method includes "ml", so switching the default
from "llm" to "ml" made _extract_from_text() reload the model once per
source in a multi-document build.
Extractors are now cached per (kind, method) on the builder. Adds tests
asserting single construction across repeated texts, that distinct
methods still get distinct extractors, and that the default path runs
end to end without any provider call.
* fix(kg): keep fallback method lists working with the extractor cache
The extractor cache keyed directly on `method`, but all three extractors
accept a list for fallback ordering (e.g. ner_method=["pattern", "ml"]),
so a list argument raised TypeError: unhashable type: 'list' before
extraction started. Lists are now converted to tuples for the cache key
only; the extractor still receives the original value.
Also seeds _extraction_stats in __init__. It was previously created only
in build(), so calling _extract_from_text() directly — as the report's
repro does — raised an AttributeError that the broad except swallowed and
logged as "Entity extraction failed".
Adds coverage for list methods on all three extractors, cache reuse for
equal lists, and distinct entries for different orderings.
* fix(kg): forward extracted relations into triplet extraction
_extract_from_text() passed only entities= to extract_triplets(), so
TripletExtractor re-derived relations itself whenever relations is None,
using a method taken from triplet_method rather than relation_method.
That duplicated work and could yield triplets inconsistent with the
relations already extracted.
relations is now initialized to None, holds the extracted list when
extract_relations=True succeeds, and is forwarded to extract_triplets().
When extraction is disabled or fails, None is passed and
TripletExtractor's existing self-derivation is unchanged.
Folded in at maintainer request rather than tracked as #944.
* docs(changelog): note that #878 documented the LLM defaults before this landed
#878 merged while this was in review and resolved the same code/docstring
mismatch in the opposite direction. Records that #930's decision makes
the code the side that changes, and that #878's docstring formatting is
retained.
|