VectorStore cannot actually migrate to or from qdrant yet. _init_backend_store constructs QdrantStore without connecting or selecting a collection, so reads raise a Collection not initialized error, and the facade store_vectors dispatches only to add/add_vectors while QdrantStore exposes insert_vectors, so writes raise NotImplementedError.
Both are pre-existing facade gaps that nothing had exposed, since migrate previously only allowed faiss/sqlite/pgvector. Adding qdrant to the allowlist claimed support that does not work end to end, so it is removed along with the dimension inference that only fires for backends missing a .dimension attribute. Tracked separately; this PR keeps just the iter_all primitive.
Addresses review feedback on this PR:
- Guard 2 and the PR-comment JS parser both required every dependency
in pip-audit's report to carry an array-valued `vulns` field. A
dependency pip-audit can't resolve/audit is reported instead as
{"name": ..., "skip_reason": ...} with no `vulns` key at all (see
pip_audit._format.json.JsonFormat._format_dep) - a normal, documented
shape, not a malformed one. That made a single unauditable package
hard-fail the whole job and show "Invalid report structure" in the PR
comment, reintroducing the same class of scan-unrelated CI break this
migration was meant to fix for Safety. Both now accept skipped
entries, treat them as zero vulns, and surface them explicitly (job
log + PR comment) instead of silently dropping or crashing on them.
Verified the fixed jq queries and JS parse logic against synthetic
pip-audit report fixtures covering the normal, skipped, and malformed
shapes.
- Restored a `workflow_dispatch` trigger on security-scan.yml. Deleting
security.yml (which had it) left no way to manually run a dependency
audit on demand.
- Updated SECURITY.md, which still described security.yml as a live
scanning workflow and Safety as an active scanner after this PR
deletes both.
The module assigned MagicMocks into sys.modules at import time and never
removed them. pytest imports every test module during collection before
running anything, so those mocks were live while later modules were
imported and each bound them into its own globals.
132 tests passed alone and failed in a full-suite run as a result. Full
suite goes from 199 failed / 5506 passed to 67 failed / 5638 passed.
A tearDownModule cannot fix this: collection has already finished by the
time it runs. The extractors resolve 'from .methods import
get_entity_method' lazily inside their methods, so the stand-in only has
to be in sys.modules while a test executes - it is now installed per test
via patch.dict in setUp and removed by addCleanup.
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
The relationship-endpoint fix merged in #1170 covers the main ontology generation pipeline, but the public property-inference path still had the same gap.
`OntologyGenerator.infer_properties()`, and the `PropertyGenerator` it delegates to, fell back to `owl:Thing` for both domain and range when a relationship used entity IDs or aliases instead of explicit `source_type` / `target_type` values. The pipeline resolved those endpoints correctly, but the public API path did not.
This moves the existing alias-building and endpoint-resolution logic out of `OntologyGenerator` and into a shared `relationship_utils` module:
* `build_entity_aliases`
* `get_relationship_endpoint`
* `resolve_relationship_endpoint_type`
`OntologyGenerator` now uses those shared helpers instead of keeping its own copies.
`PropertyGenerator._infer_object_properties()` now also receives the entity list, builds the same alias index, and uses the shared endpoint resolver. This replaces the old fallback:
```python
rel.get("source_type") or self._infer_class_from_entity(...)
```
which could only fall back to `owl:Thing` because `_infer_class_from_entity()` never actually resolved an entity.
There are two small behavior changes from centralizing the logic. `build_entity_aliases()` now converts `entity_type` to `str` before adding it to the alias set, avoiding mixed-type alias values. `resolve_relationship_endpoint_type()` returns `None` rather than `""` when there is no usable explicit type, since an empty string isn't a meaningful endpoint type.
The new `test_public_infer_properties_resolves_id_endpoints` covers the broken public API path directly. It creates entities and ID-based relationships through `infer_classes()` / `infer_properties()` and verifies that the inferred `worksFor` property resolves to `Person` for the domain and `Organization` for the range instead of falling back to `owl:Thing`.
That exercises the same endpoint-resolution behavior already covered by the pipeline tests, but through the public entry point that was still missing it.
The Security Scan workflow already scans `requirements-ci.txt` directly, but malformed Safety output could still be treated as a clean scan. If the report existed on disk but `vulnerabilities` was missing, `null`, or the wrong type, the workflow could end up counting it as zero findings.
This adds a structural check immediately after the report is written. `vulnerabilities` must be an array; otherwise the step fails closed with a clear error instead of treating a broken report as a successful scan.
There was a related problem in the PR reporting path. The comment step already knew how to render an `Invalid report structure` warning, but that branch was effectively unreachable. In GitHub Actions, a custom `if:` is implicitly gated by `success()` unless it includes a status function such as `always()` or `failure()`. Once the Safety step exited non-zero, the Upload and Comment steps were skipped, so the warning could never be posted.
Fixing that required changing how Safety failures flow through the job rather than just adding another guard. The Safety step now uses `continue-on-error: true`, which lets Bandit and Semgrep continue running and allows the Upload and Comment steps to process the failed or malformed Safety result.
Because `continue-on-error` means the Safety step no longer carries the job's final failure signal itself, the workflow now tracks that state explicitly with `SAFETY_SCAN_STATUS`. It is set to `failed` at the start of the Safety step, before any validation runs, and changes to `passed` only when the report is valid and contains zero vulnerabilities.
That default-failed behavior covers every other exit path: a missing report, malformed `vulnerabilities` field, invalid vulnerability count, Safety failure, or an actual vulnerability finding all leave the status as `failed`.
A final `Enforce Safety Gate` step checks `SAFETY_SCAN_STATUS` and fails the job unless it is exactly `passed`. This keeps the same merge-blocking behavior while still allowing the rest of the security checks and reporting steps to run after a Safety failure.
This is a follow-up to #1356. The overlapping Safety behavior changes and duplicate pip-audit path from that PR were dropped after `main` picked up the canonical fix for the underlying `cuda-toolkit` crash. This change keeps only the report-validation hardening that remains independent of that fix.
`FAISSIndex.save()` previously wrote only the raw FAISS index. `load()` then rebuilt the wrapper with empty `vector_ids` and `metadata`, so that state was never restored.
That made a save/load round trip effectively unusable through the wrapper API: `scan_vectors()` returned no vectors, `count()` returned `0`, and `get_vector(id)` returned `None` for IDs that were present in the underlying FAISS index.
This also affected migration. `semantica store migrate --from faiss` could load a valid FAISS index, see zero vectors through `scan_vectors()`, migrate nothing, and still exit successfully. Since FAISS is a supported migration source, this was a silent data-loss path rather than just a persistence bug.
The fix adds a `.meta.json` sidecar next to the FAISS binary. It stores:
* `vector_ids`
* `metadata`
* `dimension`
* `index_type`
The sidecar is written atomically using a temporary file and rename. `save()` also serializes the metadata before writing the FAISS binary, so a serialization error fails before either persistence artifact is created. That avoids leaving a valid-looking index file behind without the metadata needed to use it correctly.
On load, `dimension` and `index_type` come from the sidecar rather than the caller's arguments. This makes the reconstructed wrapper reflect the index that was actually saved instead of relying on the caller to provide matching values.
There are also explicit checks for incomplete or inconsistent persisted state. If the sidecar is missing, which can happen with indexes written by older versions or when only the FAISS binary was copied, `load()` emits a `RuntimeWarning` instead of silently returning an apparently usable wrapper with no IDs or metadata. If the number of saved vector IDs doesn't match the FAISS index's `ntotal`, `load()` raises `ProcessingError` rather than returning a state where vectors exist in FAISS but can't be reached through `scan_vectors()`.
Metadata serialization changed during review as well. The first version used `json.dumps(..., default=str)`. That avoided failures for values such as `datetime`, `UUID`, and `set`, but it was lossy: those values came back as strings instead of their original Python types.
That was replaced with a tagged encoder/decoder that preserves the supported types across a round trip. It currently handles sets, datetimes, dates, UUIDs, NumPy scalars and arrays, and bytes, with bytes stored as base64.
The decoder also uses an exact-schema check for tagged values. A normal dictionary that happens to contain a reserved tag key alongside other fields is left alone instead of being interpreted as an encoded type.
The final implementation was spread across fourteen commits, mostly following review feedback. Those changes included cleaning up conflict markers from an unfinished stash pop, expanding round-trip and retry coverage, adding the missing-sidecar warning, adding an end-to-end `scan_vectors()` persistence test, replacing lossy metadata serialization with the tagged format, checking FAISS/sidecar count mismatches, adding `bytes` support, and reordering `save()` so metadata serialization happens before the FAISS index is written.
Step 5 explicitly avoids RDFExporter's compact projection and builds
the Turtle export from the TripletStore's own triples instead, but the
Summary cell still credited RDFExporter -- a leftover from before the
rdflib-based export replaced it. Correct the claim to match the code.
Also pin the install to >=0.6.7: earlier releases could return
ontology classes with an empty uri (#1103), which made entity_type
mappings silently resolve to None instead of raising, so the notebook
would appear to pass while never actually typing its instances.
Fixes GHSA-73wf-gq98-2v4g (prototype pollution / DoS via unguarded
browserslist-stats.json parsing) and GHSA-c83g-rgw3-j3cx (unbounded
cache growth leading to OOM), both patched upstream in 4.28.7.
`get_collection()` attached any collection right after the existence check, with no look at its schema. A collection with an INT64 primary key, or one missing the `metadata` field entirely, would attach without complaint and only fail later, inside `get_vector()` or `get_metadata()`, with an error that gave no hint the real problem was upstream at attach time.
This adds a schema check between the attach and the assignment to `self.collection`, so a mismatch is caught at the point of failure instead of surfacing three calls later as an unrelated-looking error. The check validates against exactly the shape `create_collection()` builds: a `VARCHAR` primary key named `id` with `auto_id=False`, a `FLOAT_VECTOR` field named `vector`, and a `JSON` field named `metadata`. Anything else, wrong dtype, wrong name, a missing field, or an auto-generated id, is rejected before the store ever holds a reference to it.
The auto_id and metadata-dtype checks were added in a second pass after review. A collection with `auto_id=True` still attached cleanly and only broke once the store tried to insert with the explicit ids it always sends, and a `metadata` field that existed but wasn't `JSON`-typed only broke during a later write or metadata filter, for the same reason: schema drift that looked fine at attach time and failed downstream instead of at the source.
Nine tests cover this: the one matching-schema case that should succeed, and each rejection path independently, wrong pk dtype, missing pk, wrong pk name, auto_id pk, missing vector field, wrong vector dtype, missing metadata field, and non-JSON metadata.
Closes#1331.
* feat(context): add ErasureCoordinator for cross-store entity erasure
purge_node() is graph-scope by design (#957), so an entity removed from the
graph can survive verbatim as an AgentMemory item and as an embedding. The
changelog names GDPR Article 17 as purge's motivation, and an Article 17
erasure the vector store can still answer queries from is not an erasure --
it is worse than none, because purge_node() returns True and writes a
tombstone attesting the content is gone.
ErasureCoordinator composes the existing public APIs to drive the cascade and
returns an ErasureReceipt recording what each store reported. Nothing in
context_graph.py or agent_memory.py changes behaviorally; ContextGraph keeps
its documented graph-scope contract instead of acquiring references that would
invert the dependency.
Honest partial reporting is the point. Stores report erased / not_found /
not_configured / unsupported / failed, and complete is False when any store
reports unsupported or failed. FAISS, Milvus and Weaviate expose no delete at
all, so erasure genuinely cannot be completed on them today -- the receipt
says so rather than reporting a success it did not achieve.
Erasure runs outward-in (vectors, memory, graph). The tombstone is the durable
attestation, so writing it first would let a crash mid-cascade leave a record
claiming more than happened; erasing the graph last leaves a partial failure
recoverable and honest.
The memory sweep pages until dry and re-queries afterwards rather than
trusting one find_by_entity() call, whose limit=10 default silently truncates
the very check a caller uses to decide the erasure is done. Unsupported vector
backends are detected by probing the wrapped backend, since the VectorStore
facade declares delete_vectors() for every backend and only raises
NotImplementedError once called.
27 tests against real ContextGraph/AgentMemory instances, including the
25-items-on-one-entity regression that fails against a naive single-call
sweep. Full tests/context/ suite: 596 passed.
Closes#1018
* docs(context): document ErasureCoordinator in the context API reference
* test(context): exercise ErasureCoordinator against a real VectorStore
The vector-leg tests asserted the three backend shapes the coordinator
expects -- delete_vectors / delete / neither -- against fakes, which is worth
exactly as much as the assumption that a real store looks like one of them.
VectorStore(backend="inmemory") runs without external services, so it can
hold that assumption to account.
Adds the end-to-end case the receipt actually attests to: a real ContextGraph,
AgentMemory and VectorStore, where the embedding is written by
AgentMemory.store() and has to be gone afterwards. That exercises the memory
leg's own delete_memory() vector cascade rather than the coordinator's model
of it.
The real backend also pins a limit worth knowing before trusting the receipt:
it pops the ids and returns True whether or not they were there, and no
backend offers a portable existence check, so `erased` on the vectors leg
means the store accepted the delete for the ids given -- not that embeddings
were really removed. The memory leg re-queries to confirm and so is the
stronger claim. Documented on STATUS_ERASED, _erase_vectors(), and both status
tables.
tests/context/: 599 passed.
* fix(context): address review findings on ErasureCoordinator
Timestamp drift (high). erase_entity() resolved erased_at up front but passed
the caller's original `at` down to purge_node(), so on the default at=None
path the coordinator and the graph each took their own now() and the receipt
attested to a different instant than the tombstone it points at -- breaking
the invariant this module states most loudly. The resolved value is now what
the graph receives. The existing test passed only because it supplied an
explicit `at`, which hides the drift; the regression test covers at=None,
which is what callers actually use.
Backend delete results. The vectors leg treated anything other than the
literal False as success, but no in-repo backend returns a bool -- Qdrant
returns {"status": <UpdateStatus>} and Pinecone {"deleted": True}, so every
dict read as success and the backend's own account of the delete was thrown
away. Results are now interpreted by shape and the payload is kept in the
receipt as backend_result, stringified so it stays JSON-serializable as an
audit record. Bool markers match by identity so a 0 count isn't read as
False; string markers match as substrings so an enum rendering as
"UpdateStatus.FAILED" isn't read as success.
Falsey vector store. The "at least one store" guard used `not vector_store`,
rejecting a valid store whose __bool__/__len__ makes an empty instance falsey
and then reporting vector_store=None when an object had been passed. It now
separates None (absent) from False (deliberately disabled) from provided, and
echoes what it received.
`at` annotations. Widened to int/float, matching the ContextGraph normalizer
they delegate to, so the coordinator stops advertising less than the API it
wraps.
tests/context/: 608 passed.
* fix(context): report memory-owned vectors that survive erasure (#1018)
A receipt could read complete while an embedding was still in the vector
store. The vector leg deleted `vector_ids` or `[entity_id]`, and the memory
leg relied on `AgentMemory.delete_memory()` to cascade to the vectors each
item owns. That cascade is best-effort: `_delete_vector_ids()` raises when a
backend returns False, `delete_memory()` catches it, logs a warning, and
still returns True. So `batch_delete` counted the item, the residual re-query
found no items, the memory leg reported `erased`, and nothing in the receipt
recorded that the embedding was refused.
Reproduced with a store that deletes the entity-keyed id and refuses the
memory-owned one: `receipt.complete` was True with the embedding still live.
That is the failure mode this module exists to prevent -- a receipt is a
compliance artifact, and one that overstates is worse than none.
Fix by deleting memory-owned vector ids through the coordinator's own vector
leg, which reports honestly, instead of trusting the memory leg's cascade.
The ids are collected before anything is deleted, while the items still exist
to be enumerated, and are unioned with any caller-supplied ids rather than
replacing them.
This needs one addition to AgentMemory: `vector_ids_for(memory_id)`, a
read-only accessor mirroring the fallback in `delete_memory` (an item stored
without tracked ids is keyed by its own memory id). Reaching into
`_vector_ids` from the coordinator would have been the internals-access
pattern this repo keeps getting bitten by. No existing AgentMemory behaviour
changes -- `delete_memory()` still cascades best-effort, so other callers are
unaffected; the coordinator simply no longer depends on that being reliable.
It does mean the vectors are attempted twice, which is a no-op on a working
store and only ever costs a log line.
Note this deviates from the PR's stated "nothing in agent_memory.py changes"
constraint. The constraint could not hold: with `_vector_ids` private and no
portable way to ask a vector store what it still holds, the coordinator had
no way to make the claim truthful without it.
Four tests: the refused-vector case (receipt must be incomplete), that
memory-owned ids reach the store, that explicit `vector_ids` do not displace
them, and the accessor's fallback. The first three were confirmed to fail
against the previous coordinator, on the `receipt.complete` assertion rather
than incidentally. 655 tests pass across tests/context and the agno
integration.
* fix(context): make erasure receipt vector failures honest
* fix(context): optimize erasure pagination handling
* fix(context): use one timestamp for batch erasure
* style: strip trailing whitespace from erasure.py and test file
* docs(changelog): correct test counts to 48 / 738 after review rounds
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* fix(ci): drop --ignore from Safety check, filter accepted CVEs in jq instead
The follow-up to #1370: adding `--ignore SFTY-20260120-40557` to the
`safety check` invocation reintroduced the exact crash #1131/#1157 had
just fixed - "Unhandled exception happened: 'cuda-toolkit'" - but only
once Safety actually has a live vulnerability match to apply the ignore
against (the plain, un-ignored scan against the same requirements-ci.txt
had already succeeded and correctly reported that same match on main,
per the run right before this one).
I couldn't reproduce this locally: my local Safety installation doesn't
surface the live cuda-toolkit CVE match at all (its open-source
vulnerability DB appears to lag CI's), so --ignore never had a real
match to crash on in my testing. That's on me - I should have caught
that my "0 vulnerabilities" local result meant the DB hadn't even seen
the finding yet, not that the fix worked.
Since I can't safely iterate against Safety's own --ignore path without
live-DB access, this moves the "should we still fail on ID X" decision
out of Safety entirely: run the plain scan (the one path an actual CI
run has now proven doesn't crash), then filter the accepted vulnerability
ID out of the report ourselves in jq before counting/printing. Verified
the jq expression directly against a synthetic report shaped like a real
one (id present + one other unrelated id): filters exactly the intended
entry, and - as a bonus - iterating over a null/missing "vulnerabilities"
key with jq now raises inside jq the way the existing guard comment always
assumed it did, rather than silently coming back as 0.
* fix(ci): apply the accepted-CVE exclusion list to the PR comment too
Qodo caught a real gap on this PR: the jq-based exclusion I added only
covers the CI gate (the VULNS count and the failure-path detail print).
The "Comment PR with Security Results" step reads safety-report.json
independently in its own JS, with no filtering at all, so a PR touching
only the accepted cuda-toolkit CVE would still get a comment saying
"Found 1" even though the gate itself correctly treats it as
non-actionable and passes.
Export IGNORED_VULN_IDS via $GITHUB_ENV from the shell step so the JS
step can read the same list, and filter data.vulnerabilities there
before rendering - with a footnote naming what was excluded and why,
so the comment stays transparent about the accepted finding rather than
just silently hiding it.
Verified the JS logic standalone against two synthetic reports: one with
the accepted CVE plus an unrelated real one (shows only the real one,
plus the footnote), and one with only the accepted CVE (shows "No
findings" plus the footnote, rather than misleadingly looking identical
to a clean scan with no explanation).
Merging #1357 surfaced a real (not crashed) Safety finding: cuda-toolkit
13.0.3.0 < 13.1.0 is affected by SFTY-20260120-40557 / CVE-2025-33228.
This can't be fixed with a version bump on our end: torch 2.13.0 (the
latest release on PyPI - there is no newer one) hard-pins
`cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,
nvjitlink,nvrtc,nvtx]==13.0.3` on Linux via its own METADATA, not a loose
transitive requirement we control.
The underlying CVE is OS command injection in NVIDIA Nsight Systems'
gfx_hotspot recipe (process_nsys_rep_cli.py), which requires a human to
manually invoke that script with an attacker-supplied string. It isn't
reachable from any Semantica code path, and Nsight Systems isn't even
part of the extras torch requests here (cublas/cudart/cufft/cufile/
cupti/curand/cusolver/cusparse/nvjitlink/nvrtc/nvtx - no Nsight extra
among them).
Ignoring this one vulnerability ID only (not the whole package or a
blanket policy) so CI reflects actionable risk. Re-evaluate once torch
ships a release that pins a patched cuda-toolkit.
* feat(explorer): serve the RDF export formats the MCP tool already offers (#1131)
`POST /api/export` accepted only `json` and `csv` and answered 422 for everything
else, while the MCP `export_graph` tool resolved Turtle, N-Triples, RDF/XML,
JSON-LD, GraphML and Parquet through `semantica.export`. Two surfaces of one
product disagreeing about what the product can do — and for an RDF-native project,
a graph that loads as JSON-LD and cannot be exported as RDF is a one-way door.
The route now reaches the same exporters the MCP tool uses. Nothing is
reimplemented: `RDFExporter.export_to_rdf` and `GraphMLExporter.export` receive the
dict `session.build_graph_dict()` already builds.
The alias table is a copy of `mcp/tools/export.py::_FORMAT_ALIASES` plus the
spellings the issue mentioned (`ntriples`, `rdf-xml`), and a test asserts the two
tables agree — if either drifts, the formats a caller can use would depend on which
door they came through.
Media types and extensions per serializer, so a Turtle export is `text/turtle` and
not `application/json` with a `.json` name.
The 422 message now names what IS supported. The old one said only that the format
was unsupported, which reads as "this format does not exist" rather than "this door
does not open it" — that is what sent me looking through the library.
Parquet is left out on purpose: `ParquetExporter.export` writes a file and returns a
path, so serving it over HTTP is a different shape of change and deserves its own
review.
Tests, in `TestImportExport`: the seven RDF spellings, each **parsed with rdflib**
rather than asserted on strings — a response that merely looks like Turtle is what
lets this class of gap survive a suite. Plus the alias-agreement canary and the
error message. Three mutations (rejecting RDF again, breaking one alias, emptying
the message) each turn the matching tests red.
110 tests in `tests/explorer/test_explorer_api.py` pass.
* fix(explorer): complete RDF export support
* fix(explorer): secure GraphML temporary file handling
* fix(ci): scan declared dependencies with Safety
---------
Co-authored-by: 13g4d0 <13g4d0@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
ClassInferrer.infer_classes() groups entities by their type string, then
normalizes each group name (PascalCase + singularize) when it builds the
ontology class. Two source types that only differ in casing or plurality,
like Person and person, both normalize to the same class name. Nothing
caught that, so the second type's entities silently got treated as
instances of the first type's class, and property inference downstream
picked up whichever properties happened to win.
Added a pass right after entities get grouped by type: normalize every
type name that meets min_occurrences, and if two different source types
land on the same normalized name, raise ValidationError before any class
gets built. The check reuses the exact same min_occurrences filter the
real class-emission loop uses, so it only fires on collisions that would
actually produce duplicate classes, not on types that get filtered out
anyway.
The error carries validation_context with the normalized name mapped to
every source type that collided into it, so whoever's calling this can
see exactly what to rename instead of just getting a generic message.
Added test_class_inference_rejects_normalized_type_collisions covering
Person/person landing on the same class.
Follow-up to #1171.
Claude Code's plugin schema rejects "agents": "./agents" (a bare
directory string) with:
Validation errors: agents: Invalid input
so the bundled plugin has never been installable. Unlike "skills",
which accepts a directory string, "agents" must be an array of .md
file paths.
Replaced the string with the explicit list of the three agent files.
Verified with `claude plugin validate plugins` (2.1.231): fails on the
old manifest with the error above, passes after this change.
Added tests/test_plugin_manifest.py to guard the manifest shape: agents
is a non-empty array of existing .md paths that stays in sync with
plugins/agents/, and the skills directory exists.
Fixes#1350
Advanced chapter 09 labeled an introductory composition of
already-taught APIs as an enterprise semantic layer: TripletStore was
imported but never used, property_mappings stayed empty, mappings were
derived by fragile name matching (works_for never matched worksFor),
and the exported RDF was the original graph rather than an
ontology-aligned one.
Replace it with introduction/26_Semantic_Layer_Basics.ipynb, which
demonstrates the minimal semantic-layer composition honestly:
- build a small graph, generate an ontology (min_occurrences=1 so all
demo classes are inferred, base_uri in the user's namespace)
- explicit entity-type, relationship-type, and property mappings read
from the ontology's inferred_from metadata instead of name matching
- apply the mappings to produce an ontology-aligned graph, export it
as Turtle, and note the file exporter's property projection
- store the aligned graph in the embedded Oxigraph TripletStore and
answer a business question with one SPARQL query
- distinguish teaching mappings from governed production mappings and
point to Advanced 13 as the production continuation
Advanced 13 gains a positioning note naming the new lesson as its
prerequisite; introduction/14_Ontology links forward to the new
lesson. All cells execute top to bottom (verified with the embedded
Oxigraph backend).
Closes#1325
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first-knowledge-graph lesson read the parser output from a key it
never returns (content vs text), then masked the failure with
hard-coded entities, a manually assembled NetworkX graph, and an
uninvoked KGVisualizer; the final cell deleted the sample file, so
rerunning intermediate cells failed. Rework the notebook so every
stage consumes the previous stage's output:
- parse via parsed_document["text"] with an assertion that content
was actually extracted
- real NERExtractor/RelationExtractor output replaces the simulated
entities and wrong hard-coded offsets
- GraphBuilder builds the graph from actual relation endpoints via a
mention-span -> graph-ID map (consistent with notebook 07)
- KGVisualizer.visualize_network renders the graph and saves HTML
- deletion moved to an explicit optional cleanup cell, so parsing and
downstream cells stay rerunnable
Closes#1289
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The knowledge-graph lesson fabricated relationship endpoints from loop
indices, hid the corruption behind count-only output, and displayed
only merged duplicate groups as the deduplicated result. Rework the
notebook so that:
- graph edges come from Relation.subject/Relation.object mapped
through a mention-span -> graph-ID table
- the sample text keeps two separate "Apple Inc." mentions without the
sentence-boundary merge edge case
- entity resolution shows which mentions merged (merged_from) and
remaps relationship endpoints onto the canonical entity
- deduplication reports merge operations separately from the complete
deduplicated set (merged + untouched entities)
- each stage prints its transformed records, and lightweight
assertions pin the expected canonical entities and edges
Closes#1287
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lead the README with Semantica's identity as a semantic/context/knowledge
layer (Context Graph, KG, ontology and vocabulary governance via OWL/SHACL/SKOS),
with decision provenance and audit trails framed as a property of that
structure rather than the flagship pitch.
The Neo4j example in the persistent graph store section was passing a raw
Neo4jStore straight into GraphBuilder. GraphBuilder calls add_nodes() and
add_edges() on whatever it's given, and those only exist on the GraphStore
facade, not on Neo4jStore itself. Anyone who copied the example got:
AttributeError: 'Neo4jStore' object has no attribute 'add_nodes'
Swapped the import and construction to GraphStore(backend="neo4j", uri=...,
user=..., password=...), which wraps Neo4jStore internally and actually has
the methods GraphBuilder needs.
Added a test in tests/kg/test_graph_builder_with_graph_store.py that builds
a small graph through GraphBuilder with a mocked GraphStore and checks
add_nodes/add_edges get called. Also kept a test for GraphBuilder without a
graph_store at all, so that path doesn't regress either.
Closes#1135
* fix(ci): drop unpinnable benchmarks/requirements.txt install
Scorecard flagged this pip install as unpinned-by-hash (#6082). Can't
hash-pin it - benchmarks/requirements.txt doesn't exist in this repo, so
there's nothing to compile a lockfile from. Dropping it instead of
leaving it unpinned: the job already fails on the next real step
(benchmarks/benchmarks_runner.py, also missing), so this line wasn't
doing anything useful to begin with.
* fix(ci): hash-pin the spacy model download in benchmark.yml
Qodo review on this PR: dropping the benchmarks/requirements.txt install
(the previous failure point) let the job actually reach
`python -m spacy download en_core_web_sm`, which fetches an unpinned,
unhashed wheel from spacy-models' GitHub releases - undoing the point of
this PR by exposing a real unpinned-install path instead of a dead one.
Replaced with a hash-pinned direct-URL entry in benchmark-extra.in/.txt
for en_core_web_sm-3.8.0 (matches the spacy==3.8.15 already pinned in
base-deps.txt). uv independently computed the same sha256 I got via a
manual curl+sha256 of the release asset, and a --require-hashes dry-run
install verifies clean.
* fix(ci): stop checkov's suppressed checks from reopening as new alerts
Root cause found, not just worked around: checkov's SARIF exporter
includes every evaluated check as an ordinary result, including ones it
internally marked SKIPPED via the inline # checkov:skip= comments and
checkov.io/skipN annotations already on the Helm chart. It never uses
SARIF's own `suppressions` field and never drops them - so the exact same
already-suppressed finding reopens as a brand-new code scanning alert
number on every single run, forever (#6035/#6036, #6112-6115,
#6128-6131 are all the same 4 findings, manually dismissed 3 times now).
checkov's JSON output *does* correctly record which checks were skipped.
Added .github/scripts/filter_checkov_skipped.py, which cross-references
the JSON's skipped_checks against the SARIF's results (matched by check
ID + the last two path segments, since the two outputs use different path
roots) and drops anything checkov itself already decided to suppress,
before upload. Verified locally against a real checkov+helm run: removed
exactly the 4 known-suppressed helm chart results, left the 2 genuinely
real findings (deploy/gcp/cloudrun-service.yaml, deploy/kubernetes/
deployment.yaml) untouched.
* fix(ci): drop unpinnable benchmarks/requirements.txt install
Scorecard flagged this pip install as unpinned-by-hash (#6082). Can't
hash-pin it - benchmarks/requirements.txt doesn't exist in this repo, so
there's nothing to compile a lockfile from. Dropping it instead of
leaving it unpinned: the job already fails on the next real step
(benchmarks/benchmarks_runner.py, also missing), so this line wasn't
doing anything useful to begin with.
* fix(ci): hash-pin the spacy model download in benchmark.yml
Qodo review on this PR: dropping the benchmarks/requirements.txt install
(the previous failure point) let the job actually reach
`python -m spacy download en_core_web_sm`, which fetches an unpinned,
unhashed wheel from spacy-models' GitHub releases - undoing the point of
this PR by exposing a real unpinned-install path instead of a dead one.
Replaced with a hash-pinned direct-URL entry in benchmark-extra.in/.txt
for en_core_web_sm-3.8.0 (matches the spacy==3.8.15 already pinned in
base-deps.txt). uv independently computed the same sha256 I got via a
manual curl+sha256 of the release asset, and a --require-hashes dry-run
install verifies clean.
Dependabot flagged 12 aiohttp advisories (1 high, rest moderate/low - CVE
range covering request smuggling, websocket/parser bugs, cookie/redirect
issues) against aiohttp==3.13.5 pinned in checkov.txt. checkov==3.3.1
itself pinned `aiohttp<3.14.0`, which excludes every fixed release;
3.3.16 (latest) relaxes that to `<3.15.0`, so bumping checkov also lets
aiohttp resolve to 3.14.3 (fixes all of them).
Two alerts remain open, both genuinely blocked upstream rather than
something a version bump here can fix:
- asteval: checkov 3.3.16 (latest, still) hard-pins asteval==1.0.6 with
no range; the fix (1.0.9) is unresolvable without violating checkov's
own declared dependency - confirmed via `uv pip compile` refusing to
solve it. Needs checkov itself to bump the pin upstream.
- ecdsa: 0.19.2 is already the latest release; the Minerva timing-attack
advisory has no patched version, since python-ecdsa's maintainers have
stated side-channel attacks are out of scope for the project.
Both are checkov's own transitive deps, used only for local static IaC
analysis in defender-for-devops.yml (no network signing/cloud-auth calls
that would actually exercise ecdsa's signing path) - dismissing on
GitHub with that reasoning as a separate step.
* fix(docker): split explorer-extra.txt by Python version, fix broken build
main's container-scan.yml has been failing since PR #1338 merged:
ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
standard-aifc from .../standard_aifc-3.13.0-py3-none-any.whl
(from audioread==3.1.0->-r explorer-extra.txt (line 30))
Root cause: explorer-extra.txt was compiled with `--python-version 3.11`
but is installed on the Dockerfile's actual python:3.13-slim interpreter.
librosa's audioread dependency needs standard-aifc/standard-sunau only
under `python_version >= "3.13"` (Python 3.13 dropped aifc/sunau from
stdlib) - a file resolved for 3.11 has no hash for those packages at all,
so --require-hashes fails outright once pip resolves against the real
3.13 environment instead of silently under-pinning.
Splits the file in two: explorer-extra-py311.txt (ci.yml, unchanged
resolution) and explorer-extra-py313.txt (Dockerfile, newly compiled for
--python-version 3.13). They aren't interchangeable and shouldn't be
recombined - documented in .github/requirements/README.md, including how
to catch this class of bug before it ships again.
* fix(ci): correct stale -o path in explorer-extra-py311.txt header
Qodo review on this PR: the autogenerated header comment still said
-o .github/requirements/explorer-extra.txt (the pre-rename path), which
would silently regenerate the wrong file if someone copy-pasted it.
Zohaib Hassnain