convert_kg_to_rdf copies metadata into the RDF-ready dictionary at
rdf_exporter.py:302 and no serializer has ever read it back out. Turtle,
N-Triples, RDF/XML and RDFExporter's JSON-LD each write an entity's id,
type, text and confidence and nothing else, so an entity keeps its
confidence score and loses what produced it. JSONExporter's json-ld path
keeps the same fields, which is how one knowledge graph exported two ways
carried the user's data through one exporter and none through the other.
Measured on e3405ebc with an entity carrying four metadata keys: 3 triples
per format, 0 of them metadata. With this change: 7 triples per format,
4 of them metadata, and the same four in all four formats.
The keys Semantica itself writes are mapped to declared terms in
DEFAULT_METADATA_TERMS and declared in semantica-ns.ttl. A key the caller
supplied is not: which namespace an arbitrary key belongs in is #1146, and
that issue is open on the maintainer's modelling call, so the exporter
warns and skips rather than inventing an IRI. Callers who already know the
answer pass metadata_terms={key: iri}.
Two keys cannot keep their own name. sem:source is already the
ObjectProperty holding the subject of a reified relationship, so the Neo4j
loader's "source" is written as sem:sourceSystem and its "uri" as
sem:sourceUri, the one term whose value is a node rather than a literal.
sem:builtAt and sem:snapshotAt have range xsd:string, not xsd:dateTime.
GraphBuilder stamps with a timezone-naive datetime.now(), and #1114 is the
demonstration of what typing such a value as xsd:dateTime costs: a
timezone-qualified SPARQL filter over it silently drops the row. #1121
swept export and provenance and deliberately left kg/ alone.
Graph-level metadata is written only when the caller names the graph with
graph_uri, because this serializer has never minted a document node and
#1147 is where that default belongs once it lands.
The lexical form and datatype of a value are chosen once, in
_typed_literal_parts, so the four serializers cannot come to disagree
about them the way they disagreed about confidence in #1100. The JSON-LD
path writes explicit @value/@type rather than JSON's native numbers,
which would have made an integer xsd:double there and xsd:integer
everywhere else.
21 tests, asserting on the parsed graph in all four formats. Output is
unchanged when no metadata is present. Full-suite failure set is identical
to the parent commit: 512 = 512.
A JSON-LD document with a top-level `@id` *and* `@graph` places its terms in a named
graph. `rdflib.Graph.parse()` loads only the default graph and discards the rest
without raising, so every class and property in such a document was dropped while
the load reported success.
`OntologyIngestor.ingest_ontology` now parses into a `Dataset` and flattens the
quads into the working `Graph`, keeping both the default and the named graphs. This
is the same `Graph` -> `Dataset` migration #757 made for `JenaStore` (#756); the
ingest path was not covered by it.
Measured on the 12-line reproduction from the issue:
before classes=0 properties=0
after classes=2 properties=0
On a real ontology the gap is larger: 25 triples / 1 subject against 719 / 88 for
the document that surfaced this.
Tests: `tests/ingest/test_ontology_named_graph.py` covers the named-graph document,
keeps a canary on the default-graph document so the fix cannot trade one blind spot
for another, and asserts that the reported result matches the terms returned.
Reverting `Dataset()` to `Graph()` turns all four red.
`tests/ontology`, `tests/export` and `tests/ingest` pass apart from six failures in
web/feed/database/API ingestion, unrelated to this change and failing the same way
on an unmodified checkout.
Not included, and happy to add here or as a follow-up: making a load that yields
zero classes stop returning `status: "success"`. That value is what made this take
an afternoon to find, but it is a behaviour change on a different layer and seemed
worth reviewing on its own.
* test(visualization): isolate optional dependency mocks
* test(visualization): stop requiring Plotly in unit tests
Removing the global sys.modules stubs left the tests that patch
`...go.Bar`, or call a visualizer, with nothing standing in for the
module level `px` and `go` aliases. Those are None when Plotly is
missing, so patch resolution and _check_dependencies() both failed.
Add a helper that substitutes a double only for the aliases that are
None, leaving the real module in place when Plotly is installed.
---------
* test(ingest): track relationship provenance via ProvenanceManager
kg.ProvenanceTracker has no track_relationship and never did, so
patch.object raised AttributeError before the test body ran.
Closes#1055
* test(ingest): disambiguate relationship keys and pin provenance storage
Addresses review feedback on #1071.
---------
The MCP server's export_graph tool was broken on all formats in 0.6.5/0.6.6:
- json: JSONExporter().export(graph) was called without the required
file_path argument -> TypeError surfaced as {"error": ...}.
- RDF branches: RDFExporter().export_to_rdf(graph, ...) received the
ContextGraph object instead of the canonical kg dict -> AttributeError
(ContextGraph has no 'get').
- All branches: the RDF path printed a rich progress bar to stdout,
corrupting the stdio JSON-RPC framing and hanging the client (observed:
300s timeout over MCP while the same call returns in <1s directly).
Fix: convert via ContextGraph.to_kg_dict() before exporting, serialize the
json branch to a string, and force SEMANTICA_DISABLE_PROGRESS=1 for the
server process — stdout is the protocol channel, not a console.
Tests: tests/test_mcp_server_export_graph.py covers every format, the json
payload shape (entities/relationships), and the progress-disable env var.
All four are in the branch that recognises an already-converted document,
which has to survive every shape JSON-LD allows rather than the one shape
Semantica happens to produce.
A knowledge graph carrying a context of its own took the already-JSON-LD
branch and skipped its own conversion, leaving entity ids, relationship
endpoints, types and confidences as raw keys. The entities/relationships
test now runs first, and a converted document never has those keys, so the
double-conversion guard is unaffected.
A context that is a URL or an array cannot be merged key by key, and was
being dropped in favour of Semantica's defaults, silently changing how every
term expands. Both are kept as an array now, the caller's winning, which is
the same precedence the dictionary branch already used. An explicit null is
left alone on purpose: in an array it resets the active context and would
take the semantica prefix with it.
@graph may be a single node object as well as an array. list() on a
dictionary yields its keys, so an object-valued graph was replaced by a list
of strings.
A caller may hand us a document that is deliberately a named graph. That name
is theirs to keep, so it is no longer flattened; it is nested one level and
the export's own provenance goes beside it, in the default graph, where a
plain reader can see it.
Four tests, one per case, all failing before this commit.
A JSON-LD document with a top-level @id and a top-level @graph is a named
graph. Its members become quads named by that @id, and the default graph is
left empty. rdflib.Graph.parse() keeps the default graph and discards the
rest without reporting anything, so every consumer that loads an export the
ordinary way saw the document header and none of the data.
_convert_to_jsonld wrote the payload into @graph and then stamped a document
@id beside it, which named every list export and every generic-dict export.
export_knowledge_graph made it worse: it converted the graph to JSON-LD and
handed the finished document back to export(), which converted it a second
time. The converted document no longer carries entities/relationships keys,
so the second pass treated it as opaque and buried the whole knowledge graph
inside @graph, under a name that is a wall-clock timestamp.
A two-entity, one-relationship graph exported to JSON-LD parsed as 2 triples
with Graph() and 21 quads with Dataset(). The 19 missing triples were the
entire knowledge graph.
The document node now goes inside @graph when the payload lives there, and is
the document itself otherwise, so no export names its own graph by accident.
An already-converted document is merged rather than nested, which also stops
the export carrying two document nodes and two @context blocks.
Semantica's reader has the mirror of this bug (#1129), so these exports could
not be read back by Semantica either.
Bump version, cut CHANGELOG's Unreleased section into 0.6.6, backfill
changelog entries for merged PRs missing from it, and refresh
version-dependent references in README/docs.
Consolidates the remaining #994 fixes into this PR so it can fully close
the issue, per maintainer request.
From #1005 (yzxcj797):
- EmbeddingGeneratorWithProvenance.__getattr__ self-recursion guard:
accessing self._generator via attribute syntax re-entered __getattr__
forever when _generator was absent (failed __init__, pickle/copy probes
like __deepcopy__). Private-name lookups now raise AttributeError.
- 4 regression tests in TestMethodDispatchRecursion: default dispatch no
longer self-recurses for generation/text, a user-registered custom
method still takes precedence, and a bare provenance wrapper raises
AttributeError instead of RecursionError.
(The methods.py identity guards from #1005 are already present here.)
From #1006 (yzxcj797):
- doctor gains two embedding backend checks, "Embeddings
(sentence-transformers)" and "Embeddings (fastembed)". Default is a
cheap import+version check (uninstalled backend now reports fail with a
pip hint instead of invisible). --deep-embeddings (or
SEMANTICA_DOCTOR_DEEP_EMBEDDINGS=1) instantiates via TextEmbedder and
embeds a probe, catching backends that import cleanly but cannot load
(the #994 failure mode) via the hash-fallback-active signal.
_DeepEmbeddingFailure marks post-import runtime/model-load failures so
they get a remediation hint instead of a misleading pip-install hint.
- 7 tests in TestDoctorEmbeddings and TestDoctorEmbeddingHintsAndEnv.
Validation:
- tests/test_cli_commands.py: 237 passed (7 new)
- tests/test_embedding_providers.py: 9 passed (4 new)
- AST parse + import of all four modules OK
Address Qodo review on #1113:
- Escape entity text for Turtle, RDF/XML and N-Triples so names containing
quotes, XML markup, backslashes or control chars cannot break out of the
literal or inject RDF/XML (High/Security).
- Replace colon-only id split with URI-aware local-name extraction so an id
like https://example.org/acme yields 'acme', not '//example.org/acme'
(Medium/Correctness).
- Add regression tests: escaping (quotes/XML/backslash/CR/LF), parseability
via rdflib, and exact id local-name assertions.
convert_kg_to_rdf() maps an entity's 'name' to 'label'/'text' but was
never invoked from export_to_rdf(), so graphs produced by GraphBuilder
(which emit 'name') exported with an empty semantica:text on every RDF
format (turtle, ntriples, rdfxml, jsonld). Call convert_kg_to_rdf() at
the export boundary before validation/serialization so all formats
benefit from a single normalization step.
Add regression tests asserting a name-only entity exports a non-empty
label across all four serializers and the file-writing entry point,
plus that an explicit 'text' is not clobbered and an id tail is used
as a fallback label.
Closes#1097
Review finding, reproduced. `call_custom_method(..., **kwargs)` builds a
fresh dict from the unpacking, so popping `fallback_on_custom_error`
inside the helper left the caller's own kwargs untouched. On the fallback
path the flag was then forwarded straight into the default
implementation, which is exactly the case the flag exists for.
Instrumenting the default exporter shows it arriving:
config handed to the default exporter: {'fallback_on_custom_error': True}
Most defaults take **kwargs and ignore it, which is why nothing failed
loudly, but any default with a fixed signature raises TypeError on it.
The helper's docstring promised the flag was never forwarded, so the
promise was false rather than merely untidy.
All 58 sites now pop the flag from their own bag and pass it explicitly.
One site in normalize/methods.py names its bag `**context` rather than
`**kwargs`, and is handled too.
3 further tests: the flag reaches neither the default implementation nor
a successful custom method, and a per-module guard that every call site
has a matching pop, since a site that forgets one reintroduces the leak
silently.
Failure set across the six affected modules is unchanged against
upstream/main: 37 pre-existing, none new.
Review finding, reproduced. The reified node reduced the relationship
type to its last fragment or path component, so
https://a.example/ns#employs and https://b.example/ns#employs both became
semantica:type "employs". The temporal node no longer said which
predicate it described, and it disagreed with the direct triple written
beside it, which carries the full IRI.
The full predicate is written instead. I had flagged the local-name form
as a deliberate simplification in the PR description; the collision case
shows it was the wrong call.
2 further tests.
1. An absurd magnitude expanded instead of being rejected. xsd:decimal
has no exponent notation, so the value has to be written out in full,
and "1e100000000" is eleven characters that expand to a hundred
million digits. "1e100000" already produced a 100,001 character string
here. The export path continues past validation errors, so one
malformed field could exhaust memory. Values beyond
MAX_CONFIDENCE_EXPONENT are now omitted like any other unusable value.
1e-9 still round-trips.
2. Decimal keeps the sign of zero, so 0.0 and -0.0 serialised as "0" and
"-0", which are two distinct RDF terms. That is exactly the duplicate
this PR exists to remove, so zero is normalised.
4 further tests.
Four findings from the automated review, all reproduced first.
1. The fix only reached Turtle. `_uri` was the single place I corrected,
and JSON-LD and N-Triples build sh:targetClass, sh:path and sh:class
straight from graph.base_uri, so two of the three formats went on
emitting shapes that match nothing. That is the defect this PR claims
to close, still live wherever the output is not Turtle. All three
serializers now resolve through one `_term_iri`, and the pySHACL
violation test runs against each of them.
2. Classes and properties shared one name-keyed index built with
setdefault, so a property named after a class was permanently mapped
to the class IRI and its sh:path validated the wrong predicate. The
index is now split into class_iris and property_iris, and each call
site says which it wants.
3. OntologyEngine.to_shacl forwarded target_namespace and
attach_domainless_properties through generate(**options), which never
reads them, so both were silently dropped on the public path. They are
now named parameters passed to the constructor, and documented.
4. The opt-in attachment logged at debug. It broadens constraint
generation, so it warns.
7 further tests, including the target-namespace and real-violation checks
parametrised across Turtle, N-Triples and JSON-LD.
Four findings from the automated review, all reproduced first.
1. The name fallback minted invalid IRIs. `_term_iri` pasted a raw name
onto the ontology base, so a class named "Customer Account" produced
<https://example.org/onto/Customer Account>. rdflib only warns about
the space, Oxigraph rejects it with "Invalid IRI code point". That is
the same class of defect this PR set out to fix, introduced by the fix
itself. Local names are now percent-encoded.
2. `improve_coherence` raised AttributeError. It lives on
OntologyOptimizer, which holds no namespace manager, so the URI
fallback I added there crashed on any ontology carrying a class
without a URI. It now mints from the ontology's own base through a
shared module-level helper.
3. `owl:Thing` was treated as an absolute IRI. It matches the generic
scheme grammar, so `_is_absolute_iri` accepted it and domains and
ranges came out as the term <owl:Thing> rather than
<http://www.w3.org/2002/07/owl#Thing>. This is the live path: stage 4
of the generator assigns ["owl:Thing"] to object properties with no
inferred endpoints. Absoluteness is now decided on a real scheme, and
the well-known prefixes expand.
4. Unusable property entries were dropped in silence. Non-dictionary
entries and definitions carrying no type are now named in a warning.
6 further tests, including a strict-parser check through Oxigraph, which
is what catches the space that rdflib waves through.
Every module supporting custom methods wrapped the registered callable in
a bare `except Exception`, logged a warning, and carried on into the
built-in implementation:
try:
return custom_method(data, file_path, format=format, **kwargs)
except Exception as e:
logger.warning(f"Custom method {method} failed: {e}, falling back to default")
That makes a registered method advisory. It can add behaviour, but it
cannot decline. For a gate, a validator or a policy check, declining is
the entire purpose: raising is how such a method says "do not produce
this output". Catching the exception and running the default produces
exactly the output the method was registered to prevent, and the only
trace is a warning.
Demonstrated with a verifier that rejects invalid RDF and deletes the
file. The fallback wrote it straight back.
`call_custom_method` in utils/custom_methods.py now holds the policy in
one place: an exception from a registered method propagates. Callers who
relied on the old behaviour can pass `fallback_on_custom_error=True`,
which restores warn-and-continue for that call and is consumed by the
policy rather than forwarded to the method.
The swallow was in six modules, not only the one the issue was filed
against, so all 58 sites are converted: export 13, ingest 13, normalize
13, parse 12, embeddings 4, kg 3. The rewrite is mechanical and uniform.
Sentinel comparison is by identity, so a custom method returning None, 0,
"" or an empty list is not mistaken for a failure.
13 tests in tests/utils/test_custom_method_can_refuse.py, including the
issue's own demonstration and a guard that no module still carries the
swallow. Across the six affected modules the failure set is identical to
upstream/main: 37 pre-existing failures before and after, none new, with
869 passing against 856 on the baseline.
include_temporal=True emitted a well formed OWL-Time interval hanging off
a relationship IRI that appears nowhere else in the graph. A relationship
is written as a single triple, <e1> <employs> <e2>, so there is no node
for the time to attach to:
<...#rel_0_0940a860> time:hasTime <...#rel_0_0940a860__valid_interval> .
Counting inbound arcs to that subject gives zero. The timestamps parse,
they validate, and no query can reach them from the relationship they
describe, which is the only thing they are for.
The JSON-LD path already reifies relationships as sem:Relationship with
sem:source, sem:target and sem:type, and the vocabulary declares all four
terms. Turtle now emits the same shape when it has temporal data to
attach, so the two serializations describe relationships the same way and
the interval has a reachable subject.
The direct triple is unchanged, and nothing is reified when a
relationship carries no temporal data, so default output is untouched.
7 tests in tests/export/test_owl_time_reachability.py, including a SPARQL
walk from the edge to its validity interval, which is what the dangling
node made impossible, and a check that every emitted term is declared in
the shipped vocabulary. Export and ontology suites pass at 228 tests.
#1100 — the four serializers rendered the same confidence four different
ways. Turtle wrote it bare, which the Turtle grammar reads as
xsd:decimal. N-Triples typed it xsd:float. RDF/XML wrote a plain literal
with no datatype. JSON-LD wrote a native JSON number, which expands to
xsd:double. For confidence 0.9 that is four distinct RDF terms, so a
FILTER matches at most one of them, and merging two exports of one graph
gives an entity two different confidence values.
N-Triples also omitted the triple entirely when confidence was absent,
while the other three wrote the 1.0 default, so the two serializations
differed in the number of triples as well as in their datatype.
`normalize_confidence` now produces one canonical lexical form and every
path writes it with CONFIDENCE_DATATYPE. xsd:decimal is the choice
because it is what the Turtle path already produced, so the most used
output is unchanged, and because it is exact: xsd:float is 32 bit binary
and cannot represent 0.9 at all. Values that arrive in exponent notation
are reformatted, since 1e-05 is not a valid xsd:decimal.
#1102 — the Turtle path interpolated the value with no type check, so a
confidence of "high" produced `semantica:confidence high .` and made the
entire document unparseable. One bad field cost the whole export. A value
that cannot be a decimal is now omitted with a warning naming the entity,
rather than written as something the vocabulary contradicts. Numeric
strings are still accepted. Booleans are not, since bool subclasses int
and True would otherwise become a confidence of 1.
sem:confidence in the shipped vocabulary declared no rdfs:range,
deliberately, because declaring one would have contradicted three of the
four exporters. It now declares xsd:decimal, and a drift guard asserts
the vocabulary and the serializers agree.
20 tests in tests/export/test_confidence_literal_typing.py, comparing the
parsed graphs of all four formats rather than their text. Export and
ontology suites pass at 240 tests.
#1104 — SHACLGenerator used one namespace for two jobs. `base_uri` says
where the shape resources live, and it was also used to expand every
sh:targetClass and sh:path. With the default "https://semantica.dev/shapes/"
that made shapes target <https://semantica.dev/shapes/Person>, while data
carries the ontology's own class IRI or the semantica:ns# vocabulary. The
shapes matched nothing.
That failure is silent. A shape with no focus nodes is vacuously
satisfied, so pySHACL reports conforms=True on data that plainly breaks
the stated constraints. The shipped validator agrees the file is fine.
The two namespaces are now separate. `target_namespace` resolves in this
order: an explicit argument, the ontology's declared namespace, the
namespace of any absolute IRI a term already carries, the ontology URI,
and finally the vocabulary namespace the package ships rather than the
shapes namespace. Every class and property name is indexed to the IRI it
expands to, and `_uri` resolves through that index, so shapes always name
the terms the data uses.
#1105 — a property with no declared domain was attached to every node
shape. That states a constraint the ontology does not, and with minCount 1
it makes every instance of every class invalid. Such a property is now
left unattached, with a warning naming it. Passing
attach_domainless_properties=True restores the old behaviour.
tests/ontology/test_shacl_target_namespace.py adds 17 tests that validate
real data through pySHACL rather than reading the shapes text, so a shape
that targets nothing cannot pass by being ignored. They cover a generated
ontology, one that declares only a namespace, and one that carries only
class URIs.
tests/ontology/test_ontology_advanced.py::test_no_domain_property_attaches_to_all_shapes
asserted the #1105 behaviour, so it pinned the defect in place. It is now
two tests: the old expectation against the explicit opt-in, and the new
default.
Export and ontology suites pass at 239 tests.
OWLExporter read `object_properties` and `data_properties`, while
OntologyGenerator emits one combined `properties` list tagged with
type/@type. Every generated property was therefore dropped, and a
generated ontology exported as classes alone.
Class IRIs were worse. ClassInferrer writes `"uri": None` when it is
given no namespace manager, so the stage 3 guard `if "uri" not in cls`
never fired: the key is present, only its value is missing. The exporter
then interpolated the empty string into `<>`, which is a relative IRI
that resolves against the parser's base. Under rdflib that base is the
current working directory, so a two-class ontology parsed as one subject
carrying two rdfs:label values, and the identity of that subject changed
with the directory the export ran from. Oxigraph rejects the same file
outright with "No scheme found in an absolute IRI".
Changes:
- Accept both dict shapes. `_split_properties` classifies the combined
`properties` list by type/@type and merges it with any explicit
`object_properties` and `data_properties`.
- Resolve class and property IRIs through `_term_iri`, falling back from
uri to iri to id to a name joined onto the ontology base. A term with
none of those is skipped with a warning rather than emitted as `<>`.
- Resolve domain and range references through the class index, so a bare
name such as "Person" lands on the IRI that class was exported under
instead of staying relative.
- Resolve data property ranges properly. "string", "xsd:string" and a
full IRI now all give one well formed datatype. The previous
`rdfs:range xsd:{range}` produced `xsd:xsd:string` for generator output,
which no parser accepts. Turtle keeps the compact xsd: form the module
already used.
- Fix the two `not in` guards in the generator so a present-but-None uri
is minted, and mint an absolute IRI rather than assigning a bare name.
- Escape XML text and attribute values, which were interpolated raw, so a
label containing & or < no longer breaks the document.
Turtle and RDF/XML now serialise the same 25 triples for the same
ontology, and both are accepted by rdflib and by Oxigraph.
10 regression tests in tests/export/test_owl_exporter_generator_schema.py,
driven by a real OntologyGenerator run and asserting on the parsed graph
rather than on serialised text. All 10 fail on the parent commit. The
export and ontology suites pass at 231 tests.
Saurabh Meena