_turtle_object() wrote an IRI-valued metadata value (currently only
sem:sourceUri, from the "uri" metadata key) straight into `<{value}>`
with no escaping. Turtle/N-Triples IRIREFs exclude control
characters, space, and <>"{}|^`\ unescaped, so a value shaped like
`<goodIRI> . <injected> <p> <o>` closed the reference early and let
the rest of the string be parsed as an attacker-chosen extra triple:
metadata={"uri": "https://x> . <https://injected> <https://p> <https://o"}
produced a well-formed Turtle/N-Triples document containing a triple
the caller never asked for.
RDF/XML was already safe (_rdfxml_metadata_lines runs the value
through _escape_xml before putting it in an rdf:resource attribute),
and JSON-LD is safe by construction (json.dumps makes structural
injection impossible) — only the Turtle/N-Triples "iri" literal path
in _turtle_object was unguarded.
Adds _safe_iri_ref(), a narrow percent-encoder for exactly the
characters an IRIREF may not contain unescaped. It's deliberately not
_as_turtle_iri: that also resolves registered prefixes, which a
metadata value never needs, so a dedicated guard stays simpler than
threading namespaces into a module-level helper that has no `self`.
Two regression tests, parametrised over turtle/ntriples: the `>`
delimiter-breaking payload from the report, and a control-character
(newline/tab) variant covering the other half of the excluded set.
Resolves the conflict in semantica/export/rdf_exporter.py between this
branch's metadata clauses (entity/graph metadata statements) and
main's IRI-normalization and XML-escaping hardening
(_as_turtle_iri / xml_escape, landed after this branch's last sync).
Kept both: entity/relationship/graph subjects and objects now go
through _as_turtle_iri (Turtle) or _as_turtle_iri + xml_escape
(RDF/XML), same as every other identifier in these serializers,
while the metadata-clause list building and graph_uri handling from
this branch are preserved unchanged. graph_uri is now normalized the
same way for consistency with the rest of the file.
Verified: tests/export + tests/ontology (411 tests) and the existing
Turtle-IRI regression suite (test_rdf_exporter_turtle_iris.py, 9
tests) all pass against the merged code.
ProgressTracker attached ConsoleProgressDisplay unconditionally, so any
script or CI job that piped or redirected stdout had one progress bar per
stage written into its output, escape sequences included. A plain
`python demo.py > out.txt` captured 173 bytes of progress-bar noise around
10 bytes of the program's own output.
Console progress is now attached only when stdout is an interactive
terminal, when running under Jupyter, or when SEMANTICA_FORCE_PROGRESS is
set. FileProgressDisplay is untouched, so progress logging still works in
pipelines, and SEMANTICA_DISABLE_PROGRESS keeps its existing meaning and
still takes precedence.
Both progress environment variables are now documented in the README and
the utils reference; SEMANTICA_DISABLE_PROGRESS previously existed only in
the reference page.
Deviations from the issue: the issue suggested disabling the tracker on
non-TTY stdout. This gates the display instead, because disabling the
tracker would short-circuit before FileProgressDisplay and take file
progress logging down with it, and the ~20 modules that set
`progress_tracker.enabled = True` in __init__ would need the property
setter taught about TTY state to avoid undoing it. Gating the display
leaves both alone.
Design note: the claim comment on the issue proposed an
`enabled: Optional[bool] = None` constructor opt-in; during implementation
the opt-in became SEMANTICA_FORCE_PROGRESS, which needs no signature change
and follows the NO_COLOR/FORCE_COLOR convention. Known limitation: TTY
detection runs once at tracker construction (the tracker is a process-wide
singleton), so a process that redirects stdout after first use needs the
env vars to change behaviour.
Fixes#1185
_as_turtle_iri() re-encoded absolute IRIs wholesale, turning already-valid
percent-escapes like %20 into %2520. Only spans outside existing valid
%XX escapes are quoted now, so malformed escapes (%zz) still get repaired
while valid ones pass through unchanged.
serialize_to_ntriples()/serialize_to_rdfxml() also passed only the
@context-derived namespaces into _as_turtle_iri(), which shadowed the
built-in semantica:/rdf:/rdfs:/owl: prefixes entirely whenever any
@context was present. _as_turtle_iri() now always merges the built-ins
with whatever namespaces the caller passes.
vector_store_config.get_all() always includes a "dimension" key, so
forwarding it via **config into VectorIndexer(dimension=dimension, **config)
raised "got multiple values for keyword argument 'dimension'" any time the
default index-creation path ran with the default config — including
`semantica embed index`, which is exactly the second half of the #994
quick-start pipeline this PR fixes.
Review feedback: analyze_decision_influence(), trace_decision_causality(),
and find_precedents() had the same vocabulary split as get_causal_chain().
The first two read edge_type_index, which is keyed by the RAW edge_type
string, so they now filter index keys by normalized type; find_precedents()
accepts the analyzer's 'precedes' spelling alongside PRECEDENT_FOR.
Adds regression tests for all three call sites.
Review feedback: normalization must not turn invalid inputs into
AttributeError. Non-string relationship types now raise ValueError before
normalization, matching the pre-change behavior; strings are stripped
before alias lookup.
get_causal_chain() matched only the canonical uppercase spellings
(CAUSED, INFLUENCED, PRECEDENT_FOR), while CausalChainAnalyzer's
vocabulary includes the present-tense forms (causes, influences,
leads_to, supports) — and the two differ in word form, not just case,
so case-insensitive matching alone would still miss them. An edge
recorded as "causes" produced an empty audit chain.
Storage normalizes both vocabularies onto the canonical types via
_CAUSAL_EDGE_ALIASES; traversal accepts the union (_CAUSAL_TRAVERSAL_TYPES).
add_causal_relationship() now accepts either spelling and stores the
canonical form.
CodeQL (py/incomplete-url-substring-sanitization) flagged the "https://schema.org/"
in flattened check because it pattern-matches on URL-ish strings tested with `in`.
flattened is always a list here, so the check was already exact membership, not a
substring test on untrusted input, but the ambiguous idiom tripped the scanner.
Rewrite as an explicit equality comparison so the intent is unambiguous.
analyze_evolution() previously appended a constant placeholder (durations.append(1)) for every bounded relationship, so the stability metric was always 1.0 when any bounded relationship existed and 0 otherwise, never reflecting actual valid-time durations.
Stability now computes the mean valid-time duration in seconds ((valid_until - valid_from).total_seconds()) across relationships with both bounds set; unbounded/half-open intervals are skipped and non-positive intervals clamped to 0. Adds unit tests and a CHANGELOG entry.
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
* fix(dedup): never merge entities with different explicit types (fixes#1137)
The duplicate candidate confidence scoring only rewarded same-type pairs
but never penalized different-type pairs, so a Person 'Alice' and an
Organization 'Acme' (different id, type, and name) passed the confidence
threshold and were merged, silently dropping one entity. Add a type guard:
when both entities carry a non-empty type and they differ, the pair is
never a duplicate candidate (confidence 0, reason 'type_mismatch').
Untyped entities and genuinely duplicate same-type pairs keep their
previous behavior. Regression tests cover all three cases.
* fix(dedup): honor Entity.type and exclude mismatch structurally (review fixes)
Two gaps from code review (#1149):
1. _get_entity_value mapped object 'type' exclusively to .label, which
Entity objects never have — their type lives on .type. The mismatch
guard therefore never saw the type of Entity objects, and differently
typed objects could still merge. Read .type first, fall back to .label.
2. The mismatch branch returned a normal candidate with confidence 0.0,
but detection filters with >= confidence_threshold, and 0.0 is a
documented valid threshold, so mismatches slipped through. Exclude
type_mismatch candidates structurally at both filter sites regardless
of threshold.
Adds tests for Entity objects with different types and for
confidence_threshold=0.0. 94 dedup tests pass.
---------
* fix(reasoning): refuse SPARQL query execution instead of returning empty results (#1083)
SPARQLReasoner.execute_query() never executed the query: both branches
returned an empty SPARQLQueryResult, with or without a triplet store, so
callers that trust an empty result as "no matches" silently drew wrong
conclusions. Until a real triplet-store execution path lands, the method
raises NotImplementedError with an explanation, per the issue's
suggestion. The dead cache/inference scaffolding after the execution
point is removed along with it.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(reasoning): align execute_query() docs with the NotImplementedError contract (#1087)
Review feedback: the docstring still carried a "Returns" section and the
reasoning guide showed execute_query() returning bindings, both of which
now mislead. The docstring documents Raises only, the guide demonstrates
expand_query() and points to rdflib for execution until the triplet-store
path lands, and query_cache/clear_cache() are marked as reserved for that
future execution path.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(utils): bound caller-controlled keys in validation error messages (#1001)
_require_recognized_keys() and _require_nothing_dropped() interpolated
supplied keys directly into ValidationError messages, so a megabyte-long
key produced a megabyte-long exception and, through the export wrappers
that log the full exception, an equally large log entry. Keys are now
rendered through _truncate_key(), which bounds the display at 64
characters with an ellipsis; the supplied payload is never modified.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(utils): bound the count of keys shown in validation error messages (#1001)
Review feedback: per-key truncation did not bound the number of keys
shown, so a payload carrying many short unknown keys could still size the
message (and the log entry that records it). _truncate_key_list() caps
the display at 8 keys and appends "and N more", keeping the message
actionable without letting the payload size it.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Convert_units() was validating categories on raw input like "kg" or "ft"
instead of the normalized unit name, so aliases got checked against a
category list that only has canonical names in it. Any alias-based
conversion that should've worked just raised ValidationError instead.
Fixed by normalizing both units before the category check runs.
Also added foot/yard/mile/gallon to the alias map - they already had
conversion factors but weren't mapped to their canonical names, so they'd
still have failed even after the above fix.
Turned out there was a second bug hiding behind the first one: the category
check defaults both sides to None, and None == None is True, so two aliases
from different categories that neither resolved to a real category would
silently pass instead of raising. kg -> ft would just return a number
instead of erroring. Normalizing first fixes this too, since aliases now
resolve to their actual categories and the mismatch gets caught.
Added a regression test locking that second one down - kg->ft and gal->lb
now raise ValidationError instead of silently converting.
Fixes#931.
#1123 through #1127 landed while this was open, and #1125 rewrote the same
four entity loops this branch extends. Confidence is now normalised through
normalize_confidence, which returns None for a value that has no xsd:decimal
form, so the clause can be absent.
Resolved by folding that into the clause list this branch already builds:
the Turtle path assembles its predicate-object clauses and then terminates
the last one, which is what makes a variable-length list work at all, and
an omitted confidence is simply one clause fewer. RDF/XML and JSON-LD take
the upstream conditional as written, with the metadata call after it.
Sameer Kadam