fix(ontology): coalesce normalized property collisions
Different raw property spellings can normalize to the same ontology
name and IRI. works_for and worksFor, for example, both normalize to
worksFor, but property inference emitted a separate definition for
each spelling, so the generated ontology declared two distinct
properties under what would become the same IRI once minted. The same
collapse could also happen across kinds: a relationship type and an
entity attribute that normalize to the same name would previously
produce a data property and an object property sharing one name, with
no signal that anything was wrong.
infer_properties() now runs a coalescing pass after object and data
properties are both inferred. Properties are grouped by (kind, name).
Object properties that collide are merged in occurrence order:
domains and ranges are unioned rather than overwritten, so a property
seen across several source classes keeps every domain instead of
losing all but the first, and occurrence_count is summed across the
merged spellings so downstream confidence/frequency signals stay
correct. Data properties merge domains the same way and reconcile
differing ranges through the existing _get_more_general_type()
widening logic already used elsewhere in this file, rather than a new
implementation.
A name that resolves to both an object property and a data property
is not silently coalesced into either one, since the two kinds mean
different things in the emitted ontology. That case raises a
ValidationError up front, naming every colliding name and which kinds
collided, so the conflict surfaces before an ambiguous ontology is
written rather than after.
Verified beyond the two cases in the new test file: a data property
colliding across two different domain classes correctly unions the
domain instead of keeping only the first class, and three distinct
spellings of the same relationship type collapse into one property
with the occurrence count correctly summed across all three.
Follow-up to #1170 (relationship endpoint types) and #1171 (retained
data properties for normalized class names).
Bump version, cut CHANGELOG's Unreleased section into 0.6.7, backfill
changelog entries for merged PRs missing from it, and refresh
version-dependent references in README/docs.
Adds `SAPODataConnector`, `SAPODataEntity`, and `SAPIngestor` for ingesting master and transactional data from SAP OData services, mainly things like Business Partners and Sales Orders.
Tested around S/4HANA Cloud, SuccessFactors, and on-prem NetWeaver Gateway style OData endpoints.
Main pieces included:
* OAuth2 client credentials and Basic auth support. Both go through the existing `ssrf.py` checks, including the OAuth token request.
* Small EDMX parser used by `discover_service()` so we don't need to pull in `pyodata`.
* Server-side pagination support for both OData versions:
* v2: `__next`, including plain string and `__deferred` formats
* v4: `@odata.nextLink`
* Keeps the service path in the base URL correctly whether the URL has a trailing slash or not. This is normalized in `SAPIngestor.__init__`.
* Adds an `ingest-sap` extra with just `requests`, so there is no SAP/proprietary SDK dependency.
This is meant to be a fairly small first version of the connector without adding a lot of SAP-specific dependencies.
Closes#1228
PipelineBuilder.set_parallelism() validated and stored a level in
pipeline config, but ExecutionEngine._execute_steps() had no parallel
code path and no code ever read it back, so steps always ran strictly
sequentially regardless of the configured value. parallelism was also
lost across a serialize/deserialize round trip, since the nested
config key was never promoted to the top-level dict build_pipeline()
reads.
Steps are now grouped into dependency layers (declaration order
preserved within each layer). A layer runs concurrently, bounded by
ThreadPoolExecutor(max_workers=min(configured parallelism, engine
max_workers)), only when every one of the following holds: more than
one step in the layer, the shared input is a dict, every step is
opted in via the new PipelineStep.parallel_safe flag, and no step is
in delta_mode. Any layer that doesn't meet all four falls back to the
existing sequential path unchanged.
Each step's input is deep-copied before any handler in the layer
starts, so concurrent steps never share mutable state. Layer results
are merged back in declaration order, not completion order; keys
whose value is unchanged from the shared input are treated as an
echo rather than a write, so two handlers both returning {**data, ...}
don't spuriously conflict on keys neither of them actually touched.
Genuinely conflicting values for the same key raise ProcessingError
naming both the key and the two steps involved. Retry policy, step
status, result/error tracking, and progress reporting are shared
between the sequential and parallel paths so behavior stays identical
either way. On step failure, not yet started futures in the same
layer are cancelled and the error propagates, so no downstream layer
ever runs.
parallel_safe is opt-in per step because handlers that share mutable
state or depend on strict ordering are not safe to run concurrently.
ParallelismManager.execute_pipeline_steps_parallel() is deliberately
not reused here; the engine implements its own bounded layer
scheduler so retry/status semantics stay identical between the
sequential and parallel code paths instead of diverging.
fix(pipeline): address qodo review findings on PR #1226
- detect circular/unknown dependencies in parallel grouping
(ValidationError instead of RecursionError/KeyError)
- skip unchanged echoed keys in parallel result merging to
avoid false conflicts
- fail before COMPLETED status when a parallel step returns a
non-dict; never retry such contract violations
- require strict bool parallel_safe in builder and engine gate
- make per-step progress tracking IDs unique across same-type
parallel steps
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
* test(ner): fix NER configuration tests for the typed LLM extraction API
Two of the three failing tests tracked in #1059 were still red after
#1070 was closed because the mocks targeted the pre-typed provider API:
- test_ner_llm_config mocked generate_structured, but the LLM path now
goes through generate_typed with a Pydantic schema. Mock the typed
response (namespace items with .text/.label/.start/.end/.confidence)
and expect extraction_method 'llm_typed'.
- test_ner_pattern_config asserted 'Apple Inc' without the trailing
dot, but the ORG pattern captures it via (?:\.|\b). Assert 'Apple
Inc.' to match current production behavior.
Verified locally: 8/8 pass in test_ner_configurations.py; the
performance-test failures in tests/semantic_extract/ reproduce on a
clean main checkout and are unrelated.
Fixes#1059
Signed-off-by: Yunare Maia <yunare@gmail.com>
* refactor(ner): remove dead _extract_with_spacy method and unused self.nlp
_extract_with_spacy() had no callers: the ML dispatch path goes through
get_entity_method('ml') -> extract_entities_ml(), which loads the spaCy
model lazily via the process-level cache in methods.py. The instance
attribute self.nlp was only read by that dead method, so __init__ now
just validates the runtime (keeping the _ml_runtime_usable gate) instead
of eagerly loading a model that was never used.
Fixes#1058
Signed-off-by: Yunare Maia <yunare@gmail.com>
* test(split): rewrite NERExtractor cache tests to not rely on removed .nlp attribute
NERExtractor.nlp was removed in this PR as part of dead-code cleanup
(the attribute was only used by the equally-dead _extract_with_spacy()).
The three affected tests in TestNERExtractorSpacyModelCache previously
verified cache behavior through .nlp identity comparisons; rewrite them
to use load-call counts and direct se_methods.load_spacy_model() cache
queries instead:
- test_ner_extractor_reuses_cached_model_across_instances: drop the
e1.nlp is e2.nlp is e3.nlp assertion; len(calls)==1 already proves
reuse; add a cache query to confirm the cached object is non-None.
- test_ner_extractor_distinct_model_names_load_separately: store each
mock nlp in a dict keyed by name, then query the cache to assert
sm_cached is loaded['en_core_web_sm'] and sm_cached is not lg_cached.
- test_ner_extractor_failed_load_not_cached_and_retried: replace
extractor.nlp is None/not None with is-not-None construction checks
and a final cache query that verifies the recovered model is the
exact object returned by working_load.
All three tests still exercise the original behavioral contract (no
crash on missing model, failures not cached / retried, successful load
shared across instances); they just no longer rely on a private
instance attribute that no longer exists.
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
fix(config): honor boolean env overrides in Config.get() (#1038)
Config.get() checked int before bool. Since bool subclasses int, boolean
environment values could be ignored or returned as integers.
Check bool first and strip whitespace before parsing boolean environment
values. This applies to the config modules for conflicts, deduplication,
split, embeddings, export, ingest, kg, normalize, ontology, and parse.
Also make _load_env_vars() use the same whitespace handling for mapped and
generic environment variables.
Fixes#1035
* fix(parse): import email.message and repair pdfplumber test mock
- email_parser.py uses email.message.Message at class-definition time but
only did 'import email', so 'import semantica.parse' fails in a fresh
Python process unless something else imported email.message first
- test_pdf_parser patched semantica.parse.pdf_parser.pdfplumber, which
never exists as a module attribute (pdfplumber is imported inside
PDFParser.parse); inject a fake module via sys.modules instead
* fix(parse): warn when PDF parse yields no text layer (scanned PDFs)
Scanned (image-only) PDFs parsed via the default pdfplumber route
returned an empty full_text with progress status 'completed' - no error,
no warning - so the failure only surfaced far downstream. Warn in
PDFParser.parse() when every parsed page yields no text (and extract_text
is enabled), pointing users to method='docling' with enable_ocr=True.
* fix(parse): improve scanned PDF detection
---------
Co-authored-by: shanyu910 <208111055+shanyu910@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(provenance): use timezone-aware UTC and assert stored records (#946)
Replace datetime.utcnow() in ProvenanceManager, ProvenanceEntry,
BridgeAxiom, and GraphBuilderWithProvenance with
datetime.now(timezone.utc), matching PipelineWithProvenance.
KG workflow and integration tests now read provenance back through
get_provenance() and assert algorithm metadata instead of generated
IDs, and call tracker methods that actually persist records.
* fix(provenance): compare provenance timestamps as instants, not strings
query_recorded_between() and audit_log() filtered and sorted on raw ISO
strings. With the timezone-aware change, a store can hold both pre-existing
naive stamps and offset-bearing ones, and the two are not string-comparable:
"...500000+00:00" sorts above "...500000", so a record at the identical
instant as a naive bound falls outside the range that should contain it.
Both now parse through _parse_timestamp() before comparing, reading naive
values as UTC. This mirrors ProvenanceTracker._parse_dt() in kg/, the class
ProvenanceManager replaces, so both sides of the migration answer a range
query the same way. Unparseable stored timestamps are skipped and logged
rather than silently dropped; unparseable bounds raise ValueError.
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Index the four module notebooks merged via #989-#992 (Provenance
Tracking, Reasoning, Change Management, Seed Data) in the cookbook
landing page, as committed in tracking issue #1032.
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
* feat(reasoning): rule-driven actions with provenance
Add a structured Action layer so matched rules can trigger side effects
instead of only deriving new facts, turning the reasoner into a
production-rule system.
L1 - Action type system:
- Action base class with execute(bindings, reasoner) + ?var substitution
- AssertAction (optional write-back to KnowledgeGraph), RetractAction,
CallAction (structured replacement for the unused Rule.handler),
EmitEventAction (delivers to a registered event sink)
- Rule.actions field; wired into Reasoner.forward_chain() and
ReteEngine.execute_matches() (via optional bind_reasoner)
L2 - Provenance-aware actions:
- Reasoner records fired actions (rule, bindings, confidence) to
action_log when provenance is enabled
- Fix dangling import in reasoning_provenance.py (ReasoningEngine ->
Reasoner, infer -> infer_facts)
Backward compatible: rules using the legacy handler still fire (wrapped
as a CallAction); rules without actions behave exactly as before.
Adds tests/reasoning/test_rule_actions.py (9 tests).
Closes#1095
* fix(reasoning): address qodo review findings on rule actions
- Token-aware variable substitution to avoid ?x/?xy prefix collision
- KnowledgeGraph write-back protocol (explicit API -> canonical translation -> ValueError)
- Structured action_log entries with timestamp
- Decouple action firing from conclusion dedup via per-activation tracking
(fires known conclusions once; retract-self no longer loops to max_iterations)
- Add Reasoner.infer_with_results preserving confidence; infer_facts delegates
- Forward provenance flag in ReasoningProvenance; drop **kwargs; propagate confidence
- Populate Rete Match.bindings from rule conditions
- Add regression tests for each fix
* fix(reasoning): persist fired action activations
* fix(reasoning): deduplicate Rete action execution
* fix(reasoning): canonicalize action activation identity
* docs(reasoning): explain action replay controls
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
* docs(cookbook): add Seed Data module notebook
Add cookbook/introduction/25_Seed_Data.ipynb covering the seed module
with verified, executable examples:
- SeedDataManager.register_source with a CSV source
- load_source record enrichment (entity_type/source provenance)
- create_foundation_graph entity/relationship/metadata structure
- validate_quality gating
The seed module ships seed_usage.md but has no cookbook coverage. All
API calls and outputs were executed against
semantica/seed/seed_manager.py.
Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
* docs(cookbook): isolate seed CSV in a temp dir and execute notebook in Jupyter
- Write companies.csv into a session-scoped tempfile.mkdtemp() directory
instead of the working directory, so a user's existing companies.csv
can never be silently clobbered (review finding)
- Run the notebook through a fresh Jupyter kernel (restart + run all +
save): real execution counts, print() cells saved as stream outputs
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
---------
Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
* fix(ingest): import sqlalchemy text where DBIngestor and DataExporter use it
sqlalchemy.text was imported function-locally in DatabaseConnector.connect
and test_connection, but called in DataExporter.export_table_data and
DBIngestor.execute_query, which never imported it. Both raised NameError,
re-wrapped by their except handlers into a ProcessingError reading
'Failed to execute query: name text is not defined' -- a message that
looks like a database fault rather than a missing import.
No test exercised either method, so this also repairs a pre-existing
failure in tests/ingest/test_notebook_02.py::test_08_database_ingestion.
Add SQLite-backed coverage for all three call sites, including the
SELECT COUNT(*) branch that only runs when no limit is passed and would
otherwise stay untested.
Closes#1015
* test(ingest): register setUp cleanups with addCleanup
TemporaryDirectory and the SQLAlchemy engine were released only in tearDown, which unittest skips when setUp raises partway through. Register each cleanup as soon as its resource exists so a failed setUp still disposes the engine and removes the temp directory. LIFO ordering keeps dispose before cleanup, as tearDown had it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
* fix(ontology): preserve #-terminated namespaces in SHACLGenerator base_uri (#1082)
SHACLGenerator.__init__ normalized base_uri with rstrip('/') + '/', turning a #-terminated RDF namespace (e.g. http://example.org/manufacturing#) into ...#/. Every generated URI then landed in a different namespace than the instance data, so SHACL validation silently passed because the shapes targeted nothing.
__init__ now preserves a base_uri already ending in '/' or '#', matching the #-aware normalization generate() already applies. shapes_uri inherits the fix.
Adds test_hash_namespace_base_uri_is_not_mangled (fails on the old normalization), plus a CHANGELOG entry. Full ontology suite green.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ontology): collapse slash runs, only preserve #-terminated base_uri
Qodo review caught that preserving any endswith('/') base left redundant
trailing slashes (e.g. .../ns////) intact, leaking a different namespace
into emitted IRIs. Now only '#'-terminated bases are kept verbatim; slash
runs are collapsed to a single '/', matching generate() normalization.
Adds test_slash_run_normalization_regression.
---------
Co-authored-by: changshenhan <217217832+changshenhan@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
track_relationship() has no dedicated subject/object fields, so the
Step 2 example only stored relationship_id + type, leaving readers
unable to reconstruct which two entities the relationship connects.
Encode subject_entity_id/object_entity_id in metadata by convention,
and note the lack of dedicated fields in the prose.
* fix(cli): write embed generate output in the format embed index reads
* Address review: structured results get their own --output writer
deduplicate --output and ontology align --output were routed through
_write_embeddings_output, a helper for numeric matrices: it rejects the
dict/list shapes these commands produce and the .csv extension deduplicate
documents. New _write_result_output serializes structured results — JSON,
JSON-lines for lists, CSV for rows — and both commands use it. embed
generate keeps the embeddings writer, whose strictness is what #994 fixed.
On the pyarrow gap: the parquet writer already fails with an actionable
message (install pyarrow or use .json). Silently writing JSON bytes to a
.parquet path would recreate #994's magic-bytes failure, so the error stays
an error and the default suggestion stays .json.
* fix(cli): improve structured output serialization
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(export): consolidate duplicate Turtle/N-Triples literal escapers
_escape_literal (module-level) and RDFSerializer._escape_turtle_literal did
identical work in the same order (backslash, double-quote, newline, CR, tab).
Drop the newer static helper added in #1148 and route all call sites through
_escape_literal instead. Behaviour no-op.
Closes#1218.
* fix(export): handle datetime/None temporal bounds safely in OWL-Time
_escape_literal is str-only, so routing datetime or None temporal bounds
through it raised AttributeError during Turtle export. Stringify non-str
bounds (plain f-string semantics) before escaping, and render None as an
empty bound. Add regression tests for datetime bounds and end-only
intervals. Addresses Qodo high-priority finding #2 on #1221.
* fix(export): use isoformat for datetime temporal bounds
str() on a datetime drops the ISO-8601 T separator, producing a lexically invalid xsd:dateTimeStamp. Use isoformat() when available; strengthen the test to assert the exact T-separated form.
---------
_calculate_decision_content_similarity's character-bigram fallback was
unconditional, so ordinary multi-word English queries could pick up
incidental bigram overlap with unrelated decisions via max(word_sim,
bigram_sim). Gate it to only activate for CJK-like scripts or queries
with at most one whitespace token, matching its documented purpose.
Separately, _add_decision_to_graph never persisted recorded_at as a
node property, so _rebuild_decision_indexes/_sync_decision_from_node
(which already read it back) always recovered "" after any reload.
* fix(explorer): coerce decision timestamp to str to prevent 422 on /api/decisions
ContextGraph stores decision timestamps as POSIX floats (e.g. 1786513069.69),
but DecisionResponse.timestamp is typed Optional[str]. Pydantic strict
validation rejects the float and the whole /api/decisions endpoint returns
HTTP 422 "Invalid input", which breaks the Decisions workspace in the
Knowledge Explorer entirely (no decision can be listed).
Coerce the value to str (preserving None) in _node_to_decision so the
response validates. Verified: /api/decisions now returns 200 and the 3
sample decisions render in the Decisions workspace.
* test(explorer): cover decision timestamp coercion in _node_to_decision
Regression tests for the 422 fix in _node_to_decision. Covers the cases
that produced HTTP 422 (float / int timestamps from ContextGraph) and
the ones that must keep working (None, already-string, missing key).
Verified the suite catches the regression: with the fix reverted, the
float / int / nan / inf cases fail with the same ValidationError that
caused the 422; with the fix applied all 6 pass.
* fix(explorer): preserve decision timestamp normalization
The route-level str() cast introduced in the initial fix bypasses
DecisionResponse._normalize_timestamp, the field validator on main that
converts POSIX float epochs to ISO-8601 strings via
datetime.fromtimestamp(value, tz=UTC).isoformat().
With the cast in place the API emits raw numeric strings such as
'1786513069.69' instead of '2026-08-12T05:37:49+00:00', breaking
datetime.fromisoformat() for every caller and failing
TestRecordedDecisions::test_list_decisions_serializes_float_timestamp.
It also silently accepts nan/inf/out-of-range epochs that the validator
is designed to reject.
Restore _node_to_decision() to pass the raw stored value through
unchanged so DecisionResponse._normalize_timestamp remains the single
normalization boundary for all three affected endpoints:
GET /api/decisions
GET /api/decisions/{id}
GET /api/decisions/{id}/precedents
Rewrite test_decision_route_timestamp.py so every assertion uses
datetime.fromisoformat() to verify ISO-8601 output and explicitly
asserts ValidationError for nan, inf, -inf and out-of-range epochs.
Add three TestClient integration tests covering the full production
path: record_decision() -> float stored in graph -> HTTP GET -> JSON.
---------
Co-authored-by: administrator <administrator@administratordeMac-mini.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
RelationExtractor.extract_relations(text, entities, ...) requires
entities, but the tool called it with only text, raising TypeError
on every invocation. Run NER first and pass the resulting entities
through, matching how the rest of the pipeline extracts relations.
* Add tests for max_tokens propagation in LLM methods
This test verifies that the max_tokens parameter is correctly propagated to the generate_typed method for different extraction functions.
* fix(tests): make issue-176 regression tests discoverable by pytest
The contributor's PR added tests/optimize reproduce_issue_176.py — a file
with a space in its name that never matched pytest's test_*.py discovery
pattern, so the regression would have been silently skipped in CI/local runs.
The repository already contained a richer canonical regression file at
tests/reproduce_issue_176.py (11 tests across three classes) which had
the same naming problem: it was also never auto-discovered.
The contributor's file added only TestMaxTokensPropagation (3 tests), which
is a strict subset of what the canonical file already covers. No unique
coverage is lost by removing it.
Changes:
- Rename tests/reproduce_issue_176.py -> tests/test_reproduce_issue_176.py
so all 11 regression tests are collected by 'pytest tests/'
- Remove tests/optimize reproduce_issue_176.py (redundant strict subset)
No production code changes. All 11 regression tests pass.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(export): escape Turtle/N-Triples string literals (fixes#1098)
Add RDFSerializer._escape_turtle_literal and apply it to the semantica:text
literal in serialize_to_turtle and the N-Triples text triple. Backslash,
double quote, newline, CR, and tab are escaped per the RDF 1.1 Turtle
STRING_LITERAL_QUOTE grammar, so entity text containing quotes or control
characters no longer emits invalid Turtle/N-Triples.
N-Triples previously escaped only quotes and newlines; now it also handles
backslashes and tabs via the shared escaper.
* fix(export): escape OWL-Time timestamp literals in Turtle output
Addresses Qodo finding on #1148: the OWL-Time branch of
serialize_to_turtle interpolated from_val/until_val directly into quoted
literals. Apply _escape_turtle_literal there too so timestamps containing
quotes, backslashes, or control characters cannot produce invalid Turtle.
* chore: remove stray local files (AGENTS.md, evals superpowers docs) from PR branch
---------
Profiling the viewer in headless Chromium (real DOM, production React)
separated remark parse time, React commit time and DOM node count across
large-prose, large-code-block, deep-nested-list and GFM-table fixtures.
Two findings, one of which is fixed here.
1. Every re-render re-parsed the whole document and remounted the whole
subtree. remarkPlugins and the ~20-entry components map were inline
literals, so each render allocated fresh arrow components; React saw a new
element type per mapped tag and replaced the DOM rather than updating it. A
DOM-identity probe confirmed the remount on every fixture. Because
react-markdown runs the remark pipeline inside its own render, an unrelated
state change -- clicking Copy, toggling Preview/Source -- re-paid the full
parse. Measured 364ms for a 1000-row GFM table and 1121ms for 2000 rows.
Hoisting both props to module scope and memoising the rendered element on
rawContent drops re-render cost to ~0.1ms across every fixture and removes
the remount (DOM identity now survives). Initial mount and node switching
are unchanged, since those are genuine parses.
2. Initial parse of large GFM tables is quadratic and lives upstream in
remark-gfm: the same table text parses in 12.5ms without the plugin and
1156ms with it at 2000 rows. Not addressed here -- any mitigation is a
product decision and is tracked on the issue.
Note that document size is the wrong threshold for this: 562KB of prose parses
in 85ms while a 27KB GFM table takes 102ms. Row count, not bytes, predicts cost.
Rendered output is unchanged; the components map is moved verbatim. All 66
Explorer graph-workspace tests pass.
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Sameer6305