fix(config): honor boolean env overrides in Config.get() (#1038)
Config.get() checked int before bool. Since bool subclasses int, boolean
environment values could be ignored or returned as integers.
Check bool first and strip whitespace before parsing boolean environment
values. This applies to the config modules for conflicts, deduplication,
split, embeddings, export, ingest, kg, normalize, ontology, and parse.
Also make _load_env_vars() use the same whitespace handling for mapped and
generic environment variables.
Fixes#1035
* fix(parse): import email.message and repair pdfplumber test mock
- email_parser.py uses email.message.Message at class-definition time but
only did 'import email', so 'import semantica.parse' fails in a fresh
Python process unless something else imported email.message first
- test_pdf_parser patched semantica.parse.pdf_parser.pdfplumber, which
never exists as a module attribute (pdfplumber is imported inside
PDFParser.parse); inject a fake module via sys.modules instead
* fix(parse): warn when PDF parse yields no text layer (scanned PDFs)
Scanned (image-only) PDFs parsed via the default pdfplumber route
returned an empty full_text with progress status 'completed' - no error,
no warning - so the failure only surfaced far downstream. Warn in
PDFParser.parse() when every parsed page yields no text (and extract_text
is enabled), pointing users to method='docling' with enable_ocr=True.
* fix(parse): improve scanned PDF detection
---------
Co-authored-by: shanyu910 <208111055+shanyu910@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(provenance): use timezone-aware UTC and assert stored records (#946)
Replace datetime.utcnow() in ProvenanceManager, ProvenanceEntry,
BridgeAxiom, and GraphBuilderWithProvenance with
datetime.now(timezone.utc), matching PipelineWithProvenance.
KG workflow and integration tests now read provenance back through
get_provenance() and assert algorithm metadata instead of generated
IDs, and call tracker methods that actually persist records.
* fix(provenance): compare provenance timestamps as instants, not strings
query_recorded_between() and audit_log() filtered and sorted on raw ISO
strings. With the timezone-aware change, a store can hold both pre-existing
naive stamps and offset-bearing ones, and the two are not string-comparable:
"...500000+00:00" sorts above "...500000", so a record at the identical
instant as a naive bound falls outside the range that should contain it.
Both now parse through _parse_timestamp() before comparing, reading naive
values as UTC. This mirrors ProvenanceTracker._parse_dt() in kg/, the class
ProvenanceManager replaces, so both sides of the migration answer a range
query the same way. Unparseable stored timestamps are skipped and logged
rather than silently dropped; unparseable bounds raise ValueError.
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Index the four module notebooks merged via #989-#992 (Provenance
Tracking, Reasoning, Change Management, Seed Data) in the cookbook
landing page, as committed in tracking issue #1032.
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
* feat(reasoning): rule-driven actions with provenance
Add a structured Action layer so matched rules can trigger side effects
instead of only deriving new facts, turning the reasoner into a
production-rule system.
L1 - Action type system:
- Action base class with execute(bindings, reasoner) + ?var substitution
- AssertAction (optional write-back to KnowledgeGraph), RetractAction,
CallAction (structured replacement for the unused Rule.handler),
EmitEventAction (delivers to a registered event sink)
- Rule.actions field; wired into Reasoner.forward_chain() and
ReteEngine.execute_matches() (via optional bind_reasoner)
L2 - Provenance-aware actions:
- Reasoner records fired actions (rule, bindings, confidence) to
action_log when provenance is enabled
- Fix dangling import in reasoning_provenance.py (ReasoningEngine ->
Reasoner, infer -> infer_facts)
Backward compatible: rules using the legacy handler still fire (wrapped
as a CallAction); rules without actions behave exactly as before.
Adds tests/reasoning/test_rule_actions.py (9 tests).
Closes#1095
* fix(reasoning): address qodo review findings on rule actions
- Token-aware variable substitution to avoid ?x/?xy prefix collision
- KnowledgeGraph write-back protocol (explicit API -> canonical translation -> ValueError)
- Structured action_log entries with timestamp
- Decouple action firing from conclusion dedup via per-activation tracking
(fires known conclusions once; retract-self no longer loops to max_iterations)
- Add Reasoner.infer_with_results preserving confidence; infer_facts delegates
- Forward provenance flag in ReasoningProvenance; drop **kwargs; propagate confidence
- Populate Rete Match.bindings from rule conditions
- Add regression tests for each fix
* fix(reasoning): persist fired action activations
* fix(reasoning): deduplicate Rete action execution
* fix(reasoning): canonicalize action activation identity
* docs(reasoning): explain action replay controls
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
* docs(cookbook): add Seed Data module notebook
Add cookbook/introduction/25_Seed_Data.ipynb covering the seed module
with verified, executable examples:
- SeedDataManager.register_source with a CSV source
- load_source record enrichment (entity_type/source provenance)
- create_foundation_graph entity/relationship/metadata structure
- validate_quality gating
The seed module ships seed_usage.md but has no cookbook coverage. All
API calls and outputs were executed against
semantica/seed/seed_manager.py.
Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
* docs(cookbook): isolate seed CSV in a temp dir and execute notebook in Jupyter
- Write companies.csv into a session-scoped tempfile.mkdtemp() directory
instead of the working directory, so a user's existing companies.csv
can never be silently clobbered (review finding)
- Run the notebook through a fresh Jupyter kernel (restart + run all +
save): real execution counts, print() cells saved as stream outputs
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
---------
Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
* fix(ingest): import sqlalchemy text where DBIngestor and DataExporter use it
sqlalchemy.text was imported function-locally in DatabaseConnector.connect
and test_connection, but called in DataExporter.export_table_data and
DBIngestor.execute_query, which never imported it. Both raised NameError,
re-wrapped by their except handlers into a ProcessingError reading
'Failed to execute query: name text is not defined' -- a message that
looks like a database fault rather than a missing import.
No test exercised either method, so this also repairs a pre-existing
failure in tests/ingest/test_notebook_02.py::test_08_database_ingestion.
Add SQLite-backed coverage for all three call sites, including the
SELECT COUNT(*) branch that only runs when no limit is passed and would
otherwise stay untested.
Closes#1015
* test(ingest): register setUp cleanups with addCleanup
TemporaryDirectory and the SQLAlchemy engine were released only in tearDown, which unittest skips when setUp raises partway through. Register each cleanup as soon as its resource exists so a failed setUp still disposes the engine and removes the temp directory. LIFO ordering keeps dispose before cleanup, as tearDown had it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
* fix(ontology): preserve #-terminated namespaces in SHACLGenerator base_uri (#1082)
SHACLGenerator.__init__ normalized base_uri with rstrip('/') + '/', turning a #-terminated RDF namespace (e.g. http://example.org/manufacturing#) into ...#/. Every generated URI then landed in a different namespace than the instance data, so SHACL validation silently passed because the shapes targeted nothing.
__init__ now preserves a base_uri already ending in '/' or '#', matching the #-aware normalization generate() already applies. shapes_uri inherits the fix.
Adds test_hash_namespace_base_uri_is_not_mangled (fails on the old normalization), plus a CHANGELOG entry. Full ontology suite green.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ontology): collapse slash runs, only preserve #-terminated base_uri
Qodo review caught that preserving any endswith('/') base left redundant
trailing slashes (e.g. .../ns////) intact, leaking a different namespace
into emitted IRIs. Now only '#'-terminated bases are kept verbatim; slash
runs are collapsed to a single '/', matching generate() normalization.
Adds test_slash_run_normalization_regression.
---------
Co-authored-by: changshenhan <217217832+changshenhan@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
track_relationship() has no dedicated subject/object fields, so the
Step 2 example only stored relationship_id + type, leaving readers
unable to reconstruct which two entities the relationship connects.
Encode subject_entity_id/object_entity_id in metadata by convention,
and note the lack of dedicated fields in the prose.
* fix(cli): write embed generate output in the format embed index reads
* Address review: structured results get their own --output writer
deduplicate --output and ontology align --output were routed through
_write_embeddings_output, a helper for numeric matrices: it rejects the
dict/list shapes these commands produce and the .csv extension deduplicate
documents. New _write_result_output serializes structured results — JSON,
JSON-lines for lists, CSV for rows — and both commands use it. embed
generate keeps the embeddings writer, whose strictness is what #994 fixed.
On the pyarrow gap: the parquet writer already fails with an actionable
message (install pyarrow or use .json). Silently writing JSON bytes to a
.parquet path would recreate #994's magic-bytes failure, so the error stays
an error and the default suggestion stays .json.
* fix(cli): improve structured output serialization
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(export): consolidate duplicate Turtle/N-Triples literal escapers
_escape_literal (module-level) and RDFSerializer._escape_turtle_literal did
identical work in the same order (backslash, double-quote, newline, CR, tab).
Drop the newer static helper added in #1148 and route all call sites through
_escape_literal instead. Behaviour no-op.
Closes#1218.
* fix(export): handle datetime/None temporal bounds safely in OWL-Time
_escape_literal is str-only, so routing datetime or None temporal bounds
through it raised AttributeError during Turtle export. Stringify non-str
bounds (plain f-string semantics) before escaping, and render None as an
empty bound. Add regression tests for datetime bounds and end-only
intervals. Addresses Qodo high-priority finding #2 on #1221.
* fix(export): use isoformat for datetime temporal bounds
str() on a datetime drops the ISO-8601 T separator, producing a lexically invalid xsd:dateTimeStamp. Use isoformat() when available; strengthen the test to assert the exact T-separated form.
---------
_calculate_decision_content_similarity's character-bigram fallback was
unconditional, so ordinary multi-word English queries could pick up
incidental bigram overlap with unrelated decisions via max(word_sim,
bigram_sim). Gate it to only activate for CJK-like scripts or queries
with at most one whitespace token, matching its documented purpose.
Separately, _add_decision_to_graph never persisted recorded_at as a
node property, so _rebuild_decision_indexes/_sync_decision_from_node
(which already read it back) always recovered "" after any reload.
* fix(explorer): coerce decision timestamp to str to prevent 422 on /api/decisions
ContextGraph stores decision timestamps as POSIX floats (e.g. 1786513069.69),
but DecisionResponse.timestamp is typed Optional[str]. Pydantic strict
validation rejects the float and the whole /api/decisions endpoint returns
HTTP 422 "Invalid input", which breaks the Decisions workspace in the
Knowledge Explorer entirely (no decision can be listed).
Coerce the value to str (preserving None) in _node_to_decision so the
response validates. Verified: /api/decisions now returns 200 and the 3
sample decisions render in the Decisions workspace.
* test(explorer): cover decision timestamp coercion in _node_to_decision
Regression tests for the 422 fix in _node_to_decision. Covers the cases
that produced HTTP 422 (float / int timestamps from ContextGraph) and
the ones that must keep working (None, already-string, missing key).
Verified the suite catches the regression: with the fix reverted, the
float / int / nan / inf cases fail with the same ValidationError that
caused the 422; with the fix applied all 6 pass.
* fix(explorer): preserve decision timestamp normalization
The route-level str() cast introduced in the initial fix bypasses
DecisionResponse._normalize_timestamp, the field validator on main that
converts POSIX float epochs to ISO-8601 strings via
datetime.fromtimestamp(value, tz=UTC).isoformat().
With the cast in place the API emits raw numeric strings such as
'1786513069.69' instead of '2026-08-12T05:37:49+00:00', breaking
datetime.fromisoformat() for every caller and failing
TestRecordedDecisions::test_list_decisions_serializes_float_timestamp.
It also silently accepts nan/inf/out-of-range epochs that the validator
is designed to reject.
Restore _node_to_decision() to pass the raw stored value through
unchanged so DecisionResponse._normalize_timestamp remains the single
normalization boundary for all three affected endpoints:
GET /api/decisions
GET /api/decisions/{id}
GET /api/decisions/{id}/precedents
Rewrite test_decision_route_timestamp.py so every assertion uses
datetime.fromisoformat() to verify ISO-8601 output and explicitly
asserts ValidationError for nan, inf, -inf and out-of-range epochs.
Add three TestClient integration tests covering the full production
path: record_decision() -> float stored in graph -> HTTP GET -> JSON.
---------
Co-authored-by: administrator <administrator@administratordeMac-mini.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
RelationExtractor.extract_relations(text, entities, ...) requires
entities, but the tool called it with only text, raising TypeError
on every invocation. Run NER first and pass the resulting entities
through, matching how the rest of the pipeline extracts relations.
* Add tests for max_tokens propagation in LLM methods
This test verifies that the max_tokens parameter is correctly propagated to the generate_typed method for different extraction functions.
* fix(tests): make issue-176 regression tests discoverable by pytest
The contributor's PR added tests/optimize reproduce_issue_176.py — a file
with a space in its name that never matched pytest's test_*.py discovery
pattern, so the regression would have been silently skipped in CI/local runs.
The repository already contained a richer canonical regression file at
tests/reproduce_issue_176.py (11 tests across three classes) which had
the same naming problem: it was also never auto-discovered.
The contributor's file added only TestMaxTokensPropagation (3 tests), which
is a strict subset of what the canonical file already covers. No unique
coverage is lost by removing it.
Changes:
- Rename tests/reproduce_issue_176.py -> tests/test_reproduce_issue_176.py
so all 11 regression tests are collected by 'pytest tests/'
- Remove tests/optimize reproduce_issue_176.py (redundant strict subset)
No production code changes. All 11 regression tests pass.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(export): escape Turtle/N-Triples string literals (fixes#1098)
Add RDFSerializer._escape_turtle_literal and apply it to the semantica:text
literal in serialize_to_turtle and the N-Triples text triple. Backslash,
double quote, newline, CR, and tab are escaped per the RDF 1.1 Turtle
STRING_LITERAL_QUOTE grammar, so entity text containing quotes or control
characters no longer emits invalid Turtle/N-Triples.
N-Triples previously escaped only quotes and newlines; now it also handles
backslashes and tabs via the shared escaper.
* fix(export): escape OWL-Time timestamp literals in Turtle output
Addresses Qodo finding on #1148: the OWL-Time branch of
serialize_to_turtle interpolated from_val/until_val directly into quoted
literals. Apply _escape_turtle_literal there too so timestamps containing
quotes, backslashes, or control characters cannot produce invalid Turtle.
* chore: remove stray local files (AGENTS.md, evals superpowers docs) from PR branch
---------
Profiling the viewer in headless Chromium (real DOM, production React)
separated remark parse time, React commit time and DOM node count across
large-prose, large-code-block, deep-nested-list and GFM-table fixtures.
Two findings, one of which is fixed here.
1. Every re-render re-parsed the whole document and remounted the whole
subtree. remarkPlugins and the ~20-entry components map were inline
literals, so each render allocated fresh arrow components; React saw a new
element type per mapped tag and replaced the DOM rather than updating it. A
DOM-identity probe confirmed the remount on every fixture. Because
react-markdown runs the remark pipeline inside its own render, an unrelated
state change -- clicking Copy, toggling Preview/Source -- re-paid the full
parse. Measured 364ms for a 1000-row GFM table and 1121ms for 2000 rows.
Hoisting both props to module scope and memoising the rendered element on
rawContent drops re-render cost to ~0.1ms across every fixture and removes
the remount (DOM identity now survives). Initial mount and node switching
are unchanged, since those are genuine parses.
2. Initial parse of large GFM tables is quadratic and lives upstream in
remark-gfm: the same table text parses in 12.5ms without the plugin and
1156ms with it at 2000 rows. Not addressed here -- any mitigation is a
product decision and is tracked on the issue.
Note that document size is the wrong threshold for this: 562KB of prose parses
in 85ms while a 27KB GFM table takes 102ms. Row count, not bytes, predicts cost.
Rendered output is unchanged; the components map is moved verbatim. All 66
Explorer graph-workspace tests pass.
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
MarkdownContentViewer.tsx exported the isSafeUrl helper alongside the
component so it could be unit tested, which tripped
react-refresh/only-export-components.
Move the helper into a sibling pure module, markdownUrlSafety.ts,
following the existing GraphWorkspace convention for testable non-component
logic (graphAnalytics.ts, pluginRegistryPredicates.ts,
temporalLifecyclePredicates.ts). The function body is moved verbatim — the
scheme allowlist, protocol-relative rejection, whitespace-only guard and
malformed-URL handling are unchanged — so the existing URL-safety tests pass
untouched apart from the import path.
The component module now exports only its component and prop type, clearing
the lint error without any change to the lint configuration.
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Dataset(default_union=True) presents triples from every named graph as a
single merged view and is itself an rdflib.Graph subclass, so it satisfies
_convert_to_dict()'s Graph-typed contract directly. Drops the O(n) manual
quad-copy loop while keeping the same named-graph fix and behavior.
_tool_export_graph fell through to json.dumps(kg) for any format outside
the RDF set, including values never declared in the tool's own inputSchema
enum. Nothing in this server validates tool-call args against inputSchema
before dispatch, so a typo'd or unsupported format (e.g. "yaml") silently
returned JSON data labeled with the wrong format and no error.
Validate against the declared format list up front and reuse the same
constant for the inputSchema enum so the two can't drift apart again.
* fix(triplet_store): OxigraphStore silently ignores storage_path and skips flush
Two persistence bugs in OxigraphStore:
1. `storage_path=...` was silently swallowed by **config. The __init__
parameter is named `path`, so passing the project-conventional
`storage_path` (used by ProvenanceManager and other stores) left
self.path = None and the store silently degraded to in-memory —
no error, no warning, data gone on exit. Accept `storage_path` as
an alias for `path`.
2. add_triplets never called flush(). pyoxigraph auto-flushes via
background threads but, per its docs, "might lag a little bit" —
that lag is a race where reopening or crashing immediately after a
write observes fewer triples. Call flush() explicitly for on-disk
stores to close the window.
Both verified: with the fix, `OxigraphStore(storage_path=...)` persists
across reopen; without it, data is lost.
* fix(triplet_store): improve oxigraph persistence
* test(triplet_store): clarify oxigraph persistence test
---------
Co-authored-by: administrator <administrator@administratordeMac-mini.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix: preserve generation kwargs in relation extraction
* fix: include generation params in extraction cache keys
* fix: cover provider-specific generation params in extraction cache key
_GENERATION_CACHE_KEYS only covered the common OpenAI-shaped generation
params, so calls that differed only in Anthropic's system/stop_sequences,
Gemini's candidate_count, or Ollama's repeat_penalty/num_ctx/context_window
could still return a stale cached result generated under different settings.
Add these provider-specific keys to the cache key and add regression tests
covering system prompt, stop_sequences, and repeat_penalty.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
``requests.exceptions.RequestException`` subclasses ``OSError``, so the
``except (ImportError, OSError)`` handler in ``load_from_api`` swallowed
genuine network failures (connection errors, timeouts, HTTP errors) and
reported them as "requests library not available", hiding the real cause.
Remove the obsolete handler so those failures fall through to the generic
handler, which reports "Failed to load from API: ..." and chains the real
exception as ``__cause__``. Update the docstring's ``Raises`` section to
match the actual behavior.
Fixes#949
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
* Remove unreachable dead code
Delete symbols with no callers anywhere in the codebase, tests, or docs,
confirmed by a repo-wide search. These are internal/private or app-layer
(explorer) symbols, not part of the importable library's public API
(no __all__ / package re-export), so there is no user-facing change.
Removed:
- poc_runner.py: parse_import_csv_row (unused nested helper)
- change_management/version_storage.py: create_graph_snapshot_record
- context/graph_schema.py: drop_decision_schema
- explorer/dependencies.py: get_ws_manager (+ now-unused ConnectionManager import)
- explorer/routes/graph.py: _extract_node_embeddings (+ stale cross-ref comment)
- explorer/routes/ontology.py: ProposalState
- explorer/schemas.py: ErrorResponse, TemporalSnapshotResponse, ExportResponse,
StandardMessageResponse
- semantic_extract/methods.py: _parse_entity_result, _parse_triplet_result
- triplet_store/methods.py: _get_query_engine (+ now-unused _global_query_engine)
Co-Authored-By: Vinv-AI <309466812+Vinv-AI@users.noreply.github.com>
* Address review: drop now-orphaned helper and fix stale docstring
- Remove _coerce_embedding_vector from explorer/routes/graph.py: its only
non-recursive caller was _extract_node_embeddings (removed in this PR), so
it is now dead. The live coercion logic lives in
GraphSession._coerce_embedding_vector.
- Update explorer/dependencies.py module docstring: it no longer injects
ConnectionManager (get_ws_manager was removed); note that websocket manager
access is via app.state.ws_manager.
Co-Authored-By: Vinv-AI <309466812+Vinv-AI@users.noreply.github.com>
* Keep public helpers with a DeprecationWarning instead of removing them
create_graph_snapshot_record() and drop_decision_schema() are not
underscore-prefixed, so downstream users can import them directly from
their modules even though they are not re-exported from the package
__init__.py. A repo search only proves there are no in-tree callers.
Restore both unchanged and emit a DeprecationWarning on call, with a
matching ".. deprecated::" note in each docstring pointing at the
replacement. This keeps the PR non-breaking; the actual removal can
happen in a future major version.
The underscore-prefixed helper removals are unaffected.
---------
Co-authored-by: noQbot <noQbot@users.noreply.github.com>
Co-authored-by: Vinv-AI <309466812+Vinv-AI@users.noreply.github.com>
Co-authored-by: noQbot <anshul@vinv.ai>
Kyou