Compare commits

...
22 Commits
Author SHA1 Message Date
Yunare MaiaandSameer Kadam e12eec40a1 refactor(ner): remove dead _extract_with_spacy method and unused self.nlp (#1220)
* test(ner): fix NER configuration tests for the typed LLM extraction API

Two of the three failing tests tracked in #1059 were still red after
#1070 was closed because the mocks targeted the pre-typed provider API:

- test_ner_llm_config mocked generate_structured, but the LLM path now
  goes through generate_typed with a Pydantic schema. Mock the typed
  response (namespace items with .text/.label/.start/.end/.confidence)
  and expect extraction_method 'llm_typed'.
- test_ner_pattern_config asserted 'Apple Inc' without the trailing
  dot, but the ORG pattern captures it via (?:\.|\b). Assert 'Apple
  Inc.' to match current production behavior.

Verified locally: 8/8 pass in test_ner_configurations.py; the
performance-test failures in tests/semantic_extract/ reproduce on a
clean main checkout and are unrelated.

Fixes #1059

Signed-off-by: Yunare Maia <yunare@gmail.com>

* refactor(ner): remove dead _extract_with_spacy method and unused self.nlp

_extract_with_spacy() had no callers: the ML dispatch path goes through
get_entity_method('ml') -> extract_entities_ml(), which loads the spaCy
model lazily via the process-level cache in methods.py. The instance
attribute self.nlp was only read by that dead method, so __init__ now
just validates the runtime (keeping the _ml_runtime_usable gate) instead
of eagerly loading a model that was never used.

Fixes #1058

Signed-off-by: Yunare Maia <yunare@gmail.com>

* test(split): rewrite NERExtractor cache tests to not rely on removed .nlp attribute

NERExtractor.nlp was removed in this PR as part of dead-code cleanup
(the attribute was only used by the equally-dead _extract_with_spacy()).
The three affected tests in TestNERExtractorSpacyModelCache previously
verified cache behavior through .nlp identity comparisons; rewrite them
to use load-call counts and direct se_methods.load_spacy_model() cache
queries instead:

- test_ner_extractor_reuses_cached_model_across_instances: drop the
  e1.nlp is e2.nlp is e3.nlp assertion; len(calls)==1 already proves
  reuse; add a cache query to confirm the cached object is non-None.

- test_ner_extractor_distinct_model_names_load_separately: store each
  mock nlp in a dict keyed by name, then query the cache to assert
  sm_cached is loaded['en_core_web_sm'] and sm_cached is not lg_cached.

- test_ner_extractor_failed_load_not_cached_and_retried: replace
  extractor.nlp is None/not None with is-not-None construction checks
  and a final cache query that verifies the recovered model is the
  exact object returned by working_load.

All three tests still exercise the original behavioral contract (no
crash on missing model, failures not cached / retried, successful load
shared across instances); they just no longer rely on a private
instance attribute that no longer exists.

---------

Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
2026-08-27 17:56:03 +05:30
Kyou 4da27c38bb fix(config): honor boolean env overrides in Config.get() (#1038)
fix(config): honor boolean env overrides in Config.get() (#1038)

Config.get() checked int before bool. Since bool subclasses int, boolean
environment values could be ignored or returned as integers.

Check bool first and strip whitespace before parsing boolean environment
values. This applies to the config modules for conflicts, deduplication,
split, embeddings, export, ingest, kg, normalize, ontology, and parse.

Also make _load_env_vars() use the same whitespace handling for mapped and
generic environment variables.

Fixes #1035
2026-08-27 16:33:09 +05:00
7f928f9f8e fix(parse): warn when PDF parse yields no text layer (scanned PDFs) (#1021)
* fix(parse): import email.message and repair pdfplumber test mock

- email_parser.py uses email.message.Message at class-definition time but
  only did 'import email', so 'import semantica.parse' fails in a fresh
  Python process unless something else imported email.message first
- test_pdf_parser patched semantica.parse.pdf_parser.pdfplumber, which
  never exists as a module attribute (pdfplumber is imported inside
  PDFParser.parse); inject a fake module via sys.modules instead

* fix(parse): warn when PDF parse yields no text layer (scanned PDFs)

Scanned (image-only) PDFs parsed via the default pdfplumber route
returned an empty full_text with progress status 'completed' - no error,
no warning - so the failure only surfaced far downstream. Warn in
PDFParser.parse() when every parsed page yields no text (and extract_text
is enabled), pointing users to method='docling' with enable_ocr=True.

* fix(parse): improve scanned PDF detection

---------

Co-authored-by: shanyu910 <208111055+shanyu910@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
2026-08-27 16:51:04 +05:30
aoright 65e6dcfef5 fix(worker): remove unused sys import and organize imports (#1061)
Signed-off-by: aoright <102943475+aoright@users.noreply.github.com>
2026-08-27 16:08:33 +05:00
pravit-ampandPravit Ampapathini 0775b0114e test(provenance): assert stored records in KG provenance suites (#946) (#1132)
* fix(provenance): use timezone-aware UTC and assert stored records (#946)

Replace datetime.utcnow() in ProvenanceManager, ProvenanceEntry,
BridgeAxiom, and GraphBuilderWithProvenance with
datetime.now(timezone.utc), matching PipelineWithProvenance.

KG workflow and integration tests now read provenance back through
get_provenance() and assert algorithm metadata instead of generated
IDs, and call tracker methods that actually persist records.

* fix(provenance): compare provenance timestamps as instants, not strings

query_recorded_between() and audit_log() filtered and sorted on raw ISO
strings. With the timezone-aware change, a store can hold both pre-existing
naive stamps and offset-bearing ones, and the two are not string-comparable:
"...500000+00:00" sorts above "...500000", so a record at the identical
instant as a naive bound falls outside the range that should contain it.

Both now parse through _parse_timestamp() before comparing, reading naive
values as UTC. This mirrors ProvenanceTracker._parse_dt() in kg/, the class
ProvenanceManager replaces, so both sides of the migration answer a range
query the same way. Unparseable stored timestamps are skipped and logged
rather than silently dropped; unparseable bounds raise ValueError.

---------

Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
2026-08-27 15:40:38 +05:00
Mohd Kaif f4c3064571 Merge pull request #1113 from cxzg007/fix/rdf-name-label-normalization
fix(export): normalize entity name to label on all RDF paths
2026-08-27 16:08:53 +05:30
KaifAhmad1 b2dc633796 Merge remote-tracking branch 'origin/main' into pr-1113-work
# Conflicts:
#	semantica/export/rdf_exporter.py
2026-08-27 15:51:50 +05:30
Mohd Kaif 13b287b974 Merge pull request #1173 from yzxcj797/fix/neo4j-edge-id-space-1136
fix(graph_store): resolve application ids to internal ids when creating relationships
2026-08-27 15:39:02 +05:30
Mohd Kaif cec9bee099 Merge branch 'main' into fix/neo4j-edge-id-space-1136 2026-08-27 15:31:51 +05:30
Mohd Kaif 36ced4e826 Merge pull request #1225 from LeonSGP43/cookbook-index-22-25
docs(cookbook): add index entries for notebooks 22-25
2026-08-27 15:24:02 +05:30
LeonSGP43 6032b4e0bc docs(cookbook): add index entries for notebooks 22-25
Index the four module notebooks merged via #989-#992 (Provenance
Tracking, Reasoning, Change Management, Seed Data) in the cookbook
landing page, as committed in tracking issue #1032.

Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
2026-08-27 17:39:48 +08:00
yzxcj797 8db95f00c6 fix(utils): raise on key collision in flatten_dict instead of silently dropping values (#1012) 2026-08-27 15:07:53 +05:30
Guofang.Tang 23baf21d5a fix(ontology): retain data properties for normalized class names (#1171)
* fix(ontology): retain properties for normalized class names

* perf(ontology): precompute normalized class lookup
2026-08-27 13:32:14 +05:00
cxzg007and江俊杰 5d54919804 feat(reasoning): rule-driven actions with provenance (#1096)
* feat(reasoning): rule-driven actions with provenance

Add a structured Action layer so matched rules can trigger side effects
instead of only deriving new facts, turning the reasoner into a
production-rule system.

L1 - Action type system:
- Action base class with execute(bindings, reasoner) + ?var substitution
- AssertAction (optional write-back to KnowledgeGraph), RetractAction,
  CallAction (structured replacement for the unused Rule.handler),
  EmitEventAction (delivers to a registered event sink)
- Rule.actions field; wired into Reasoner.forward_chain() and
  ReteEngine.execute_matches() (via optional bind_reasoner)

L2 - Provenance-aware actions:
- Reasoner records fired actions (rule, bindings, confidence) to
  action_log when provenance is enabled
- Fix dangling import in reasoning_provenance.py (ReasoningEngine ->
  Reasoner, infer -> infer_facts)

Backward compatible: rules using the legacy handler still fire (wrapped
as a CallAction); rules without actions behave exactly as before.

Adds tests/reasoning/test_rule_actions.py (9 tests).

Closes #1095

* fix(reasoning): address qodo review findings on rule actions

- Token-aware variable substitution to avoid ?x/?xy prefix collision
- KnowledgeGraph write-back protocol (explicit API -> canonical translation -> ValueError)
- Structured action_log entries with timestamp
- Decouple action firing from conclusion dedup via per-activation tracking
  (fires known conclusions once; retract-self no longer loops to max_iterations)
- Add Reasoner.infer_with_results preserving confidence; infer_facts delegates
- Forward provenance flag in ReasoningProvenance; drop **kwargs; propagate confidence
- Populate Rete Match.bindings from rule conditions
- Add regression tests for each fix

* fix(reasoning): persist fired action activations

* fix(reasoning): deduplicate Rete action execution

* fix(reasoning): canonicalize action activation identity

* docs(reasoning): explain action replay controls

---------

Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
2026-08-27 13:18:07 +05:00
cxzg007and江俊杰 c9c777993b fix(pipeline): wire registered step handlers (#1215)
* fix(pipeline): wire registered step handlers

Resolve handlers registered by step type, keep explicit handlers authoritative, and prevent builder control fields from leaking into runtime kwargs.

Refs #1214

* fix(pipeline): preserve dependencies on deserialize

* fix(pipeline): dispatch falsy handlers via identity check

---------

Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
2026-08-27 13:11:42 +05:00
LeonSGPandLeonSGP43 9cec305a75 docs(cookbook): add Seed Data module notebook (#992)
* docs(cookbook): add Seed Data module notebook

Add cookbook/introduction/25_Seed_Data.ipynb covering the seed module
with verified, executable examples:

- SeedDataManager.register_source with a CSV source
- load_source record enrichment (entity_type/source provenance)
- create_foundation_graph entity/relationship/metadata structure
- validate_quality gating

The seed module ships seed_usage.md but has no cookbook coverage. All
API calls and outputs were executed against
semantica/seed/seed_manager.py.

Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>

* docs(cookbook): isolate seed CSV in a temp dir and execute notebook in Jupyter

- Write companies.csv into a session-scoped tempfile.mkdtemp() directory
  instead of the working directory, so a user's existing companies.csv
  can never be silently clobbered (review finding)
- Run the notebook through a fresh Jupyter kernel (restart + run all +
  save): real execution counts, print() cells saved as stream outputs

Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>

---------

Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
2026-08-27 13:06:23 +05:00
LeonSGPandLeonSGP43 3d0ce55fd7 docs(cookbook): add Change Management module notebook (#991)
* docs(cookbook): add Change Management module notebook

Add cookbook/introduction/24_Change_Management.ipynb covering the
change_management module with verified, executable examples:

- ChangeLogEntry with email-validated author field
- InMemoryVersionStorage save/get/list_all/exists/delete round trip
- named tags (save_tag/get_tag) for release pinning
- compute_checksum / verify_checksum integrity verification with
  tamper detection

The change_management module currently has no cookbook coverage. All
API calls and outputs were executed against
semantica/change_management/change_log.py and version_storage.py.

Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>

* docs(cookbook): clarify outputs verified against repo source, not PyPI release

Signed-off-by: LeonSGP43 <leonsgp43@users.noreply.github.com>

* docs(cookbook): execute change management notebook in Jupyter (real kernel run, stream outputs, execution counts)

Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>

---------

Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <leonsgp43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
2026-08-27 13:00:36 +05:00
LeonSGPandLeonSGP43 b13cc1cca2 docs(cookbook): add Reasoning module notebook (#990)
* docs(cookbook): add Reasoning module notebook

Add cookbook/introduction/23_Reasoning.ipynb covering the reasoning
module with verified, executable examples:

- Reasoner facade: add_fact / add_rule / forward_chain
- one-shot infer_facts(facts, rules)
- backward_chain goal proving with premises
- re-run-safe rule deduplication (#732)
- DatalogReasoner: semi-naive fixpoint evaluation + variable queries
- ExplanationGenerator: Explanation / ReasoningPath records

The reasoning module currently has no cookbook coverage even though it
ships reasoning_usage.md in the package. All API calls and outputs were
verified against semantica/reasoning/reasoner.py,
datalog_reasoner.py, and explanation_generator.py.

Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>

* docs(cookbook): correct infer_facts semantics description (appends to instance state, no reset)

Signed-off-by: LeonSGP43 <leonsgp43@users.noreply.github.com>

* docs(cookbook): execute reasoning notebook in Jupyter (real kernel run, stream outputs, execution counts)

Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>

---------

Signed-off-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <leonsgp43@users.noreply.github.com>
Signed-off-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
2026-08-27 12:55:02 +05:00
yzxcj797 92ad7bc2df Address review: string-only application id resolution
Per the Qodo review: only string application ids are recorded in (and
resolved through) _app_node_id_map. Internal ids are commonly integers,
so an integer application id could collide with — and silently remap —
a caller-supplied internal id of the same value. Also pass a labels list
to create_node in the regression test, matching the API signature.
2026-08-22 02:07:55 +08:00
yzxcj797 db81136b0a fix(graph_store): resolve application ids to internal ids when creating relationships
GraphStore.add_edges reads application-level string ids from
source_id/target_id and passed them straight to the backend, while
Neo4jStore.create_relationship matches on internal integer ids (id(n)).
Nothing resolved one to the other, so persisting a graph created every
node and zero relationships — each edge failed with 'nodes not found'
as a logger.warning and the call appeared to succeed (#1136).

add_nodes already receives the application-id/internal-id pair from
create_nodes (the app id is preserved in properties['id']) and discarded
it one statement before add_edges needed it. Keep the map on the store,
populate it from both add_nodes and create_node, and resolve known
application ids in create_relationship. Unknown ids pass through
unchanged, so direct internal-id callers and backends whose ids are the
application ids keep their existing behavior.
2026-08-22 01:23:46 +08:00
江俊杰 1c27a0ae7e fix(export): escape RDF literals and use URI-aware id fallback
Address Qodo review on #1113:
- Escape entity text for Turtle, RDF/XML and N-Triples so names containing
  quotes, XML markup, backslashes or control chars cannot break out of the
  literal or inject RDF/XML (High/Security).
- Replace colon-only id split with URI-aware local-name extraction so an id
  like https://example.org/acme yields 'acme', not '//example.org/acme'
  (Medium/Correctness).
- Add regression tests: escaping (quotes/XML/backslash/CR/LF), parseability
  via rdflib, and exact id local-name assertions.
2026-08-20 10:19:52 +08:00
江俊杰 9e2f349221 fix(export): normalize entity name to label on all RDF paths
convert_kg_to_rdf() maps an entity's 'name' to 'label'/'text' but was
never invoked from export_to_rdf(), so graphs produced by GraphBuilder
(which emit 'name') exported with an empty semantica:text on every RDF
format (turtle, ntriples, rdfxml, jsonld). Call convert_kg_to_rdf() at
the export boundary before validation/serialization so all formats
benefit from a single normalization step.

Add regression tests asserting a name-only entity exports a non-empty
label across all four serializers and the file-writing entry point,
plus that an explicit 'text' is not clobbered and an id tail is used
as a fallback label.

Closes #1097
2026-08-20 10:19:52 +08:00
49 changed files with 3999 additions and 580 deletions
+6
View File
@@ -119,6 +119,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **KG provenance tests asserted on generated ID strings instead of stored records, and `kg_provenance.py` was missed by the `utcnow` sweep** (closes #946) by @pravit-amp
- The KG workflow and integration suites checked that a tracker call returned an ID matching a prefix (`assert cent_id.startswith("centrality_")`) without ever reading the record back, so an ID generator that returned a well-formed string and wrote nothing would have passed. Worse, some of those calls named tracker methods that do not exist anywhere in `semantica/` (`track_layer_analysis`, `track_centrality_score`), so the assertions were satisfied with no real interaction behind them
- Those tests now read provenance back through `get_provenance()` and assert on algorithm metadata, and call the methods that actually persist records. Verified by mutation rather than by a green run alone: neutering the manager's storage write (`self.storage.store(...)` → no-op) fails 10 tests
- `GraphBuilderWithProvenance` in `semantica/kg/kg_provenance.py` still stamped `activity_started_at_time`/`activity_ended_at_time` with the deprecated `datetime.utcnow()`; it was outside the `export/`+`provenance/` scope of the #1114 sweep below and now uses the same `utc_now_iso()` helper. `docs/guides/provenance.md` and `docs/reference/provenance.md` were still documenting `utcnow()` and a naive timestamp example, and now show the helper and the offset-bearing form
- 16 tests across the affected suites ended in `return <value>` instead of asserting, which pytest reports as `PytestReturnNotNoneWarning`; now zero
- **The temporal-evolution `stability` metric was a hardcoded placeholder, not a duration**
- `TemporalGraphQuery.analyze_evolution()` documents `stability` as a "relationship duration/stability measure", but the implementation appended a constant `1` for every relationship with both `valid_from` and `valid_until` set (`durations.append(1) # Placeholder`). The reported stability was therefore always `1.0` when any bounded relationship existed and `0` otherwise — it never reflected how long relationships actually stayed valid, so it could not distinguish a graph of decade-long relationships from one of one-second relationships
- `stability` now computes the mean valid-time duration in seconds (`(valid_until - valid_from).total_seconds()`) across relationships that have both bounds set. Relationships with a missing or open `valid_from`/`valid_until` are skipped (their duration is unbounded), and non-positive intervals are clamped to `0`; an empty set still reports `0`
+383
View File
@@ -0,0 +1,383 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b76a5997",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)\n",
"\n",
"# Reasoning Module — Practical Guide\n",
"\n",
"Semantica's `reasoning` module derives new knowledge from existing facts and knowledge graphs. It ships several strategies behind one facade:\n",
"\n",
"- **`Reasoner`** — unified facade with forward chaining, backward chaining, and one-shot `infer_facts`\n",
"- **`DatalogReasoner`** — semi-naive Datalog fixpoint evaluation with variable queries\n",
"- **`ExplanationGenerator`** — human-readable explanations and reasoning paths for inferred conclusions\n",
"- Plus lower-level engines: `ReteEngine`, `SPARQLReasoner`, `GraphReasoner`, temporal reasoning\n",
"\n",
"This notebook walks through the facade, the Datalog engine, and explanations. All APIs are verified against `semantica/reasoning/`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "52073af7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:45:55.427457Z",
"iopub.status.busy": "2026-08-26T18:45:55.427247Z",
"iopub.status.idle": "2026-08-26T18:45:57.266607Z",
"shell.execute_reply": "2026-08-26T18:45:57.264783Z"
}
},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "markdown",
"id": "06deb916",
"metadata": {},
"source": [
"## 1) Forward chaining with the `Reasoner` facade\n",
"\n",
"Facts are simple `Predicate(args)` strings. Rules use `IF <conditions> THEN <conclusion>` with `?x`-style variables. `forward_chain()` derives everything possible and returns a list of `InferenceResult` objects."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "519ca92d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:45:57.270791Z",
"iopub.status.busy": "2026-08-26T18:45:57.270352Z",
"iopub.status.idle": "2026-08-26T18:45:59.991941Z",
"shell.execute_reply": "2026-08-26T18:45:59.990678Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>Reasoner</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>DatalogReasoner</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>ExplanationGenerator</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"🔄 Semantica is reasoning: Performing forward chaining 🤔 reasoning Reasoner |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Inferred 2 new facts\n",
" Human(Jane) (rule: Rule 1, confidence: 1.0)\n",
" Human(John) (rule: Rule 1, confidence: 1.0)\n"
]
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"\n",
"reasoner.add_fact(\"Person(John)\")\n",
"reasoner.add_fact(\"Person(Jane)\")\n",
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"results = reasoner.forward_chain()\n",
"print(f\"Inferred {len(results)} new facts\")\n",
"for res in results:\n",
" print(f\" {res.conclusion} (rule: {res.rule_used.name}, confidence: {res.confidence})\")"
]
},
{
"cell_type": "markdown",
"id": "c1131c45",
"metadata": {},
"source": [
"## 2) One-shot inference with `infer_facts`\n",
"\n",
"`infer_facts(facts, rules)` **adds** the given facts and rules to this `Reasoner` instance, runs forward chaining to fixpoint, and returns the derived facts as strings. It does not reset the instance's existing state — create a fresh `Reasoner()` first if you need isolation between runs."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "26249990",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:45:59.995447Z",
"iopub.status.busy": "2026-08-26T18:45:59.995069Z",
"iopub.status.idle": "2026-08-26T18:46:00.004107Z",
"shell.execute_reply": "2026-08-26T18:46:00.002873Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['Employee(Jane, Acme)', 'Employee(John, Acme)']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"derived = Reasoner().infer_facts(\n",
" facts=[\"WorksFor(John, Acme)\", \"WorksFor(Jane, Acme)\"],\n",
" rules=[\"IF WorksFor(?x, ?y) THEN Employee(?x, ?y)\"],\n",
")\n",
"derived"
]
},
{
"cell_type": "markdown",
"id": "d5504a38",
"metadata": {},
"source": [
"## 3) Backward chaining: proving a goal\n",
"\n",
"`backward_chain(goal)` works backwards from a conclusion through the rules. It returns the `InferenceResult` that proves the goal, or `None`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c4ef85dd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.007740Z",
"iopub.status.busy": "2026-08-26T18:46:00.007346Z",
"iopub.status.idle": "2026-08-26T18:46:00.015561Z",
"shell.execute_reply": "2026-08-26T18:46:00.014145Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Human(John)\n",
"premises: ['Person(John)']\n"
]
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"proof = reasoner.backward_chain(\"Human(John)\")\n",
"print(proof.conclusion if proof else \"not provable\")\n",
"print(\"premises:\", proof.premises if proof else None)"
]
},
{
"cell_type": "markdown",
"id": "b245581d",
"metadata": {},
"source": [
"## 4) Re-run safety\n",
"\n",
"`add_rule` deduplicates rules with identical conditions and conclusion, so re-executing a setup cell (the common Jupyter re-run) does not duplicate rules — see issue #732."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "fb2aeb39",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.019091Z",
"iopub.status.busy": "2026-08-26T18:46:00.018881Z",
"iopub.status.idle": "2026-08-26T18:46:00.024042Z",
"shell.execute_reply": "2026-08-26T18:46:00.022836Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Skipping duplicate rule (same conditions/conclusion as 'rule_1'): IF Person(?x) THEN Human(?x)\n"
]
},
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"\n",
"# Simulate a Jupyter cell re-run: add the same rule twice\n",
"r1 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"r2 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"\n",
"len(reasoner.rules)"
]
},
{
"cell_type": "markdown",
"id": "ba2e5c4a",
"metadata": {},
"source": [
"## 5) Datalog reasoning\n",
"\n",
"`DatalogReasoner` uses classic Datalog syntax (`head :- body.`) and semi-naive fixpoint evaluation. Queries return variable bindings as a list of dicts — use uppercase variables to ask *which* facts hold."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9ec5c0c4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.026769Z",
"iopub.status.busy": "2026-08-26T18:46:00.026588Z",
"iopub.status.idle": "2026-08-26T18:46:00.034963Z",
"shell.execute_reply": "2026-08-26T18:46:00.032672Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[{'X': 'tom', 'Z': 'ann'}]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import DatalogReasoner\n",
"\n",
"datalog = DatalogReasoner()\n",
"datalog.add_fact(\"parent(tom, mary)\")\n",
"datalog.add_fact(\"parent(mary, ann)\")\n",
"datalog.add_rule(\"grandparent(X, Z) :- parent(X, Y), parent(Y, Z)\")\n",
"\n",
"datalog.derive_all()\n",
"datalog.query(\"grandparent(X, Z)\")"
]
},
{
"cell_type": "markdown",
"id": "d4f0689b",
"metadata": {},
"source": [
"## 6) Explanations for inferred conclusions\n",
"\n",
"`ExplanationGenerator` turns `InferenceResult` objects into structured `Explanation` and `ReasoningPath` records, so agents can show *why* they believe a derived fact."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "19dcd3a7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:00.038649Z",
"iopub.status.busy": "2026-08-26T18:46:00.038396Z",
"iopub.status.idle": "2026-08-26T18:46:00.059188Z",
"shell.execute_reply": "2026-08-26T18:46:00.057805Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"('Explanation', 'ReasoningPath')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.reasoning import Reasoner, ExplanationGenerator\n",
"\n",
"reasoner = Reasoner()\n",
"reasoner.add_fact(\"Person(John)\")\n",
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
"results = reasoner.forward_chain()\n",
"\n",
"gen = ExplanationGenerator()\n",
"explanation = gen.generate_explanation(results[0])\n",
"path = gen.show_reasoning_path(results[0])\n",
"\n",
"type(explanation).__name__, type(path).__name__"
]
},
{
"cell_type": "markdown",
"id": "fb882ee4",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Task | API |\n",
"|---|---|\n",
"| Derive all new facts | `Reasoner.forward_chain()` |\n",
"| One-shot inference | `Reasoner.infer_facts(facts, rules)` |\n",
"| Prove a goal | `Reasoner.backward_chain(goal)` |\n",
"| Datalog fixpoint | `DatalogReasoner.derive_all()` + `query(\"p(X, Y)\")` |\n",
"| Explain a conclusion | `ExplanationGenerator.generate_explanation(result)` |\n",
"\n",
"See also `semantica/reasoning/reasoning_usage.md` and the module docstrings for `ReteEngine`, `SPARQLReasoner`, and temporal reasoning."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
@@ -0,0 +1,299 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "8d7096ea",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)\n",
"\n",
"# Change Management — Practical Guide\n",
"\n",
"Semantica's `change_management` module provides versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies:\n",
"\n",
"- **`ChangeLogEntry`** — standardized change metadata (validated timestamp/author)\n",
"- **`InMemoryVersionStorage` / `SQLiteVersionStorage`** — version snapshot storage with named tags\n",
"- **`compute_checksum` / `verify_checksum`** — SHA-256 integrity verification\n",
"\n",
"This notebook runs a complete save → tag → verify → tamper-detect cycle. All outputs are real executed results verified against the repository's `semantica/change_management/` source at the time of writing (the `pip install` cell may fetch a newer release with slightly different behavior)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "7bdffec1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:37.171333Z",
"iopub.status.busy": "2026-08-26T18:46:37.171183Z",
"iopub.status.idle": "2026-08-26T18:46:39.060860Z",
"shell.execute_reply": "2026-08-26T18:46:39.059594Z"
}
},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "markdown",
"id": "169efee1",
"metadata": {},
"source": [
"## 1) A `ChangeLogEntry` records *who* changed *what*, *when*\n",
"\n",
"`author` must be a valid email — the dataclass validates on construction (`ValidationError` otherwise), which keeps audit trails clean."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5b17acdb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.064077Z",
"iopub.status.busy": "2026-08-26T18:46:39.063818Z",
"iopub.status.idle": "2026-08-26T18:46:39.321881Z",
"shell.execute_reply": "2026-08-26T18:46:39.321036Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"ChangeLogEntry(timestamp='2026-08-15T09:00:00Z', author='demo@example.com', description='initial version', change_id=None, related_changes=[])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.change_management import ChangeLogEntry\n",
"\n",
"entry = ChangeLogEntry(\n",
" timestamp=\"2026-08-15T09:00:00Z\",\n",
" author=\"demo@example.com\",\n",
" description=\"initial version\",\n",
")\n",
"entry"
]
},
{
"cell_type": "markdown",
"id": "53d8df5c",
"metadata": {},
"source": [
"## 2) Save a versioned snapshot\n",
"\n",
"A snapshot is a dict with a required `label` plus your payload. Here we attach the KG data, the change log, and a SHA-256 `checksum` computed over everything except the checksum field itself."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "fec16f24",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.325528Z",
"iopub.status.busy": "2026-08-26T18:46:39.325140Z",
"iopub.status.idle": "2026-08-26T18:46:39.331480Z",
"shell.execute_reply": "2026-08-26T18:46:39.330586Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from semantica.change_management import InMemoryVersionStorage, compute_checksum\n",
"\n",
"storage = InMemoryVersionStorage()\n",
"\n",
"snapshot = {\n",
" \"label\": \"v1.0.0\",\n",
" \"data\": {\"entities\": {\"acme\": {\"type\": \"Company\"}}},\n",
" \"change_log\": {\n",
" \"timestamp\": entry.timestamp,\n",
" \"author\": entry.author,\n",
" \"description\": entry.description,\n",
" },\n",
"}\n",
"snapshot[\"checksum\"] = compute_checksum({k: v for k, v in snapshot.items() if k != \"checksum\"})\n",
"\n",
"storage.save(snapshot)\n",
"storage.exists(\"v1.0.0\")"
]
},
{
"cell_type": "markdown",
"id": "0f1c603b",
"metadata": {},
"source": [
"## 3) Named tags pin a version for releases\n",
"\n",
"`save_tag` / `get_tag` map stable names (e.g. `release`) to version labels, decoupling consumers from label churn."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "62f7643e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.335182Z",
"iopub.status.busy": "2026-08-26T18:46:39.334886Z",
"iopub.status.idle": "2026-08-26T18:46:39.339586Z",
"shell.execute_reply": "2026-08-26T18:46:39.338568Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"('v1.0.0', ['v1.0.0'])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"storage.save_tag(\"release\", \"v1.0.0\")\n",
"\n",
"storage.get_tag(\"release\"), [s[\"label\"] for s in storage.list_all()]"
]
},
{
"cell_type": "markdown",
"id": "96df12da",
"metadata": {},
"source": [
"## 4) Verify integrity — and catch tampering\n",
"\n",
"`verify_checksum(snapshot)` recomputes the SHA-256 over the snapshot (minus its `checksum` field) and compares. A single mutated character in the data flips the result to `False`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "26d0de85",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.342653Z",
"iopub.status.busy": "2026-08-26T18:46:39.342466Z",
"iopub.status.idle": "2026-08-26T18:46:39.346714Z",
"shell.execute_reply": "2026-08-26T18:46:39.345623Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"intact: True\n",
"tampered: False\n"
]
}
],
"source": [
"from semantica.change_management import verify_checksum\n",
"\n",
"stored = storage.get(\"v1.0.0\")\n",
"print(\"intact:\", verify_checksum(stored))\n",
"\n",
"tampered = storage.get(\"v1.0.0\")\n",
"tampered[\"data\"][\"entities\"][\"acme\"][\"note\"] = \"mutated after the fact\"\n",
"print(\"tampered:\", verify_checksum(tampered))"
]
},
{
"cell_type": "markdown",
"id": "bd14c3e4",
"metadata": {},
"source": [
"## 5) Retiring a version\n",
"\n",
"`delete(label)` removes a snapshot; tags pointing at it are your responsibility to update."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "de9fe3e5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:46:39.349814Z",
"iopub.status.busy": "2026-08-26T18:46:39.349513Z",
"iopub.status.idle": "2026-08-26T18:46:39.354710Z",
"shell.execute_reply": "2026-08-26T18:46:39.353669Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"storage.delete(\"v1.0.0\")\n",
"storage.exists(\"v1.0.0\")"
]
},
{
"cell_type": "markdown",
"id": "ab667b32",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Task | API |\n",
"|---|---|\n",
"| Record audit metadata | `ChangeLogEntry(timestamp, author=email, description)` |\n",
"| Persist a version | `InMemoryVersionStorage().save({\"label\": ..., ...})` |\n",
"| Pin a release name | `save_tag(\"release\", \"v1.0.0\")` / `get_tag(\"release\")` |\n",
"| Integrity check | `compute_checksum(snap)` / `verify_checksum(snap)` |\n",
"| Persistent backend | `SQLiteVersionStorage(path)` — same interface |\n",
"\n",
"See also `semantica/change_management/change_management_usage.md` for the manager classes (`TemporalVersionManager`, `OntologyVersionManager`)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
+314
View File
@@ -0,0 +1,314 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "6eb4dfba",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)\n",
"\n",
"# Seed Data — Practical Guide\n",
"\n",
"The `seed` module bootstraps a knowledge graph from **trusted, pre-known data** (CSV/JSON/database/API sources) before any extraction runs. This gives extraction a foundation to link against instead of starting from an empty graph.\n",
"\n",
"Key pieces:\n",
"\n",
"- **`SeedDataManager`** — registers data sources and builds foundation graphs\n",
"- **`create_foundation_graph()`** — turns registered sources into `entities` + `relationships` + `metadata`\n",
"- **`validate_quality()`** — checks a foundation graph before you commit it\n",
"\n",
"All examples below were executed against `semantica/seed/seed_manager.py`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "32f80cc6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:18.716466Z",
"iopub.status.busy": "2026-08-26T18:51:18.716264Z",
"iopub.status.idle": "2026-08-26T18:51:20.533828Z",
"shell.execute_reply": "2026-08-26T18:51:20.531402Z"
}
},
"outputs": [],
"source": [
"!pip install -q semantica"
]
},
{
"cell_type": "markdown",
"id": "75136e5f",
"metadata": {},
"source": [
"## 1) Prepare a seed CSV and register the source\n",
"\n",
"`register_source(name, format, location, entity_type=...)` records where trusted data lives. `verified=True` (the default) marks the source as pre-validated."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a8089e1f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.538772Z",
"iopub.status.busy": "2026-08-26T18:51:20.538323Z",
"iopub.status.idle": "2026-08-26T18:51:20.675403Z",
"shell.execute_reply": "2026-08-26T18:51:20.674060Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import csv\n",
"import tempfile\n",
"from pathlib import Path\n",
"from semantica.seed import SeedDataManager\n",
"\n",
"# Write the sample CSV into a session-scoped temp directory so we never\n",
"# clobber a companies.csv that might exist in the user's working directory.\n",
"seed_csv = Path(tempfile.mkdtemp(prefix=\"semantica-seed-\")) / \"companies.csv\"\n",
"with open(seed_csv, \"w\", newline=\"\") as f:\n",
" writer = csv.DictWriter(f, fieldnames=[\"id\", \"name\", \"type\", \"industry\"])\n",
" writer.writeheader()\n",
" writer.writerow({\"id\": \"c1\", \"name\": \"Acme\", \"type\": \"Company\", \"industry\": \"robotics\"})\n",
" writer.writerow({\"id\": \"c2\", \"name\": \"Globex\", \"type\": \"Company\", \"industry\": \"energy\"})\n",
"\n",
"manager = SeedDataManager()\n",
"manager.register_source(\"companies\", format=\"csv\", location=str(seed_csv), entity_type=\"Company\")\n"
]
},
{
"cell_type": "markdown",
"id": "e87221ba",
"metadata": {},
"source": [
"## 2) Load records from a registered source\n",
"\n",
"`load_source(name)` reads the source and enriches each record with `entity_type` and `source` provenance keys."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "f932e550",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.679424Z",
"iopub.status.busy": "2026-08-26T18:51:20.679156Z",
"iopub.status.idle": "2026-08-26T18:51:20.690659Z",
"shell.execute_reply": "2026-08-26T18:51:20.688812Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is seeding</td><td>🌱 seed</td><td>SeedDataManager</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"🔄 Semantica is seeding: Loading seed data from CSV: /var/folders/7s/bvvstgs10y963tz6_4bbnklr0000gn/T/semantica-seed-eu9__ep1/companies.csv 🌱 seed SeedDataManager |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"loaded 2 records\n"
]
},
{
"data": {
"text/plain": [
"{'id': 'c1',\n",
" 'name': 'Acme',\n",
" 'type': 'Company',\n",
" 'industry': 'robotics',\n",
" 'entity_type': 'Company',\n",
" 'source': 'companies'}"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"records = manager.load_source(\"companies\")\n",
"print(f\"loaded {len(records)} records\")\n",
"records[0]"
]
},
{
"cell_type": "markdown",
"id": "f2ebce64",
"metadata": {},
"source": [
"## 3) Build the foundation graph\n",
"\n",
"`create_foundation_graph()` converts every registered source into graph-ready entities and relationships. Entities carry `confidence: 1.0` — seed data is trusted by definition."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "09388c31",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.695259Z",
"iopub.status.busy": "2026-08-26T18:51:20.694928Z",
"iopub.status.idle": "2026-08-26T18:51:20.708595Z",
"shell.execute_reply": "2026-08-26T18:51:20.707072Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['entities', 'metadata', 'relationships']"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"foundation = manager.create_foundation_graph()\n",
"sorted(foundation.keys())"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4610a59f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.713136Z",
"iopub.status.busy": "2026-08-26T18:51:20.712795Z",
"iopub.status.idle": "2026-08-26T18:51:20.718637Z",
"shell.execute_reply": "2026-08-26T18:51:20.716835Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 'c1',\n",
" 'text': 'Acme',\n",
" 'type': 'Company',\n",
" 'confidence': 1.0,\n",
" 'metadata': {'industry': 'robotics', 'source': 'companies'}}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"foundation[\"entities\"][0]"
]
},
{
"cell_type": "markdown",
"id": "f3a52dc7",
"metadata": {},
"source": [
"## 4) Validate quality before committing\n",
"\n",
"`validate_quality(foundation_graph)` returns `valid`, `errors`, `warnings`, and `metrics` so you can gate bad seed data before it pollutes the graph."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "4eb7e664",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-26T18:51:20.722674Z",
"iopub.status.busy": "2026-08-26T18:51:20.722118Z",
"iopub.status.idle": "2026-08-26T18:51:20.732003Z",
"shell.execute_reply": "2026-08-26T18:51:20.730170Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"quality = manager.validate_quality(foundation)\n",
"quality[\"valid\"]"
]
},
{
"cell_type": "markdown",
"id": "b534be89",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"| Task | API |\n",
"|---|---|\n",
"| Register a trusted source | `register_source(name, format, location, entity_type=...)` |\n",
"| Load records | `load_source(name)` — adds `entity_type` / `source` keys |\n",
"| Direct file load | `load_from_csv(path)` / `load_from_json(path)` |\n",
"| Build the graph | `create_foundation_graph()` → `entities` / `relationships` / `metadata` |\n",
"| Gate bad data | `validate_quality(graph)` → `valid` / `errors` / `warnings` / `metrics` |\n",
"\n",
"See also `semantica/seed/seed_usage.md` for `load_from_database`, `load_from_api`, and `integrate_with_extracted`."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
+4
View File
@@ -35,6 +35,7 @@ Essential guides to master the Semantica framework.
- **[Vector Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)** — Setting up vector stores for similarity search and retrieval. *Intermediate*
- **[Graph Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Graph_Store.ipynb)** — Persisting knowledge graphs in Neo4j or FalkorDB. Topics: Neo4j, Cypher, Persistence · *Intermediate*
- **[Ontology](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)** — Defining domain schemas and ontologies to structure your data. Topics: OWL, RDF, Schema Design · *Intermediate*
- **[Seed Data](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)** — Bootstrapping a knowledge graph from trusted CSV, JSON, database, and API sources before extraction runs. Topics: SeedDataManager, Foundation Graphs · *Intermediate*
## Advanced Concepts
@@ -50,6 +51,9 @@ Deep dive into advanced features, customization, and complex workflows.
- **[Multi-Source Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)** — Merging data from disparate sources into a unified graph. Topics: Entity Resolution, Merging, Fusion · *Advanced*
- **[Reasoning and Inference](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)** — Using logical reasoning to infer new knowledge from existing facts. Topics: Logic Rules, Inference Engines · *Advanced*
- **[Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)** — Modeling and querying data that changes over time. Topics: Time Series, Temporal Logic, Allen Algebra · *Advanced*
- **[Provenance Tracking](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/22_Provenance_Tracking.ipynb)** — Audit-grade, W3C PROV-O-aligned tracking of where every entity, relationship, and chunk came from. Topics: PROV-O, Lineage, Checksums, Invalidation · *Advanced*
- **[Reasoning Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)** — Deriving new knowledge from existing facts with forward chaining, backward chaining, and Datalog strategies. Topics: Reasoner, Datalog, Explanations · *Advanced*
- **[Change Management](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)** — Versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies. Topics: ChangeLogEntry, Version Storage, Data Integrity · *Advanced*
## How to Run
+4 -4
View File
@@ -222,10 +222,10 @@ builder.register_step_handler("ner_extract", run_ner)
builder.register_step_handler("triplet_extract", run_triplets)
builder.register_step_handler("kg_merge", merge_into_graph)
builder.add_step("ingest", "file_ingest", handler=ingest_stix_bundles, path="./stix_bundles/")
builder.add_step("ner", "ner_extract", handler=run_ner, confidence_threshold=0.75)
builder.add_step("triplets", "triplet_extract", handler=run_triplets, include_temporal=True)
builder.add_step("store", "kg_merge", handler=merge_into_graph, output_path="./cti_output/")
builder.add_step("ingest", "file_ingest", path="./stix_bundles/")
builder.add_step("ner", "ner_extract", confidence_threshold=0.75)
builder.add_step("triplets", "triplet_extract", include_temporal=True)
builder.add_step("store", "kg_merge", output_path="./cti_output/")
# ingest feeds both ner and triplets in parallel
builder.connect_steps("ingest", "ner")
+1 -1
View File
@@ -639,7 +639,7 @@ Every `ProvenanceEntry` maps directly to W3C PROV-O terms. If your compliance te
| — | `previous_version_id` | This entry corrects/replaces a prior version of the *same* fact |
| `prov:wasDerivedFrom` | `derived_from_id` | This entry was derived from a *different* source entity |
| `prov:used` | `used_entities` | Entity IDs consumed to produce this one |
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `datetime.utcnow()` at write time |
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `utc_now_iso()` at write time |
| `prov:qualifiedInvalidation` | `invalidated`, `invalidated_at_time`, `invalidated_by`, `invalidation_reason` | A retraction/correction recorded as a tombstone via `ProvenanceManager.invalidate()`, never a hard delete |
| `prov:startedAtTime` / `prov:endedAtTime` | `activity_started_at_time`, `activity_ended_at_time` | Typed Activity timing — pass an `ActivityRecord` via the `activity=` kwarg to set these together with `activity_id` |
| `prov:qualifiedGeneration`/`Generation`, `qualifiedUsage`/`Usage`, `qualifiedDerivation`/`Derivation` | (derived from the fields above) | Additive qualified forms of `wasGeneratedBy`/`used`/`wasDerivedFrom`, emitted automatically alongside the plain triples |
+13
View File
@@ -150,6 +150,12 @@ HighRiskSupplier(DELTA-3) conf=100% rule=Rule 3
DELTA-3 is flagged even though no document described it that way — the system traced: DELTA-3 supplied GAMMA-7, and GAMMA-7 exploits critical CVEs. For rules that need priority ordering or graded confidence, use the `Rule` dataclass:
If a rule has side-effecting actions, one concrete activation runs those
actions at most once on a Reasoner instance. Re-running `forward_chain()` is
therefore safe: already-attempted actions are not repeated. Use
`reasoner.reset_action_history()` when you intentionally want to replay them;
`reasoner.clear()` and `reasoner.reset()` also clear the history.
```python
# Higher priority rules fire first; confidence propagates into InferenceResult.confidence
reasoner.add_rule(Rule(
@@ -360,6 +366,13 @@ engine.reset()
The rule network is compiled once by `build_network()`. Each subsequent `add_fact()` call propagates incrementally through only the nodes whose conditions it satisfies — not the full rule set — which keeps evaluation cost proportional to the number of new activations rather than the total rule count.
With a Reasoner bound, Rete action side effects are attempted once per rule,
bindings, and matched fact identity. Passing the same match to
`execute_matches()` again still returns the same conclusion, but does not repeat
its actions. Call `engine.reset_action_history()` to replay actions without
clearing working memory. `engine.reset()` and `engine.build_network()` also
clear the action history.
## Step 7 — Temporal interval reasoning
`TemporalReasoningEngine` computes Allen interval relations between time windows, letting you identify whether two events overlap, one contains the other, they meet at a boundary, and so on across your graph:
+1 -1
View File
@@ -250,7 +250,7 @@ entry = ProvenanceEntry(
source_document="report.pdf", # str: default ""
source_location="Page 4", # Optional[str]: default None
source_quote="Relevant text...", # Optional[str]: default None
timestamp="2024-01-01T12:00:00", # str: auto-set to utcnow()
timestamp="2024-01-01T12:00:00+00:00", # str: auto-set to utc_now_iso()
first_seen=None, # Optional[str]: ISO timestamp
last_updated=None, # Optional[str]: ISO timestamp
confidence=0.9, # float: default 1.0
+19 -2
View File
@@ -127,9 +127,19 @@ conclusions = reasoner.infer_facts(
| `forward_chain()` | `List[InferenceResult]` | Derive all possible conclusions iteratively until fixpoint |
| `backward_chain(goal, max_depth)` | `InferenceResult \| None` | Prove a specific goal string, returns `None` if unprovable |
| `infer_facts(facts, rules)` | `List[str]` | Load facts and rules then run `forward_chain()`, returns conclusion strings |
| `clear()` | `None` | Clear all facts and rules |
| `reset_action_history()` | `None` | Allow actions for previously fired activations to run again |
| `clear()` | `None` | Clear all facts, rules, and action activation history |
| `reset()` | `None` | Alias for `clear()` |
Rules with actions use at-most-once attempt semantics per concrete activation
(rule ID, bindings, and matched facts). Calling `forward_chain()` again on the
same instance does not repeat side effects for an activation that was already
attempted, even when an action raised an exception. Call
`reset_action_history()` to deliberately retry without clearing facts or rules;
`clear()` and `reset()` also clear this history. Replacing a rule's actions in
place does not invalidate an existing activation; reset the history explicitly
when the replacement should be replayed.
### Rule and Fact dataclass fields
```python
@@ -230,9 +240,16 @@ engine.reset()
| `add_fact(fact)` | `None` | Add a `Fact` to working memory and propagate through the network |
| `match_patterns(facts)` | `List[Match]` | Match all patterns; optionally add facts before matching |
| `execute_matches(matches)` | `List[Any]` | Execute matched rules and return their conclusion values |
| `reset()` | `None` | Clear facts and all node activation state |
| `reset_action_history()` | `None` | Allow actions for previously executed activations to run again |
| `reset()` | `None` | Clear facts, node activation state, and action activation history |
| `get_network_stats()` | `dict` | Return counts of alpha, beta, terminal nodes and facts |
When a Reasoner is bound, `execute_matches()` deduplicates action side effects
by rule ID, bindings, and matched fact identity. Re-executing a match still
returns its conclusion for compatibility, but its actions are skipped after the
first attempt. `reset_action_history()`, `reset()`, and `build_network()` allow
those actions to run again.
## SPARQLReasoner
+4 -4
View File
@@ -109,7 +109,7 @@ class ConflictsConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -136,12 +136,12 @@ class ConflictsConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+4 -4
View File
@@ -110,7 +110,7 @@ class DeduplicationConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -137,12 +137,12 @@ class DeduplicationConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+6 -6
View File
@@ -106,7 +106,7 @@ class EmbeddingsConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -123,8 +123,8 @@ class EmbeddingsConfig:
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
# Try to convert to appropriate type
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -149,12 +149,12 @@ class EmbeddingsConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+6 -6
View File
@@ -105,7 +105,7 @@ class ExportConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -122,8 +122,8 @@ class ExportConfig:
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
# Try to convert to appropriate type
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -148,12 +148,12 @@ class ExportConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+48 -5
View File
@@ -642,6 +642,37 @@ class RDFSerializer:
self.logger.debug("RDF serializer initialized")
@staticmethod
def _local_name_from_id(identifier: str) -> str:
"""Derive a human-readable local name from an entity identifier.
Handles HTTP(S)/IRI identifiers (path segments and fragments, tolerating
trailing slashes) as well as compact/CURIE and URN-style identifiers.
"""
raw = str(identifier).strip()
if not raw:
return ""
# Prefer a fragment if present (e.g. http://ex.org/onto#acme -> acme).
if "#" in raw:
candidate = raw.rsplit("#", 1)[-1]
if candidate:
return candidate
# For IRIs/paths, take the last non-empty path segment.
if "/" in raw:
segment = raw.rstrip("/").rsplit("/", 1)[-1]
if segment:
return segment
# Fall back to the tail of a CURIE/URN (e.g. urn:x:acme, semantica:acme).
if ":" in raw:
candidate = raw.rsplit(":", 1)[-1]
if candidate:
return candidate
return raw
def convert_kg_to_rdf(self, knowledge_graph: Dict[str, Any]) -> Dict[str, Any]:
"""
Convert knowledge graph to RDF data structure.
@@ -681,8 +712,11 @@ class RDFSerializer:
if "name" in norm_entity:
norm_entity["label"] = norm_entity["name"]
elif "id" in norm_entity:
# Use ID part as label if no name/text
norm_entity["label"] = str(norm_entity["id"]).split(":")[-1]
# Derive a readable label from the identifier's local name
# (fragment/last path segment/CURIE tail). See #1097.
local_name = self._local_name_from_id(norm_entity["id"])
if local_name:
norm_entity["label"] = local_name
rdf_data["entities"].append(norm_entity)
@@ -1082,8 +1116,9 @@ class RDFSerializer:
# RDF/XML syntax: rdf:Description with rdf:about
# Attribute values are delimited by quotes, and both of these
# are caller input. Element text is left alone deliberately: that
# is #1098, and it is being fixed on its own path.
# are caller input. Element text (semantica:text) is caller input
# too, so it needs the same escaping to avoid injecting markup
# or breaking out of the element (#1097 / #1113).
entity_iri = xml_escape(
self._as_turtle_iri(entity_id, namespaces), quote=True
)
@@ -1092,7 +1127,9 @@ class RDFSerializer:
)
lines.append(f' <rdf:Description rdf:about="{entity_iri}">')
lines.append(f' <rdf:type rdf:resource="{entity_type_iri}"/>')
lines.append(f" <semantica:text>{text}</semantica:text>")
lines.append(
f" <semantica:text>{xml_escape(text)}</semantica:text>"
)
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
@@ -1714,6 +1751,12 @@ class RDFExporter:
self.logger.debug(f"Exporting to RDF format: {format}")
# Normalize the graph before serialization so every format benefits
# from field normalization (e.g. mapping 'name' -> 'label'/'text').
# Without this, graphs produced by GraphBuilder (which emit 'name')
# export with an empty semantica:text on all RDF paths. See #1097.
data = self.serializer.convert_kg_to_rdf(data)
self.progress_tracker.update_tracking(
tracking_id, message="Validating RDF data..."
)
+55 -3
View File
@@ -556,6 +556,13 @@ class GraphStore:
)
self.config = config
# Application-id -> backend-internal-id map for nodes added through
# the compatibility layer (#1136). add_nodes()/create_node() record
# the internal ids the backend returns; create_relationship()
# resolves known application ids through it so string ids stop
# mismatching backends that match on internal ids (Neo4j id(n)).
self._app_node_id_map: Dict[Any, Any] = {}
# Initialize store backend
self._store_backend = None
self._manager = None
@@ -630,7 +637,9 @@ class GraphStore:
**options,
) -> Dict[str, Any]:
"""Create a node."""
return self._manager.nodes.create(labels, properties, **options)
created = self._manager.nodes.create(labels, properties, **options)
self._record_app_node_id(created)
return created
def create_nodes(
self,
@@ -688,9 +697,25 @@ class GraphStore:
properties: Optional[Dict[str, Any]] = None,
**options,
) -> Dict[str, Any]:
"""Create a relationship."""
"""Create a relationship.
Node ids added through the compatibility layer are application-level
strings, while backends such as Neo4j match on internal integer ids
(#1136). Known application ids are resolved to the internal ids the
backend returned at creation time; unknown ids pass through
unchanged, so direct internal-id callers keep working. Only string
application ids participate: internal ids are commonly integers, so
recording or resolving an integer key could remap a caller-supplied
internal id to a different node.
"""
return self._manager.relationships.create(
start_node_id, end_node_id, rel_type, properties, **options
self._app_node_id_map.get(start_node_id, start_node_id)
if isinstance(start_node_id, str) else start_node_id,
self._app_node_id_map.get(end_node_id, end_node_id)
if isinstance(end_node_id, str) else end_node_id,
rel_type,
properties,
**options,
)
def get_relationships(
@@ -794,6 +819,24 @@ class GraphStore:
"""Create an index."""
return self._manager.create_index(label, property_name, index_type, **options)
def _record_app_node_id(self, created: Optional[Dict[str, Any]]) -> None:
"""Record the application-id -> internal-id pair of a created node.
Backends return their own internal id alongside the stored properties;
when the caller supplied an application id it is preserved in
``properties["id"]`` by the compatibility layer, which makes the pair
recoverable (#1136). Only STRING application ids are recorded:
internal ids are commonly integers, and an integer application id
would collide with (and silently remap) a caller-supplied internal id
of the same value in ``create_relationship``.
"""
if not isinstance(created, dict):
return
app_id = (created.get("properties") or {}).get("id")
internal_id = created.get("id")
if isinstance(app_id, str) and internal_id is not None:
self._app_node_id_map[app_id] = internal_id
# Compatibility with AgentMemory / ContextGraph interface
def add_nodes(self, nodes: List[Dict[str, Any]], **options) -> int:
"""
@@ -853,12 +896,21 @@ class GraphStore:
# and properties.
result = self.create_nodes(graph_nodes, **options)
# Keep the application-id -> internal-id pairs instead of discarding
# them, so add_edges()/create_relationship() can resolve the string
# ids callers actually use (#1136).
for created in result:
self._record_app_node_id(created)
return len(result)
def add_edges(self, edges: List[Dict[str, Any]], **options) -> int:
"""
Add edges (Compatibility method).
``source_id``/``target_id`` are application-level string ids; they are
resolved to the backend's internal ids via the map ``add_nodes``
populated when the nodes were created (#1136).
Args:
edges: List of edge dictionaries
**options: Additional options
+8 -8
View File
@@ -110,7 +110,7 @@ class IngestConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -127,8 +127,8 @@ class IngestConfig:
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
# Try to convert to appropriate type
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -143,8 +143,8 @@ class IngestConfig:
if key.startswith(mcp_prefix) and key not in env_mappings:
config_key = key[len(mcp_prefix) :].lower()
# Try to convert to appropriate type
if value.lower() in ("true", "false"):
self._configs[f"mcp_{config_key}"] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[f"mcp_{config_key}"] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[f"mcp_{config_key}"] = int(value)
else:
@@ -169,12 +169,12 @@ class IngestConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+6 -6
View File
@@ -106,7 +106,7 @@ class KGConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -123,8 +123,8 @@ class KGConfig:
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
# Try to convert to appropriate type
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -149,12 +149,12 @@ class KGConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+6 -5
View File
@@ -54,10 +54,11 @@ Version: 1.0.0
"""
from typing import Any, Dict, List, Optional
from datetime import datetime
import uuid
import time
from ..utils.helpers import utc_now_iso
class GraphBuilderWithProvenance:
"""
@@ -103,7 +104,7 @@ class GraphBuilderWithProvenance:
def build(self, sources, **kwargs):
"""Build graph with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
activity_started_at_time = utc_now_iso()
# Track the build operation (recorded before the build runs, so it
# has no end time yet — this is the "in progress" marker).
if self.provenance and self._prov_manager:
@@ -124,7 +125,7 @@ class GraphBuilderWithProvenance:
)
result = self._builder.build(sources, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
activity_ended_at_time = utc_now_iso()
# Track individual entities and relationships if available
if self.provenance and self._prov_manager and hasattr(result, 'get'):
@@ -180,7 +181,7 @@ class GraphBuilderWithProvenance:
def build_single_source(self, kg_data, **kwargs):
"""Build graph from single source with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
activity_started_at_time = utc_now_iso()
# Track the build operation (recorded before the build runs, so it
# has no end time yet — this is the "in progress" marker).
if self.provenance and self._prov_manager:
@@ -200,7 +201,7 @@ class GraphBuilderWithProvenance:
)
result = self._builder.build_single_source(kg_data, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
activity_ended_at_time = utc_now_iso()
# Track entities and relationships if available
if self.provenance and self._prov_manager and isinstance(result, dict):
+6 -6
View File
@@ -98,7 +98,7 @@ class NormalizeConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -113,8 +113,8 @@ class NormalizeConfig:
for key, value in os.environ.items():
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -136,12 +136,12 @@ class NormalizeConfig:
value = os.getenv(env_key)
if value:
try:
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+6 -6
View File
@@ -99,7 +99,7 @@ class OntologyConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -114,8 +114,8 @@ class OntologyConfig:
for key, value in os.environ.items():
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -137,12 +137,12 @@ class OntologyConfig:
value = os.getenv(env_key)
if value:
try:
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+43 -12
View File
@@ -197,23 +197,22 @@ class PropertyGenerator:
self, entities: List[Dict[str, Any]], classes: List[Dict[str, Any]], **options
) -> List[Dict[str, Any]]:
"""Infer data properties from entity attributes."""
# Group entities by type
entity_types = defaultdict(list)
# Group entities by their inferred class so normalized class names remain
# aligned with the class definitions emitted by ClassInferrer.
class_entities = defaultdict(list)
class_lookup = self._build_class_type_lookup(classes)
for entity in entities:
entity_type = entity.get("type") or entity.get("entity_type", "Entity")
entity_types[entity_type].append(entity)
class_def = self._find_class_for_entity_type(entity_type, class_lookup)
if not class_def:
continue
class_name = class_def["name"]
class_entities[class_name].append(entity)
# Extract data properties for each class
properties = []
for entity_type, type_entities in entity_types.items():
# Find corresponding class
class_def = next(
(cls for cls in classes if cls["name"] == entity_type), None
)
if not class_def:
continue
for class_name, type_entities in class_entities.items():
# Extract data properties
data_props = self._extract_data_properties(type_entities)
@@ -233,7 +232,7 @@ class PropertyGenerator:
else None,
"label": normalized_name,
"comment": f"Data property for {prop_name}",
"domain": [entity_type],
"domain": [class_name],
"range": prop_type,
"metadata": {"inferred_from": prop_name},
}
@@ -242,6 +241,38 @@ class PropertyGenerator:
return properties
def _build_class_type_lookup(
self, classes: List[Dict[str, Any]]
) -> Dict[str, Dict[str, Any]]:
"""Build a lookup for raw, normalized, and recorded source type names."""
lookup: Dict[str, Dict[str, Any]] = {}
for class_def in classes:
class_name = class_def.get("name")
if class_name:
lookup.setdefault(str(class_name), class_def)
lookup.setdefault(
self.naming_conventions.normalize_class_name(str(class_name)),
class_def,
)
inferred_from = class_def.get("metadata", {}).get("inferred_from")
if inferred_from is not None:
lookup.setdefault(str(inferred_from), class_def)
lookup.setdefault(
self.naming_conventions.normalize_class_name(str(inferred_from)),
class_def,
)
return lookup
def _find_class_for_entity_type(
self, entity_type: Any, class_lookup: Dict[str, Dict[str, Any]]
) -> Optional[Dict[str, Any]]:
"""Find a class using the precomputed type lookup."""
raw_type = str(entity_type)
normalized_type = self.naming_conventions.normalize_class_name(raw_type)
return class_lookup.get(raw_type) or class_lookup.get(normalized_type)
def _extract_data_properties(
self, entities: List[Dict[str, Any]]
) -> Dict[str, str]:
+6 -6
View File
@@ -98,7 +98,7 @@ class ParseConfig:
if value:
try:
if type_func == bool:
self._configs[config_key] = value.lower() in (
self._configs[config_key] = value.strip().lower() in (
"true",
"1",
"yes",
@@ -113,8 +113,8 @@ class ParseConfig:
for key, value in os.environ.items():
if key.startswith(env_prefix) and key not in env_mappings:
config_key = key[len(env_prefix) :].lower()
if value.lower() in ("true", "false"):
self._configs[config_key] = value.lower() == "true"
if value.strip().lower() in ("true", "false"):
self._configs[config_key] = value.strip().lower() == "true"
elif value.isdigit():
self._configs[config_key] = int(value)
else:
@@ -136,12 +136,12 @@ class ParseConfig:
value = os.getenv(env_key)
if value:
try:
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+1
View File
@@ -33,6 +33,7 @@ License: MIT
"""
import email
import email.message
from dataclasses import dataclass, field
from email import message_from_bytes, message_from_string
from email.header import decode_header
+26 -2
View File
@@ -143,15 +143,39 @@ class PDFParser:
)
pages.append(page_data)
full_text = "\n\n".join(page.text for page in pages)
# Scanned/image-only PDFs have no text layer; warn so the
# failure surfaces at parse time instead of downstream.
# Check any() over page texts to avoid a temporary stripped
# copy of the full concatenation for large documents.
no_text = (
options.get("extract_text", True)
and pages
and not any(page.text.strip() for page in pages)
)
if no_text:
self.logger.warning(
f"PDF {file_path.name}: parsed {len(pages)} page(s) but "
f"extracted no text. This is likely a scanned "
f"(image-only) PDF. Retry with "
f"parse_pdf(..., method='docling', enable_ocr=True) "
f"for OCR-based extraction."
)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Parsed {len(pages)} pages",
message=(
f"Parsed {len(pages)} page(s) (no text layer detected)"
if no_text
else f"Parsed {len(pages)} page(s)"
),
)
return {
"metadata": metadata.__dict__,
"pages": [page.__dict__ for page in pages],
"full_text": "\n\n".join(page.text for page in pages),
"full_text": full_text,
"total_pages": len(pdf.pages),
}
+1 -1
View File
@@ -379,7 +379,7 @@ class ExecutionEngine:
data = delta_result
if step.handler:
if step.handler is not None:
return step.handler(data, **step.config, **options)
else:
return data
+6 -2
View File
@@ -131,13 +131,17 @@ class PipelineBuilder:
delta_mode = config.pop("delta_mode", False)
base_version_id = config.pop("base_version_id", None)
target_version_id = config.pop("target_version_id", None)
dependencies = config.pop("dependencies", [])
handler = config.pop("handler", None)
if handler is None:
handler = self.step_registry.get(step_type)
step = PipelineStep(
name=step_name,
step_type=step_type,
config=config,
dependencies=config.get("dependencies", []),
handler=config.get("handler"),
dependencies=dependencies,
handler=handler,
delta_mode = delta_mode,
base_version_id=base_version_id,
target_version_id=target_version_id,
+13
View File
@@ -8,6 +8,13 @@ and native Datalog evaluation.
"""
from .reasoner import Reasoner, InferenceResult, Rule, Fact, RuleType
from .reasoner import (
Action,
AssertAction,
RetractAction,
CallAction,
EmitEventAction,
)
from .graph_reasoner import GraphReasoner
from .explanation_generator import (
Explanation,
@@ -37,6 +44,12 @@ __all__ = [
"Rule",
"Fact",
"RuleType",
# Rule-driven actions
"Action",
"AssertAction",
"RetractAction",
"CallAction",
"EmitEventAction",
# Rete engine
"ReteEngine",
"ReteNode",
+424 -19
View File
@@ -7,13 +7,16 @@ supported by the Semantica framework. It serves as a facade for different reason
import re
import uuid
from collections.abc import Mapping, Sequence, Set as AbstractSet
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Dict, List, Optional, Set, Tuple, Union, Callable
from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Union
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
class RuleType(Enum):
"""Rule types."""
IMPLICATION = "implication"
@@ -21,6 +24,306 @@ class RuleType(Enum):
CONSTRAINT = "constraint"
TRANSFORMATION = "transformation"
def _substitute_variables(template: str, bindings: Dict[str, str]) -> str:
"""Substitute ``?var`` placeholders with their bound values, token-aware.
A naive ``str.replace(f"?{var}", value)`` corrupts placeholders that share
a prefix -- e.g. binding ``?x`` would also rewrite the ``?x`` inside ``?xy``.
We replace every ``?word`` token in a single regex pass so that only whole
variable names are matched (``\\w+`` never partially matches a longer name),
leaving unbound placeholders untouched.
"""
if not bindings:
return template
def _replace(match: "re.Match") -> str:
var_name = match.group(1)
# Preserve unbound placeholders verbatim.
return str(bindings[var_name]) if var_name in bindings else match.group(0)
return re.sub(r"\?(\w+)", _replace, template)
def _canonicalize_activation_value(
value: Any, active_containers: Optional[Dict[int, int]] = None
) -> Tuple[Any, ...]:
"""Convert nested activation data into a deterministic, hashable value."""
if active_containers is None:
active_containers = {}
value_type = (type(value).__module__, type(value).__qualname__)
is_mapping = isinstance(value, Mapping)
is_sequence = isinstance(value, Sequence) and not isinstance(
value, (str, bytes, bytearray)
)
is_set = isinstance(value, AbstractSet) and not isinstance(
value, (str, bytes, bytearray)
)
if not (is_mapping or is_sequence or is_set):
return ("scalar", value_type, repr(value))
object_id = id(value)
if object_id in active_containers:
return ("reference", active_containers[object_id])
active_containers[object_id] = len(active_containers)
try:
if is_mapping:
keyed_items = [
(
_canonicalize_activation_value(key, active_containers),
item,
)
for key, item in value.items()
]
keyed_items.sort(key=lambda entry: repr(entry[0]))
entries = tuple(
(
key,
_canonicalize_activation_value(item, active_containers),
)
for key, item in keyed_items
)
return ("mapping", value_type, entries)
if is_sequence:
return (
"sequence",
value_type,
tuple(
_canonicalize_activation_value(item, active_containers)
for item in value
),
)
items = tuple(
sorted(
(
_canonicalize_activation_value(item, active_containers)
for item in value
),
key=repr,
)
)
return ("set", value_type, items)
finally:
del active_containers[object_id]
def _make_activation_key(
rule_id: str, bindings: Dict[str, Any], fact_tokens: List[Any]
) -> Tuple[Any, ...]:
"""Return a stable identity for one concrete rule activation."""
canonical_bindings = tuple(
sorted(
(str(name), _canonicalize_activation_value(value))
for name, value in bindings.items()
)
)
return (
rule_id,
canonical_bindings,
tuple(
sorted(
(_canonicalize_activation_value(token) for token in fact_tokens),
key=repr,
)
),
)
def _parse_fact(fact: str) -> Optional[Tuple[str, List[str]]]:
"""Parse a ``Predicate(arg1, arg2, ...)`` fact string.
Returns ``(predicate, [args])`` or ``None`` when the fact is not in the
canonical predicate form (e.g. a bare atom). Whitespace around args is
stripped and empty arg lists are supported (``Foo()`` -> ``("Foo", [])``).
"""
match = re.match(r"^\s*([^()\s]+)\s*\((.*)\)\s*$", fact)
if not match:
return None
predicate = match.group(1)
inner = match.group(2).strip()
if not inner:
return predicate, []
args = [arg.strip() for arg in inner.split(",")]
return predicate, args
def _write_fact_to_graph(graph: Any, fact: str, *, retract: bool = False) -> None:
"""Persist (or remove) a fact against a knowledge-graph-like target.
Write-back follows an explicit, ordered protocol so that
``AssertAction(write_back=True)`` never silently no-ops:
1. If the target exposes an explicit fact API (``add_fact`` / ``assert_fact``
for asserts, ``remove_fact`` / ``retract_fact`` / ``discard_fact`` for
retracts), that is used verbatim.
2. Otherwise, if the target looks like the canonical
:class:`~semantica.kg.knowledge_graph.KnowledgeGraph` (has ``entities``
and ``relationships`` lists), the fact is translated into a node
(single-arg predicate) or relationship (two-arg predicate) and
added/removed accordingly.
3. Any other target, or a fact that cannot be translated, raises
:class:`ValueError` so the failure surfaces instead of being swallowed.
"""
if retract:
for method_name in ("retract_fact", "remove_fact", "discard_fact"):
method = getattr(graph, method_name, None)
if callable(method):
method(fact)
return
else:
for method_name in ("add_fact", "assert_fact"):
method = getattr(graph, method_name, None)
if callable(method):
method(fact)
return
entities = getattr(graph, "entities", None)
relationships = getattr(graph, "relationships", None)
if isinstance(entities, list) and isinstance(relationships, list):
parsed = _parse_fact(fact)
if parsed is None:
raise ValueError(
f"Cannot translate fact {fact!r} into graph node/relationship: "
"expected canonical Predicate(args) form."
)
predicate, args = parsed
if len(args) == 1:
node = {"id": args[0], "type": predicate}
if retract:
_remove_matching(
entities,
lambda e: e.get("id") == args[0] and e.get("type") == predicate,
)
elif node not in entities:
entities.append(node)
return
if len(args) == 2:
rel = {"source": args[0], "target": args[1], "type": predicate}
if retract:
_remove_matching(
relationships,
lambda r: r.get("source") == args[0]
and r.get("target") == args[1]
and r.get("type") == predicate,
)
elif rel not in relationships:
relationships.append(rel)
return
raise ValueError(
f"Cannot write fact {fact!r} to graph: only unary (node) and binary "
"(relationship) predicates are supported by the default adapter."
)
raise ValueError(
f"knowledge_graph target {type(graph).__name__!r} does not expose a "
"supported write-back API (add_fact/assert_fact or entities/relationships)."
)
def _remove_matching(items: List[Dict[str, Any]], predicate: Callable[[Dict[str, Any]], bool]) -> None:
"""Remove in place every dict in ``items`` for which ``predicate`` is True."""
items[:] = [item for item in items if not predicate(item)]
class Action:
"""Base class for an action fired when a rule matches.
Actions turn the reasoner from a pure inference engine into a
production-rule system: when a rule's conditions match, its actions run
with the match's variable bindings, allowing side effects (asserting or
retracting facts, calling external tools, emitting events) rather than
only deriving a new fact.
Subclasses implement :meth:`execute`, which receives the substituted
``bindings`` and the owning ``reasoner`` and returns an optional
description of what happened (used for provenance / explanation).
"""
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
raise NotImplementedError
@staticmethod
def _substitute(template: str, bindings: Dict[str, str]) -> str:
return _substitute_variables(template, bindings)
@dataclass
class AssertAction(Action):
"""Assert a new fact when the rule fires.
``fact`` may contain ``?var`` placeholders that are substituted with the
match bindings. If ``write_back`` is set and the reasoner exposes a
knowledge graph, the fact is also written there.
"""
fact: str
write_back: bool = False
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
concrete = self._substitute(self.fact, bindings)
reasoner.facts.add(concrete)
if self.write_back and getattr(reasoner, "knowledge_graph", None) is not None:
_write_fact_to_graph(reasoner.knowledge_graph, concrete)
return f"assert {concrete}"
@dataclass
class RetractAction(Action):
"""Retract a fact when the rule fires (basic truth maintenance).
If ``write_back`` is set and the reasoner exposes a knowledge graph, the
fact is also removed there using the graph's delete semantics (mirroring
:class:`AssertAction`'s write-back).
"""
fact: str
write_back: bool = False
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
concrete = self._substitute(self.fact, bindings)
reasoner.facts.discard(concrete)
if self.write_back and getattr(reasoner, "knowledge_graph", None) is not None:
_write_fact_to_graph(reasoner.knowledge_graph, concrete, retract=True)
return f"retract {concrete}"
@dataclass
class CallAction(Action):
"""Call an external function/tool when the rule fires.
Wraps an arbitrary callable, which is invoked as ``func(bindings,
reasoner)``. This is the structured replacement for the previously
unused ``Rule.handler`` callback.
"""
func: Callable
name: str = "call"
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
self.func(bindings, reasoner)
return f"call {self.name}"
@dataclass
class EmitEventAction(Action):
"""Emit an event to the reasoner's registered event sink when fired.
The event name may contain ``?var`` placeholders. Events are delivered to
any callable registered via :meth:`Reasoner.on_event`.
"""
event: str
payload: Dict[str, Any] = field(default_factory=dict)
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
concrete = self._substitute(self.event, bindings)
sink = getattr(reasoner, "_event_sink", None)
if callable(sink):
sink(concrete, {**self.payload, "bindings": dict(bindings)})
return f"emit {concrete}"
@dataclass
class Rule:
"""Simplified rule definition."""
@@ -32,6 +335,7 @@ class Rule:
confidence: float = 1.0
priority: int = 0
handler: Optional[Callable] = None
actions: List[Action] = field(default_factory=list)
metadata: Dict[str, Any] = field(default_factory=dict)
@dataclass
@@ -79,7 +383,70 @@ class Reasoner:
self.rules: List[Rule] = []
self.facts: Set[str] = set()
self.rule_counter = 0
self._fired_activations: Set[Tuple[Any, ...]] = set()
# Optional knowledge graph for AssertAction(write_back=True) targets.
self.knowledge_graph = kwargs.get("knowledge_graph")
# Optional event sink for EmitEventAction; register via on_event().
self._event_sink: Optional[Callable] = None
# When True, action-induced fact changes are recorded for provenance
# via _record_action() -> self.action_log.
self.provenance: bool = bool(kwargs.get("provenance", False))
self.action_log: List[Dict[str, Any]] = []
def on_event(self, sink: Callable) -> None:
"""Register a callable ``sink(event_name, payload)`` for EmitEventAction."""
self._event_sink = sink
def _record_action(
self, rule: "Rule", action: "Action", description: Optional[str], bindings: Dict[str, str]
) -> None:
"""Record a fired action for provenance / explanation when enabled.
Each entry is a structured dict carrying an ISO-8601 ``timestamp`` and a
parsed ``operation``/``fact`` split (when the description follows the
``"<op> <fact>"`` convention used by the built-in actions) so that
downstream consumers such as :class:`ExplanationGenerator` and the
provenance layer can reason about *what changed* without re-parsing the
free-text description.
"""
if not self.provenance or description is None:
return
operation, _, subject = description.partition(" ")
self.action_log.append(
{
"action_id": uuid.uuid4().hex[:8],
"rule_id": rule.rule_id,
"action": type(action).__name__,
"operation": operation or None,
"fact": subject or None,
"description": description,
"bindings": dict(bindings),
"confidence": rule.confidence,
"timestamp": datetime.now(timezone.utc).isoformat(),
}
)
def _fire_actions(self, rule: "Rule", bindings: Dict[str, str]) -> None:
"""Run a fired rule's actions (and legacy handler) with match bindings.
Backward compatible: a rule with an old-style ``handler`` but no
``actions`` still has its handler invoked, so pre-existing rules keep
working while new rules use the structured Action layer.
"""
actions = list(rule.actions)
if rule.handler is not None:
actions.append(CallAction(rule.handler, name=f"handler:{rule.rule_id}"))
for action in actions:
try:
description = action.execute(bindings, self)
self._record_action(rule, action, description, bindings)
except Exception as exc: # noqa: BLE001
self.logger.error(
f"Error executing action {type(action).__name__} "
f"for rule '{rule.rule_id}': {exc}"
)
def add_rule(self, rule_def: Union[str, Rule]) -> Rule:
"""Add a rule to the reasoner.
@@ -161,6 +528,20 @@ class Reasoner:
Returns:
List of inferred facts (conclusions)
"""
return [result.conclusion for result in self.infer_with_results(facts, rules)]
def infer_with_results(
self,
facts: Union[List[Any], Dict[str, Any]],
rules: Optional[List[Union[str, Rule]]] = None,
) -> List[InferenceResult]:
"""Infer new facts and return the full :class:`InferenceResult` objects.
Unlike :meth:`infer_facts` (which returns only conclusion strings for
backward compatibility), this preserves each result's ``rule_used``,
``premises`` and ``confidence`` so callers such as the provenance
wrapper can record real confidence values instead of ``None``.
"""
tracking_id = self.progress_tracker.start_tracking(
module="reasoning",
submodule="Reasoner",
@@ -180,17 +561,14 @@ class Reasoner:
# Perform inference
results = self.forward_chain()
# Extract conclusions from results
inferred_facts = [result.conclusion for result in results]
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Inferred {len(inferred_facts)} new facts"
message=f"Inferred {len(results)} new facts"
)
return inferred_facts
return results
except Exception as e:
self.progress_tracker.stop_tracking(
@@ -213,7 +591,15 @@ class Reasoner:
new_facts_added = True
max_iterations = self.config.get("max_iterations", 50)
iteration = 0
# Activations (rule + concrete bindings + matched facts) whose actions
# have already fired. Actions are side-effecting and must
# fire exactly once per distinct match, decoupled from whether the
# rule's *conclusion* is new. This fixes two failure modes:
# * A valid binding whose conclusion is already known (or duplicated
# within a pass) previously never fired its actions.
# * A RetractAction that removes a premise of its own rule previously
# re-fired every pass, iterating to max_iterations. Recording the
# activation means it fires once and stops driving iterations.
while new_facts_added and iteration < max_iterations:
new_facts_added = False
iteration += 1
@@ -236,7 +622,21 @@ class Reasoner:
pass_results: Dict[str, InferenceResult] = {}
for rule in self.rules:
for conclusion, matched_facts in self._match_rule(rule):
for conclusion, matched_facts, bindings in self._match_rule(rule):
# Fire this activation's actions exactly once, independent
# of the conclusion-dedup below. Keyed by rule id + the
# concrete bindings so distinct matches each fire, but a
# repeated match (same bindings across passes) does not.
if rule.actions or rule.handler is not None:
activation_key = _make_activation_key(
rule.rule_id,
bindings,
matched_facts,
)
if activation_key not in self._fired_activations:
self._fired_activations.add(activation_key)
self._fire_actions(rule, bindings)
if conclusion in pass_results:
# Another derivation of a conclusion already produced
# earlier in this same pass: merge premises, dedup.
@@ -372,14 +772,17 @@ class Reasoner:
conclusion=conclusion_str.strip()
)
def _match_rule(self, rule: Rule) -> List[Tuple[str, List[str]]]:
def _match_rule(self, rule: Rule) -> List[Tuple[str, List[str], Dict[str, str]]]:
"""
Match rule conditions against facts and return instantiated conclusions
paired with the facts that satisfied each condition.
paired with the facts that satisfied each condition and the variable
bindings that produced them.
Returns:
List of (conclusion, matched_facts) tuples, where matched_facts is
the ordered list of facts bound to this rule's conditions.
List of (conclusion, matched_facts, bindings) tuples, where
matched_facts is the ordered list of facts bound to this rule's
conditions and bindings maps variable name -> matched value (used
to fire the rule's actions).
"""
if not rule.conditions:
return []
@@ -410,7 +813,7 @@ class Reasoner:
results = []
for bindings, matched_facts in bindings_list:
instantiated_conclusion = self._substitute(rule.conclusion, bindings)
results.append((instantiated_conclusion, matched_facts))
results.append((instantiated_conclusion, matched_facts, bindings))
return results
@@ -456,16 +859,18 @@ class Reasoner:
def _substitute(self, pattern: str, bindings: Dict[str, str]) -> str:
"""Substitute variables in a pattern with bound values."""
result = pattern
for var, value in bindings.items():
result = result.replace(f"?{var}", value)
return result
return _substitute_variables(pattern, bindings)
def reset_action_history(self) -> None:
"""Allow previously fired rule activations to execute their actions again."""
self._fired_activations.clear()
def clear(self) -> None:
"""Clear facts and rules."""
"""Clear facts, rules, and action activation history."""
self.facts.clear()
self.rules.clear()
self.rule_counter = 0
self.reset_action_history()
def reset(self) -> None:
"""Alias for clear()."""
+24 -9
View File
@@ -13,9 +13,9 @@ Author: Semantica Contributors
License: MIT
"""
from typing import Any, Optional
from datetime import datetime
import uuid
from datetime import datetime
from typing import Any, Optional
class ReasoningEngineWithProvenance:
@@ -28,10 +28,10 @@ class ReasoningEngineWithProvenance:
is_automated: bool = True,
**config,
):
from .reasoning_engine import ReasoningEngine
from .reasoner import Reasoner
self.provenance = provenance
self._engine = ReasoningEngine(**config)
self._engine = Reasoner(provenance=provenance, **config)
self._prov_manager = None
self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated
@@ -43,12 +43,27 @@ class ReasoningEngineWithProvenance:
except ImportError:
self.provenance = False
def infer(self, premises: Any, source: str = None, **kwargs):
"""Perform inference with provenance tracking."""
def infer(self, premises: Any, source: str = None, rules: Any = None):
"""Perform inference with provenance tracking.
Only the reasoner's real parameters (``premises`` and ``rules``) are
forwarded to the underlying engine; arbitrary keyword arguments are no
longer passed through (they previously reached
``Reasoner.infer_facts`` -- which accepts only ``facts``/``rules`` --
and raised ``TypeError``).
"""
activity_started_at_time = datetime.utcnow().isoformat()
result = self._engine.infer(premises, **kwargs)
results = self._engine.infer_with_results(premises, rules)
activity_ended_at_time = datetime.utcnow().isoformat()
# Aggregate confidence across the derived results (min = weakest link);
# None only when nothing was inferred.
confidence = (
min(r.confidence for r in results) if results else None
)
# Preserve the historical return shape: a list of conclusion strings.
inferred = [r.conclusion for r in results]
if self.provenance and self._prov_manager:
self._prov_manager.track_entity(
entity_id=f"inference_{uuid.uuid4().hex[:8]}",
@@ -61,11 +76,11 @@ class ReasoningEngineWithProvenance:
activity_ended_at_time=activity_ended_at_time,
metadata={
"premises_count": len(premises) if hasattr(premises, '__len__') else 1,
"confidence": getattr(result, 'confidence', None)
"confidence": confidence,
}
)
return result
return inferred
def __getattr__(self, name):
return getattr(self._engine, name)
+117 -14
View File
@@ -33,14 +33,75 @@ Author: Semantica Contributors
License: MIT
"""
from collections import defaultdict
import re
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Set, Tuple
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .reasoner import Fact, Rule
from .reasoner import Fact, Rule, _make_activation_key
def _extract_bindings(condition: Any, fact: Fact) -> Dict[str, Any]:
"""Extract ``?var`` bindings by matching a condition pattern against a fact.
``condition`` is the pattern stored on the alpha node (typically a string
like ``"Person(?x)"``); ``fact`` is the working-memory :class:`Fact`. The
fact's canonical string form (``Predicate(arg1, arg2, ...)``) is matched
against the pattern using the same ``?\\w+`` placeholder convention as the
Reasoner, so downstream actions receive real bindings (e.g. ``{"x": "John"}``)
instead of the empty dict that previously left ``?x`` placeholders
unsubstituted.
Returns an empty dict when the condition is not a string pattern or does
not match -- callers treat that as "no bindings extracted".
"""
if not isinstance(condition, str):
return {}
segments = re.split(r"(\?\w+)", condition)
seen_vars: Set[str] = set()
p_regex = ""
for seg in segments:
if seg.startswith("?"):
var_name = seg[1:]
if var_name in seen_vars:
p_regex += f"(?P={var_name})"
else:
p_regex += f"(?P<{var_name}>.+?)"
seen_vars.add(var_name)
else:
p_regex += re.escape(seg)
p_regex = f"^{p_regex}$"
try:
match = re.match(p_regex, str(fact))
except re.error:
return {}
if not match:
return {}
return {k: v for k, v in match.groupdict().items() if v is not None}
def _bindings_for_rule(rule: Rule, facts: List[Fact]) -> Dict[str, Any]:
"""Merge ``?var`` bindings from matching a rule's conditions against facts.
Each fact is matched against every condition of the rule; the first
condition that yields bindings for a fact contributes them. Bindings from
all facts are merged so multi-condition (joined) rules receive the full
variable environment. Later conflicting values do not overwrite earlier
ones, preserving the binding that a join already validated.
"""
bindings: Dict[str, Any] = {}
for fact in facts:
for condition in rule.conditions:
extracted = _extract_bindings(condition, fact)
if not extracted:
continue
for key, value in extracted.items():
bindings.setdefault(key, value)
break
return bindings
@dataclass
@@ -58,7 +119,7 @@ class ReteNode:
def __init__(self, node_id: str):
self.node_id = node_id
self.children: List["ReteNode"] = []
self.children: List[ReteNode] = []
class AlphaNode(ReteNode):
@@ -151,6 +212,17 @@ class ReteEngine:
self.facts: List[Fact] = []
self.fact_counter = 0
self.node_counter = 0
self._executed_activations: Set[Tuple[Any, ...]] = set()
# Optional Reasoner used to fire rule-driven actions on match. When
# set, execute_matches() runs each matched rule's ``actions`` (and any
# legacy ``handler``) through the Reasoner's action machinery so that
# Rete-based matching benefits from the same production-rule behaviour
# as forward_chain(). Left None keeps the pure-matching mode.
self.reasoner: Optional[Any] = self.config.get("reasoner")
def bind_reasoner(self, reasoner: Any) -> None:
"""Attach a Reasoner so matched rules can fire their actions."""
self.reasoner = reasoner
def build_network(self, rules: List[Rule]) -> None:
"""
@@ -166,6 +238,7 @@ class ReteEngine:
)
try:
self.reset_action_history()
self.network.clear()
self.progress_tracker.update_tracking(
@@ -252,15 +325,24 @@ class ReteEngine:
# Propagate to children
for grandchild in child.children:
if isinstance(grandchild, TerminalNode):
facts = [left_fact, fact]
match = Match(
rule=grandchild.rule,
facts=[left_fact, fact],
facts=facts,
bindings=_bindings_for_rule(
grandchild.rule, facts
),
confidence=1.0,
)
grandchild.activate(match)
elif isinstance(child, TerminalNode):
# Direct activation
match = Match(rule=child.rule, facts=[fact], confidence=1.0)
match = Match(
rule=child.rule,
facts=[fact],
bindings=_bindings_for_rule(child.rule, [fact]),
confidence=1.0,
)
child.activate(match)
def match_patterns(self, facts: Optional[List[Fact]] = None) -> List[Match]:
@@ -276,7 +358,7 @@ class ReteEngine:
tracking_id = self.progress_tracker.start_tracking(
module="reasoning",
submodule="ReteEngine",
message=f"Matching patterns using Rete algorithm",
message="Matching patterns using Rete algorithm",
)
try:
@@ -339,10 +421,28 @@ class ReteEngine:
)
results = []
for match in matches:
# Conclusions are the pure inference result and remain
# independent from optional side-effect execution below.
results.append(match.rule.conclusion)
try:
# Execute rule
result = match.rule.conclusion
results.append(result)
# Fire the rule's actions (and any legacy handler) through
# the bound Reasoner so Rete matching produces the same
# side effects / provenance as forward_chain(). Falls back
# to just recording the conclusion when no Reasoner is bound.
if self.reasoner is not None and (
match.rule.actions or match.rule.handler is not None
):
activation_key = _make_activation_key(
match.rule.rule_id,
match.bindings,
[
(fact.fact_id, fact.predicate, fact.arguments)
for fact in match.facts
],
)
if activation_key not in self._executed_activations:
self._executed_activations.add(activation_key)
self.reasoner._fire_actions(match.rule, match.bindings)
except Exception as e:
self.logger.error(f"Error executing match: {e}")
@@ -359,13 +459,16 @@ class ReteEngine:
)
raise
def reset_action_history(self) -> None:
"""Allow previously executed activations to fire their actions again."""
self._executed_activations.clear()
def reset(self) -> None:
"""Reset Rete engine."""
"""Reset Rete working memory and action activation history."""
self.facts.clear()
self.reset_action_history()
for node in self.network.values():
if isinstance(node, AlphaNode):
node.matches.clear()
elif isinstance(node, BetaNode):
if isinstance(node, AlphaNode) or isinstance(node, BetaNode):
node.matches.clear()
elif isinstance(node, TerminalNode):
node.activations.clear()
+6 -40
View File
@@ -139,17 +139,19 @@ class NERExtractor:
if not self.progress_tracker.enabled:
self.progress_tracker.enabled = True
# Initialize spaCy model if ML method is used
self.nlp = None
# Validate the spaCy runtime up front if ML method is used. The model
# itself is loaded lazily by extract_entities_ml() through the
# process-level cache in methods.py; this instance only tracks whether
# ML dispatch should be attempted at all.
self._ml_runtime_usable = True
if "ml" in self.method and SPACY_AVAILABLE:
try:
# Deferred import: keeps semantic_extract.methods out of the
# module-level import graph and routes loading through the
# module-level import graph and routes validation through the
# process-level cache so repeated NERExtractor constructions
# never pay the ~120 ms spacy.load() cost more than once.
from .methods import load_spacy_model
self.nlp = load_spacy_model(self.model_name)
load_spacy_model(self.model_name)
except OSError:
self.logger.warning(
f"spaCy model {self.model_name} not found. ML method will fallback."
@@ -535,42 +537,6 @@ class NERExtractor:
return processed
def _extract_with_spacy(
self, text: str, min_confidence: float, entity_types: Optional[List[str]]
) -> List[Entity]:
"""Extract entities using spaCy."""
entities = []
doc = self.nlp(text)
for ent in doc.ents:
# Filter by entity types if specified
if entity_types and ent.label_ not in entity_types:
continue
# Get confidence if available
confidence = 1.0
if hasattr(ent, "confidence"):
confidence = ent.confidence
elif hasattr(ent, "score"):
confidence = ent.score
if confidence >= min_confidence:
entities.append(
Entity(
text=ent.text,
label=ent.label_,
start_char=ent.start_char,
end_char=ent.end_char,
confidence=confidence,
metadata={
"lemma": ent.lemma_ if hasattr(ent, "lemma_") else ent.text
},
)
)
return entities
def _extract_fallback(self, text: str) -> List[Entity]:
"""Fallback entity extraction using simple patterns."""
entities = []
+3 -3
View File
@@ -123,12 +123,12 @@ class SplitConfig:
if value:
try:
# Try to convert to appropriate type
if isinstance(default, int):
if isinstance(default, bool):
return value.strip().lower() in ("true", "1", "yes", "on")
elif isinstance(default, int):
return int(value)
elif isinstance(default, float):
return float(value)
elif isinstance(default, bool):
return value.lower() in ("true", "1", "yes", "on")
return value
except (ValueError, TypeError):
pass
+3 -3
View File
@@ -5,11 +5,11 @@ This module provides the worker process for the Semantica framework,
enabling distributed and background task processing.
"""
import time
import signal
import sys
from .utils.logging import get_logger, setup_logging
import time
from .core.orchestrator import Semantica
from .utils.logging import get_logger, setup_logging
# Initialize logging
setup_logging()
+168
View File
@@ -0,0 +1,168 @@
'''Regression tests for issue #1035.
Config.get() must honor boolean environment overrides. Previously the type
dispatch checked isinstance(default, int) before isinstance(default, bool);
since bool subclasses int the bool branch was unreachable, so a boolean
default silently ignored the environment variable (or returned an int when the
value happened to parse).
'''
import os
import unittest
from unittest.mock import patch
from semantica.conflicts.config import ConflictsConfig
from semantica.deduplication.config import DeduplicationConfig
from semantica.embeddings.config import EmbeddingsConfig
from semantica.export.config import ExportConfig
from semantica.ingest.config import IngestConfig
from semantica.kg.config import KGConfig
from semantica.normalize.config import NormalizeConfig
from semantica.ontology.config import OntologyConfig
from semantica.parse.config import ParseConfig
from semantica.split.config import SplitConfig
# The three modules whose get() reaches the type dispatch (no generic
# PREFIX_* scanner in _load_env_vars pre-populates _configs).
_DISPATCH_MODULES = [
("conflicts", "CONFLICT", ConflictsConfig),
("deduplication", "DEDUP", DeduplicationConfig),
("split", "SPLIT", SplitConfig),
]
class TestConfigBoolEnvOverride(unittest.TestCase):
"""Boolean env overrides must reach Config.get() on the dispatch path."""
def test_conflicts_bool_true(self):
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "true"}):
self.assertIs(ConflictsConfig().get("zztestflag", False), True)
def test_conflicts_bool_false(self):
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "false"}):
self.assertIs(ConflictsConfig().get("zztestflag", True), False)
def test_conflicts_int_still_parsed(self):
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "42"}):
self.assertEqual(ConflictsConfig().get("zztestflag", 0), 42)
def test_conflicts_float_still_parsed(self):
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "1.5"}):
self.assertEqual(ConflictsConfig().get("zztestflag", 0.0), 1.5)
def test_conflicts_str_still_returned(self):
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "some-value"}):
self.assertEqual(ConflictsConfig().get("zztestflag", "x"), "some-value")
def test_truthy_spellings(self):
for value in ["true", "1", "yes", "on", "TRUE", "True", "YES", "ON", " true "]:
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": value}):
self.assertIs(
ConflictsConfig().get("zztestflag", False),
True,
msg="value {0!r} should parse as True".format(value),
)
def test_falsy_spellings(self):
for value in ["false", "0", "no", "off", "FALSE", "False", "NO", "OFF", " false "]:
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": value}):
self.assertIs(
ConflictsConfig().get("zztestflag", True),
False,
msg="value {0!r} should parse as False".format(value),
)
def test_programmatic_config_takes_precedence(self):
cfg = ConflictsConfig()
cfg.set("zztestflag", "nope")
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "true"}):
self.assertEqual(cfg.get("zztestflag", False), "nope")
def test_non_bool_value_falls_back_to_default(self):
# "not-a-number" is not in the truthy set -> False for a bool default;
# the int branch would have raised and swallowed the env var entirely.
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "not-a-number"}):
self.assertIs(ConflictsConfig().get("zztestflag", True), False)
def test_bool_dispatch_across_three_modules(self):
for name, prefix, cls in _DISPATCH_MODULES:
env_key = prefix + "_ZZTESTFLAG"
with self.subTest(module=name, env_key=env_key):
with patch.dict(os.environ, {env_key: "true"}):
self.assertIs(cls().get("zztestflag", False), True)
with patch.dict(os.environ, {env_key: "false"}):
self.assertIs(cls().get("zztestflag", True), False)
with patch.dict(os.environ, {env_key: "1"}):
self.assertIs(cls().get("zztestflag", False), True)
with patch.dict(os.environ, {env_key: "0"}):
self.assertIs(cls().get("zztestflag", True), False)
# Every module config also maps known boolean env vars in _load_env_vars;
# those must parse as real bools too (not ints or strings).
_MAPPED_BOOL_ENV = [
("conflicts", "CONFLICT_AUTO_RESOLVE", ConflictsConfig),
("deduplication", "DEDUP_USE_CLUSTERING", DeduplicationConfig),
("embeddings", "EMBEDDING_NORMALIZE", EmbeddingsConfig),
("export", "EXPORT_VALIDATE", ExportConfig),
("ingest", "INGEST_RECURSIVE", IngestConfig),
("kg", "KG_MERGE_ENTITIES", KGConfig),
("ontology", "ONTOLOGY_CHECK_CONSISTENCY", OntologyConfig),
("parse", "PARSE_EXTRACT_TABLES", ParseConfig),
]
# Modules that only reach bool env parsing via the generic PREFIX_* scanner
# in _load_env_vars (no bool entry in env_mappings).
_SCANNER_BOOL_ENV = [
("split", "SPLIT_ZZTESTFLAG", SplitConfig),
("normalize", "NORMALIZE_ZZTESTFLAG", NormalizeConfig),
]
class TestMappedBoolEnvVars(unittest.TestCase):
def test_mapped_bool_true(self):
for module, env_key, cls in _MAPPED_BOOL_ENV:
with self.subTest(module=module, env_key=env_key):
with patch.dict(os.environ, {env_key: "true"}):
cfg = cls()
key = env_key.split("_", 1)[1].lower()
self.assertIs(cfg.get(key, False), True)
def test_mapped_bool_false(self):
for module, env_key, cls in _MAPPED_BOOL_ENV:
with self.subTest(module=module, env_key=env_key):
with patch.dict(os.environ, {env_key: "false"}):
cfg = cls()
key = env_key.split("_", 1)[1].lower()
self.assertIs(cfg.get(key, True), False)
def test_mapped_bool_with_whitespace(self):
# Qodo follow-up: _load_env_vars() must strip like get() does, so a
# padded value (" true ") is not silently parsed as False.
for module, env_key, cls in _MAPPED_BOOL_ENV:
with self.subTest(module=module, env_key=env_key):
with patch.dict(os.environ, {env_key: " true "}):
cfg = cls()
key = env_key.split("_", 1)[1].lower()
self.assertIs(cfg.get(key, False), True)
def test_scanner_bool_true(self):
for module, env_key, cls in _SCANNER_BOOL_ENV:
with self.subTest(module=module, env_key=env_key):
with patch.dict(os.environ, {env_key: "true"}):
cfg = cls()
key = env_key.split("_", 1)[1].lower()
self.assertIs(cfg.get(key, False), True)
def test_scanner_bool_with_whitespace(self):
for module, env_key, cls in _SCANNER_BOOL_ENV:
with self.subTest(module=module, env_key=env_key):
with patch.dict(os.environ, {env_key: " true "}):
cfg = cls()
key = env_key.split("_", 1)[1].lower()
self.assertIs(cfg.get(key, False), True)
if __name__ == "__main__":
unittest.main()
+152 -1
View File
@@ -1,8 +1,8 @@
"""Tests for RDFExporter format alias resolution (issue #355)."""
import pytest
from semantica.export import RDFExporter
from semantica.export import RDFExporter
RDF_DATA = {
"entities": [
@@ -89,3 +89,154 @@ def test_validate_rdf_returns_overall_valid_key(exporter):
result = exporter.validate_rdf(RDF_DATA)
assert "overall_valid" in result
assert isinstance(result["overall_valid"], bool)
# --- Regression tests for #1097 -------------------------------------------
# convert_kg_to_rdf() normalizes an entity's 'name' into 'label'/'text' but was
# never called from the export path, so GraphBuilder graphs (which emit 'name')
# exported with an empty semantica:text on every RDF format. These tests assert
# the human-readable label survives export on all four serializers.
NAME_ONLY_DATA = {
"entities": [
{
"id": "https://example.org/acme",
"name": "Acme Corp",
"type": "https://example.org/Org",
"confidence": 0.91,
},
],
"relationships": [],
}
@pytest.mark.parametrize("fmt", ["turtle", "ntriples", "rdfxml", "jsonld"])
def test_name_only_entity_exports_nonempty_label(exporter, fmt):
"""A GraphBuilder-style 'name'-only entity must export a non-empty label.
Regression for #1097: previously every RDF path dropped the label because
convert_kg_to_rdf() was never invoked from export_to_rdf().
"""
result = exporter.export_to_rdf(NAME_ONLY_DATA, format=fmt)
assert "Acme Corp" in result
# The empty-text pattern that the bug produced must not appear.
assert 'semantica:text ""' not in result
def test_name_only_export_to_file_contains_label(exporter, tmp_path):
"""The file-writing entry point must also normalize name -> label (#1097)."""
out = tmp_path / "acme.ttl"
exporter.export(NAME_ONLY_DATA, str(out), format="turtle")
content = out.read_text()
assert "Acme Corp" in content
assert 'semantica:text ""' not in content
def test_existing_text_not_overwritten_by_name(exporter):
"""An entity that already has 'text' must keep it, not be clobbered by 'name' (#1097)."""
data = {
"entities": [
{
"id": "https://example.org/acme",
"name": "Acme Corp",
"text": "Explicit Text",
"type": "https://example.org/Org",
}
],
"relationships": [],
}
result = exporter.export_to_rdf(data, format="turtle")
assert "Explicit Text" in result
assert "Acme Corp" not in result
def test_id_fallback_label_when_no_name(exporter):
"""With neither 'name' nor 'text', the id local-name is used as label (#1097)."""
data = {
"entities": [
{"id": "https://example.org/acme", "type": "https://example.org/Org"}
],
"relationships": [],
}
result = exporter.export_to_rdf(data, format="turtle")
assert 'semantica:text ""' not in result
# The label must be the URI local name 'acme', not '//example.org/acme'.
assert 'semantica:text "acme"' in result
@pytest.mark.parametrize(
"identifier,expected",
[
("https://example.org/acme", "acme"),
("https://example.org/path/acme", "acme"),
("https://example.org/onto#acme", "acme"),
("https://example.org/acme/", "acme"),
("urn:example:acme", "acme"),
("semantica:acme", "acme"),
("acme", "acme"),
],
)
def test_id_fallback_local_name_extraction(exporter, identifier, expected):
"""The id fallback must extract a URI-aware local name, not a colon split (#1113)."""
data = {
"entities": [{"id": identifier, "type": "https://example.org/Org"}],
"relationships": [],
}
result = exporter.export_to_rdf(data, format="turtle")
assert f'semantica:text "{expected}"' in result
def _export(exporter, name, fmt):
data = {
"entities": [
{"id": "https://example.org/e1", "name": name, "type": "ORG"}
],
"relationships": [],
}
return exporter.export_to_rdf(data, format=fmt)
def test_turtle_escapes_quotes_and_control_chars(exporter):
"""Turtle literals must escape quotes/backslashes/newlines (#1113 security)."""
result = _export(exporter, 'Acme "Best" \\ Corp\nLine2\tTab\rCR', "turtle")
# The raw closing-quote breakout must not appear inside the literal.
assert '"Acme "Best"' not in result
assert '\\"Best\\"' in result
assert "\\\\ Corp" in result
assert "\\n" in result and "\\t" in result and "\\r" in result
# No unescaped newline leaked into the literal value.
assert "Line2" in result and "\nLine2" not in result.split("semantica:text")[1]
def test_ntriples_escapes_quotes_and_control_chars(exporter):
"""N-Triples literals must escape backslash first, then quotes/controls (#1113)."""
result = _export(exporter, 'Quote " Back \\ New\nTab\t', "ntriples")
assert '\\"' in result
assert "\\\\" in result
assert "\\n" in result and "\\t" in result
# Each triple must be a single physical line: the literal value carrying the
# escaped text must not have leaked a bare newline that splits it in two.
text_lines = [ln for ln in result.splitlines() if "Quote" in ln]
assert len(text_lines) == 1
assert text_lines[0].rstrip().endswith(" .")
def test_rdfxml_escapes_markup(exporter):
"""RDF/XML character data must escape &, <, > so names cannot inject markup (#1113)."""
result = _export(exporter, 'Acme <script>&"x"', "rdfxml")
assert "<script>" not in result
assert "&lt;script&gt;" in result
assert "&amp;" in result
# The document must still parse as well-formed XML.
import xml.dom.minidom
xml.dom.minidom.parseString(result)
def test_turtle_output_is_parseable_with_special_name(exporter):
"""A name full of metacharacters must still yield parseable Turtle (#1113)."""
rdflib = pytest.importorskip("rdflib")
result = _export(exporter, 'Tricky "quote" \\ and <angle> & amp', "turtle")
graph = rdflib.Graph()
# Should not raise a parser error.
graph.parse(data=result, format="turtle")
@@ -0,0 +1,176 @@
"""Regression tests for the application-id / internal-id mismatch (#1136).
`GraphStore.add_edges` reads application-level string ids from
``source_id``/``target_id`` while backends such as Neo4j match relationships
on their internal integer ids (``id(n)``). Every edge therefore failed with
"nodes not found" and a graph persisted with all nodes and zero
relationships.
The fix keeps the application-id -> internal-id map that ``add_nodes``
already receives from the backend (instead of discarding it) and resolves
known string ids in ``create_relationship``. These tests pin that contract
with a fake manager no live database needed.
"""
import unittest
from typing import Any, Dict, List, Optional, Tuple
from semantica.graph_store.graph_store import GraphStore
class FakeNodeManager:
"""Mimics NodeManager against a backend that mints internal integer ids."""
def __init__(self) -> None:
self._next_internal_id = 100
self.created: List[Dict[str, Any]] = []
def create_batch(
self, nodes: List[Dict[str, Any]], **options: Any
) -> List[Dict[str, Any]]:
created: List[Dict[str, Any]] = []
for node in nodes:
internal_id = self._next_internal_id
self._next_internal_id += 1
record = {
"id": internal_id,
"labels": node.get("labels", []),
"properties": dict(node.get("properties", {})),
}
created.append(record)
self.created.append(record)
return created
def create(
self, labels: List[str], properties: Dict[str, Any], **options: Any
) -> Dict[str, Any]:
return self.create_batch(
[{"labels": labels, "properties": properties}], **options
)[0]
class FakeRelationshipManager:
"""Records the resolved ids create_relationship hands to the backend."""
def __init__(self) -> None:
self.calls: List[Tuple[Any, Any, str]] = []
def create(
self,
start_node_id: Any,
end_node_id: Any,
rel_type: str,
properties: Optional[Dict[str, Any]] = None,
**options: Any,
) -> Dict[str, Any]:
self.calls.append((start_node_id, end_node_id, rel_type))
return {"start": start_node_id, "end": end_node_id, "type": rel_type}
class FakeManager:
def __init__(self) -> None:
self.nodes = FakeNodeManager()
self.relationships = FakeRelationshipManager()
def make_store() -> Tuple[GraphStore, FakeManager]:
"""Build a GraphStore around fakes, skipping backend initialization."""
store = GraphStore.__new__(GraphStore)
store.logger = None # type: ignore[assignment]
store.progress_tracker = None # type: ignore[assignment]
store.backend = "fake"
store.config = {}
store._app_node_id_map = {}
manager = FakeManager()
store._store_backend = None
store._manager = manager # type: ignore[assignment]
return store, manager
class AppIdResolutionTests(unittest.TestCase):
def test_add_edges_resolves_application_ids_to_internal_ids(self) -> None:
store, manager = make_store()
node_count = store.add_nodes(
[
{"id": "e1", "type": "Person", "properties": {}},
{"id": "e2", "type": "Organization", "properties": {}},
]
)
self.assertEqual(node_count, 2)
edge_count = store.add_edges(
[{"source_id": "e1", "target_id": "e2", "type": "knows"}]
)
self.assertEqual(edge_count, 1)
self.assertEqual(len(manager.relationships.calls), 1)
start, end, rel_type = manager.relationships.calls[0]
self.assertEqual(start, 100, "source app id must resolve to the internal id")
self.assertEqual(end, 101, "target app id must resolve to the internal id")
self.assertEqual(rel_type, "knows")
def test_build_from_entities_and_relationships_creates_the_edge(self) -> None:
"""The exact shape of the #1136 reproduction, against fakes."""
store, manager = make_store()
stats = store.build_from_entities_and_relationships(
[
{"id": "e1", "type": "Person", "text": "Alice"},
{"id": "e2", "type": "Organization", "text": "Acme"},
],
[{"source_id": "e1", "target_id": "e2", "type": "knows"}],
)
self.assertEqual(stats["statistics"]["node_count"], 2)
self.assertEqual(
stats["statistics"]["edge_count"], 1, "the edge must be created, not dropped"
)
self.assertEqual(manager.relationships.calls, [(100, 101, "knows")])
def test_unknown_ids_pass_through_unchanged(self) -> None:
"""Ids the map does not know keep the pre-fix pass-through behavior."""
store, manager = make_store()
store.create_relationship(7, 8, "RELATES_TO")
self.assertEqual(manager.relationships.calls, [(7, 8, "RELATES_TO")])
def test_create_node_also_populates_the_map(self) -> None:
store, manager = make_store()
store.create_node(["Person"], {"id": "app-1", "name": "Alice"})
store.add_edges([{"source_id": "app-1", "target_id": 42, "type": "knows"}])
self.assertEqual(manager.relationships.calls, [(100, 42, "knows")])
def test_integer_application_ids_never_collide_with_internal_ids(self) -> None:
# Qodo review: an int application id must not be recorded/resolved,
# or a caller passing that same int as an internal id would be
# silently remapped to a different node.
store, manager = make_store()
store.add_nodes(
[
{"id": 100, "type": "Person", "properties": {}},
{"id": "str-app", "type": "Person", "properties": {}},
]
)
# Internal id 100 legitimately targets the FIRST node; the integer
# app id of that same value must not redirect it to the second.
store.create_relationship(100, 101, "RELATES_TO")
self.assertEqual(manager.relationships.calls, [(100, 101, "RELATES_TO")])
self.assertNotIn(100, store._app_node_id_map)
def test_nodes_without_application_id_do_not_pollute_the_map(self) -> None:
store, manager = make_store()
store.add_nodes([{"labels": ["Person"], "properties": {"name": "Anon"}}])
store.create_relationship("not-mapped", 5, "RELATES_TO")
self.assertEqual(manager.relationships.calls, [("not-mapped", 5, "RELATES_TO")])
if __name__ == "__main__":
unittest.main()
+127 -94
View File
@@ -7,6 +7,7 @@ Tests integration between all KG components and algorithms.
import pytest
import networkx as nx
import numpy as np
from datetime import datetime, timedelta
from typing import Dict, List, Any, Tuple
import time
import json
@@ -14,7 +15,6 @@ import json
from semantica.kg import (
GraphBuilderWithProvenance,
AlgorithmTrackerWithProvenance,
NodeEmbedder,
SimilarityCalculator,
PathFinder,
LinkPredictor,
@@ -24,6 +24,40 @@ from semantica.kg import (
)
def _stored(owner, entity_id):
"""Read a provenance record back. Fails if tracking only minted an ID."""
record = owner._prov_manager.get_provenance(entity_id)
assert record is not None, (
f"no stored provenance for {entity_id!r} — an ID was generated without a write"
)
assert record.get("entity_id") == entity_id
return record
def _assert_utc_iso(value, field="timestamp"):
assert value, f"missing {field}"
parsed = datetime.fromisoformat(value)
assert parsed.tzinfo is not None, (
f"{field}={value!r} is naive (datetime.utcnow leftover)"
)
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
return parsed
def _assert_tracked(owner, entity_id, source, **metadata):
record = _stored(owner, entity_id)
assert record["source_document"] == source
actual = record.get("metadata") or {}
for key, expected in metadata.items():
assert actual.get(key) == expected, (
f"{entity_id} metadata[{key!r}]={actual.get(key)!r}, expected {expected!r}"
)
_assert_utc_iso(record["timestamp"], "timestamp")
if record.get("last_updated"):
_assert_utc_iso(record["last_updated"], "last_updated")
return record
class TestComprehensiveIntegration:
"""Comprehensive integration tests for KG module."""
@@ -157,8 +191,6 @@ class TestComprehensiveIntegration:
# Initialize all components
builder = GraphBuilderWithProvenance(provenance=True)
tracker = AlgorithmTrackerWithProvenance(provenance=True)
embedder = NodeEmbedder()
embedder.enable_provenance = True
sim_calc = SimilarityCalculator()
path_finder = PathFinder()
link_predictor = LinkPredictor()
@@ -351,14 +383,34 @@ class TestComprehensiveIntegration:
source='comprehensive_integration_test'
)
# Verify all phases completed successfully
expected = {
'construction': 'graph_construction',
'centrality': 'centrality_calculation',
'connectivity': 'connectivity_analysis',
'community': 'community_detection',
'similarity': 'similarity_calculation',
'link_prediction': 'link_prediction',
'path_analysis': 'path_analysis',
'cross_layer': 'cross_layer_analysis',
}
assert len(execution_ids) == 8
for phase, exec_id in execution_ids.items():
assert exec_id is not None
assert len(exec_id) > 10
print(f"Full pipeline integration completed: {pipeline_id}")
return pipeline_id
_assert_tracked(
tracker,
exec_id,
source=pipeline_id,
entity_type=expected[phase],
)
summary_record = _assert_tracked(
tracker,
summary_id,
source='comprehensive_integration_test',
entity_type='pipeline_summary',
pipeline_id=pipeline_id,
phases_count=8,
)
assert summary_id.startswith('pipeline_summary_')
assert summary_record['metadata']['input_data_size'] == len(complex_graph_data)
def test_multi_layer_network_analysis(self, multi_layer_network, realistic_embeddings):
"""Test multi-layer network analysis."""
@@ -382,39 +434,39 @@ class TestComprehensiveIntegration:
# Centrality analysis
if graph.number_of_nodes() > 0:
try:
degree_cent = centrality_calc.calculate_degree_centrality(graph_dict)
layer_results[f"{layer_name}_centrality"] = degree_cent
# Track with provenance
cent_id = tracker.track_layer_analysis(
layer_name=layer_name,
graph=graph,
analysis_type='centrality',
results=degree_cent,
source=multi_layer_id
)
except Exception as e:
print(f"Centrality analysis failed for {layer_name}: {e}")
degree_cent = centrality_calc.calculate_degree_centrality(graph_dict)
layer_results[f"{layer_name}_centrality"] = degree_cent
cent_id = tracker.track_centrality_calculation(
graph=graph,
centrality_scores=degree_cent['centrality'],
method='degree',
source=multi_layer_id
)
_assert_tracked(
tracker,
cent_id,
source=multi_layer_id,
method='degree',
scores_count=len(degree_cent['centrality']),
)
# Community detection
if graph.number_of_edges() > 0:
try:
communities = community_detector.detect_communities(graph_dict, method='label_propagation')
layer_results[f"{layer_name}_communities"] = communities
# Track with provenance
comm_id = tracker.track_layer_analysis(
layer_name=layer_name,
graph=graph,
analysis_type='communities',
results=communities,
source=multi_layer_id
)
except Exception as e:
print(f"Community detection failed for {layer_name}: {e}")
communities = community_detector.detect_communities(graph_dict, method='label_propagation')
layer_results[f"{layer_name}_communities"] = communities
comm_id = tracker.track_community_detection(
graph=graph,
communities=communities['communities'],
method='label_propagation',
source=multi_layer_id
)
_assert_tracked(
tracker,
comm_id,
source=multi_layer_id,
method='label_propagation',
communities_count=len(communities['communities']),
)
# Cross-layer similarity analysis
print("Cross-layer similarity analysis")
@@ -433,39 +485,43 @@ class TestComprehensiveIntegration:
similarity_score = len(common_nodes) / max(len(graph1.nodes()), len(graph2.nodes()))
layer_similarities[f"{layer1_name}_{layer2_name}"] = similarity_score
# Track cross-layer analysis
cross_layer_id = tracker.track_cross_layer_analysis(
multi_layer_network=multi_layer_network,
layer_similarities=layer_similarities,
graph_data=multi_layer_network,
cross_layer_results=layer_similarities,
source='multi_layer_test'
)
_assert_tracked(
tracker,
cross_layer_id,
source='multi_layer_test',
entity_type='cross_layer_analysis',
layers_count=len(layer_similarities),
)
# Embedding-based entity similarity
print("Embedding-based entity similarity")
entity_similarities = sim_calc.pairwise_similarity(realistic_embeddings)
# Track embedding analysis
embed_id = tracker.track_embedding_analysis(
embeddings=realistic_embeddings,
similarities=entity_similarities,
analysis_results=entity_similarities,
source='multi_layer_test'
)
_assert_tracked(
tracker,
embed_id,
source='multi_layer_test',
entity_type='embedding_analysis',
embeddings_count=len(realistic_embeddings),
)
# Verify results
assert len(layer_results) > 0
assert len(layer_similarities) > 0
assert len(entity_similarities) > 0
print(f"Multi-layer analysis completed")
print(f"Layers analyzed: {list(multi_layer_network.keys())}")
print(f"Layer similarities: {list(layer_similarities.keys())}")
return multi_layer_id
def test_error_handling_and_recovery(self):
"""Test error handling and recovery mechanisms."""
tracker = AlgorithmTrackerWithProvenance(provenance=True)
centrality_calc = CentralityCalculator()
# Test with invalid graph data
invalid_graph = {
@@ -473,46 +529,24 @@ class TestComprehensiveIntegration:
'edges': []
}
# Should handle gracefully
try:
centrality_calc = CentralityCalculator()
result = centrality_calc.calculate_degree_centrality(invalid_graph)
# Should return empty result or handle gracefully
assert isinstance(result, dict)
except Exception as e:
# Should be a controlled exception
assert isinstance(e, (ValueError, RuntimeError))
result = centrality_calc.calculate_degree_centrality(invalid_graph)
assert isinstance(result, dict)
# Test with invalid embeddings
invalid_embeddings = {
'node1': [1, 2], # Different dimensions
'node2': [1, 2, 3, 4] # Different dimensions
}
try:
sim_calc = SimilarityCalculator()
result = sim_calc.batch_similarity(
embeddings=invalid_embeddings,
query_embedding=[1, 2, 3, 4],
method='cosine'
)
# Should handle dimension mismatch
except Exception as e:
# Should handle gracefully
assert isinstance(e, ValueError)
# Test provenance tracking with invalid data
try:
result = tracker.track_embedding_computation(
graph=None, # Invalid graph
algorithm='test',
embeddings={},
parameters={}
)
# Should either return None or handle gracefully
except Exception as e:
# Should be a controlled exception
assert isinstance(e, (ValueError, TypeError))
# Provenance tracking with a None graph still writes a record.
result = tracker.track_embedding_computation(
graph=None,
algorithm='test',
embeddings={},
parameters={},
source='integration_error_recovery'
)
_assert_tracked(
tracker,
result,
source='integration_error_recovery',
algorithm='test',
input_data_type='NoneType',
)
# Test graceful degradation when provenance is disabled
tracker_no_prov = AlgorithmTrackerWithProvenance(provenance=False)
@@ -526,8 +560,7 @@ class TestComprehensiveIntegration:
# Should return None when provenance is disabled
assert result is None
print("Error handling and recovery test completed")
assert tracker_no_prov._prov_manager is None
def test_performance_benchmarks(self, realistic_embeddings):
"""Test performance benchmarks with realistic data."""
+262 -166
View File
@@ -7,13 +7,13 @@ Tests complete provenance tracking workflows across multiple algorithms.
import pytest
import networkx as nx
import time
from datetime import datetime, timedelta
from typing import Dict, List, Any
import uuid
from semantica.kg import (
GraphBuilderWithProvenance,
AlgorithmTrackerWithProvenance,
NodeEmbedder,
SimilarityCalculator,
LinkPredictor,
CentralityCalculator,
@@ -21,6 +21,40 @@ from semantica.kg import (
)
def _stored(owner, entity_id):
"""Read a provenance record back. Fails if tracking only minted an ID."""
record = owner._prov_manager.get_provenance(entity_id)
assert record is not None, (
f"no stored provenance for {entity_id!r} — an ID was generated without a write"
)
assert record.get("entity_id") == entity_id
return record
def _assert_utc_iso(value, field="timestamp"):
assert value, f"missing {field}"
parsed = datetime.fromisoformat(value)
assert parsed.tzinfo is not None, (
f"{field}={value!r} is naive (datetime.utcnow leftover)"
)
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
return parsed
def _assert_tracked(owner, entity_id, source, **metadata):
record = _stored(owner, entity_id)
assert record["source_document"] == source
actual = record.get("metadata") or {}
for key, expected in metadata.items():
assert actual.get(key) == expected, (
f"{entity_id} metadata[{key!r}]={actual.get(key)!r}, expected {expected!r}"
)
_assert_utc_iso(record["timestamp"], "timestamp")
if record.get("last_updated"):
_assert_utc_iso(record["last_updated"], "last_updated")
return record
class TestProvenanceWorkflows:
"""Test provenance tracking workflows."""
@@ -116,10 +150,25 @@ class TestProvenanceWorkflows:
source=workflow_id
)
assert construction_id is not None
_assert_tracked(
tracker,
construction_id,
source=workflow_id,
entity_type="graph_construction",
entities_count=8,
relationships_count=12,
)
assert construction_id.startswith('graph_construction_')
for entity in graph_result['entities']:
built = _stored(builder, entity['id'])
assert built['metadata']['operation'] == 'build_entity'
assert built['metadata']['entity_type'] == entity['type']
_assert_utc_iso(built['activity_started_at_time'], 'activity_started_at_time')
_assert_utc_iso(built['activity_ended_at_time'], 'activity_ended_at_time')
# Step 3: Track entity processing
processed = []
for entity in graph_result['entities']:
entity_id = tracker.track_entity_processing(
entity_id=entity['id'],
@@ -127,9 +176,19 @@ class TestProvenanceWorkflows:
entity_data=entity,
source=workflow_id
)
assert entity_id is not None
_assert_tracked(
tracker,
entity_id,
source=workflow_id,
processed_entity_id=entity['id'],
processed_entity_type=entity['type'],
)
processed.append(entity_id)
assert len(processed) == 8
assert len(set(processed)) == 8
# Step 4: Track relationship processing
rel_ids = []
for relationship in graph_result['relationships']:
rel_id = tracker.track_relationship_processing(
relationship_id=f"{relationship['source']}-{relationship['target']}",
@@ -137,10 +196,15 @@ class TestProvenanceWorkflows:
relationship_data=relationship,
source=workflow_id
)
assert rel_id is not None
print(f"Graph construction workflow completed: {workflow_id}")
return workflow_id
_assert_tracked(
tracker,
rel_id,
source=workflow_id,
processed_relationship_type=relationship['type'],
)
rel_ids.append(rel_id)
assert len(rel_ids) == 12
assert len(set(rel_ids)) == 12
def test_embedding_workflow(self, workflow_graph, workflow_embeddings):
"""Test complete embedding workflow with provenance."""
@@ -176,8 +240,20 @@ class TestProvenanceWorkflows:
source=workflow_id
)
assert embed_id is not None
_assert_tracked(
tracker,
embed_id,
source=workflow_id,
algorithm='node2vec',
node_count=6,
embedding_dimension=4,
)
assert embed_id.startswith('embedding_')
for node_id, vector in computed_embeddings.items():
node_record = _stored(tracker, f"embedding_{node_id}")
assert node_record['metadata']['node_id'] == node_id
assert node_record['metadata']['execution_id'] == embed_id
assert node_record['metadata']['embedding_dimension'] == len(vector)
# Step 2: Track embedding quality metrics
quality_metrics = {
@@ -196,10 +272,13 @@ class TestProvenanceWorkflows:
source=workflow_id
)
assert quality_id is not None
print(f"Embedding workflow completed: {workflow_id}")
return workflow_id
_assert_tracked(
tracker,
quality_id,
source=workflow_id,
algorithm='node2vec_quality_check',
)
assert quality_id != embed_id
def test_similarity_analysis_workflow(self, workflow_embeddings):
"""Test complete similarity analysis workflow with provenance."""
@@ -229,10 +308,18 @@ class TestProvenanceWorkflows:
source=workflow_id
)
assert sim_id is not None
_assert_tracked(
tracker,
sim_id,
source=workflow_id,
method='cosine',
similarities_count=len(similarities),
)
assert sim_id.startswith('similarity_')
assert len(similarities) == 3
# Step 2: Track individual similarity results
result_ids = []
for node_id, similarity_score in similarities.items():
result_id = tracker.track_similarity_result(
node_id=node_id,
@@ -241,7 +328,17 @@ class TestProvenanceWorkflows:
execution_id=sim_id,
source=workflow_id
)
assert result_id is not None
record = _assert_tracked(
tracker,
result_id,
source=workflow_id,
node_id=node_id,
method='cosine',
execution_id=sim_id,
)
assert record['metadata']['similarity_score'] == similarity_score
result_ids.append(result_id)
assert len(result_ids) == len(similarities)
# Step 3: Track similarity threshold analysis
threshold = 0.7
@@ -254,10 +351,14 @@ class TestProvenanceWorkflows:
source=workflow_id
)
assert threshold_id is not None
print(f"Similarity analysis workflow completed: {workflow_id}")
return workflow_id
_assert_tracked(
tracker,
threshold_id,
source=workflow_id,
threshold=threshold,
execution_id=sim_id,
)
assert _stored(tracker, threshold_id)['metadata']['high_similarity_count'] == len(high_similarity)
def test_link_prediction_workflow(self, workflow_graph):
"""Test complete link prediction workflow with provenance."""
@@ -287,7 +388,13 @@ class TestProvenanceWorkflows:
source=workflow_id
)
assert pred_id is not None
_assert_tracked(
tracker,
pred_id,
source=workflow_id,
method=method,
predictions_count=len(predictions),
)
assert pred_id.startswith('link_prediction_')
# Step 2: Track individual predictions
@@ -300,10 +407,17 @@ class TestProvenanceWorkflows:
execution_id=pred_id,
source=workflow_id
)
assert result_id is not None
print(f"Link prediction workflow completed: {workflow_id}")
return workflow_id
record = _assert_tracked(
tracker,
result_id,
source=workflow_id,
source_node=source,
target_node=target,
method=method,
execution_id=pred_id,
)
assert record['metadata']['prediction_score'] == score
assert len(methods) == 3
def test_centrality_analysis_workflow(self, workflow_graph):
"""Test complete centrality analysis workflow with provenance."""
@@ -326,54 +440,42 @@ class TestProvenanceWorkflows:
('eigenvector', centrality_calc.calculate_eigenvector_centrality)
]
tracked_methods = []
for method_name, method_func in centrality_methods:
try:
start_time = time.time()
result = method_func(graph_dict)
calculation_time = time.time() - start_time
cent_id = tracker.track_centrality_calculation(
graph=workflow_graph,
centrality_scores=result['centrality'],
method=method_name,
parameters={},
calculation_time=calculation_time,
source=workflow_id
)
assert cent_id is not None
assert cent_id.startswith('centrality_')
# Step 2: Track individual centrality scores
for node_id, score in result['centrality'].items():
score_id = tracker.track_centrality_score(
node_id=node_id,
centrality_score=score,
method=method_name,
execution_id=cent_id,
source=workflow_id
)
assert score_id is not None
# Step 3: Track centrality ranking analysis
rankings = result['rankings']
top_nodes = rankings[:3] # Top 3 nodes
ranking_id = tracker.track_centrality_ranking(
execution_id=cent_id,
rankings=rankings,
top_nodes=top_nodes,
method=method_name,
source=workflow_id
)
assert ranking_id is not None
except Exception as e:
print(f"Warning: {method_name} centrality failed: {e}")
print(f"Centrality analysis workflow completed: {workflow_id}")
return workflow_id
except Exception:
# Algorithm failure is allowed for optional methods, not for degree.
if method_name == 'degree':
raise
continue
calculation_time = 0.0
cent_id = tracker.track_centrality_calculation(
graph=workflow_graph,
centrality_scores=result['centrality'],
method=method_name,
parameters={},
calculation_time=calculation_time,
source=workflow_id
)
_assert_tracked(
tracker,
cent_id,
source=workflow_id,
method=method_name,
scores_count=len(result['centrality']),
)
assert cent_id.startswith('centrality_')
for node_id, score in result['centrality'].items():
score_record = _stored(tracker, f"centrality_{node_id}_{cent_id}")
assert score_record['metadata']['node_id'] == node_id
assert score_record['metadata']['method'] == method_name
assert score_record['metadata']['centrality_score'] == score
tracked_methods.append(method_name)
assert 'degree' in tracked_methods
assert len(tracked_methods) >= 1
def test_community_detection_workflow(self, workflow_graph):
"""Test complete community detection workflow with provenance."""
@@ -391,56 +493,39 @@ class TestProvenanceWorkflows:
# Step 1: Track community detection
methods = ['label_propagation', 'louvain']
tracked_methods = []
for method in methods:
try:
start_time = time.time()
result = community_detector.detect_communities(graph_dict, method=method)
detection_time = time.time() - start_time
comm_id = tracker.track_community_detection(
graph=workflow_graph,
communities=result['communities'],
method=method,
parameters={},
detection_time=detection_time,
source=workflow_id
)
assert comm_id is not None
assert comm_id.startswith('community_')
# Step 2: Track individual communities
for i, community in enumerate(result['communities']):
comm_result_id = tracker.track_community_result(
community_id=i,
nodes=community,
method=method,
execution_id=comm_id,
source=workflow_id
)
assert comm_result_id is not None
# Step 3: Track community quality metrics
quality_metrics = {
'modularity': 0.3,
'num_communities': len(result['communities']),
'avg_community_size': len(result['communities']) / len(result['communities']) if result['communities'] else 0
}
quality_id = tracker.track_community_quality(
execution_id=comm_id,
metrics=quality_metrics,
method=method,
source=workflow_id
)
assert quality_id is not None
except Exception as e:
print(f"Warning: {method} community detection failed: {e}")
print(f"Community detection workflow completed: {workflow_id}")
return workflow_id
except Exception:
if method == 'label_propagation':
raise
continue
comm_id = tracker.track_community_detection(
graph=workflow_graph,
communities=result['communities'],
method=method,
parameters={},
detection_time=0.0,
source=workflow_id
)
_assert_tracked(
tracker,
comm_id,
source=workflow_id,
method=method,
communities_count=len(result['communities']),
)
assert comm_id.startswith('community_')
for i, community in enumerate(result['communities']):
comm_record = _stored(tracker, f"community_{comm_id}_{i}")
assert comm_record['metadata']['community_id'] == i
assert comm_record['metadata']['method'] == method
assert comm_record['metadata']['nodes'] == community
tracked_methods.append(method)
assert 'label_propagation' in tracked_methods
def test_comprehensive_provenance_workflow(self, workflow_data, workflow_graph, workflow_embeddings):
"""Test comprehensive provenance workflow combining all algorithms."""
@@ -555,29 +640,37 @@ class TestProvenanceWorkflows:
source='comprehensive_test'
)
# Verify all execution IDs
# Verify all execution IDs were stored with the expected payload
assert len(execution_ids) == 6
expected_types = {
'construction': ('graph_construction_', 'graph_construction'),
'embedding': ('embedding_', 'embedding_computation'),
'similarity': ('similarity_', 'similarity_calculation'),
'link_prediction': ('link_prediction_', 'link_prediction'),
'centrality': ('centrality_', 'centrality_calculation'),
'community_detection': ('community_', 'community_detection'),
}
for phase, exec_id in execution_ids.items():
assert exec_id is not None
assert len(exec_id) > 10
# Verify all IDs are unique
prefix, entity_type = expected_types[phase]
assert exec_id.startswith(prefix)
_assert_tracked(
tracker,
exec_id,
source=master_workflow_id,
entity_type=entity_type,
)
summary_record = _assert_tracked(
tracker,
summary_id,
source='comprehensive_test',
entity_type='workflow_summary',
)
assert summary_id.startswith('workflow_summary_')
assert summary_record['metadata']['master_workflow_id'] == master_workflow_id
all_ids = list(execution_ids.values()) + [summary_id]
assert len(set(all_ids)) == len(all_ids)
# Verify ID prefixes
assert execution_ids['construction'].startswith('graph_construction_')
assert execution_ids['embedding'].startswith('embedding_')
assert execution_ids['similarity'].startswith('similarity_')
assert execution_ids['link_prediction'].startswith('link_prediction_')
assert execution_ids['centrality'].startswith('centrality_')
assert execution_ids['community_detection'].startswith('community_')
assert summary_id.startswith('workflow_summary_')
print(f"Comprehensive provenance workflow completed: {master_workflow_id}")
print(f"Execution IDs: {list(execution_ids.keys())}")
return master_workflow_id
def test_provenance_data_integrity(self, workflow_graph, workflow_embeddings):
"""Test provenance data integrity and consistency."""
@@ -618,24 +711,24 @@ class TestProvenanceWorkflows:
)
operations.append(('link_prediction', link_id))
# Verify data integrity
expected = {
'embedding': ('embedding_', 'embedding_computation', embed_id),
'similarity': ('similarity_', 'similarity_calculation', sim_id),
'link_prediction': ('link_prediction_', 'link_prediction', link_id),
}
for op_type, op_id in operations:
assert op_id is not None
assert len(op_id) > 10
# Verify ID format consistency
if op_type == 'embedding':
assert op_id.startswith('embedding_')
elif op_type == 'similarity':
assert op_id.startswith('similarity_')
elif op_type == 'link_prediction':
assert op_id.startswith('link_prediction_')
# Verify workflow consistency
prefix, entity_type, expected_id = expected[op_type]
assert op_id == expected_id
assert op_id.startswith(prefix)
_assert_tracked(
tracker,
op_id,
source=workflow_id,
entity_type=entity_type,
)
workflow_ids = [op_id for _, op_id in operations]
assert len(set(workflow_ids)) == len(workflow_ids) # All unique
print(f"Provenance data integrity test completed: {workflow_id}")
assert len(set(workflow_ids)) == len(workflow_ids)
def test_provenance_error_recovery(self):
"""Test provenance system error recovery."""
@@ -650,24 +743,27 @@ class TestProvenanceWorkflows:
)
assert result is None # Should return None when provenance is disabled
assert tracker_no_prov._prov_manager is None
# Test error handling during tracking
# Tracking still records an execution when the graph is None — that is
# current production behavior, so assert the write rather than
# "either returns or raises".
tracker = AlgorithmTrackerWithProvenance(provenance=True)
# This should not raise exceptions even with invalid data
try:
result = tracker.track_embedding_computation(
graph=None, # Invalid graph
algorithm='test',
embeddings={},
parameters={}
)
# Should either return None or handle gracefully
except Exception as e:
# If it raises, it should be a controlled exception
assert isinstance(e, (ValueError, TypeError))
print("Provenance error recovery test completed")
result = tracker.track_embedding_computation(
graph=None,
algorithm='test',
embeddings={},
parameters={},
source='error_recovery'
)
_assert_tracked(
tracker,
result,
source='error_recovery',
algorithm='test',
input_data_type='NoneType',
node_count=0,
)
if __name__ == '__main__':
+151 -95
View File
@@ -7,13 +7,13 @@ Tests provenance tracking workflows using only available methods.
import pytest
import networkx as nx
import time
from datetime import datetime, timedelta
from typing import Dict, List, Any
import uuid
from semantica.kg import (
GraphBuilderWithProvenance,
AlgorithmTrackerWithProvenance,
NodeEmbedder,
SimilarityCalculator,
LinkPredictor,
CentralityCalculator,
@@ -21,6 +21,40 @@ from semantica.kg import (
)
def _stored(owner, entity_id):
"""Read a provenance record back. Fails if tracking only minted an ID."""
record = owner._prov_manager.get_provenance(entity_id)
assert record is not None, (
f"no stored provenance for {entity_id!r} — an ID was generated without a write"
)
assert record.get("entity_id") == entity_id
return record
def _assert_utc_iso(value, field="timestamp"):
assert value, f"missing {field}"
parsed = datetime.fromisoformat(value)
assert parsed.tzinfo is not None, (
f"{field}={value!r} is naive (datetime.utcnow leftover)"
)
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
return parsed
def _assert_tracked(owner, entity_id, source, **metadata):
record = _stored(owner, entity_id)
assert record["source_document"] == source
actual = record.get("metadata") or {}
for key, expected in metadata.items():
assert actual.get(key) == expected, (
f"{entity_id} metadata[{key!r}]={actual.get(key)!r}, expected {expected!r}"
)
_assert_utc_iso(record["timestamp"], "timestamp")
if record.get("last_updated"):
_assert_utc_iso(record["last_updated"], "last_updated")
return record
class TestProvenanceWorkflowsSimple:
"""Test provenance tracking workflows with available methods."""
@@ -93,11 +127,19 @@ class TestProvenanceWorkflowsSimple:
source=workflow_id
)
assert embed_id is not None
_assert_tracked(
tracker,
embed_id,
source=workflow_id,
algorithm='node2vec',
node_count=6,
embedding_dimension=4,
)
assert embed_id.startswith('embedding_')
print(f"Simple embedding workflow completed: {workflow_id}")
return workflow_id
for node_id in workflow_embeddings:
node_record = _stored(tracker, f"embedding_{node_id}")
assert node_record['metadata']['execution_id'] == embed_id
assert node_record['metadata']['node_id'] == node_id
def test_similarity_workflow_simple(self, workflow_embeddings):
"""Test simple similarity workflow with provenance."""
@@ -124,11 +166,15 @@ class TestProvenanceWorkflowsSimple:
source=workflow_id
)
assert sim_id is not None
_assert_tracked(
tracker,
sim_id,
source=workflow_id,
method='cosine',
similarities_count=len(similarities),
)
assert sim_id.startswith('similarity_')
print(f"Simple similarity workflow completed: {workflow_id}")
return workflow_id
assert len(similarities) == 3
def test_link_prediction_workflow_simple(self, workflow_graph):
"""Test simple link prediction workflow with provenance."""
@@ -153,11 +199,15 @@ class TestProvenanceWorkflowsSimple:
source=workflow_id
)
assert link_id is not None
_assert_tracked(
tracker,
link_id,
source=workflow_id,
method='preferential_attachment',
predictions_count=len(predictions),
)
assert link_id.startswith('link_prediction_')
print(f"Simple link prediction workflow completed: {workflow_id}")
return workflow_id
assert len(predictions) >= 1
def test_centrality_workflow_simple(self, workflow_graph):
"""Test simple centrality workflow with provenance."""
@@ -184,11 +234,15 @@ class TestProvenanceWorkflowsSimple:
source=workflow_id
)
assert cent_id is not None
_assert_tracked(
tracker,
cent_id,
source=workflow_id,
method='degree',
scores_count=len(degree_cent['centrality']),
)
assert cent_id.startswith('centrality_')
print(f"Simple centrality workflow completed: {workflow_id}")
return workflow_id
assert set(degree_cent['centrality']) == set(workflow_graph.nodes())
def test_community_detection_workflow_simple(self, workflow_graph):
"""Test simple community detection workflow with provenance."""
@@ -215,11 +269,15 @@ class TestProvenanceWorkflowsSimple:
source=workflow_id
)
assert comm_id is not None
_assert_tracked(
tracker,
comm_id,
source=workflow_id,
method='label_propagation',
communities_count=len(communities['communities']),
)
assert comm_id.startswith('community_')
print(f"Simple community detection workflow completed: {workflow_id}")
return workflow_id
assert len(communities['communities']) >= 1
def test_graph_construction_workflow_simple(self, workflow_data):
"""Test simple graph construction workflow with provenance."""
@@ -237,23 +295,27 @@ class TestProvenanceWorkflowsSimple:
assert len(graph_result['entities']) == 4
assert len(graph_result['relationships']) == 3
# Track graph construction using embedding computation method as proxy
construction_id = tracker.track_embedding_computation(
graph={'nodes': list(graph_result['entities']), 'edges': list(graph_result['relationships'])},
algorithm='graph_construction',
embeddings={'graph_size': len(graph_result['entities'])},
parameters={
'entities_count': len(graph_result['entities']),
'relationships_count': len(graph_result['relationships'])
},
construction_id = tracker.track_graph_construction(
input_data=workflow_data,
output_graph=graph_result,
entities_count=len(graph_result['entities']),
relationships_count=len(graph_result['relationships']),
construction_time=0.0,
source=workflow_id
)
assert construction_id is not None
assert construction_id.startswith('embedding_') # Using embedding method as proxy
print(f"Simple graph construction workflow completed: {workflow_id}")
return workflow_id
_assert_tracked(
tracker,
construction_id,
source=workflow_id,
entity_type='graph_construction',
entities_count=4,
relationships_count=3,
)
assert construction_id.startswith('graph_construction_')
for entity in graph_result['entities']:
built = _stored(builder, entity['id'])
assert built['metadata']['operation'] == 'build_entity'
_assert_utc_iso(built['activity_started_at_time'], 'activity_started_at_time')
def test_comprehensive_workflow_simple(self, workflow_data, workflow_graph, workflow_embeddings):
"""Test comprehensive workflow with all available methods."""
@@ -271,11 +333,12 @@ class TestProvenanceWorkflowsSimple:
# Phase 1: Graph Construction
graph_result = builder.build_single_source(workflow_data)
construction_id = tracker.track_embedding_computation(
graph={'nodes': list(graph_result['entities']), 'edges': list(graph_result['relationships'])},
algorithm='graph_construction',
embeddings={'graph_size': len(graph_result['entities'])},
parameters={'entities_count': len(graph_result['entities'])},
construction_id = tracker.track_graph_construction(
input_data=workflow_data,
output_graph=graph_result,
entities_count=len(graph_result['entities']),
relationships_count=len(graph_result['relationships']),
construction_time=0.0,
source=master_workflow_id
)
execution_ids['construction'] = construction_id
@@ -351,27 +414,25 @@ class TestProvenanceWorkflowsSimple:
)
execution_ids['community_detection'] = comm_id
# Verify all execution IDs
expected = {
'construction': ('graph_construction_', 'graph_construction'),
'embedding': ('embedding_', 'embedding_computation'),
'similarity': ('similarity_', 'similarity_calculation'),
'link_prediction': ('link_prediction_', 'link_prediction'),
'centrality': ('centrality_', 'centrality_calculation'),
'community_detection': ('community_', 'community_detection'),
}
assert len(execution_ids) == 6
for phase, exec_id in execution_ids.items():
assert exec_id is not None
assert len(exec_id) > 10
# Verify all IDs are unique
all_ids = list(execution_ids.values())
assert len(set(all_ids)) == len(all_ids)
# Verify ID prefixes
assert execution_ids['embedding'].startswith('embedding_')
assert execution_ids['similarity'].startswith('similarity_')
assert execution_ids['link_prediction'].startswith('link_prediction_')
assert execution_ids['centrality'].startswith('centrality_')
assert execution_ids['community_detection'].startswith('community_')
print(f"Comprehensive workflow completed: {master_workflow_id}")
print(f"Execution IDs: {list(execution_ids.keys())}")
return master_workflow_id
prefix, entity_type = expected[phase]
assert exec_id.startswith(prefix)
_assert_tracked(
tracker,
exec_id,
source=master_workflow_id,
entity_type=entity_type,
)
assert len(set(execution_ids.values())) == len(execution_ids)
def test_provenance_data_integrity_simple(self, workflow_graph, workflow_embeddings):
"""Test provenance data integrity with available methods."""
@@ -412,24 +473,19 @@ class TestProvenanceWorkflowsSimple:
)
operations.append(('link_prediction', link_id))
# Verify data integrity
expected = {
'embedding': 'embedding_computation',
'similarity': 'similarity_calculation',
'link_prediction': 'link_prediction',
}
for op_type, op_id in operations:
assert op_id is not None
assert len(op_id) > 10
# Verify ID format consistency
if op_type == 'embedding':
assert op_id.startswith('embedding_')
elif op_type == 'similarity':
assert op_id.startswith('similarity_')
elif op_type == 'link_prediction':
assert op_id.startswith('link_prediction_')
# Verify workflow consistency
workflow_ids = [op_id for _, op_id in operations]
assert len(set(workflow_ids)) == len(workflow_ids) # All unique
print(f"Provenance data integrity test completed: {workflow_id}")
_assert_tracked(
tracker,
op_id,
source=workflow_id,
entity_type=expected[op_type],
)
assert len(set(op_id for _, op_id in operations)) == 3
def test_provenance_error_recovery_simple(self):
"""Test provenance system error recovery."""
@@ -443,25 +499,25 @@ class TestProvenanceWorkflowsSimple:
parameters={}
)
assert result is None # Should return None when provenance is disabled
# Test error handling during tracking
assert result is None
assert tracker_no_prov._prov_manager is None
tracker = AlgorithmTrackerWithProvenance(provenance=True)
# This should not raise exceptions even with invalid data
try:
result = tracker.track_embedding_computation(
graph=None, # Invalid graph
algorithm='test',
embeddings={},
parameters={}
)
# Should either return None or handle gracefully
except Exception as e:
# If it raises, it should be a controlled exception
assert isinstance(e, (ValueError, TypeError))
print("Provenance error recovery test completed")
result = tracker.track_embedding_computation(
graph=None,
algorithm='test',
embeddings={},
parameters={},
source='error_recovery'
)
_assert_tracked(
tracker,
result,
source='error_recovery',
algorithm='test',
input_data_type='NoneType',
node_count=0,
)
if __name__ == '__main__':
@@ -0,0 +1,41 @@
from semantica.ontology.class_inferrer import ClassInferrer
from semantica.ontology.ontology_generator import OntologyGenerator
from semantica.ontology.property_generator import PropertyGenerator
def _entities():
return [
{
"id": "e1",
"type": "software engineer",
"name": "Alice",
"email": "alice@example.org",
},
{
"id": "e2",
"type": "software engineer",
"name": "Bob",
"email": "bob@example.org",
},
]
def test_property_generator_matches_normalized_class_names():
entities = _entities()
classes = ClassInferrer().infer_classes(entities)
properties = PropertyGenerator().infer_properties(entities, [], classes)
email = next(prop for prop in properties if prop["name"] == "email")
assert email["domain"] == ["SoftwareEngineer"]
assert email["range"] == "xsd:string"
def test_ontology_pipeline_emits_data_properties_for_normalized_types():
ontology = OntologyGenerator().generate_ontology(
{"entities": _entities(), "relationships": []}
)
email = next(prop for prop in ontology["properties"] if prop["name"] == "email")
assert email["domain"] == ["SoftwareEngineer"]
assert email["range"] == "xsd:string"
+10 -6
View File
@@ -4,6 +4,7 @@ import tempfile
import os
import json
import csv
import sys
from pathlib import Path
import pytest
@@ -137,29 +138,32 @@ class TestParseComprehensive(unittest.TestCase):
# --- Document Parser Tests ---
@patch('semantica.parse.pdf_parser.pdfplumber')
def test_pdf_parser(self, mock_pdfplumber):
def test_pdf_parser(self):
parser = PDFParser()
# pdfplumber is imported inside PDFParser.parse, so inject a fake
# module into sys.modules instead of patching a module attribute
mock_pdfplumber = MagicMock()
mock_pdf = MagicMock()
mock_page = MagicMock()
mock_page.extract_text.return_value = "Page text"
mock_pdf.pages = [mock_page]
# Ensure metadata is a dict, not a property object if that's an issue
mock_pdf.metadata = {"Title": "Test PDF"}
# Setup the context manager
mock_context_manager = MagicMock()
mock_context_manager.__enter__.return_value = mock_pdf
mock_context_manager.__exit__.return_value = None
mock_pdfplumber.open.return_value = mock_context_manager
# We don't need a real file if we mock open, but the parser likely checks file existence
with tempfile.NamedTemporaryFile(mode='wb', delete=False, suffix='.pdf') as tmp:
tmp.write(b"dummy pdf content")
tmp_path = tmp.name
try:
result = parser.parse(tmp_path)
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
result = parser.parse(tmp_path)
# Returns dict with full_text
self.assertIn("Page text", result["full_text"])
self.assertEqual(result["metadata"].get("title"), "Test PDF")
+145
View File
@@ -0,0 +1,145 @@
"""
Tests for scanned-PDF (empty text layer) detection in PDFParser.
Covers:
- Warning when all parsed pages lack a text layer (scanned PDFs)
- No warning when text is present or text extraction is disabled
"""
import os
import sys
import tempfile
import unittest
from unittest.mock import MagicMock, patch
from semantica.parse.pdf_parser import PDFParser
class TestScannedPdfWarning(unittest.TestCase):
"""PDFParser warns when a PDF has pages but no extractable text."""
def setUp(self):
self.mock_logger = MagicMock()
logger_patcher = patch(
"semantica.parse.pdf_parser.get_logger", return_value=self.mock_logger
)
tracker_patcher = patch("semantica.parse.pdf_parser.get_progress_tracker")
logger_patcher.start()
tracker_patcher.start()
self.addCleanup(logger_patcher.stop)
self.addCleanup(tracker_patcher.stop)
def _make_pdf_path(self):
with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=".pdf") as tmp:
tmp.write(b"dummy pdf content")
return tmp.name
def _mock_pdfplumber(self, page_text):
"""Build a fake pdfplumber module whose single page yields page_text."""
mock_page = MagicMock()
mock_page.extract_text.return_value = page_text
mock_page.extract_tables.return_value = []
mock_page.width = 612
mock_page.height = 792
mock_pdf = MagicMock()
mock_pdf.pages = [mock_page]
mock_pdf.metadata = {}
mock_pdfplumber = MagicMock()
context_manager = MagicMock()
context_manager.__enter__.return_value = mock_pdf
mock_pdfplumber.open.return_value = context_manager
return mock_pdfplumber
def test_warns_when_no_text_layer(self):
mock_pdfplumber = self._mock_pdfplumber(page_text=None) # scanned page
parser = PDFParser()
path = self._make_pdf_path()
try:
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
result = parser.parse(path)
self.assertEqual(result["full_text"], "")
self.mock_logger.warning.assert_called_once()
message = self.mock_logger.warning.call_args[0][0]
self.assertIn("scanned", message)
self.assertIn("enable_ocr", message)
finally:
os.unlink(path)
def test_no_warning_when_text_present(self):
mock_pdfplumber = self._mock_pdfplumber(page_text="Real digital text")
parser = PDFParser()
path = self._make_pdf_path()
try:
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
result = parser.parse(path)
self.assertIn("Real digital text", result["full_text"])
self.mock_logger.warning.assert_not_called()
finally:
os.unlink(path)
def test_no_warning_when_extract_text_disabled(self):
mock_pdfplumber = self._mock_pdfplumber(page_text=None)
parser = PDFParser()
path = self._make_pdf_path()
try:
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
result = parser.parse(path, extract_text=False, extract_tables=False)
self.assertEqual(result["full_text"], "")
self.mock_logger.warning.assert_not_called()
finally:
os.unlink(path)
def test_warns_when_pages_have_only_whitespace(self):
"""Pages that return only whitespace count as no extractable text."""
mock_pdfplumber = self._mock_pdfplumber(page_text=" \n\t ")
parser = PDFParser()
path = self._make_pdf_path()
try:
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
parser.parse(path)
self.mock_logger.warning.assert_called_once()
message = self.mock_logger.warning.call_args[0][0]
self.assertIn("scanned", message)
self.assertIn("enable_ocr", message)
finally:
os.unlink(path)
def test_no_warning_for_mixed_pdf(self):
"""A PDF where at least one page has real text must not warn."""
# Build a two-page PDF: page 1 has text, page 2 is image-only
mock_page_text = MagicMock()
mock_page_text.extract_text.return_value = "Actual content"
mock_page_text.extract_tables.return_value = []
mock_page_text.width = 612
mock_page_text.height = 792
mock_page_image = MagicMock()
mock_page_image.extract_text.return_value = None
mock_page_image.extract_tables.return_value = []
mock_page_image.width = 612
mock_page_image.height = 792
mock_pdf = MagicMock()
mock_pdf.pages = [mock_page_text, mock_page_image]
mock_pdf.metadata = {}
mock_pdfplumber = MagicMock()
context_manager = MagicMock()
context_manager.__enter__.return_value = mock_pdf
mock_pdfplumber.open.return_value = context_manager
parser = PDFParser()
path = self._make_pdf_path()
try:
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
result = parser.parse(path)
self.assertIn("Actual content", result["full_text"])
self.mock_logger.warning.assert_not_called()
finally:
os.unlink(path)
if __name__ == "__main__":
unittest.main()
+88 -1
View File
@@ -66,6 +66,93 @@ class TestPipelineModule(unittest.TestCase):
self.assertEqual(result.output, 12) # (5 + 1) * 2 = 12
self.assertEqual(pipeline.steps[0].status, StepStatus.COMPLETED)
def test_registered_step_handler_executes(self):
"""A handler registered by step type should execute."""
def increment(data):
return data + 1
builder = PipelineBuilder()
builder.register_step_handler("math", increment)
builder.add_step("increment", "math")
result = ExecutionEngine().execute_pipeline(
builder.build("registered"), data=1
)
self.assertTrue(result.success)
self.assertEqual(result.output, 2)
def test_explicit_handler_does_not_receive_control_fields(self):
"""Builder-only fields should not be passed to strict handlers."""
def source(data):
return data
def increment(data, amount):
return data + amount
builder = PipelineBuilder()
builder.add_step("source", "source", handler=source)
step = builder.add_step(
"increment",
"math",
handler=increment,
dependencies=["source"],
amount=2,
)
result = ExecutionEngine().execute_pipeline(
builder.build("strict"), data=1
)
self.assertTrue(result.success)
self.assertEqual(result.output, 3)
self.assertEqual(step.config, {"amount": 2})
def test_explicit_handler_overrides_registered_handler(self):
"""An explicit step handler should take precedence over the registry."""
builder = PipelineBuilder()
builder.register_step_handler("math", lambda data: data + 100)
builder.add_step("increment", "math", handler=lambda data: data + 1)
result = ExecutionEngine().execute_pipeline(
builder.build("override"), data=1
)
self.assertTrue(result.success)
self.assertEqual(result.output, 2)
def test_step_without_handler_passes_input_through(self):
"""A step with no explicit or registered handler should be a no-op."""
builder = PipelineBuilder()
builder.add_step("passthrough", "unregistered")
result = ExecutionEngine().execute_pipeline(
builder.build("handlerless"), data={"value": 1}
)
self.assertTrue(result.success)
self.assertEqual(result.output, {"value": 1})
def test_falsy_explicit_handler_is_invoked(self):
"""A handler whose __bool__ is False must still be dispatched."""
class FalseyHandler:
def __bool__(self):
return False
def __call__(self, data):
return {"explicit": data}
builder = PipelineBuilder()
builder.add_step("step1", "mytype", handler=FalseyHandler())
result = ExecutionEngine().execute_pipeline(
builder.build("falsy"), data=1
)
self.assertTrue(result.success)
self.assertEqual(result.output, {"explicit": 1})
def test_execution_engine_failure(self):
"""Test pipeline failure handling."""
def failing_handler(data, **kwargs):
@@ -112,8 +199,8 @@ class TestPipelineModule(unittest.TestCase):
_ = semantica.pipeline
from semantica.pipeline import PipelineBuilder, PipelineValidator
from semantica.deduplication import DuplicateDetector
from semantica.pipeline import PipelineBuilder, PipelineValidator
builder = PipelineBuilder()
builder.add_step("step1", "dummy")
+196 -3
View File
@@ -6,8 +6,9 @@ chunk tracking, source tracking, and lineage tracing.
"""
import pytest
import warnings
from unittest.mock import patch
from datetime import datetime
from datetime import datetime, timedelta, timezone
from semantica.provenance import ProvenanceManager, SourceReference, ProvenanceEntry
from semantica.provenance.storage import InMemoryStorage, SQLiteStorage
@@ -733,8 +734,8 @@ class TestProvenanceManager:
entity_type="entity",
activity_id="test",
source_document="doc_1",
first_seen=datetime.utcnow().isoformat(),
last_updated=datetime.utcnow().isoformat(),
first_seen=datetime.now(timezone.utc).isoformat(),
last_updated=datetime.now(timezone.utc).isoformat(),
)
with patch.object(prov_mgr.storage, "store", side_effect=RuntimeError("storage error")), \
patch.object(prov_mgr.logger, "error") as mock_log_error:
@@ -1672,3 +1673,195 @@ class TestActivityTimingAcrossWrappers:
assert entry["activity_ended_at_time"] is not None
class TestTimezoneAwareUtcTimestamps:
"""Issue #946 — timezone-aware UTC stamps without datetime.utcnow()."""
def test_track_entity_stamps_timezone_aware_utc(self):
before = datetime.now(timezone.utc) - timedelta(seconds=1)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always", DeprecationWarning)
prov_mgr = ProvenanceManager()
entry = prov_mgr.track_entity("utc_entity", source="doc_1")
after = datetime.now(timezone.utc) + timedelta(seconds=1)
utcnow_warnings = [
w
for w in caught
if issubclass(w.category, DeprecationWarning)
and "utcnow" in str(w.message).lower()
]
assert utcnow_warnings == []
for field in ("timestamp", "first_seen", "last_updated"):
value = getattr(entry, field)
parsed = datetime.fromisoformat(value)
assert parsed.tzinfo is not None, f"{field}={value!r} is naive"
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
assert before <= parsed <= after
stored = prov_mgr.get_provenance("utc_entity")
assert stored is not None
assert stored["source_document"] == "doc_1"
assert stored["last_updated"] == entry.last_updated
def test_invalidate_stamps_timezone_aware_utc(self):
prov_mgr = ProvenanceManager()
prov_mgr.track_entity("utc_invalidate", source="doc_1")
result = prov_mgr.invalidate(
"utc_invalidate", agent_id="reviewer", reason="test"
)
parsed = datetime.fromisoformat(result.invalidated_at_time)
assert parsed.tzinfo is not None
assert parsed.utcoffset() == timedelta(0)
def test_graph_builder_activity_times_are_timezone_aware_utc(self):
from semantica.kg.kg_provenance import GraphBuilderWithProvenance
builder = GraphBuilderWithProvenance(provenance=True, agent_id="builder_svc")
result = builder.build_single_source({
"entities": [{"id": "person1", "type": "Person", "name": "Ada"}],
"relationships": [],
})
assert result["entities"][0]["id"] == "person1"
person = builder._prov_manager.get_provenance("person1")
assert person is not None
assert person["metadata"]["operation"] == "build_entity"
for field in ("activity_started_at_time", "activity_ended_at_time"):
parsed = datetime.fromisoformat(person[field])
assert parsed.tzinfo is not None, (
f"{field}={person[field]!r} is naive"
)
assert parsed.utcoffset() == timedelta(0)
started = datetime.fromisoformat(person["activity_started_at_time"])
ended = datetime.fromisoformat(person["activity_ended_at_time"])
assert started <= ended
class TestMixedFormatTimestampComparisons:
"""Issue #946 review — naive and offset-bearing stamps must compare as instants.
Records written before the timezone-aware change carry a naive UTC stamp
(``2026-08-19T22:35:38.501697``); records written after carry ``+00:00``.
Comparing the two as raw strings puts the offset-bearing form above a naive
bound at the identical instant, so the record falls outside a range that
should contain it. These tests pin the boundary in both directions.
"""
NAIVE = "2026-08-19T12:00:00.500000"
AWARE = "2026-08-19T12:00:00.500000+00:00"
def _manager_with(self, *timestamps):
prov_mgr = ProvenanceManager()
for index, stamp in enumerate(timestamps):
prov_mgr.storage.store(
ProvenanceEntry(
entity_id=f"entity_{index}",
entity_type="entity",
activity_id="test",
source_document="doc_1",
timestamp=stamp,
)
)
return prov_mgr
def test_raw_string_compare_would_exclude_the_boundary_record(self):
"""Guards the premise: the two forms are not string-comparable."""
assert not (self.AWARE <= self.NAIVE)
assert datetime.fromisoformat(self.AWARE) == datetime.fromisoformat(
self.NAIVE
).replace(tzinfo=timezone.utc)
def test_query_range_with_naive_bounds_includes_aware_record(self):
prov_mgr = self._manager_with(self.AWARE)
results = prov_mgr.query_recorded_between(
"2026-08-19T12:00:00.500000", "2026-08-19T12:00:00.500000"
)
assert [r["entity_id"] for r in results] == ["entity_0"]
def test_query_range_with_aware_bounds_includes_naive_record(self):
prov_mgr = self._manager_with(self.NAIVE)
results = prov_mgr.query_recorded_between(
"2026-08-19T12:00:00.500000+00:00", "2026-08-19T12:00:00.500000+00:00"
)
assert [r["entity_id"] for r in results] == ["entity_0"]
def test_query_range_returns_both_formats_from_a_mixed_store(self):
prov_mgr = self._manager_with(self.NAIVE, self.AWARE)
results = prov_mgr.query_recorded_between(
"2026-08-19T11:00:00", "2026-08-19T13:00:00+00:00"
)
assert {r["entity_id"] for r in results} == {"entity_0", "entity_1"}
def test_query_range_sorts_mixed_formats_chronologically(self):
prov_mgr = self._manager_with(
"2026-08-19T12:00:02+00:00", # entity_0, latest
"2026-08-19T12:00:00", # entity_1, earliest
"2026-08-19T12:00:01+00:00", # entity_2, middle
)
results = prov_mgr.query_recorded_between(
"2026-08-19T11:00:00", "2026-08-19T13:00:00"
)
assert [r["entity_id"] for r in results] == [
"entity_1",
"entity_2",
"entity_0",
]
def test_query_range_accepts_trailing_z(self):
prov_mgr = self._manager_with(self.NAIVE)
results = prov_mgr.query_recorded_between(
"2026-08-19T11:00:00Z", "2026-08-19T13:00:00Z"
)
assert len(results) == 1
def test_query_range_skips_unparseable_timestamp(self):
prov_mgr = self._manager_with("not-a-timestamp", self.AWARE)
results = prov_mgr.query_recorded_between(
"2026-08-19T11:00:00", "2026-08-19T13:00:00"
)
assert [r["entity_id"] for r in results] == ["entity_1"]
def test_audit_log_since_naive_bound_includes_aware_record(self):
prov_mgr = self._manager_with(self.AWARE)
entries = prov_mgr.audit_log(
since="2026-08-19T12:00:00.500000", format="json"
)
assert [e["entity_id"] for e in entries] == ["entity_0"]
def test_audit_log_since_aware_bound_includes_naive_record(self):
prov_mgr = self._manager_with(self.NAIVE)
entries = prov_mgr.audit_log(
since="2026-08-19T12:00:00.500000+00:00", format="json"
)
assert [e["entity_id"] for e in entries] == ["entity_0"]
def test_audit_log_excludes_records_before_since_across_formats(self):
prov_mgr = self._manager_with(
"2026-08-19T11:59:59", # entity_0, before
"2026-08-19T12:00:01+00:00", # entity_1, after
)
entries = prov_mgr.audit_log(since="2026-08-19T12:00:00+00:00", format="json")
assert [e["entity_id"] for e in entries] == ["entity_1"]
def test_audit_log_sorts_mixed_formats_chronologically(self):
prov_mgr = self._manager_with(
"2026-08-19T12:00:02+00:00",
"2026-08-19T12:00:00",
"2026-08-19T12:00:01+00:00",
)
entries = prov_mgr.audit_log(format="json")
assert [e["entity_id"] for e in entries] == [
"entity_1",
"entity_2",
"entity_0",
]
def test_audit_log_unparseable_timestamp_sorts_first(self):
prov_mgr = self._manager_with("not-a-timestamp", "2026-08-19T12:00:00+00:00")
entries = prov_mgr.audit_log(format="json")
assert [e["entity_id"] for e in entries] == ["entity_0", "entity_1"]
def test_audit_log_since_excludes_unparseable_timestamp(self):
prov_mgr = self._manager_with("not-a-timestamp", "2026-08-19T12:00:00+00:00")
entries = prov_mgr.audit_log(since="2026-08-19T11:00:00", format="json")
assert [e["entity_id"] for e in entries] == ["entity_1"]
+531
View File
@@ -0,0 +1,531 @@
"""Tests for rule-driven actions (production-rule behaviour) on the Reasoner.
Covers the L1 Action layer (Assert/Retract/Call/Emit), provenance logging of
fired actions (L2), and backward compatibility with the legacy Rule.handler
callback.
"""
import unittest
from collections import UserDict
from semantica.reasoning import (
AssertAction,
CallAction,
EmitEventAction,
Fact,
Match,
Reasoner,
ReteEngine,
RetractAction,
)
class TestRuleActions(unittest.TestCase):
def setUp(self):
self.reasoner = Reasoner()
def _add_person_parent_facts(self):
self.reasoner.add_fact("Person(John)")
self.reasoner.add_fact("Parent(John, Jane)")
def test_assert_action_fires_and_substitutes_bindings(self):
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [AssertAction("Adult(?x)")]
self._add_person_parent_facts()
self.reasoner.forward_chain()
# Action-asserted fact uses the match bindings (?x -> John).
self.assertIn("Adult(John)", self.reasoner.facts)
def test_retract_action_removes_fact(self):
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [RetractAction("Person(?x)")]
self._add_person_parent_facts()
self.reasoner.forward_chain()
self.assertNotIn("Person(John)", self.reasoner.facts)
def test_call_action_invoked_with_bindings(self):
seen = {}
def record(bindings, reasoner):
seen.update(bindings)
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [CallAction(record, name="record")]
self._add_person_parent_facts()
self.reasoner.forward_chain()
self.assertEqual(seen.get("x"), "John")
self.assertEqual(seen.get("y"), "Jane")
def test_emit_event_action_delivers_to_sink(self):
events = []
self.reasoner.on_event(lambda name, payload: events.append((name, payload)))
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [EmitEventAction("child_derived:?y")]
self._add_person_parent_facts()
self.reasoner.forward_chain()
self.assertEqual(len(events), 1)
name, payload = events[0]
self.assertEqual(name, "child_derived:Jane")
self.assertEqual(payload["bindings"]["x"], "John")
def test_assert_action_write_back_to_knowledge_graph(self):
class FakeKG:
def __init__(self):
self.added = []
def add_fact(self, fact):
self.added.append(fact)
kg = FakeKG()
reasoner = Reasoner(knowledge_graph=kg)
rule = reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [AssertAction("Adult(?x)", write_back=True)]
reasoner.add_fact("Person(John)")
reasoner.add_fact("Parent(John, Jane)")
reasoner.forward_chain()
self.assertIn("Adult(John)", kg.added)
def test_provenance_logs_fired_actions(self):
reasoner = Reasoner(provenance=True)
rule = reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [AssertAction("Adult(?x)")]
reasoner.add_fact("Person(John)")
reasoner.add_fact("Parent(John, Jane)")
reasoner.forward_chain()
self.assertEqual(len(reasoner.action_log), 1)
entry = reasoner.action_log[0]
self.assertEqual(entry["action"], "AssertAction")
self.assertEqual(entry["rule_id"], rule.rule_id)
self.assertEqual(entry["bindings"]["x"], "John")
self.assertIn("Adult(John)", entry["description"])
def test_repeated_forward_chain_fires_same_activation_once(self):
calls = []
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [
CallAction(lambda bindings, reasoner: calls.append(dict(bindings)))
]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.reasoner.forward_chain()
self.assertEqual(calls, [{"x": "John"}])
def test_repeated_forward_chain_records_provenance_once(self):
reasoner = Reasoner(provenance=True)
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [AssertAction("Verified(?x)")]
reasoner.add_fact("Person(John)")
reasoner.forward_chain()
reasoner.forward_chain()
self.assertEqual(len(reasoner.action_log), 1)
def test_new_binding_creates_a_new_activation(self):
calls = []
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [
CallAction(lambda bindings, reasoner: calls.append(bindings["x"]))
]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.reasoner.add_fact("Person(Jane)")
self.reasoner.forward_chain()
self.assertCountEqual(calls, ["John", "Jane"])
def test_reset_action_history_allows_deliberate_replay(self):
calls = []
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("called"))]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.reasoner.reset_action_history()
self.reasoner.forward_chain()
self.assertEqual(calls, ["called", "called"])
def test_clear_resets_action_history(self):
calls = []
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("called"))]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.reasoner.clear()
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("called"))]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.assertEqual(calls, ["called", "called"])
def test_failed_action_is_not_retried_without_explicit_reset(self):
attempts = []
def fail(bindings, reasoner):
attempts.append(bindings["x"])
raise RuntimeError("boom")
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [CallAction(fail)]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.reasoner.forward_chain()
self.assertEqual(attempts, ["John"])
self.reasoner.reset_action_history()
self.reasoner.forward_chain()
self.assertEqual(attempts, ["John", "John"])
def test_replacing_actions_in_place_requires_explicit_history_reset(self):
calls = []
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("first"))]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("second"))]
self.reasoner.forward_chain()
self.assertEqual(calls, ["first"])
self.reasoner.reset_action_history()
self.reasoner.forward_chain()
self.assertEqual(calls, ["first", "second"])
def test_activation_is_recorded_before_reentrant_action_execution(self):
calls = []
def reenter(bindings, reasoner):
calls.append(bindings["x"])
if len(calls) == 1:
reasoner.forward_chain()
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [CallAction(reenter)]
self.reasoner.add_fact("Person(John)")
self.reasoner.forward_chain()
self.assertEqual(calls, ["John"])
def test_no_provenance_log_when_disabled(self):
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [AssertAction("Adult(?x)")]
self._add_person_parent_facts()
self.reasoner.forward_chain()
self.assertEqual(self.reasoner.action_log, [])
def test_legacy_handler_still_invoked(self):
calls = []
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.handler = lambda bindings, reasoner: calls.append(bindings)
self._add_person_parent_facts()
self.reasoner.forward_chain()
self.assertEqual(len(calls), 1)
self.assertEqual(calls[0]["x"], "John")
def test_action_error_does_not_break_chain(self):
def boom(bindings, reasoner):
raise RuntimeError("boom")
rule = self.reasoner.add_rule(
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
)
rule.actions = [CallAction(boom, name="boom"), AssertAction("Adult(?x)")]
self._add_person_parent_facts()
# A failing action is logged but must not abort the pass; the later
# action still runs and the conclusion is still derived.
self.reasoner.forward_chain()
self.assertIn("Adult(John)", self.reasoner.facts)
self.assertIn("Child(Jane, John)", self.reasoner.facts)
class TestRuleActionRegressions(unittest.TestCase):
"""Regression coverage for the qodo-flagged bugs on PR #1096."""
def test_variable_substitution_no_prefix_collision(self):
# bug7: naive str.replace of "?x" would also corrupt "?xy". A
# token-aware substitution must bind ?x and ?xy independently.
reasoner = Reasoner()
rule = reasoner.add_rule("IF Pair(?x, ?xy) THEN Linked(?x, ?xy)")
rule.actions = [AssertAction("Tag(?x, ?xy)")]
reasoner.add_fact("Pair(John, Johny)")
reasoner.forward_chain()
self.assertIn("Tag(John, Johny)", reasoner.facts)
def test_assert_write_back_to_canonical_knowledge_graph(self):
# bug1: a KG exposing only entities/relationships (no add_fact) must
# still receive the asserted fact via canonical translation.
from semantica.kg.knowledge_graph import KnowledgeGraph
kg = KnowledgeGraph()
reasoner = Reasoner(knowledge_graph=kg)
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [AssertAction("Adult(?x)", write_back=True)]
reasoner.add_fact("Person(John)")
reasoner.forward_chain()
# A single-argument fact lands as an entity node.
self.assertTrue(any("John" in str(e) for e in kg.entities))
def test_write_back_unsupported_target_raises(self):
# bug1: an unsupported write-back target must fail loudly, not silently.
from semantica.reasoning.reasoner import _write_fact_to_graph
with self.assertRaises(ValueError):
_write_fact_to_graph(object(), "Adult(John)")
def test_provenance_entry_has_timestamp(self):
# bug3: action_log entries must be structured with a timestamp.
reasoner = Reasoner(provenance=True)
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [AssertAction("Adult(?x)")]
reasoner.add_fact("Person(John)")
reasoner.forward_chain()
entry = reasoner.action_log[0]
self.assertIn("timestamp", entry)
self.assertTrue(entry["timestamp"])
def test_action_fires_even_when_conclusion_already_known(self):
# bug4: previously an activation whose conclusion was already known
# skipped firing its actions. Now it must still fire exactly once.
reasoner = Reasoner()
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [AssertAction("Verified(?x)")]
reasoner.add_fact("Person(John)")
# Conclusion already present before the pass runs.
reasoner.add_fact("Adult(John)")
reasoner.forward_chain()
self.assertIn("Verified(John)", reasoner.facts)
def test_retract_self_conclusion_terminates(self):
# bug5: a RetractAction removing its own premise previously re-fired
# every pass up to max_iterations. It must fire once and terminate.
reasoner = Reasoner()
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.actions = [RetractAction("Person(?x)")]
reasoner.add_fact("Person(John)")
# Should return promptly without exhausting iterations.
reasoner.forward_chain()
self.assertNotIn("Person(John)", reasoner.facts)
def test_infer_with_results_preserves_confidence(self):
# bug9: confidence must survive to the InferenceResult objects.
reasoner = Reasoner()
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
rule.confidence = 0.8
results = reasoner.infer_with_results(["Person(John)"])
self.assertTrue(results)
self.assertTrue(all(0.0 <= r.confidence <= 1.0 for r in results))
self.assertAlmostEqual(
min(r.confidence for r in results), 0.8, places=6
)
class TestReteActionExecution(unittest.TestCase):
def setUp(self):
self.calls = []
self.reasoner = Reasoner()
self.rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
self.rule.actions = [
CallAction(
lambda bindings, reasoner: self.calls.append(dict(bindings))
)
]
self.match = Match(
rule=self.rule,
facts=[Fact("person-1", "Person", ["John"])],
bindings={"x": "John"},
)
self.engine = ReteEngine(reasoner=self.reasoner)
def test_rete_repeated_execute_matches_fires_activation_once(self):
first_results = self.engine.execute_matches([self.match])
second_results = self.engine.execute_matches([self.match])
self.assertEqual(first_results, ["Adult(?x)"])
self.assertEqual(second_results, ["Adult(?x)"])
self.assertEqual(self.calls, [{"x": "John"}])
def test_rete_duplicate_match_preserves_results_but_fires_once(self):
results = self.engine.execute_matches([self.match, self.match])
self.assertEqual(results, ["Adult(?x)", "Adult(?x)"])
self.assertEqual(self.calls, [{"x": "John"}])
def test_rete_distinct_fact_ids_create_distinct_activations(self):
other_match = Match(
rule=self.rule,
facts=[Fact("person-2", "Person", ["John"])],
bindings={"x": "John"},
)
self.engine.execute_matches([self.match])
self.engine.execute_matches([other_match])
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
def test_rete_equivalent_nested_bindings_share_an_activation(self):
first_match = Match(
rule=self.rule,
facts=self.match.facts,
bindings={"x": {"a": 1, "b": 2}},
)
reordered_match = Match(
rule=self.rule,
facts=self.match.facts,
bindings={"x": {"b": 2, "a": 1}},
)
self.engine.execute_matches([first_match])
self.engine.execute_matches([reordered_match])
self.assertEqual(len(self.calls), 1)
def test_rete_structured_fact_identity_avoids_separator_collisions(self):
first_match = Match(
rule=self.rule,
facts=[Fact("a", "b:C", [])],
bindings={"x": "John"},
)
colliding_text_match = Match(
rule=self.rule,
facts=[Fact("a:b", "C", [])],
bindings={"x": "John"},
)
self.engine.execute_matches([first_match])
self.engine.execute_matches([colliding_text_match])
self.assertEqual(len(self.calls), 2)
def test_rete_cyclic_binding_preserves_results_and_deduplicates_actions(self):
cyclic_value = []
cyclic_value.append(cyclic_value)
cyclic_match = Match(
rule=self.rule,
facts=self.match.facts,
bindings={"x": cyclic_value},
)
first_results = self.engine.execute_matches([cyclic_match])
second_results = self.engine.execute_matches([cyclic_match])
self.assertEqual(first_results, ["Adult(?x)"])
self.assertEqual(second_results, ["Adult(?x)"])
self.assertEqual(len(self.calls), 1)
def test_rete_equivalent_mapping_implementations_share_an_activation(self):
first_match = Match(
rule=self.rule,
facts=self.match.facts,
bindings={"x": UserDict({"a": 1, "b": 2})},
)
reordered_match = Match(
rule=self.rule,
facts=self.match.facts,
bindings={"x": UserDict({"b": 2, "a": 1})},
)
self.engine.execute_matches([first_match])
self.engine.execute_matches([reordered_match])
self.assertEqual(len(self.calls), 1)
def test_rete_key_error_does_not_suppress_conclusion(self):
class UnrepresentableValue:
def __repr__(self):
raise RuntimeError("cannot represent")
invalid_match = Match(
rule=self.rule,
facts=self.match.facts,
bindings={"x": UnrepresentableValue()},
)
results = self.engine.execute_matches([invalid_match])
self.assertEqual(results, ["Adult(?x)"])
self.assertEqual(self.calls, [])
def test_rete_reset_action_history_allows_deliberate_replay(self):
self.engine.execute_matches([self.match])
self.engine.reset_action_history()
self.engine.execute_matches([self.match])
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
def test_rete_reset_allows_action_replay(self):
self.engine.execute_matches([self.match])
self.engine.reset()
self.engine.execute_matches([self.match])
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
def test_rete_build_network_allows_action_replay(self):
self.engine.execute_matches([self.match])
self.engine.build_network([self.rule])
self.engine.execute_matches([self.match])
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
if __name__ == "__main__":
unittest.main()
+24 -20
View File
@@ -216,14 +216,13 @@ class TestNERExtractorSpacyModelCache:
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
e1 = NERExtractor(method="ml")
e2 = NERExtractor(method="ml")
e3 = NERExtractor(method="ml", model="en_core_web_sm")
NERExtractor(method="ml")
NERExtractor(method="ml")
NERExtractor(method="ml", model="en_core_web_sm")
assert len(calls) == 1, (
"repeated NERExtractor constructions should not reload the model"
)
assert e1.nlp is e2.nlp is e3.nlp
def test_ner_extractor_and_split_callers_share_one_cached_model(self, monkeypatch):
"""NERExtractor, SemanticChunker, and split_by_sentences must all use
@@ -248,20 +247,23 @@ class TestNERExtractorSpacyModelCache:
def test_ner_extractor_distinct_model_names_load_separately(self, monkeypatch):
"""Different model names must produce separate cache entries."""
calls = []
loaded = {}
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
nlp = _nlp_mock()
loaded[name] = nlp
return nlp
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
sm = NERExtractor(method="ml", model="en_core_web_sm")
lg = NERExtractor(method="ml", model="en_core_web_lg")
sm_again = NERExtractor(method="ml", model="en_core_web_sm")
NERExtractor(method="ml", model="en_core_web_sm")
NERExtractor(method="ml", model="en_core_web_lg")
NERExtractor(method="ml", model="en_core_web_sm")
assert calls == ["en_core_web_sm", "en_core_web_lg"]
assert sm.nlp is sm_again.nlp
assert sm.nlp is not lg.nlp
# Same model name -> same cached Language object; different names -> different objects.
assert loaded["en_core_web_sm"] is not loaded["en_core_web_lg"]
def test_ner_extractor_failed_load_not_cached_and_retried(self, monkeypatch):
"""A missing model must not poison the cache. A subsequent construction
@@ -274,31 +276,33 @@ class TestNERExtractorSpacyModelCache:
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
# Construction with missing model: nlp must remain None, no crash
extractor1 = NERExtractor(method="ml")
assert extractor1.nlp is None
# Construction with missing model: must not raise
NERExtractor(method="ml")
assert len(attempts) == 1, "one load attempt expected for the missing model"
# Second construction: must retry (cache must not hold the failure)
extractor2 = NERExtractor(method="ml")
assert extractor2.nlp is None
NERExtractor(method="ml")
assert len(attempts) == 2, "a failed load must not be cached"
# Now install a working model and verify recovery
loaded_models = {}
def working_load(name, **_kwargs):
attempts.append(name)
return _nlp_mock()
nlp = _nlp_mock()
loaded_models[name] = nlp
return nlp
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
extractor3 = NERExtractor(method="ml")
extractor4 = NERExtractor(method="ml")
NERExtractor(method="ml")
NERExtractor(method="ml")
assert extractor3.nlp is not None
assert extractor3.nlp is extractor4.nlp
assert len(attempts) == 3, (
"exactly one successful load expected after the model becomes available"
)
# Confirm the recovered model is cached and shared across callers.
assert se_methods.load_spacy_model("en_core_web_sm") is loaded_models["en_core_web_sm"]
def test_ner_extractor_non_ml_method_does_not_load_model(self, monkeypatch):
"""NERExtractor with a non-ml method must not touch the spaCy cache."""
+19 -6
View File
@@ -33,10 +33,24 @@ class TestNERConfigurations(unittest.TestCase):
# Mock LLM provider
mock_provider = MagicMock()
mock_provider.is_available.return_value = True
mock_provider.generate_structured.return_value = [
{"text": "Apple Inc.", "label": "ORG", "start": 0, "end": 10, "confidence": 0.95},
{"text": "Steve Jobs", "label": "PERSON", "start": 26, "end": 36, "confidence": 0.98}
# The LLM path uses generate_typed (Pydantic schema validation), not
# generate_structured. Build a typed response object whose .entities
# carries simple namespace-like items.
def _entity(text, label, start, end, confidence):
item = MagicMock()
item.text = text
item.label = label
item.start = start
item.end = end
item.confidence = confidence
return item
typed_response = MagicMock()
typed_response.entities = [
_entity("Apple Inc.", "ORG", 0, 10, 0.95),
_entity("Steve Jobs", "PERSON", 26, 36, 0.98),
]
mock_provider.generate_typed.return_value = typed_response
mock_create_provider.return_value = mock_provider
# Initialize extractor with LLM method
@@ -56,7 +70,7 @@ class TestNERConfigurations(unittest.TestCase):
self.assertEqual(len(entities), 2)
self.assertEqual(entities[0].text, "Apple Inc.")
self.assertEqual(entities[0].label, "ORG")
self.assertEqual(entities[0].metadata["extraction_method"], "llm")
self.assertEqual(entities[0].metadata["extraction_method"], "llm_typed")
self.assertEqual(entities[0].metadata["model"], "gpt-4")
@patch('semantica.semantic_extract.methods.spacy')
@@ -115,7 +129,6 @@ class TestNERConfigurations(unittest.TestCase):
with patch('semantica.semantic_extract.ner_extractor.SPACY_AVAILABLE', True):
extractor = NERExtractor(method="ml", model="en_core_web_sm")
self.assertIsNone(extractor.nlp)
self.assertFalse(extractor._ml_runtime_usable)
@patch('semantica.semantic_extract.methods.get_entity_method')
@@ -183,7 +196,7 @@ class TestNERConfigurations(unittest.TestCase):
self.assertTrue(len(entities) >= 2)
texts = [e.text for e in entities]
self.assertIn("Apple Inc", texts) # Regex pattern does not capture the trailing dot
self.assertIn("Apple Inc.", texts) # The ORG pattern includes the trailing dot via (?:\.|\b)
# Actually methods.py regex: r"\b([A-Z][a-zA-Z]+(?:\s+[A-Z][a-zA-Z]+)*\s+(?:Inc|Corp|LLC|Ltd|Company))\b"
# "Apple Inc." -> "Apple Inc" (dot is outside \b if not matched?)
# Let's check the result strictly
+37 -10
View File
@@ -31,21 +31,48 @@ class TestHelpers(unittest.TestCase):
dict2 = {"b": {"d": 3}, "e": 4}
merged = helpers.merge_dicts(dict1, dict2, deep=True)
self.assertEqual(merged, {"a": 1, "b": {"c": 2, "d": 3}, "e": 4})
def test_flatten_dict(self):
data = {"a": {"b": 1, "c": 2}}
"""Basic nested flattening with multiple sibling keys."""
data = {"a": {"b": 1, "c": 2}, "d": 3}
result = helpers.flatten_dict(data)
self.assertEqual(result, {"a.b": 1, "a.c": 2})
self.assertEqual(result, {"a.b": 1, "a.c": 2, "d": 3})
def test_flatten_dict_deeply_nested(self):
"""Deeply nested structure is fully flattened."""
self.assertEqual(
helpers.flatten_dict({"a": {"b": {"c": 1}}}),
{"a.b.c": 1},
)
def test_flatten_dict_custom_separator(self):
"""Custom separator is used in generated keys."""
self.assertEqual(
helpers.flatten_dict({"a": {"b": 1}}, sep="__"),
{"a__b": 1},
)
def test_flatten_dict_empty(self):
"""Empty input returns empty output."""
self.assertEqual(helpers.flatten_dict({}), {})
def test_flatten_dict_key_collision(self):
data = {
"a.b": 1,
"a": {
"b": 2
}
}
"""#1010 regression: a top-level key containing the separator must not
silently overwrite a value produced from a nested dict when both resolve
to the same flattened key. Before the fix, {'a.b': 1, 'a': {'b': 2}}
silently dropped one value; now it raises ValueError."""
with self.assertRaises(ValueError) as ctx:
helpers.flatten_dict({"a.b": 1, "a": {"b": 2}})
self.assertIn("Key collision", str(ctx.exception))
self.assertIn("a.b", str(ctx.exception))
with self.assertRaises(ValueError):
helpers.flatten_dict(data)
def test_flatten_dict_no_false_positive(self):
"""Similar-looking keys that produce distinct flattened keys must not
trigger the collision guard."""
self.assertEqual(
helpers.flatten_dict({"a.b": 1, "a": {"c": 2}}),
{"a.b": 1, "a.c": 2},
)
def test_safe_import_returns_module_and_flag(self):
module, available = helpers.safe_import("json")