mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Compare commits
22
Commits
f187d4b5da
...
e12eec40a1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e12eec40a1 | ||
|
|
4da27c38bb | ||
|
|
7f928f9f8e | ||
|
|
65e6dcfef5 | ||
|
|
0775b0114e | ||
|
|
f4c3064571 | ||
|
|
b2dc633796 | ||
|
|
13b287b974 | ||
|
|
cec9bee099 | ||
|
|
36ced4e826 | ||
|
|
6032b4e0bc | ||
|
|
8db95f00c6 | ||
|
|
23baf21d5a | ||
|
|
5d54919804 | ||
|
|
c9c777993b | ||
|
|
9cec305a75 | ||
|
|
3d0ce55fd7 | ||
|
|
b13cc1cca2 | ||
|
|
92ad7bc2df | ||
|
|
db81136b0a | ||
|
|
1c27a0ae7e | ||
|
|
9e2f349221 |
@@ -119,6 +119,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- **KG provenance tests asserted on generated ID strings instead of stored records, and `kg_provenance.py` was missed by the `utcnow` sweep** (closes #946) by @pravit-amp
|
||||
- The KG workflow and integration suites checked that a tracker call returned an ID matching a prefix (`assert cent_id.startswith("centrality_")`) without ever reading the record back, so an ID generator that returned a well-formed string and wrote nothing would have passed. Worse, some of those calls named tracker methods that do not exist anywhere in `semantica/` (`track_layer_analysis`, `track_centrality_score`), so the assertions were satisfied with no real interaction behind them
|
||||
- Those tests now read provenance back through `get_provenance()` and assert on algorithm metadata, and call the methods that actually persist records. Verified by mutation rather than by a green run alone: neutering the manager's storage write (`self.storage.store(...)` → no-op) fails 10 tests
|
||||
- `GraphBuilderWithProvenance` in `semantica/kg/kg_provenance.py` still stamped `activity_started_at_time`/`activity_ended_at_time` with the deprecated `datetime.utcnow()`; it was outside the `export/`+`provenance/` scope of the #1114 sweep below and now uses the same `utc_now_iso()` helper. `docs/guides/provenance.md` and `docs/reference/provenance.md` were still documenting `utcnow()` and a naive timestamp example, and now show the helper and the offset-bearing form
|
||||
- 16 tests across the affected suites ended in `return <value>` instead of asserting, which pytest reports as `PytestReturnNotNoneWarning`; now zero
|
||||
|
||||
- **The temporal-evolution `stability` metric was a hardcoded placeholder, not a duration**
|
||||
- `TemporalGraphQuery.analyze_evolution()` documents `stability` as a "relationship duration/stability measure", but the implementation appended a constant `1` for every relationship with both `valid_from` and `valid_until` set (`durations.append(1) # Placeholder`). The reported stability was therefore always `1.0` when any bounded relationship existed and `0` otherwise — it never reflected how long relationships actually stayed valid, so it could not distinguish a graph of decade-long relationships from one of one-second relationships
|
||||
- `stability` now computes the mean valid-time duration in seconds (`(valid_until - valid_from).total_seconds()`) across relationships that have both bounds set. Relationships with a missing or open `valid_from`/`valid_until` are skipped (their duration is unbounded), and non-positive intervals are clamped to `0`; an empty set still reports `0`
|
||||
|
||||
@@ -0,0 +1,383 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b76a5997",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)\n",
|
||||
"\n",
|
||||
"# Reasoning Module — Practical Guide\n",
|
||||
"\n",
|
||||
"Semantica's `reasoning` module derives new knowledge from existing facts and knowledge graphs. It ships several strategies behind one facade:\n",
|
||||
"\n",
|
||||
"- **`Reasoner`** — unified facade with forward chaining, backward chaining, and one-shot `infer_facts`\n",
|
||||
"- **`DatalogReasoner`** — semi-naive Datalog fixpoint evaluation with variable queries\n",
|
||||
"- **`ExplanationGenerator`** — human-readable explanations and reasoning paths for inferred conclusions\n",
|
||||
"- Plus lower-level engines: `ReteEngine`, `SPARQLReasoner`, `GraphReasoner`, temporal reasoning\n",
|
||||
"\n",
|
||||
"This notebook walks through the facade, the Datalog engine, and explanations. All APIs are verified against `semantica/reasoning/`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "52073af7",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:45:55.427457Z",
|
||||
"iopub.status.busy": "2026-08-26T18:45:55.427247Z",
|
||||
"iopub.status.idle": "2026-08-26T18:45:57.266607Z",
|
||||
"shell.execute_reply": "2026-08-26T18:45:57.264783Z"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install -q semantica"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "06deb916",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1) Forward chaining with the `Reasoner` facade\n",
|
||||
"\n",
|
||||
"Facts are simple `Predicate(args)` strings. Rules use `IF <conditions> THEN <conclusion>` with `?x`-style variables. `forward_chain()` derives everything possible and returns a list of `InferenceResult` objects."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "519ca92d",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:45:57.270791Z",
|
||||
"iopub.status.busy": "2026-08-26T18:45:57.270352Z",
|
||||
"iopub.status.idle": "2026-08-26T18:45:59.991941Z",
|
||||
"shell.execute_reply": "2026-08-26T18:45:59.990678Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>Reasoner</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>DatalogReasoner</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr><tr><td>✅</td><td>Semantica is reasoning</td><td>🤔 reasoning</td><td>ExplanationGenerator</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"🔄 Semantica is reasoning: Performing forward chaining 🤔 reasoning Reasoner |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Inferred 2 new facts\n",
|
||||
" Human(Jane) (rule: Rule 1, confidence: 1.0)\n",
|
||||
" Human(John) (rule: Rule 1, confidence: 1.0)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner\n",
|
||||
"\n",
|
||||
"reasoner = Reasoner()\n",
|
||||
"\n",
|
||||
"reasoner.add_fact(\"Person(John)\")\n",
|
||||
"reasoner.add_fact(\"Person(Jane)\")\n",
|
||||
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
|
||||
"\n",
|
||||
"results = reasoner.forward_chain()\n",
|
||||
"print(f\"Inferred {len(results)} new facts\")\n",
|
||||
"for res in results:\n",
|
||||
" print(f\" {res.conclusion} (rule: {res.rule_used.name}, confidence: {res.confidence})\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c1131c45",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2) One-shot inference with `infer_facts`\n",
|
||||
"\n",
|
||||
"`infer_facts(facts, rules)` **adds** the given facts and rules to this `Reasoner` instance, runs forward chaining to fixpoint, and returns the derived facts as strings. It does not reset the instance's existing state — create a fresh `Reasoner()` first if you need isolation between runs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "26249990",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:45:59.995447Z",
|
||||
"iopub.status.busy": "2026-08-26T18:45:59.995069Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:00.004107Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:00.002873Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['Employee(Jane, Acme)', 'Employee(John, Acme)']"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner\n",
|
||||
"\n",
|
||||
"derived = Reasoner().infer_facts(\n",
|
||||
" facts=[\"WorksFor(John, Acme)\", \"WorksFor(Jane, Acme)\"],\n",
|
||||
" rules=[\"IF WorksFor(?x, ?y) THEN Employee(?x, ?y)\"],\n",
|
||||
")\n",
|
||||
"derived"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d5504a38",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3) Backward chaining: proving a goal\n",
|
||||
"\n",
|
||||
"`backward_chain(goal)` works backwards from a conclusion through the rules. It returns the `InferenceResult` that proves the goal, or `None`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "c4ef85dd",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:00.007740Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:00.007346Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:00.015561Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:00.014145Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Human(John)\n",
|
||||
"premises: ['Person(John)']\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner\n",
|
||||
"\n",
|
||||
"reasoner = Reasoner()\n",
|
||||
"reasoner.add_fact(\"Person(John)\")\n",
|
||||
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
|
||||
"\n",
|
||||
"proof = reasoner.backward_chain(\"Human(John)\")\n",
|
||||
"print(proof.conclusion if proof else \"not provable\")\n",
|
||||
"print(\"premises:\", proof.premises if proof else None)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b245581d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4) Re-run safety\n",
|
||||
"\n",
|
||||
"`add_rule` deduplicates rules with identical conditions and conclusion, so re-executing a setup cell (the common Jupyter re-run) does not duplicate rules — see issue #732."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "fb2aeb39",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:00.019091Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:00.018881Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:00.024042Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:00.022836Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Skipping duplicate rule (same conditions/conclusion as 'rule_1'): IF Person(?x) THEN Human(?x)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"1"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner\n",
|
||||
"\n",
|
||||
"reasoner = Reasoner()\n",
|
||||
"reasoner.add_fact(\"Person(John)\")\n",
|
||||
"\n",
|
||||
"# Simulate a Jupyter cell re-run: add the same rule twice\n",
|
||||
"r1 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
|
||||
"r2 = reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
|
||||
"\n",
|
||||
"len(reasoner.rules)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ba2e5c4a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5) Datalog reasoning\n",
|
||||
"\n",
|
||||
"`DatalogReasoner` uses classic Datalog syntax (`head :- body.`) and semi-naive fixpoint evaluation. Queries return variable bindings as a list of dicts — use uppercase variables to ask *which* facts hold."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "9ec5c0c4",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:00.026769Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:00.026588Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:00.034963Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:00.032672Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[{'X': 'tom', 'Z': 'ann'}]"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.reasoning import DatalogReasoner\n",
|
||||
"\n",
|
||||
"datalog = DatalogReasoner()\n",
|
||||
"datalog.add_fact(\"parent(tom, mary)\")\n",
|
||||
"datalog.add_fact(\"parent(mary, ann)\")\n",
|
||||
"datalog.add_rule(\"grandparent(X, Z) :- parent(X, Y), parent(Y, Z)\")\n",
|
||||
"\n",
|
||||
"datalog.derive_all()\n",
|
||||
"datalog.query(\"grandparent(X, Z)\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d4f0689b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 6) Explanations for inferred conclusions\n",
|
||||
"\n",
|
||||
"`ExplanationGenerator` turns `InferenceResult` objects into structured `Explanation` and `ReasoningPath` records, so agents can show *why* they believe a derived fact."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "19dcd3a7",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:00.038649Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:00.038396Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:00.059188Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:00.057805Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('Explanation', 'ReasoningPath')"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner, ExplanationGenerator\n",
|
||||
"\n",
|
||||
"reasoner = Reasoner()\n",
|
||||
"reasoner.add_fact(\"Person(John)\")\n",
|
||||
"reasoner.add_rule(\"IF Person(?x) THEN Human(?x)\")\n",
|
||||
"results = reasoner.forward_chain()\n",
|
||||
"\n",
|
||||
"gen = ExplanationGenerator()\n",
|
||||
"explanation = gen.generate_explanation(results[0])\n",
|
||||
"path = gen.show_reasoning_path(results[0])\n",
|
||||
"\n",
|
||||
"type(explanation).__name__, type(path).__name__"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fb882ee4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"| Task | API |\n",
|
||||
"|---|---|\n",
|
||||
"| Derive all new facts | `Reasoner.forward_chain()` |\n",
|
||||
"| One-shot inference | `Reasoner.infer_facts(facts, rules)` |\n",
|
||||
"| Prove a goal | `Reasoner.backward_chain(goal)` |\n",
|
||||
"| Datalog fixpoint | `DatalogReasoner.derive_all()` + `query(\"p(X, Y)\")` |\n",
|
||||
"| Explain a conclusion | `ExplanationGenerator.generate_explanation(result)` |\n",
|
||||
"\n",
|
||||
"See also `semantica/reasoning/reasoning_usage.md` and the module docstrings for `ReteEngine`, `SPARQLReasoner`, and temporal reasoning."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.13.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8d7096ea",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)\n",
|
||||
"\n",
|
||||
"# Change Management — Practical Guide\n",
|
||||
"\n",
|
||||
"Semantica's `change_management` module provides versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies:\n",
|
||||
"\n",
|
||||
"- **`ChangeLogEntry`** — standardized change metadata (validated timestamp/author)\n",
|
||||
"- **`InMemoryVersionStorage` / `SQLiteVersionStorage`** — version snapshot storage with named tags\n",
|
||||
"- **`compute_checksum` / `verify_checksum`** — SHA-256 integrity verification\n",
|
||||
"\n",
|
||||
"This notebook runs a complete save → tag → verify → tamper-detect cycle. All outputs are real executed results verified against the repository's `semantica/change_management/` source at the time of writing (the `pip install` cell may fetch a newer release with slightly different behavior)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "7bdffec1",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:37.171333Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:37.171183Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:39.060860Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:39.059594Z"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install -q semantica"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "169efee1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1) A `ChangeLogEntry` records *who* changed *what*, *when*\n",
|
||||
"\n",
|
||||
"`author` must be a valid email — the dataclass validates on construction (`ValidationError` otherwise), which keeps audit trails clean."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "5b17acdb",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:39.064077Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:39.063818Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:39.321881Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:39.321036Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"ChangeLogEntry(timestamp='2026-08-15T09:00:00Z', author='demo@example.com', description='initial version', change_id=None, related_changes=[])"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.change_management import ChangeLogEntry\n",
|
||||
"\n",
|
||||
"entry = ChangeLogEntry(\n",
|
||||
" timestamp=\"2026-08-15T09:00:00Z\",\n",
|
||||
" author=\"demo@example.com\",\n",
|
||||
" description=\"initial version\",\n",
|
||||
")\n",
|
||||
"entry"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "53d8df5c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2) Save a versioned snapshot\n",
|
||||
"\n",
|
||||
"A snapshot is a dict with a required `label` plus your payload. Here we attach the KG data, the change log, and a SHA-256 `checksum` computed over everything except the checksum field itself."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "fec16f24",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:39.325528Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:39.325140Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:39.331480Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:39.330586Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"True"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.change_management import InMemoryVersionStorage, compute_checksum\n",
|
||||
"\n",
|
||||
"storage = InMemoryVersionStorage()\n",
|
||||
"\n",
|
||||
"snapshot = {\n",
|
||||
" \"label\": \"v1.0.0\",\n",
|
||||
" \"data\": {\"entities\": {\"acme\": {\"type\": \"Company\"}}},\n",
|
||||
" \"change_log\": {\n",
|
||||
" \"timestamp\": entry.timestamp,\n",
|
||||
" \"author\": entry.author,\n",
|
||||
" \"description\": entry.description,\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"snapshot[\"checksum\"] = compute_checksum({k: v for k, v in snapshot.items() if k != \"checksum\"})\n",
|
||||
"\n",
|
||||
"storage.save(snapshot)\n",
|
||||
"storage.exists(\"v1.0.0\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0f1c603b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3) Named tags pin a version for releases\n",
|
||||
"\n",
|
||||
"`save_tag` / `get_tag` map stable names (e.g. `release`) to version labels, decoupling consumers from label churn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "62f7643e",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:39.335182Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:39.334886Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:39.339586Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:39.338568Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"('v1.0.0', ['v1.0.0'])"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"storage.save_tag(\"release\", \"v1.0.0\")\n",
|
||||
"\n",
|
||||
"storage.get_tag(\"release\"), [s[\"label\"] for s in storage.list_all()]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "96df12da",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4) Verify integrity — and catch tampering\n",
|
||||
"\n",
|
||||
"`verify_checksum(snapshot)` recomputes the SHA-256 over the snapshot (minus its `checksum` field) and compares. A single mutated character in the data flips the result to `False`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "26d0de85",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:39.342653Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:39.342466Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:39.346714Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:39.345623Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"intact: True\n",
|
||||
"tampered: False\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.change_management import verify_checksum\n",
|
||||
"\n",
|
||||
"stored = storage.get(\"v1.0.0\")\n",
|
||||
"print(\"intact:\", verify_checksum(stored))\n",
|
||||
"\n",
|
||||
"tampered = storage.get(\"v1.0.0\")\n",
|
||||
"tampered[\"data\"][\"entities\"][\"acme\"][\"note\"] = \"mutated after the fact\"\n",
|
||||
"print(\"tampered:\", verify_checksum(tampered))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bd14c3e4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5) Retiring a version\n",
|
||||
"\n",
|
||||
"`delete(label)` removes a snapshot; tags pointing at it are your responsibility to update."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "de9fe3e5",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:46:39.349814Z",
|
||||
"iopub.status.busy": "2026-08-26T18:46:39.349513Z",
|
||||
"iopub.status.idle": "2026-08-26T18:46:39.354710Z",
|
||||
"shell.execute_reply": "2026-08-26T18:46:39.353669Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"False"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"storage.delete(\"v1.0.0\")\n",
|
||||
"storage.exists(\"v1.0.0\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ab667b32",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"| Task | API |\n",
|
||||
"|---|---|\n",
|
||||
"| Record audit metadata | `ChangeLogEntry(timestamp, author=email, description)` |\n",
|
||||
"| Persist a version | `InMemoryVersionStorage().save({\"label\": ..., ...})` |\n",
|
||||
"| Pin a release name | `save_tag(\"release\", \"v1.0.0\")` / `get_tag(\"release\")` |\n",
|
||||
"| Integrity check | `compute_checksum(snap)` / `verify_checksum(snap)` |\n",
|
||||
"| Persistent backend | `SQLiteVersionStorage(path)` — same interface |\n",
|
||||
"\n",
|
||||
"See also `semantica/change_management/change_management_usage.md` for the manager classes (`TemporalVersionManager`, `OntologyVersionManager`)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.13.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6eb4dfba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)\n",
|
||||
"\n",
|
||||
"# Seed Data — Practical Guide\n",
|
||||
"\n",
|
||||
"The `seed` module bootstraps a knowledge graph from **trusted, pre-known data** (CSV/JSON/database/API sources) before any extraction runs. This gives extraction a foundation to link against instead of starting from an empty graph.\n",
|
||||
"\n",
|
||||
"Key pieces:\n",
|
||||
"\n",
|
||||
"- **`SeedDataManager`** — registers data sources and builds foundation graphs\n",
|
||||
"- **`create_foundation_graph()`** — turns registered sources into `entities` + `relationships` + `metadata`\n",
|
||||
"- **`validate_quality()`** — checks a foundation graph before you commit it\n",
|
||||
"\n",
|
||||
"All examples below were executed against `semantica/seed/seed_manager.py`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "32f80cc6",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:51:18.716466Z",
|
||||
"iopub.status.busy": "2026-08-26T18:51:18.716264Z",
|
||||
"iopub.status.idle": "2026-08-26T18:51:20.533828Z",
|
||||
"shell.execute_reply": "2026-08-26T18:51:20.531402Z"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install -q semantica"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "75136e5f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1) Prepare a seed CSV and register the source\n",
|
||||
"\n",
|
||||
"`register_source(name, format, location, entity_type=...)` records where trusted data lives. `verified=True` (the default) marks the source as pre-validated."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "a8089e1f",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:51:20.538772Z",
|
||||
"iopub.status.busy": "2026-08-26T18:51:20.538323Z",
|
||||
"iopub.status.idle": "2026-08-26T18:51:20.675403Z",
|
||||
"shell.execute_reply": "2026-08-26T18:51:20.674060Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"True"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import csv\n",
|
||||
"import tempfile\n",
|
||||
"from pathlib import Path\n",
|
||||
"from semantica.seed import SeedDataManager\n",
|
||||
"\n",
|
||||
"# Write the sample CSV into a session-scoped temp directory so we never\n",
|
||||
"# clobber a companies.csv that might exist in the user's working directory.\n",
|
||||
"seed_csv = Path(tempfile.mkdtemp(prefix=\"semantica-seed-\")) / \"companies.csv\"\n",
|
||||
"with open(seed_csv, \"w\", newline=\"\") as f:\n",
|
||||
" writer = csv.DictWriter(f, fieldnames=[\"id\", \"name\", \"type\", \"industry\"])\n",
|
||||
" writer.writeheader()\n",
|
||||
" writer.writerow({\"id\": \"c1\", \"name\": \"Acme\", \"type\": \"Company\", \"industry\": \"robotics\"})\n",
|
||||
" writer.writerow({\"id\": \"c2\", \"name\": \"Globex\", \"type\": \"Company\", \"industry\": \"energy\"})\n",
|
||||
"\n",
|
||||
"manager = SeedDataManager()\n",
|
||||
"manager.register_source(\"companies\", format=\"csv\", location=str(seed_csv), entity_type=\"Company\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e87221ba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2) Load records from a registered source\n",
|
||||
"\n",
|
||||
"`load_source(name)` reads the source and enriches each record with `entity_type` and `source` provenance keys."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "f932e550",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:51:20.679424Z",
|
||||
"iopub.status.busy": "2026-08-26T18:51:20.679156Z",
|
||||
"iopub.status.idle": "2026-08-26T18:51:20.690659Z",
|
||||
"shell.execute_reply": "2026-08-26T18:51:20.688812Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>Progress</th><th>ETA</th><th>Rate</th><th>Time</th><th>Extracted</th></tr><tr><td>✅</td><td>Semantica is seeding</td><td>🌱 seed</td><td>SeedDataManager</td><td>100.0%</td><td>-</td><td>-</td><td>0.00s</td><td>-</td></tr></table></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"🔄 Semantica is seeding: Loading seed data from CSV: /var/folders/7s/bvvstgs10y963tz6_4bbnklr0000gn/T/semantica-seed-eu9__ep1/companies.csv 🌱 seed SeedDataManager |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"loaded 2 records\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'id': 'c1',\n",
|
||||
" 'name': 'Acme',\n",
|
||||
" 'type': 'Company',\n",
|
||||
" 'industry': 'robotics',\n",
|
||||
" 'entity_type': 'Company',\n",
|
||||
" 'source': 'companies'}"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"records = manager.load_source(\"companies\")\n",
|
||||
"print(f\"loaded {len(records)} records\")\n",
|
||||
"records[0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f2ebce64",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3) Build the foundation graph\n",
|
||||
"\n",
|
||||
"`create_foundation_graph()` converts every registered source into graph-ready entities and relationships. Entities carry `confidence: 1.0` — seed data is trusted by definition."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "09388c31",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:51:20.695259Z",
|
||||
"iopub.status.busy": "2026-08-26T18:51:20.694928Z",
|
||||
"iopub.status.idle": "2026-08-26T18:51:20.708595Z",
|
||||
"shell.execute_reply": "2026-08-26T18:51:20.707072Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['entities', 'metadata', 'relationships']"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"foundation = manager.create_foundation_graph()\n",
|
||||
"sorted(foundation.keys())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "4610a59f",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:51:20.713136Z",
|
||||
"iopub.status.busy": "2026-08-26T18:51:20.712795Z",
|
||||
"iopub.status.idle": "2026-08-26T18:51:20.718637Z",
|
||||
"shell.execute_reply": "2026-08-26T18:51:20.716835Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'id': 'c1',\n",
|
||||
" 'text': 'Acme',\n",
|
||||
" 'type': 'Company',\n",
|
||||
" 'confidence': 1.0,\n",
|
||||
" 'metadata': {'industry': 'robotics', 'source': 'companies'}}"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"foundation[\"entities\"][0]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f3a52dc7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4) Validate quality before committing\n",
|
||||
"\n",
|
||||
"`validate_quality(foundation_graph)` returns `valid`, `errors`, `warnings`, and `metrics` so you can gate bad seed data before it pollutes the graph."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "4eb7e664",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2026-08-26T18:51:20.722674Z",
|
||||
"iopub.status.busy": "2026-08-26T18:51:20.722118Z",
|
||||
"iopub.status.idle": "2026-08-26T18:51:20.732003Z",
|
||||
"shell.execute_reply": "2026-08-26T18:51:20.730170Z"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"True"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"quality = manager.validate_quality(foundation)\n",
|
||||
"quality[\"valid\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b534be89",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"| Task | API |\n",
|
||||
"|---|---|\n",
|
||||
"| Register a trusted source | `register_source(name, format, location, entity_type=...)` |\n",
|
||||
"| Load records | `load_source(name)` — adds `entity_type` / `source` keys |\n",
|
||||
"| Direct file load | `load_from_csv(path)` / `load_from_json(path)` |\n",
|
||||
"| Build the graph | `create_foundation_graph()` → `entities` / `relationships` / `metadata` |\n",
|
||||
"| Gate bad data | `validate_quality(graph)` → `valid` / `errors` / `warnings` / `metrics` |\n",
|
||||
"\n",
|
||||
"See also `semantica/seed/seed_usage.md` for `load_from_database`, `load_from_api`, and `integrate_with_extracted`."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.13.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -35,6 +35,7 @@ Essential guides to master the Semantica framework.
|
||||
- **[Vector Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)** — Setting up vector stores for similarity search and retrieval. *Intermediate*
|
||||
- **[Graph Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Graph_Store.ipynb)** — Persisting knowledge graphs in Neo4j or FalkorDB. Topics: Neo4j, Cypher, Persistence · *Intermediate*
|
||||
- **[Ontology](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)** — Defining domain schemas and ontologies to structure your data. Topics: OWL, RDF, Schema Design · *Intermediate*
|
||||
- **[Seed Data](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)** — Bootstrapping a knowledge graph from trusted CSV, JSON, database, and API sources before extraction runs. Topics: SeedDataManager, Foundation Graphs · *Intermediate*
|
||||
|
||||
|
||||
## Advanced Concepts
|
||||
@@ -50,6 +51,9 @@ Deep dive into advanced features, customization, and complex workflows.
|
||||
- **[Multi-Source Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)** — Merging data from disparate sources into a unified graph. Topics: Entity Resolution, Merging, Fusion · *Advanced*
|
||||
- **[Reasoning and Inference](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)** — Using logical reasoning to infer new knowledge from existing facts. Topics: Logic Rules, Inference Engines · *Advanced*
|
||||
- **[Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)** — Modeling and querying data that changes over time. Topics: Time Series, Temporal Logic, Allen Algebra · *Advanced*
|
||||
- **[Provenance Tracking](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/22_Provenance_Tracking.ipynb)** — Audit-grade, W3C PROV-O-aligned tracking of where every entity, relationship, and chunk came from. Topics: PROV-O, Lineage, Checksums, Invalidation · *Advanced*
|
||||
- **[Reasoning Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/23_Reasoning.ipynb)** — Deriving new knowledge from existing facts with forward chaining, backward chaining, and Datalog strategies. Topics: Reasoner, Datalog, Explanations · *Advanced*
|
||||
- **[Change Management](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/24_Change_Management.ipynb)** — Versioning, audit trails, and data-integrity checks for knowledge graphs and ontologies. Topics: ChangeLogEntry, Version Storage, Data Integrity · *Advanced*
|
||||
|
||||
|
||||
## How to Run
|
||||
|
||||
@@ -222,10 +222,10 @@ builder.register_step_handler("ner_extract", run_ner)
|
||||
builder.register_step_handler("triplet_extract", run_triplets)
|
||||
builder.register_step_handler("kg_merge", merge_into_graph)
|
||||
|
||||
builder.add_step("ingest", "file_ingest", handler=ingest_stix_bundles, path="./stix_bundles/")
|
||||
builder.add_step("ner", "ner_extract", handler=run_ner, confidence_threshold=0.75)
|
||||
builder.add_step("triplets", "triplet_extract", handler=run_triplets, include_temporal=True)
|
||||
builder.add_step("store", "kg_merge", handler=merge_into_graph, output_path="./cti_output/")
|
||||
builder.add_step("ingest", "file_ingest", path="./stix_bundles/")
|
||||
builder.add_step("ner", "ner_extract", confidence_threshold=0.75)
|
||||
builder.add_step("triplets", "triplet_extract", include_temporal=True)
|
||||
builder.add_step("store", "kg_merge", output_path="./cti_output/")
|
||||
|
||||
# ingest feeds both ner and triplets in parallel
|
||||
builder.connect_steps("ingest", "ner")
|
||||
|
||||
@@ -639,7 +639,7 @@ Every `ProvenanceEntry` maps directly to W3C PROV-O terms. If your compliance te
|
||||
| — | `previous_version_id` | This entry corrects/replaces a prior version of the *same* fact |
|
||||
| `prov:wasDerivedFrom` | `derived_from_id` | This entry was derived from a *different* source entity |
|
||||
| `prov:used` | `used_entities` | Entity IDs consumed to produce this one |
|
||||
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `datetime.utcnow()` at write time |
|
||||
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `utc_now_iso()` at write time |
|
||||
| `prov:qualifiedInvalidation` | `invalidated`, `invalidated_at_time`, `invalidated_by`, `invalidation_reason` | A retraction/correction recorded as a tombstone via `ProvenanceManager.invalidate()`, never a hard delete |
|
||||
| `prov:startedAtTime` / `prov:endedAtTime` | `activity_started_at_time`, `activity_ended_at_time` | Typed Activity timing — pass an `ActivityRecord` via the `activity=` kwarg to set these together with `activity_id` |
|
||||
| `prov:qualifiedGeneration`/`Generation`, `qualifiedUsage`/`Usage`, `qualifiedDerivation`/`Derivation` | (derived from the fields above) | Additive qualified forms of `wasGeneratedBy`/`used`/`wasDerivedFrom`, emitted automatically alongside the plain triples |
|
||||
|
||||
@@ -150,6 +150,12 @@ HighRiskSupplier(DELTA-3) conf=100% rule=Rule 3
|
||||
|
||||
DELTA-3 is flagged even though no document described it that way — the system traced: DELTA-3 supplied GAMMA-7, and GAMMA-7 exploits critical CVEs. For rules that need priority ordering or graded confidence, use the `Rule` dataclass:
|
||||
|
||||
If a rule has side-effecting actions, one concrete activation runs those
|
||||
actions at most once on a Reasoner instance. Re-running `forward_chain()` is
|
||||
therefore safe: already-attempted actions are not repeated. Use
|
||||
`reasoner.reset_action_history()` when you intentionally want to replay them;
|
||||
`reasoner.clear()` and `reasoner.reset()` also clear the history.
|
||||
|
||||
```python
|
||||
# Higher priority rules fire first; confidence propagates into InferenceResult.confidence
|
||||
reasoner.add_rule(Rule(
|
||||
@@ -360,6 +366,13 @@ engine.reset()
|
||||
|
||||
The rule network is compiled once by `build_network()`. Each subsequent `add_fact()` call propagates incrementally through only the nodes whose conditions it satisfies — not the full rule set — which keeps evaluation cost proportional to the number of new activations rather than the total rule count.
|
||||
|
||||
With a Reasoner bound, Rete action side effects are attempted once per rule,
|
||||
bindings, and matched fact identity. Passing the same match to
|
||||
`execute_matches()` again still returns the same conclusion, but does not repeat
|
||||
its actions. Call `engine.reset_action_history()` to replay actions without
|
||||
clearing working memory. `engine.reset()` and `engine.build_network()` also
|
||||
clear the action history.
|
||||
|
||||
## Step 7 — Temporal interval reasoning
|
||||
|
||||
`TemporalReasoningEngine` computes Allen interval relations between time windows, letting you identify whether two events overlap, one contains the other, they meet at a boundary, and so on across your graph:
|
||||
|
||||
@@ -250,7 +250,7 @@ entry = ProvenanceEntry(
|
||||
source_document="report.pdf", # str: default ""
|
||||
source_location="Page 4", # Optional[str]: default None
|
||||
source_quote="Relevant text...", # Optional[str]: default None
|
||||
timestamp="2024-01-01T12:00:00", # str: auto-set to utcnow()
|
||||
timestamp="2024-01-01T12:00:00+00:00", # str: auto-set to utc_now_iso()
|
||||
first_seen=None, # Optional[str]: ISO timestamp
|
||||
last_updated=None, # Optional[str]: ISO timestamp
|
||||
confidence=0.9, # float: default 1.0
|
||||
|
||||
@@ -127,9 +127,19 @@ conclusions = reasoner.infer_facts(
|
||||
| `forward_chain()` | `List[InferenceResult]` | Derive all possible conclusions iteratively until fixpoint |
|
||||
| `backward_chain(goal, max_depth)` | `InferenceResult \| None` | Prove a specific goal string, returns `None` if unprovable |
|
||||
| `infer_facts(facts, rules)` | `List[str]` | Load facts and rules then run `forward_chain()`, returns conclusion strings |
|
||||
| `clear()` | `None` | Clear all facts and rules |
|
||||
| `reset_action_history()` | `None` | Allow actions for previously fired activations to run again |
|
||||
| `clear()` | `None` | Clear all facts, rules, and action activation history |
|
||||
| `reset()` | `None` | Alias for `clear()` |
|
||||
|
||||
Rules with actions use at-most-once attempt semantics per concrete activation
|
||||
(rule ID, bindings, and matched facts). Calling `forward_chain()` again on the
|
||||
same instance does not repeat side effects for an activation that was already
|
||||
attempted, even when an action raised an exception. Call
|
||||
`reset_action_history()` to deliberately retry without clearing facts or rules;
|
||||
`clear()` and `reset()` also clear this history. Replacing a rule's actions in
|
||||
place does not invalidate an existing activation; reset the history explicitly
|
||||
when the replacement should be replayed.
|
||||
|
||||
### Rule and Fact dataclass fields
|
||||
|
||||
```python
|
||||
@@ -230,9 +240,16 @@ engine.reset()
|
||||
| `add_fact(fact)` | `None` | Add a `Fact` to working memory and propagate through the network |
|
||||
| `match_patterns(facts)` | `List[Match]` | Match all patterns; optionally add facts before matching |
|
||||
| `execute_matches(matches)` | `List[Any]` | Execute matched rules and return their conclusion values |
|
||||
| `reset()` | `None` | Clear facts and all node activation state |
|
||||
| `reset_action_history()` | `None` | Allow actions for previously executed activations to run again |
|
||||
| `reset()` | `None` | Clear facts, node activation state, and action activation history |
|
||||
| `get_network_stats()` | `dict` | Return counts of alpha, beta, terminal nodes and facts |
|
||||
|
||||
When a Reasoner is bound, `execute_matches()` deduplicates action side effects
|
||||
by rule ID, bindings, and matched fact identity. Re-executing a match still
|
||||
returns its conclusion for compatibility, but its actions are skipped after the
|
||||
first attempt. `reset_action_history()`, `reset()`, and `build_network()` allow
|
||||
those actions to run again.
|
||||
|
||||
|
||||
## SPARQLReasoner
|
||||
|
||||
|
||||
@@ -109,7 +109,7 @@ class ConflictsConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -136,12 +136,12 @@ class ConflictsConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -110,7 +110,7 @@ class DeduplicationConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -137,12 +137,12 @@ class DeduplicationConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -106,7 +106,7 @@ class EmbeddingsConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -123,8 +123,8 @@ class EmbeddingsConfig:
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
# Try to convert to appropriate type
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -149,12 +149,12 @@ class EmbeddingsConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -105,7 +105,7 @@ class ExportConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -122,8 +122,8 @@ class ExportConfig:
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
# Try to convert to appropriate type
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -148,12 +148,12 @@ class ExportConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -642,6 +642,37 @@ class RDFSerializer:
|
||||
|
||||
self.logger.debug("RDF serializer initialized")
|
||||
|
||||
@staticmethod
|
||||
def _local_name_from_id(identifier: str) -> str:
|
||||
"""Derive a human-readable local name from an entity identifier.
|
||||
|
||||
Handles HTTP(S)/IRI identifiers (path segments and fragments, tolerating
|
||||
trailing slashes) as well as compact/CURIE and URN-style identifiers.
|
||||
"""
|
||||
raw = str(identifier).strip()
|
||||
if not raw:
|
||||
return ""
|
||||
|
||||
# Prefer a fragment if present (e.g. http://ex.org/onto#acme -> acme).
|
||||
if "#" in raw:
|
||||
candidate = raw.rsplit("#", 1)[-1]
|
||||
if candidate:
|
||||
return candidate
|
||||
|
||||
# For IRIs/paths, take the last non-empty path segment.
|
||||
if "/" in raw:
|
||||
segment = raw.rstrip("/").rsplit("/", 1)[-1]
|
||||
if segment:
|
||||
return segment
|
||||
|
||||
# Fall back to the tail of a CURIE/URN (e.g. urn:x:acme, semantica:acme).
|
||||
if ":" in raw:
|
||||
candidate = raw.rsplit(":", 1)[-1]
|
||||
if candidate:
|
||||
return candidate
|
||||
|
||||
return raw
|
||||
|
||||
def convert_kg_to_rdf(self, knowledge_graph: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert knowledge graph to RDF data structure.
|
||||
@@ -681,8 +712,11 @@ class RDFSerializer:
|
||||
if "name" in norm_entity:
|
||||
norm_entity["label"] = norm_entity["name"]
|
||||
elif "id" in norm_entity:
|
||||
# Use ID part as label if no name/text
|
||||
norm_entity["label"] = str(norm_entity["id"]).split(":")[-1]
|
||||
# Derive a readable label from the identifier's local name
|
||||
# (fragment/last path segment/CURIE tail). See #1097.
|
||||
local_name = self._local_name_from_id(norm_entity["id"])
|
||||
if local_name:
|
||||
norm_entity["label"] = local_name
|
||||
|
||||
rdf_data["entities"].append(norm_entity)
|
||||
|
||||
@@ -1082,8 +1116,9 @@ class RDFSerializer:
|
||||
|
||||
# RDF/XML syntax: rdf:Description with rdf:about
|
||||
# Attribute values are delimited by quotes, and both of these
|
||||
# are caller input. Element text is left alone deliberately: that
|
||||
# is #1098, and it is being fixed on its own path.
|
||||
# are caller input. Element text (semantica:text) is caller input
|
||||
# too, so it needs the same escaping to avoid injecting markup
|
||||
# or breaking out of the element (#1097 / #1113).
|
||||
entity_iri = xml_escape(
|
||||
self._as_turtle_iri(entity_id, namespaces), quote=True
|
||||
)
|
||||
@@ -1092,7 +1127,9 @@ class RDFSerializer:
|
||||
)
|
||||
lines.append(f' <rdf:Description rdf:about="{entity_iri}">')
|
||||
lines.append(f' <rdf:type rdf:resource="{entity_type_iri}"/>')
|
||||
lines.append(f" <semantica:text>{text}</semantica:text>")
|
||||
lines.append(
|
||||
f" <semantica:text>{xml_escape(text)}</semantica:text>"
|
||||
)
|
||||
if confidence is None:
|
||||
self.logger.warning(
|
||||
f"Entity {entity_id} has a confidence that is not a number "
|
||||
@@ -1714,6 +1751,12 @@ class RDFExporter:
|
||||
|
||||
self.logger.debug(f"Exporting to RDF format: {format}")
|
||||
|
||||
# Normalize the graph before serialization so every format benefits
|
||||
# from field normalization (e.g. mapping 'name' -> 'label'/'text').
|
||||
# Without this, graphs produced by GraphBuilder (which emit 'name')
|
||||
# export with an empty semantica:text on all RDF paths. See #1097.
|
||||
data = self.serializer.convert_kg_to_rdf(data)
|
||||
|
||||
self.progress_tracker.update_tracking(
|
||||
tracking_id, message="Validating RDF data..."
|
||||
)
|
||||
|
||||
@@ -556,6 +556,13 @@ class GraphStore:
|
||||
)
|
||||
self.config = config
|
||||
|
||||
# Application-id -> backend-internal-id map for nodes added through
|
||||
# the compatibility layer (#1136). add_nodes()/create_node() record
|
||||
# the internal ids the backend returns; create_relationship()
|
||||
# resolves known application ids through it so string ids stop
|
||||
# mismatching backends that match on internal ids (Neo4j id(n)).
|
||||
self._app_node_id_map: Dict[Any, Any] = {}
|
||||
|
||||
# Initialize store backend
|
||||
self._store_backend = None
|
||||
self._manager = None
|
||||
@@ -630,7 +637,9 @@ class GraphStore:
|
||||
**options,
|
||||
) -> Dict[str, Any]:
|
||||
"""Create a node."""
|
||||
return self._manager.nodes.create(labels, properties, **options)
|
||||
created = self._manager.nodes.create(labels, properties, **options)
|
||||
self._record_app_node_id(created)
|
||||
return created
|
||||
|
||||
def create_nodes(
|
||||
self,
|
||||
@@ -688,9 +697,25 @@ class GraphStore:
|
||||
properties: Optional[Dict[str, Any]] = None,
|
||||
**options,
|
||||
) -> Dict[str, Any]:
|
||||
"""Create a relationship."""
|
||||
"""Create a relationship.
|
||||
|
||||
Node ids added through the compatibility layer are application-level
|
||||
strings, while backends such as Neo4j match on internal integer ids
|
||||
(#1136). Known application ids are resolved to the internal ids the
|
||||
backend returned at creation time; unknown ids pass through
|
||||
unchanged, so direct internal-id callers keep working. Only string
|
||||
application ids participate: internal ids are commonly integers, so
|
||||
recording or resolving an integer key could remap a caller-supplied
|
||||
internal id to a different node.
|
||||
"""
|
||||
return self._manager.relationships.create(
|
||||
start_node_id, end_node_id, rel_type, properties, **options
|
||||
self._app_node_id_map.get(start_node_id, start_node_id)
|
||||
if isinstance(start_node_id, str) else start_node_id,
|
||||
self._app_node_id_map.get(end_node_id, end_node_id)
|
||||
if isinstance(end_node_id, str) else end_node_id,
|
||||
rel_type,
|
||||
properties,
|
||||
**options,
|
||||
)
|
||||
|
||||
def get_relationships(
|
||||
@@ -794,6 +819,24 @@ class GraphStore:
|
||||
"""Create an index."""
|
||||
return self._manager.create_index(label, property_name, index_type, **options)
|
||||
|
||||
def _record_app_node_id(self, created: Optional[Dict[str, Any]]) -> None:
|
||||
"""Record the application-id -> internal-id pair of a created node.
|
||||
|
||||
Backends return their own internal id alongside the stored properties;
|
||||
when the caller supplied an application id it is preserved in
|
||||
``properties["id"]`` by the compatibility layer, which makes the pair
|
||||
recoverable (#1136). Only STRING application ids are recorded:
|
||||
internal ids are commonly integers, and an integer application id
|
||||
would collide with (and silently remap) a caller-supplied internal id
|
||||
of the same value in ``create_relationship``.
|
||||
"""
|
||||
if not isinstance(created, dict):
|
||||
return
|
||||
app_id = (created.get("properties") or {}).get("id")
|
||||
internal_id = created.get("id")
|
||||
if isinstance(app_id, str) and internal_id is not None:
|
||||
self._app_node_id_map[app_id] = internal_id
|
||||
|
||||
# Compatibility with AgentMemory / ContextGraph interface
|
||||
def add_nodes(self, nodes: List[Dict[str, Any]], **options) -> int:
|
||||
"""
|
||||
@@ -853,12 +896,21 @@ class GraphStore:
|
||||
# and properties.
|
||||
|
||||
result = self.create_nodes(graph_nodes, **options)
|
||||
# Keep the application-id -> internal-id pairs instead of discarding
|
||||
# them, so add_edges()/create_relationship() can resolve the string
|
||||
# ids callers actually use (#1136).
|
||||
for created in result:
|
||||
self._record_app_node_id(created)
|
||||
return len(result)
|
||||
|
||||
def add_edges(self, edges: List[Dict[str, Any]], **options) -> int:
|
||||
"""
|
||||
Add edges (Compatibility method).
|
||||
|
||||
``source_id``/``target_id`` are application-level string ids; they are
|
||||
resolved to the backend's internal ids via the map ``add_nodes``
|
||||
populated when the nodes were created (#1136).
|
||||
|
||||
Args:
|
||||
edges: List of edge dictionaries
|
||||
**options: Additional options
|
||||
|
||||
@@ -110,7 +110,7 @@ class IngestConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -127,8 +127,8 @@ class IngestConfig:
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
# Try to convert to appropriate type
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -143,8 +143,8 @@ class IngestConfig:
|
||||
if key.startswith(mcp_prefix) and key not in env_mappings:
|
||||
config_key = key[len(mcp_prefix) :].lower()
|
||||
# Try to convert to appropriate type
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[f"mcp_{config_key}"] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[f"mcp_{config_key}"] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[f"mcp_{config_key}"] = int(value)
|
||||
else:
|
||||
@@ -169,12 +169,12 @@ class IngestConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -106,7 +106,7 @@ class KGConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -123,8 +123,8 @@ class KGConfig:
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
# Try to convert to appropriate type
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -149,12 +149,12 @@ class KGConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -54,10 +54,11 @@ Version: 1.0.0
|
||||
"""
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
import time
|
||||
|
||||
from ..utils.helpers import utc_now_iso
|
||||
|
||||
|
||||
class GraphBuilderWithProvenance:
|
||||
"""
|
||||
@@ -103,7 +104,7 @@ class GraphBuilderWithProvenance:
|
||||
|
||||
def build(self, sources, **kwargs):
|
||||
"""Build graph with provenance tracking."""
|
||||
activity_started_at_time = datetime.utcnow().isoformat()
|
||||
activity_started_at_time = utc_now_iso()
|
||||
# Track the build operation (recorded before the build runs, so it
|
||||
# has no end time yet — this is the "in progress" marker).
|
||||
if self.provenance and self._prov_manager:
|
||||
@@ -124,7 +125,7 @@ class GraphBuilderWithProvenance:
|
||||
)
|
||||
|
||||
result = self._builder.build(sources, **kwargs)
|
||||
activity_ended_at_time = datetime.utcnow().isoformat()
|
||||
activity_ended_at_time = utc_now_iso()
|
||||
|
||||
# Track individual entities and relationships if available
|
||||
if self.provenance and self._prov_manager and hasattr(result, 'get'):
|
||||
@@ -180,7 +181,7 @@ class GraphBuilderWithProvenance:
|
||||
|
||||
def build_single_source(self, kg_data, **kwargs):
|
||||
"""Build graph from single source with provenance tracking."""
|
||||
activity_started_at_time = datetime.utcnow().isoformat()
|
||||
activity_started_at_time = utc_now_iso()
|
||||
# Track the build operation (recorded before the build runs, so it
|
||||
# has no end time yet — this is the "in progress" marker).
|
||||
if self.provenance and self._prov_manager:
|
||||
@@ -200,7 +201,7 @@ class GraphBuilderWithProvenance:
|
||||
)
|
||||
|
||||
result = self._builder.build_single_source(kg_data, **kwargs)
|
||||
activity_ended_at_time = datetime.utcnow().isoformat()
|
||||
activity_ended_at_time = utc_now_iso()
|
||||
|
||||
# Track entities and relationships if available
|
||||
if self.provenance and self._prov_manager and isinstance(result, dict):
|
||||
|
||||
@@ -98,7 +98,7 @@ class NormalizeConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -113,8 +113,8 @@ class NormalizeConfig:
|
||||
for key, value in os.environ.items():
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -136,12 +136,12 @@ class NormalizeConfig:
|
||||
value = os.getenv(env_key)
|
||||
if value:
|
||||
try:
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -99,7 +99,7 @@ class OntologyConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -114,8 +114,8 @@ class OntologyConfig:
|
||||
for key, value in os.environ.items():
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -137,12 +137,12 @@ class OntologyConfig:
|
||||
value = os.getenv(env_key)
|
||||
if value:
|
||||
try:
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -197,23 +197,22 @@ class PropertyGenerator:
|
||||
self, entities: List[Dict[str, Any]], classes: List[Dict[str, Any]], **options
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Infer data properties from entity attributes."""
|
||||
# Group entities by type
|
||||
entity_types = defaultdict(list)
|
||||
# Group entities by their inferred class so normalized class names remain
|
||||
# aligned with the class definitions emitted by ClassInferrer.
|
||||
class_entities = defaultdict(list)
|
||||
class_lookup = self._build_class_type_lookup(classes)
|
||||
for entity in entities:
|
||||
entity_type = entity.get("type") or entity.get("entity_type", "Entity")
|
||||
entity_types[entity_type].append(entity)
|
||||
class_def = self._find_class_for_entity_type(entity_type, class_lookup)
|
||||
if not class_def:
|
||||
continue
|
||||
class_name = class_def["name"]
|
||||
class_entities[class_name].append(entity)
|
||||
|
||||
# Extract data properties for each class
|
||||
properties = []
|
||||
|
||||
for entity_type, type_entities in entity_types.items():
|
||||
# Find corresponding class
|
||||
class_def = next(
|
||||
(cls for cls in classes if cls["name"] == entity_type), None
|
||||
)
|
||||
if not class_def:
|
||||
continue
|
||||
|
||||
for class_name, type_entities in class_entities.items():
|
||||
# Extract data properties
|
||||
data_props = self._extract_data_properties(type_entities)
|
||||
|
||||
@@ -233,7 +232,7 @@ class PropertyGenerator:
|
||||
else None,
|
||||
"label": normalized_name,
|
||||
"comment": f"Data property for {prop_name}",
|
||||
"domain": [entity_type],
|
||||
"domain": [class_name],
|
||||
"range": prop_type,
|
||||
"metadata": {"inferred_from": prop_name},
|
||||
}
|
||||
@@ -242,6 +241,38 @@ class PropertyGenerator:
|
||||
|
||||
return properties
|
||||
|
||||
def _build_class_type_lookup(
|
||||
self, classes: List[Dict[str, Any]]
|
||||
) -> Dict[str, Dict[str, Any]]:
|
||||
"""Build a lookup for raw, normalized, and recorded source type names."""
|
||||
lookup: Dict[str, Dict[str, Any]] = {}
|
||||
for class_def in classes:
|
||||
class_name = class_def.get("name")
|
||||
if class_name:
|
||||
lookup.setdefault(str(class_name), class_def)
|
||||
lookup.setdefault(
|
||||
self.naming_conventions.normalize_class_name(str(class_name)),
|
||||
class_def,
|
||||
)
|
||||
|
||||
inferred_from = class_def.get("metadata", {}).get("inferred_from")
|
||||
if inferred_from is not None:
|
||||
lookup.setdefault(str(inferred_from), class_def)
|
||||
lookup.setdefault(
|
||||
self.naming_conventions.normalize_class_name(str(inferred_from)),
|
||||
class_def,
|
||||
)
|
||||
|
||||
return lookup
|
||||
|
||||
def _find_class_for_entity_type(
|
||||
self, entity_type: Any, class_lookup: Dict[str, Dict[str, Any]]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""Find a class using the precomputed type lookup."""
|
||||
raw_type = str(entity_type)
|
||||
normalized_type = self.naming_conventions.normalize_class_name(raw_type)
|
||||
return class_lookup.get(raw_type) or class_lookup.get(normalized_type)
|
||||
|
||||
def _extract_data_properties(
|
||||
self, entities: List[Dict[str, Any]]
|
||||
) -> Dict[str, str]:
|
||||
|
||||
@@ -98,7 +98,7 @@ class ParseConfig:
|
||||
if value:
|
||||
try:
|
||||
if type_func == bool:
|
||||
self._configs[config_key] = value.lower() in (
|
||||
self._configs[config_key] = value.strip().lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
@@ -113,8 +113,8 @@ class ParseConfig:
|
||||
for key, value in os.environ.items():
|
||||
if key.startswith(env_prefix) and key not in env_mappings:
|
||||
config_key = key[len(env_prefix) :].lower()
|
||||
if value.lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.lower() == "true"
|
||||
if value.strip().lower() in ("true", "false"):
|
||||
self._configs[config_key] = value.strip().lower() == "true"
|
||||
elif value.isdigit():
|
||||
self._configs[config_key] = int(value)
|
||||
else:
|
||||
@@ -136,12 +136,12 @@ class ParseConfig:
|
||||
value = os.getenv(env_key)
|
||||
if value:
|
||||
try:
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
@@ -33,6 +33,7 @@ License: MIT
|
||||
"""
|
||||
|
||||
import email
|
||||
import email.message
|
||||
from dataclasses import dataclass, field
|
||||
from email import message_from_bytes, message_from_string
|
||||
from email.header import decode_header
|
||||
|
||||
@@ -143,15 +143,39 @@ class PDFParser:
|
||||
)
|
||||
pages.append(page_data)
|
||||
|
||||
full_text = "\n\n".join(page.text for page in pages)
|
||||
|
||||
# Scanned/image-only PDFs have no text layer; warn so the
|
||||
# failure surfaces at parse time instead of downstream.
|
||||
# Check any() over page texts to avoid a temporary stripped
|
||||
# copy of the full concatenation for large documents.
|
||||
no_text = (
|
||||
options.get("extract_text", True)
|
||||
and pages
|
||||
and not any(page.text.strip() for page in pages)
|
||||
)
|
||||
if no_text:
|
||||
self.logger.warning(
|
||||
f"PDF {file_path.name}: parsed {len(pages)} page(s) but "
|
||||
f"extracted no text. This is likely a scanned "
|
||||
f"(image-only) PDF. Retry with "
|
||||
f"parse_pdf(..., method='docling', enable_ocr=True) "
|
||||
f"for OCR-based extraction."
|
||||
)
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message=f"Parsed {len(pages)} pages",
|
||||
message=(
|
||||
f"Parsed {len(pages)} page(s) (no text layer detected)"
|
||||
if no_text
|
||||
else f"Parsed {len(pages)} page(s)"
|
||||
),
|
||||
)
|
||||
return {
|
||||
"metadata": metadata.__dict__,
|
||||
"pages": [page.__dict__ for page in pages],
|
||||
"full_text": "\n\n".join(page.text for page in pages),
|
||||
"full_text": full_text,
|
||||
"total_pages": len(pdf.pages),
|
||||
}
|
||||
|
||||
|
||||
@@ -379,7 +379,7 @@ class ExecutionEngine:
|
||||
|
||||
data = delta_result
|
||||
|
||||
if step.handler:
|
||||
if step.handler is not None:
|
||||
return step.handler(data, **step.config, **options)
|
||||
else:
|
||||
return data
|
||||
|
||||
@@ -131,13 +131,17 @@ class PipelineBuilder:
|
||||
delta_mode = config.pop("delta_mode", False)
|
||||
base_version_id = config.pop("base_version_id", None)
|
||||
target_version_id = config.pop("target_version_id", None)
|
||||
dependencies = config.pop("dependencies", [])
|
||||
handler = config.pop("handler", None)
|
||||
if handler is None:
|
||||
handler = self.step_registry.get(step_type)
|
||||
|
||||
step = PipelineStep(
|
||||
name=step_name,
|
||||
step_type=step_type,
|
||||
config=config,
|
||||
dependencies=config.get("dependencies", []),
|
||||
handler=config.get("handler"),
|
||||
dependencies=dependencies,
|
||||
handler=handler,
|
||||
delta_mode = delta_mode,
|
||||
base_version_id=base_version_id,
|
||||
target_version_id=target_version_id,
|
||||
|
||||
@@ -8,6 +8,13 @@ and native Datalog evaluation.
|
||||
"""
|
||||
|
||||
from .reasoner import Reasoner, InferenceResult, Rule, Fact, RuleType
|
||||
from .reasoner import (
|
||||
Action,
|
||||
AssertAction,
|
||||
RetractAction,
|
||||
CallAction,
|
||||
EmitEventAction,
|
||||
)
|
||||
from .graph_reasoner import GraphReasoner
|
||||
from .explanation_generator import (
|
||||
Explanation,
|
||||
@@ -37,6 +44,12 @@ __all__ = [
|
||||
"Rule",
|
||||
"Fact",
|
||||
"RuleType",
|
||||
# Rule-driven actions
|
||||
"Action",
|
||||
"AssertAction",
|
||||
"RetractAction",
|
||||
"CallAction",
|
||||
"EmitEventAction",
|
||||
# Rete engine
|
||||
"ReteEngine",
|
||||
"ReteNode",
|
||||
|
||||
+424
-19
@@ -7,13 +7,16 @@ supported by the Semantica framework. It serves as a facade for different reason
|
||||
|
||||
import re
|
||||
import uuid
|
||||
from collections.abc import Mapping, Sequence, Set as AbstractSet
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple, Union, Callable
|
||||
from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Union
|
||||
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
|
||||
class RuleType(Enum):
|
||||
"""Rule types."""
|
||||
IMPLICATION = "implication"
|
||||
@@ -21,6 +24,306 @@ class RuleType(Enum):
|
||||
CONSTRAINT = "constraint"
|
||||
TRANSFORMATION = "transformation"
|
||||
|
||||
|
||||
def _substitute_variables(template: str, bindings: Dict[str, str]) -> str:
|
||||
"""Substitute ``?var`` placeholders with their bound values, token-aware.
|
||||
|
||||
A naive ``str.replace(f"?{var}", value)`` corrupts placeholders that share
|
||||
a prefix -- e.g. binding ``?x`` would also rewrite the ``?x`` inside ``?xy``.
|
||||
We replace every ``?word`` token in a single regex pass so that only whole
|
||||
variable names are matched (``\\w+`` never partially matches a longer name),
|
||||
leaving unbound placeholders untouched.
|
||||
"""
|
||||
if not bindings:
|
||||
return template
|
||||
|
||||
def _replace(match: "re.Match") -> str:
|
||||
var_name = match.group(1)
|
||||
# Preserve unbound placeholders verbatim.
|
||||
return str(bindings[var_name]) if var_name in bindings else match.group(0)
|
||||
|
||||
return re.sub(r"\?(\w+)", _replace, template)
|
||||
|
||||
|
||||
def _canonicalize_activation_value(
|
||||
value: Any, active_containers: Optional[Dict[int, int]] = None
|
||||
) -> Tuple[Any, ...]:
|
||||
"""Convert nested activation data into a deterministic, hashable value."""
|
||||
if active_containers is None:
|
||||
active_containers = {}
|
||||
|
||||
value_type = (type(value).__module__, type(value).__qualname__)
|
||||
is_mapping = isinstance(value, Mapping)
|
||||
is_sequence = isinstance(value, Sequence) and not isinstance(
|
||||
value, (str, bytes, bytearray)
|
||||
)
|
||||
is_set = isinstance(value, AbstractSet) and not isinstance(
|
||||
value, (str, bytes, bytearray)
|
||||
)
|
||||
if not (is_mapping or is_sequence or is_set):
|
||||
return ("scalar", value_type, repr(value))
|
||||
|
||||
object_id = id(value)
|
||||
if object_id in active_containers:
|
||||
return ("reference", active_containers[object_id])
|
||||
active_containers[object_id] = len(active_containers)
|
||||
|
||||
try:
|
||||
if is_mapping:
|
||||
keyed_items = [
|
||||
(
|
||||
_canonicalize_activation_value(key, active_containers),
|
||||
item,
|
||||
)
|
||||
for key, item in value.items()
|
||||
]
|
||||
keyed_items.sort(key=lambda entry: repr(entry[0]))
|
||||
entries = tuple(
|
||||
(
|
||||
key,
|
||||
_canonicalize_activation_value(item, active_containers),
|
||||
)
|
||||
for key, item in keyed_items
|
||||
)
|
||||
return ("mapping", value_type, entries)
|
||||
if is_sequence:
|
||||
return (
|
||||
"sequence",
|
||||
value_type,
|
||||
tuple(
|
||||
_canonicalize_activation_value(item, active_containers)
|
||||
for item in value
|
||||
),
|
||||
)
|
||||
items = tuple(
|
||||
sorted(
|
||||
(
|
||||
_canonicalize_activation_value(item, active_containers)
|
||||
for item in value
|
||||
),
|
||||
key=repr,
|
||||
)
|
||||
)
|
||||
return ("set", value_type, items)
|
||||
finally:
|
||||
del active_containers[object_id]
|
||||
|
||||
|
||||
def _make_activation_key(
|
||||
rule_id: str, bindings: Dict[str, Any], fact_tokens: List[Any]
|
||||
) -> Tuple[Any, ...]:
|
||||
"""Return a stable identity for one concrete rule activation."""
|
||||
canonical_bindings = tuple(
|
||||
sorted(
|
||||
(str(name), _canonicalize_activation_value(value))
|
||||
for name, value in bindings.items()
|
||||
)
|
||||
)
|
||||
return (
|
||||
rule_id,
|
||||
canonical_bindings,
|
||||
tuple(
|
||||
sorted(
|
||||
(_canonicalize_activation_value(token) for token in fact_tokens),
|
||||
key=repr,
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _parse_fact(fact: str) -> Optional[Tuple[str, List[str]]]:
|
||||
"""Parse a ``Predicate(arg1, arg2, ...)`` fact string.
|
||||
|
||||
Returns ``(predicate, [args])`` or ``None`` when the fact is not in the
|
||||
canonical predicate form (e.g. a bare atom). Whitespace around args is
|
||||
stripped and empty arg lists are supported (``Foo()`` -> ``("Foo", [])``).
|
||||
"""
|
||||
match = re.match(r"^\s*([^()\s]+)\s*\((.*)\)\s*$", fact)
|
||||
if not match:
|
||||
return None
|
||||
predicate = match.group(1)
|
||||
inner = match.group(2).strip()
|
||||
if not inner:
|
||||
return predicate, []
|
||||
args = [arg.strip() for arg in inner.split(",")]
|
||||
return predicate, args
|
||||
|
||||
|
||||
def _write_fact_to_graph(graph: Any, fact: str, *, retract: bool = False) -> None:
|
||||
"""Persist (or remove) a fact against a knowledge-graph-like target.
|
||||
|
||||
Write-back follows an explicit, ordered protocol so that
|
||||
``AssertAction(write_back=True)`` never silently no-ops:
|
||||
|
||||
1. If the target exposes an explicit fact API (``add_fact`` / ``assert_fact``
|
||||
for asserts, ``remove_fact`` / ``retract_fact`` / ``discard_fact`` for
|
||||
retracts), that is used verbatim.
|
||||
2. Otherwise, if the target looks like the canonical
|
||||
:class:`~semantica.kg.knowledge_graph.KnowledgeGraph` (has ``entities``
|
||||
and ``relationships`` lists), the fact is translated into a node
|
||||
(single-arg predicate) or relationship (two-arg predicate) and
|
||||
added/removed accordingly.
|
||||
3. Any other target, or a fact that cannot be translated, raises
|
||||
:class:`ValueError` so the failure surfaces instead of being swallowed.
|
||||
"""
|
||||
if retract:
|
||||
for method_name in ("retract_fact", "remove_fact", "discard_fact"):
|
||||
method = getattr(graph, method_name, None)
|
||||
if callable(method):
|
||||
method(fact)
|
||||
return
|
||||
else:
|
||||
for method_name in ("add_fact", "assert_fact"):
|
||||
method = getattr(graph, method_name, None)
|
||||
if callable(method):
|
||||
method(fact)
|
||||
return
|
||||
|
||||
entities = getattr(graph, "entities", None)
|
||||
relationships = getattr(graph, "relationships", None)
|
||||
if isinstance(entities, list) and isinstance(relationships, list):
|
||||
parsed = _parse_fact(fact)
|
||||
if parsed is None:
|
||||
raise ValueError(
|
||||
f"Cannot translate fact {fact!r} into graph node/relationship: "
|
||||
"expected canonical Predicate(args) form."
|
||||
)
|
||||
predicate, args = parsed
|
||||
if len(args) == 1:
|
||||
node = {"id": args[0], "type": predicate}
|
||||
if retract:
|
||||
_remove_matching(
|
||||
entities,
|
||||
lambda e: e.get("id") == args[0] and e.get("type") == predicate,
|
||||
)
|
||||
elif node not in entities:
|
||||
entities.append(node)
|
||||
return
|
||||
if len(args) == 2:
|
||||
rel = {"source": args[0], "target": args[1], "type": predicate}
|
||||
if retract:
|
||||
_remove_matching(
|
||||
relationships,
|
||||
lambda r: r.get("source") == args[0]
|
||||
and r.get("target") == args[1]
|
||||
and r.get("type") == predicate,
|
||||
)
|
||||
elif rel not in relationships:
|
||||
relationships.append(rel)
|
||||
return
|
||||
raise ValueError(
|
||||
f"Cannot write fact {fact!r} to graph: only unary (node) and binary "
|
||||
"(relationship) predicates are supported by the default adapter."
|
||||
)
|
||||
|
||||
raise ValueError(
|
||||
f"knowledge_graph target {type(graph).__name__!r} does not expose a "
|
||||
"supported write-back API (add_fact/assert_fact or entities/relationships)."
|
||||
)
|
||||
|
||||
|
||||
def _remove_matching(items: List[Dict[str, Any]], predicate: Callable[[Dict[str, Any]], bool]) -> None:
|
||||
"""Remove in place every dict in ``items`` for which ``predicate`` is True."""
|
||||
items[:] = [item for item in items if not predicate(item)]
|
||||
|
||||
class Action:
|
||||
"""Base class for an action fired when a rule matches.
|
||||
|
||||
Actions turn the reasoner from a pure inference engine into a
|
||||
production-rule system: when a rule's conditions match, its actions run
|
||||
with the match's variable bindings, allowing side effects (asserting or
|
||||
retracting facts, calling external tools, emitting events) rather than
|
||||
only deriving a new fact.
|
||||
|
||||
Subclasses implement :meth:`execute`, which receives the substituted
|
||||
``bindings`` and the owning ``reasoner`` and returns an optional
|
||||
description of what happened (used for provenance / explanation).
|
||||
"""
|
||||
|
||||
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
|
||||
raise NotImplementedError
|
||||
|
||||
@staticmethod
|
||||
def _substitute(template: str, bindings: Dict[str, str]) -> str:
|
||||
return _substitute_variables(template, bindings)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AssertAction(Action):
|
||||
"""Assert a new fact when the rule fires.
|
||||
|
||||
``fact`` may contain ``?var`` placeholders that are substituted with the
|
||||
match bindings. If ``write_back`` is set and the reasoner exposes a
|
||||
knowledge graph, the fact is also written there.
|
||||
"""
|
||||
|
||||
fact: str
|
||||
write_back: bool = False
|
||||
|
||||
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
|
||||
concrete = self._substitute(self.fact, bindings)
|
||||
reasoner.facts.add(concrete)
|
||||
if self.write_back and getattr(reasoner, "knowledge_graph", None) is not None:
|
||||
_write_fact_to_graph(reasoner.knowledge_graph, concrete)
|
||||
return f"assert {concrete}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class RetractAction(Action):
|
||||
"""Retract a fact when the rule fires (basic truth maintenance).
|
||||
|
||||
If ``write_back`` is set and the reasoner exposes a knowledge graph, the
|
||||
fact is also removed there using the graph's delete semantics (mirroring
|
||||
:class:`AssertAction`'s write-back).
|
||||
"""
|
||||
|
||||
fact: str
|
||||
write_back: bool = False
|
||||
|
||||
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
|
||||
concrete = self._substitute(self.fact, bindings)
|
||||
reasoner.facts.discard(concrete)
|
||||
if self.write_back and getattr(reasoner, "knowledge_graph", None) is not None:
|
||||
_write_fact_to_graph(reasoner.knowledge_graph, concrete, retract=True)
|
||||
return f"retract {concrete}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CallAction(Action):
|
||||
"""Call an external function/tool when the rule fires.
|
||||
|
||||
Wraps an arbitrary callable, which is invoked as ``func(bindings,
|
||||
reasoner)``. This is the structured replacement for the previously
|
||||
unused ``Rule.handler`` callback.
|
||||
"""
|
||||
|
||||
func: Callable
|
||||
name: str = "call"
|
||||
|
||||
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
|
||||
self.func(bindings, reasoner)
|
||||
return f"call {self.name}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class EmitEventAction(Action):
|
||||
"""Emit an event to the reasoner's registered event sink when fired.
|
||||
|
||||
The event name may contain ``?var`` placeholders. Events are delivered to
|
||||
any callable registered via :meth:`Reasoner.on_event`.
|
||||
"""
|
||||
|
||||
event: str
|
||||
payload: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def execute(self, bindings: Dict[str, str], reasoner: "Reasoner") -> Optional[str]:
|
||||
concrete = self._substitute(self.event, bindings)
|
||||
sink = getattr(reasoner, "_event_sink", None)
|
||||
if callable(sink):
|
||||
sink(concrete, {**self.payload, "bindings": dict(bindings)})
|
||||
return f"emit {concrete}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Rule:
|
||||
"""Simplified rule definition."""
|
||||
@@ -32,6 +335,7 @@ class Rule:
|
||||
confidence: float = 1.0
|
||||
priority: int = 0
|
||||
handler: Optional[Callable] = None
|
||||
actions: List[Action] = field(default_factory=list)
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
@dataclass
|
||||
@@ -79,7 +383,70 @@ class Reasoner:
|
||||
self.rules: List[Rule] = []
|
||||
self.facts: Set[str] = set()
|
||||
self.rule_counter = 0
|
||||
|
||||
self._fired_activations: Set[Tuple[Any, ...]] = set()
|
||||
|
||||
# Optional knowledge graph for AssertAction(write_back=True) targets.
|
||||
self.knowledge_graph = kwargs.get("knowledge_graph")
|
||||
# Optional event sink for EmitEventAction; register via on_event().
|
||||
self._event_sink: Optional[Callable] = None
|
||||
# When True, action-induced fact changes are recorded for provenance
|
||||
# via _record_action() -> self.action_log.
|
||||
self.provenance: bool = bool(kwargs.get("provenance", False))
|
||||
self.action_log: List[Dict[str, Any]] = []
|
||||
|
||||
def on_event(self, sink: Callable) -> None:
|
||||
"""Register a callable ``sink(event_name, payload)`` for EmitEventAction."""
|
||||
self._event_sink = sink
|
||||
|
||||
def _record_action(
|
||||
self, rule: "Rule", action: "Action", description: Optional[str], bindings: Dict[str, str]
|
||||
) -> None:
|
||||
"""Record a fired action for provenance / explanation when enabled.
|
||||
|
||||
Each entry is a structured dict carrying an ISO-8601 ``timestamp`` and a
|
||||
parsed ``operation``/``fact`` split (when the description follows the
|
||||
``"<op> <fact>"`` convention used by the built-in actions) so that
|
||||
downstream consumers such as :class:`ExplanationGenerator` and the
|
||||
provenance layer can reason about *what changed* without re-parsing the
|
||||
free-text description.
|
||||
"""
|
||||
if not self.provenance or description is None:
|
||||
return
|
||||
operation, _, subject = description.partition(" ")
|
||||
self.action_log.append(
|
||||
{
|
||||
"action_id": uuid.uuid4().hex[:8],
|
||||
"rule_id": rule.rule_id,
|
||||
"action": type(action).__name__,
|
||||
"operation": operation or None,
|
||||
"fact": subject or None,
|
||||
"description": description,
|
||||
"bindings": dict(bindings),
|
||||
"confidence": rule.confidence,
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
def _fire_actions(self, rule: "Rule", bindings: Dict[str, str]) -> None:
|
||||
"""Run a fired rule's actions (and legacy handler) with match bindings.
|
||||
|
||||
Backward compatible: a rule with an old-style ``handler`` but no
|
||||
``actions`` still has its handler invoked, so pre-existing rules keep
|
||||
working while new rules use the structured Action layer.
|
||||
"""
|
||||
actions = list(rule.actions)
|
||||
if rule.handler is not None:
|
||||
actions.append(CallAction(rule.handler, name=f"handler:{rule.rule_id}"))
|
||||
for action in actions:
|
||||
try:
|
||||
description = action.execute(bindings, self)
|
||||
self._record_action(rule, action, description, bindings)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self.logger.error(
|
||||
f"Error executing action {type(action).__name__} "
|
||||
f"for rule '{rule.rule_id}': {exc}"
|
||||
)
|
||||
|
||||
def add_rule(self, rule_def: Union[str, Rule]) -> Rule:
|
||||
"""Add a rule to the reasoner.
|
||||
|
||||
@@ -161,6 +528,20 @@ class Reasoner:
|
||||
Returns:
|
||||
List of inferred facts (conclusions)
|
||||
"""
|
||||
return [result.conclusion for result in self.infer_with_results(facts, rules)]
|
||||
|
||||
def infer_with_results(
|
||||
self,
|
||||
facts: Union[List[Any], Dict[str, Any]],
|
||||
rules: Optional[List[Union[str, Rule]]] = None,
|
||||
) -> List[InferenceResult]:
|
||||
"""Infer new facts and return the full :class:`InferenceResult` objects.
|
||||
|
||||
Unlike :meth:`infer_facts` (which returns only conclusion strings for
|
||||
backward compatibility), this preserves each result's ``rule_used``,
|
||||
``premises`` and ``confidence`` so callers such as the provenance
|
||||
wrapper can record real confidence values instead of ``None``.
|
||||
"""
|
||||
tracking_id = self.progress_tracker.start_tracking(
|
||||
module="reasoning",
|
||||
submodule="Reasoner",
|
||||
@@ -180,17 +561,14 @@ class Reasoner:
|
||||
|
||||
# Perform inference
|
||||
results = self.forward_chain()
|
||||
|
||||
# Extract conclusions from results
|
||||
inferred_facts = [result.conclusion for result in results]
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message=f"Inferred {len(inferred_facts)} new facts"
|
||||
message=f"Inferred {len(results)} new facts"
|
||||
)
|
||||
|
||||
return inferred_facts
|
||||
return results
|
||||
|
||||
except Exception as e:
|
||||
self.progress_tracker.stop_tracking(
|
||||
@@ -213,7 +591,15 @@ class Reasoner:
|
||||
new_facts_added = True
|
||||
max_iterations = self.config.get("max_iterations", 50)
|
||||
iteration = 0
|
||||
|
||||
# Activations (rule + concrete bindings + matched facts) whose actions
|
||||
# have already fired. Actions are side-effecting and must
|
||||
# fire exactly once per distinct match, decoupled from whether the
|
||||
# rule's *conclusion* is new. This fixes two failure modes:
|
||||
# * A valid binding whose conclusion is already known (or duplicated
|
||||
# within a pass) previously never fired its actions.
|
||||
# * A RetractAction that removes a premise of its own rule previously
|
||||
# re-fired every pass, iterating to max_iterations. Recording the
|
||||
# activation means it fires once and stops driving iterations.
|
||||
while new_facts_added and iteration < max_iterations:
|
||||
new_facts_added = False
|
||||
iteration += 1
|
||||
@@ -236,7 +622,21 @@ class Reasoner:
|
||||
pass_results: Dict[str, InferenceResult] = {}
|
||||
|
||||
for rule in self.rules:
|
||||
for conclusion, matched_facts in self._match_rule(rule):
|
||||
for conclusion, matched_facts, bindings in self._match_rule(rule):
|
||||
# Fire this activation's actions exactly once, independent
|
||||
# of the conclusion-dedup below. Keyed by rule id + the
|
||||
# concrete bindings so distinct matches each fire, but a
|
||||
# repeated match (same bindings across passes) does not.
|
||||
if rule.actions or rule.handler is not None:
|
||||
activation_key = _make_activation_key(
|
||||
rule.rule_id,
|
||||
bindings,
|
||||
matched_facts,
|
||||
)
|
||||
if activation_key not in self._fired_activations:
|
||||
self._fired_activations.add(activation_key)
|
||||
self._fire_actions(rule, bindings)
|
||||
|
||||
if conclusion in pass_results:
|
||||
# Another derivation of a conclusion already produced
|
||||
# earlier in this same pass: merge premises, dedup.
|
||||
@@ -372,14 +772,17 @@ class Reasoner:
|
||||
conclusion=conclusion_str.strip()
|
||||
)
|
||||
|
||||
def _match_rule(self, rule: Rule) -> List[Tuple[str, List[str]]]:
|
||||
def _match_rule(self, rule: Rule) -> List[Tuple[str, List[str], Dict[str, str]]]:
|
||||
"""
|
||||
Match rule conditions against facts and return instantiated conclusions
|
||||
paired with the facts that satisfied each condition.
|
||||
paired with the facts that satisfied each condition and the variable
|
||||
bindings that produced them.
|
||||
|
||||
Returns:
|
||||
List of (conclusion, matched_facts) tuples, where matched_facts is
|
||||
the ordered list of facts bound to this rule's conditions.
|
||||
List of (conclusion, matched_facts, bindings) tuples, where
|
||||
matched_facts is the ordered list of facts bound to this rule's
|
||||
conditions and bindings maps variable name -> matched value (used
|
||||
to fire the rule's actions).
|
||||
"""
|
||||
if not rule.conditions:
|
||||
return []
|
||||
@@ -410,7 +813,7 @@ class Reasoner:
|
||||
results = []
|
||||
for bindings, matched_facts in bindings_list:
|
||||
instantiated_conclusion = self._substitute(rule.conclusion, bindings)
|
||||
results.append((instantiated_conclusion, matched_facts))
|
||||
results.append((instantiated_conclusion, matched_facts, bindings))
|
||||
|
||||
return results
|
||||
|
||||
@@ -456,16 +859,18 @@ class Reasoner:
|
||||
|
||||
def _substitute(self, pattern: str, bindings: Dict[str, str]) -> str:
|
||||
"""Substitute variables in a pattern with bound values."""
|
||||
result = pattern
|
||||
for var, value in bindings.items():
|
||||
result = result.replace(f"?{var}", value)
|
||||
return result
|
||||
return _substitute_variables(pattern, bindings)
|
||||
|
||||
def reset_action_history(self) -> None:
|
||||
"""Allow previously fired rule activations to execute their actions again."""
|
||||
self._fired_activations.clear()
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Clear facts and rules."""
|
||||
"""Clear facts, rules, and action activation history."""
|
||||
self.facts.clear()
|
||||
self.rules.clear()
|
||||
self.rule_counter = 0
|
||||
self.reset_action_history()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Alias for clear()."""
|
||||
|
||||
@@ -13,9 +13,9 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
from typing import Any, Optional
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
class ReasoningEngineWithProvenance:
|
||||
@@ -28,10 +28,10 @@ class ReasoningEngineWithProvenance:
|
||||
is_automated: bool = True,
|
||||
**config,
|
||||
):
|
||||
from .reasoning_engine import ReasoningEngine
|
||||
from .reasoner import Reasoner
|
||||
|
||||
self.provenance = provenance
|
||||
self._engine = ReasoningEngine(**config)
|
||||
self._engine = Reasoner(provenance=provenance, **config)
|
||||
self._prov_manager = None
|
||||
self._agent_id = agent_id or self.__class__.__name__
|
||||
self._is_automated = is_automated
|
||||
@@ -43,12 +43,27 @@ class ReasoningEngineWithProvenance:
|
||||
except ImportError:
|
||||
self.provenance = False
|
||||
|
||||
def infer(self, premises: Any, source: str = None, **kwargs):
|
||||
"""Perform inference with provenance tracking."""
|
||||
def infer(self, premises: Any, source: str = None, rules: Any = None):
|
||||
"""Perform inference with provenance tracking.
|
||||
|
||||
Only the reasoner's real parameters (``premises`` and ``rules``) are
|
||||
forwarded to the underlying engine; arbitrary keyword arguments are no
|
||||
longer passed through (they previously reached
|
||||
``Reasoner.infer_facts`` -- which accepts only ``facts``/``rules`` --
|
||||
and raised ``TypeError``).
|
||||
"""
|
||||
activity_started_at_time = datetime.utcnow().isoformat()
|
||||
result = self._engine.infer(premises, **kwargs)
|
||||
results = self._engine.infer_with_results(premises, rules)
|
||||
activity_ended_at_time = datetime.utcnow().isoformat()
|
||||
|
||||
# Aggregate confidence across the derived results (min = weakest link);
|
||||
# None only when nothing was inferred.
|
||||
confidence = (
|
||||
min(r.confidence for r in results) if results else None
|
||||
)
|
||||
# Preserve the historical return shape: a list of conclusion strings.
|
||||
inferred = [r.conclusion for r in results]
|
||||
|
||||
if self.provenance and self._prov_manager:
|
||||
self._prov_manager.track_entity(
|
||||
entity_id=f"inference_{uuid.uuid4().hex[:8]}",
|
||||
@@ -61,11 +76,11 @@ class ReasoningEngineWithProvenance:
|
||||
activity_ended_at_time=activity_ended_at_time,
|
||||
metadata={
|
||||
"premises_count": len(premises) if hasattr(premises, '__len__') else 1,
|
||||
"confidence": getattr(result, 'confidence', None)
|
||||
"confidence": confidence,
|
||||
}
|
||||
)
|
||||
|
||||
return result
|
||||
return inferred
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._engine, name)
|
||||
|
||||
@@ -33,14 +33,75 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
from collections import defaultdict
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .reasoner import Fact, Rule
|
||||
from .reasoner import Fact, Rule, _make_activation_key
|
||||
|
||||
|
||||
def _extract_bindings(condition: Any, fact: Fact) -> Dict[str, Any]:
|
||||
"""Extract ``?var`` bindings by matching a condition pattern against a fact.
|
||||
|
||||
``condition`` is the pattern stored on the alpha node (typically a string
|
||||
like ``"Person(?x)"``); ``fact`` is the working-memory :class:`Fact`. The
|
||||
fact's canonical string form (``Predicate(arg1, arg2, ...)``) is matched
|
||||
against the pattern using the same ``?\\w+`` placeholder convention as the
|
||||
Reasoner, so downstream actions receive real bindings (e.g. ``{"x": "John"}``)
|
||||
instead of the empty dict that previously left ``?x`` placeholders
|
||||
unsubstituted.
|
||||
|
||||
Returns an empty dict when the condition is not a string pattern or does
|
||||
not match -- callers treat that as "no bindings extracted".
|
||||
"""
|
||||
if not isinstance(condition, str):
|
||||
return {}
|
||||
|
||||
segments = re.split(r"(\?\w+)", condition)
|
||||
seen_vars: Set[str] = set()
|
||||
p_regex = ""
|
||||
for seg in segments:
|
||||
if seg.startswith("?"):
|
||||
var_name = seg[1:]
|
||||
if var_name in seen_vars:
|
||||
p_regex += f"(?P={var_name})"
|
||||
else:
|
||||
p_regex += f"(?P<{var_name}>.+?)"
|
||||
seen_vars.add(var_name)
|
||||
else:
|
||||
p_regex += re.escape(seg)
|
||||
p_regex = f"^{p_regex}$"
|
||||
|
||||
try:
|
||||
match = re.match(p_regex, str(fact))
|
||||
except re.error:
|
||||
return {}
|
||||
if not match:
|
||||
return {}
|
||||
return {k: v for k, v in match.groupdict().items() if v is not None}
|
||||
|
||||
|
||||
def _bindings_for_rule(rule: Rule, facts: List[Fact]) -> Dict[str, Any]:
|
||||
"""Merge ``?var`` bindings from matching a rule's conditions against facts.
|
||||
|
||||
Each fact is matched against every condition of the rule; the first
|
||||
condition that yields bindings for a fact contributes them. Bindings from
|
||||
all facts are merged so multi-condition (joined) rules receive the full
|
||||
variable environment. Later conflicting values do not overwrite earlier
|
||||
ones, preserving the binding that a join already validated.
|
||||
"""
|
||||
bindings: Dict[str, Any] = {}
|
||||
for fact in facts:
|
||||
for condition in rule.conditions:
|
||||
extracted = _extract_bindings(condition, fact)
|
||||
if not extracted:
|
||||
continue
|
||||
for key, value in extracted.items():
|
||||
bindings.setdefault(key, value)
|
||||
break
|
||||
return bindings
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -58,7 +119,7 @@ class ReteNode:
|
||||
|
||||
def __init__(self, node_id: str):
|
||||
self.node_id = node_id
|
||||
self.children: List["ReteNode"] = []
|
||||
self.children: List[ReteNode] = []
|
||||
|
||||
|
||||
class AlphaNode(ReteNode):
|
||||
@@ -151,6 +212,17 @@ class ReteEngine:
|
||||
self.facts: List[Fact] = []
|
||||
self.fact_counter = 0
|
||||
self.node_counter = 0
|
||||
self._executed_activations: Set[Tuple[Any, ...]] = set()
|
||||
# Optional Reasoner used to fire rule-driven actions on match. When
|
||||
# set, execute_matches() runs each matched rule's ``actions`` (and any
|
||||
# legacy ``handler``) through the Reasoner's action machinery so that
|
||||
# Rete-based matching benefits from the same production-rule behaviour
|
||||
# as forward_chain(). Left None keeps the pure-matching mode.
|
||||
self.reasoner: Optional[Any] = self.config.get("reasoner")
|
||||
|
||||
def bind_reasoner(self, reasoner: Any) -> None:
|
||||
"""Attach a Reasoner so matched rules can fire their actions."""
|
||||
self.reasoner = reasoner
|
||||
|
||||
def build_network(self, rules: List[Rule]) -> None:
|
||||
"""
|
||||
@@ -166,6 +238,7 @@ class ReteEngine:
|
||||
)
|
||||
|
||||
try:
|
||||
self.reset_action_history()
|
||||
self.network.clear()
|
||||
|
||||
self.progress_tracker.update_tracking(
|
||||
@@ -252,15 +325,24 @@ class ReteEngine:
|
||||
# Propagate to children
|
||||
for grandchild in child.children:
|
||||
if isinstance(grandchild, TerminalNode):
|
||||
facts = [left_fact, fact]
|
||||
match = Match(
|
||||
rule=grandchild.rule,
|
||||
facts=[left_fact, fact],
|
||||
facts=facts,
|
||||
bindings=_bindings_for_rule(
|
||||
grandchild.rule, facts
|
||||
),
|
||||
confidence=1.0,
|
||||
)
|
||||
grandchild.activate(match)
|
||||
elif isinstance(child, TerminalNode):
|
||||
# Direct activation
|
||||
match = Match(rule=child.rule, facts=[fact], confidence=1.0)
|
||||
match = Match(
|
||||
rule=child.rule,
|
||||
facts=[fact],
|
||||
bindings=_bindings_for_rule(child.rule, [fact]),
|
||||
confidence=1.0,
|
||||
)
|
||||
child.activate(match)
|
||||
|
||||
def match_patterns(self, facts: Optional[List[Fact]] = None) -> List[Match]:
|
||||
@@ -276,7 +358,7 @@ class ReteEngine:
|
||||
tracking_id = self.progress_tracker.start_tracking(
|
||||
module="reasoning",
|
||||
submodule="ReteEngine",
|
||||
message=f"Matching patterns using Rete algorithm",
|
||||
message="Matching patterns using Rete algorithm",
|
||||
)
|
||||
|
||||
try:
|
||||
@@ -339,10 +421,28 @@ class ReteEngine:
|
||||
)
|
||||
results = []
|
||||
for match in matches:
|
||||
# Conclusions are the pure inference result and remain
|
||||
# independent from optional side-effect execution below.
|
||||
results.append(match.rule.conclusion)
|
||||
try:
|
||||
# Execute rule
|
||||
result = match.rule.conclusion
|
||||
results.append(result)
|
||||
# Fire the rule's actions (and any legacy handler) through
|
||||
# the bound Reasoner so Rete matching produces the same
|
||||
# side effects / provenance as forward_chain(). Falls back
|
||||
# to just recording the conclusion when no Reasoner is bound.
|
||||
if self.reasoner is not None and (
|
||||
match.rule.actions or match.rule.handler is not None
|
||||
):
|
||||
activation_key = _make_activation_key(
|
||||
match.rule.rule_id,
|
||||
match.bindings,
|
||||
[
|
||||
(fact.fact_id, fact.predicate, fact.arguments)
|
||||
for fact in match.facts
|
||||
],
|
||||
)
|
||||
if activation_key not in self._executed_activations:
|
||||
self._executed_activations.add(activation_key)
|
||||
self.reasoner._fire_actions(match.rule, match.bindings)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error executing match: {e}")
|
||||
|
||||
@@ -359,13 +459,16 @@ class ReteEngine:
|
||||
)
|
||||
raise
|
||||
|
||||
def reset_action_history(self) -> None:
|
||||
"""Allow previously executed activations to fire their actions again."""
|
||||
self._executed_activations.clear()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Reset Rete engine."""
|
||||
"""Reset Rete working memory and action activation history."""
|
||||
self.facts.clear()
|
||||
self.reset_action_history()
|
||||
for node in self.network.values():
|
||||
if isinstance(node, AlphaNode):
|
||||
node.matches.clear()
|
||||
elif isinstance(node, BetaNode):
|
||||
if isinstance(node, AlphaNode) or isinstance(node, BetaNode):
|
||||
node.matches.clear()
|
||||
elif isinstance(node, TerminalNode):
|
||||
node.activations.clear()
|
||||
|
||||
@@ -139,17 +139,19 @@ class NERExtractor:
|
||||
if not self.progress_tracker.enabled:
|
||||
self.progress_tracker.enabled = True
|
||||
|
||||
# Initialize spaCy model if ML method is used
|
||||
self.nlp = None
|
||||
# Validate the spaCy runtime up front if ML method is used. The model
|
||||
# itself is loaded lazily by extract_entities_ml() through the
|
||||
# process-level cache in methods.py; this instance only tracks whether
|
||||
# ML dispatch should be attempted at all.
|
||||
self._ml_runtime_usable = True
|
||||
if "ml" in self.method and SPACY_AVAILABLE:
|
||||
try:
|
||||
# Deferred import: keeps semantic_extract.methods out of the
|
||||
# module-level import graph and routes loading through the
|
||||
# module-level import graph and routes validation through the
|
||||
# process-level cache so repeated NERExtractor constructions
|
||||
# never pay the ~120 ms spacy.load() cost more than once.
|
||||
from .methods import load_spacy_model
|
||||
self.nlp = load_spacy_model(self.model_name)
|
||||
load_spacy_model(self.model_name)
|
||||
except OSError:
|
||||
self.logger.warning(
|
||||
f"spaCy model {self.model_name} not found. ML method will fallback."
|
||||
@@ -535,42 +537,6 @@ class NERExtractor:
|
||||
|
||||
return processed
|
||||
|
||||
def _extract_with_spacy(
|
||||
self, text: str, min_confidence: float, entity_types: Optional[List[str]]
|
||||
) -> List[Entity]:
|
||||
"""Extract entities using spaCy."""
|
||||
entities = []
|
||||
|
||||
doc = self.nlp(text)
|
||||
|
||||
for ent in doc.ents:
|
||||
# Filter by entity types if specified
|
||||
if entity_types and ent.label_ not in entity_types:
|
||||
continue
|
||||
|
||||
# Get confidence if available
|
||||
confidence = 1.0
|
||||
if hasattr(ent, "confidence"):
|
||||
confidence = ent.confidence
|
||||
elif hasattr(ent, "score"):
|
||||
confidence = ent.score
|
||||
|
||||
if confidence >= min_confidence:
|
||||
entities.append(
|
||||
Entity(
|
||||
text=ent.text,
|
||||
label=ent.label_,
|
||||
start_char=ent.start_char,
|
||||
end_char=ent.end_char,
|
||||
confidence=confidence,
|
||||
metadata={
|
||||
"lemma": ent.lemma_ if hasattr(ent, "lemma_") else ent.text
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
return entities
|
||||
|
||||
def _extract_fallback(self, text: str) -> List[Entity]:
|
||||
"""Fallback entity extraction using simple patterns."""
|
||||
entities = []
|
||||
|
||||
@@ -123,12 +123,12 @@ class SplitConfig:
|
||||
if value:
|
||||
try:
|
||||
# Try to convert to appropriate type
|
||||
if isinstance(default, int):
|
||||
if isinstance(default, bool):
|
||||
return value.strip().lower() in ("true", "1", "yes", "on")
|
||||
elif isinstance(default, int):
|
||||
return int(value)
|
||||
elif isinstance(default, float):
|
||||
return float(value)
|
||||
elif isinstance(default, bool):
|
||||
return value.lower() in ("true", "1", "yes", "on")
|
||||
return value
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
+3
-3
@@ -5,11 +5,11 @@ This module provides the worker process for the Semantica framework,
|
||||
enabling distributed and background task processing.
|
||||
"""
|
||||
|
||||
import time
|
||||
import signal
|
||||
import sys
|
||||
from .utils.logging import get_logger, setup_logging
|
||||
import time
|
||||
|
||||
from .core.orchestrator import Semantica
|
||||
from .utils.logging import get_logger, setup_logging
|
||||
|
||||
# Initialize logging
|
||||
setup_logging()
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
'''Regression tests for issue #1035.
|
||||
|
||||
Config.get() must honor boolean environment overrides. Previously the type
|
||||
dispatch checked isinstance(default, int) before isinstance(default, bool);
|
||||
since bool subclasses int the bool branch was unreachable, so a boolean
|
||||
default silently ignored the environment variable (or returned an int when the
|
||||
value happened to parse).
|
||||
'''
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from semantica.conflicts.config import ConflictsConfig
|
||||
from semantica.deduplication.config import DeduplicationConfig
|
||||
from semantica.embeddings.config import EmbeddingsConfig
|
||||
from semantica.export.config import ExportConfig
|
||||
from semantica.ingest.config import IngestConfig
|
||||
from semantica.kg.config import KGConfig
|
||||
from semantica.normalize.config import NormalizeConfig
|
||||
from semantica.ontology.config import OntologyConfig
|
||||
from semantica.parse.config import ParseConfig
|
||||
from semantica.split.config import SplitConfig
|
||||
|
||||
|
||||
# The three modules whose get() reaches the type dispatch (no generic
|
||||
# PREFIX_* scanner in _load_env_vars pre-populates _configs).
|
||||
_DISPATCH_MODULES = [
|
||||
("conflicts", "CONFLICT", ConflictsConfig),
|
||||
("deduplication", "DEDUP", DeduplicationConfig),
|
||||
("split", "SPLIT", SplitConfig),
|
||||
]
|
||||
|
||||
|
||||
class TestConfigBoolEnvOverride(unittest.TestCase):
|
||||
"""Boolean env overrides must reach Config.get() on the dispatch path."""
|
||||
|
||||
def test_conflicts_bool_true(self):
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "true"}):
|
||||
self.assertIs(ConflictsConfig().get("zztestflag", False), True)
|
||||
|
||||
def test_conflicts_bool_false(self):
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "false"}):
|
||||
self.assertIs(ConflictsConfig().get("zztestflag", True), False)
|
||||
|
||||
def test_conflicts_int_still_parsed(self):
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "42"}):
|
||||
self.assertEqual(ConflictsConfig().get("zztestflag", 0), 42)
|
||||
|
||||
def test_conflicts_float_still_parsed(self):
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "1.5"}):
|
||||
self.assertEqual(ConflictsConfig().get("zztestflag", 0.0), 1.5)
|
||||
|
||||
def test_conflicts_str_still_returned(self):
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "some-value"}):
|
||||
self.assertEqual(ConflictsConfig().get("zztestflag", "x"), "some-value")
|
||||
|
||||
def test_truthy_spellings(self):
|
||||
for value in ["true", "1", "yes", "on", "TRUE", "True", "YES", "ON", " true "]:
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": value}):
|
||||
self.assertIs(
|
||||
ConflictsConfig().get("zztestflag", False),
|
||||
True,
|
||||
msg="value {0!r} should parse as True".format(value),
|
||||
)
|
||||
|
||||
def test_falsy_spellings(self):
|
||||
for value in ["false", "0", "no", "off", "FALSE", "False", "NO", "OFF", " false "]:
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": value}):
|
||||
self.assertIs(
|
||||
ConflictsConfig().get("zztestflag", True),
|
||||
False,
|
||||
msg="value {0!r} should parse as False".format(value),
|
||||
)
|
||||
|
||||
def test_programmatic_config_takes_precedence(self):
|
||||
cfg = ConflictsConfig()
|
||||
cfg.set("zztestflag", "nope")
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "true"}):
|
||||
self.assertEqual(cfg.get("zztestflag", False), "nope")
|
||||
|
||||
def test_non_bool_value_falls_back_to_default(self):
|
||||
# "not-a-number" is not in the truthy set -> False for a bool default;
|
||||
# the int branch would have raised and swallowed the env var entirely.
|
||||
with patch.dict(os.environ, {"CONFLICT_ZZTESTFLAG": "not-a-number"}):
|
||||
self.assertIs(ConflictsConfig().get("zztestflag", True), False)
|
||||
|
||||
def test_bool_dispatch_across_three_modules(self):
|
||||
for name, prefix, cls in _DISPATCH_MODULES:
|
||||
env_key = prefix + "_ZZTESTFLAG"
|
||||
with self.subTest(module=name, env_key=env_key):
|
||||
with patch.dict(os.environ, {env_key: "true"}):
|
||||
self.assertIs(cls().get("zztestflag", False), True)
|
||||
with patch.dict(os.environ, {env_key: "false"}):
|
||||
self.assertIs(cls().get("zztestflag", True), False)
|
||||
with patch.dict(os.environ, {env_key: "1"}):
|
||||
self.assertIs(cls().get("zztestflag", False), True)
|
||||
with patch.dict(os.environ, {env_key: "0"}):
|
||||
self.assertIs(cls().get("zztestflag", True), False)
|
||||
|
||||
|
||||
# Every module config also maps known boolean env vars in _load_env_vars;
|
||||
# those must parse as real bools too (not ints or strings).
|
||||
_MAPPED_BOOL_ENV = [
|
||||
("conflicts", "CONFLICT_AUTO_RESOLVE", ConflictsConfig),
|
||||
("deduplication", "DEDUP_USE_CLUSTERING", DeduplicationConfig),
|
||||
("embeddings", "EMBEDDING_NORMALIZE", EmbeddingsConfig),
|
||||
("export", "EXPORT_VALIDATE", ExportConfig),
|
||||
("ingest", "INGEST_RECURSIVE", IngestConfig),
|
||||
("kg", "KG_MERGE_ENTITIES", KGConfig),
|
||||
("ontology", "ONTOLOGY_CHECK_CONSISTENCY", OntologyConfig),
|
||||
("parse", "PARSE_EXTRACT_TABLES", ParseConfig),
|
||||
]
|
||||
|
||||
# Modules that only reach bool env parsing via the generic PREFIX_* scanner
|
||||
# in _load_env_vars (no bool entry in env_mappings).
|
||||
_SCANNER_BOOL_ENV = [
|
||||
("split", "SPLIT_ZZTESTFLAG", SplitConfig),
|
||||
("normalize", "NORMALIZE_ZZTESTFLAG", NormalizeConfig),
|
||||
]
|
||||
|
||||
|
||||
class TestMappedBoolEnvVars(unittest.TestCase):
|
||||
def test_mapped_bool_true(self):
|
||||
for module, env_key, cls in _MAPPED_BOOL_ENV:
|
||||
with self.subTest(module=module, env_key=env_key):
|
||||
with patch.dict(os.environ, {env_key: "true"}):
|
||||
cfg = cls()
|
||||
key = env_key.split("_", 1)[1].lower()
|
||||
self.assertIs(cfg.get(key, False), True)
|
||||
|
||||
def test_mapped_bool_false(self):
|
||||
for module, env_key, cls in _MAPPED_BOOL_ENV:
|
||||
with self.subTest(module=module, env_key=env_key):
|
||||
with patch.dict(os.environ, {env_key: "false"}):
|
||||
cfg = cls()
|
||||
key = env_key.split("_", 1)[1].lower()
|
||||
self.assertIs(cfg.get(key, True), False)
|
||||
|
||||
def test_mapped_bool_with_whitespace(self):
|
||||
# Qodo follow-up: _load_env_vars() must strip like get() does, so a
|
||||
# padded value (" true ") is not silently parsed as False.
|
||||
for module, env_key, cls in _MAPPED_BOOL_ENV:
|
||||
with self.subTest(module=module, env_key=env_key):
|
||||
with patch.dict(os.environ, {env_key: " true "}):
|
||||
cfg = cls()
|
||||
key = env_key.split("_", 1)[1].lower()
|
||||
self.assertIs(cfg.get(key, False), True)
|
||||
|
||||
def test_scanner_bool_true(self):
|
||||
for module, env_key, cls in _SCANNER_BOOL_ENV:
|
||||
with self.subTest(module=module, env_key=env_key):
|
||||
with patch.dict(os.environ, {env_key: "true"}):
|
||||
cfg = cls()
|
||||
key = env_key.split("_", 1)[1].lower()
|
||||
self.assertIs(cfg.get(key, False), True)
|
||||
|
||||
def test_scanner_bool_with_whitespace(self):
|
||||
for module, env_key, cls in _SCANNER_BOOL_ENV:
|
||||
with self.subTest(module=module, env_key=env_key):
|
||||
with patch.dict(os.environ, {env_key: " true "}):
|
||||
cfg = cls()
|
||||
key = env_key.split("_", 1)[1].lower()
|
||||
self.assertIs(cfg.get(key, False), True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,8 +1,8 @@
|
||||
"""Tests for RDFExporter format alias resolution (issue #355)."""
|
||||
|
||||
import pytest
|
||||
from semantica.export import RDFExporter
|
||||
|
||||
from semantica.export import RDFExporter
|
||||
|
||||
RDF_DATA = {
|
||||
"entities": [
|
||||
@@ -89,3 +89,154 @@ def test_validate_rdf_returns_overall_valid_key(exporter):
|
||||
result = exporter.validate_rdf(RDF_DATA)
|
||||
assert "overall_valid" in result
|
||||
assert isinstance(result["overall_valid"], bool)
|
||||
|
||||
|
||||
# --- Regression tests for #1097 -------------------------------------------
|
||||
# convert_kg_to_rdf() normalizes an entity's 'name' into 'label'/'text' but was
|
||||
# never called from the export path, so GraphBuilder graphs (which emit 'name')
|
||||
# exported with an empty semantica:text on every RDF format. These tests assert
|
||||
# the human-readable label survives export on all four serializers.
|
||||
|
||||
NAME_ONLY_DATA = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "https://example.org/acme",
|
||||
"name": "Acme Corp",
|
||||
"type": "https://example.org/Org",
|
||||
"confidence": 0.91,
|
||||
},
|
||||
],
|
||||
"relationships": [],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fmt", ["turtle", "ntriples", "rdfxml", "jsonld"])
|
||||
def test_name_only_entity_exports_nonempty_label(exporter, fmt):
|
||||
"""A GraphBuilder-style 'name'-only entity must export a non-empty label.
|
||||
|
||||
Regression for #1097: previously every RDF path dropped the label because
|
||||
convert_kg_to_rdf() was never invoked from export_to_rdf().
|
||||
"""
|
||||
result = exporter.export_to_rdf(NAME_ONLY_DATA, format=fmt)
|
||||
assert "Acme Corp" in result
|
||||
# The empty-text pattern that the bug produced must not appear.
|
||||
assert 'semantica:text ""' not in result
|
||||
|
||||
|
||||
def test_name_only_export_to_file_contains_label(exporter, tmp_path):
|
||||
"""The file-writing entry point must also normalize name -> label (#1097)."""
|
||||
out = tmp_path / "acme.ttl"
|
||||
exporter.export(NAME_ONLY_DATA, str(out), format="turtle")
|
||||
content = out.read_text()
|
||||
assert "Acme Corp" in content
|
||||
assert 'semantica:text ""' not in content
|
||||
|
||||
|
||||
def test_existing_text_not_overwritten_by_name(exporter):
|
||||
"""An entity that already has 'text' must keep it, not be clobbered by 'name' (#1097)."""
|
||||
data = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "https://example.org/acme",
|
||||
"name": "Acme Corp",
|
||||
"text": "Explicit Text",
|
||||
"type": "https://example.org/Org",
|
||||
}
|
||||
],
|
||||
"relationships": [],
|
||||
}
|
||||
result = exporter.export_to_rdf(data, format="turtle")
|
||||
assert "Explicit Text" in result
|
||||
assert "Acme Corp" not in result
|
||||
|
||||
|
||||
def test_id_fallback_label_when_no_name(exporter):
|
||||
"""With neither 'name' nor 'text', the id local-name is used as label (#1097)."""
|
||||
data = {
|
||||
"entities": [
|
||||
{"id": "https://example.org/acme", "type": "https://example.org/Org"}
|
||||
],
|
||||
"relationships": [],
|
||||
}
|
||||
result = exporter.export_to_rdf(data, format="turtle")
|
||||
assert 'semantica:text ""' not in result
|
||||
# The label must be the URI local name 'acme', not '//example.org/acme'.
|
||||
assert 'semantica:text "acme"' in result
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"identifier,expected",
|
||||
[
|
||||
("https://example.org/acme", "acme"),
|
||||
("https://example.org/path/acme", "acme"),
|
||||
("https://example.org/onto#acme", "acme"),
|
||||
("https://example.org/acme/", "acme"),
|
||||
("urn:example:acme", "acme"),
|
||||
("semantica:acme", "acme"),
|
||||
("acme", "acme"),
|
||||
],
|
||||
)
|
||||
def test_id_fallback_local_name_extraction(exporter, identifier, expected):
|
||||
"""The id fallback must extract a URI-aware local name, not a colon split (#1113)."""
|
||||
data = {
|
||||
"entities": [{"id": identifier, "type": "https://example.org/Org"}],
|
||||
"relationships": [],
|
||||
}
|
||||
result = exporter.export_to_rdf(data, format="turtle")
|
||||
assert f'semantica:text "{expected}"' in result
|
||||
|
||||
|
||||
def _export(exporter, name, fmt):
|
||||
data = {
|
||||
"entities": [
|
||||
{"id": "https://example.org/e1", "name": name, "type": "ORG"}
|
||||
],
|
||||
"relationships": [],
|
||||
}
|
||||
return exporter.export_to_rdf(data, format=fmt)
|
||||
|
||||
|
||||
def test_turtle_escapes_quotes_and_control_chars(exporter):
|
||||
"""Turtle literals must escape quotes/backslashes/newlines (#1113 security)."""
|
||||
result = _export(exporter, 'Acme "Best" \\ Corp\nLine2\tTab\rCR', "turtle")
|
||||
# The raw closing-quote breakout must not appear inside the literal.
|
||||
assert '"Acme "Best"' not in result
|
||||
assert '\\"Best\\"' in result
|
||||
assert "\\\\ Corp" in result
|
||||
assert "\\n" in result and "\\t" in result and "\\r" in result
|
||||
# No unescaped newline leaked into the literal value.
|
||||
assert "Line2" in result and "\nLine2" not in result.split("semantica:text")[1]
|
||||
|
||||
|
||||
def test_ntriples_escapes_quotes_and_control_chars(exporter):
|
||||
"""N-Triples literals must escape backslash first, then quotes/controls (#1113)."""
|
||||
result = _export(exporter, 'Quote " Back \\ New\nTab\t', "ntriples")
|
||||
assert '\\"' in result
|
||||
assert "\\\\" in result
|
||||
assert "\\n" in result and "\\t" in result
|
||||
# Each triple must be a single physical line: the literal value carrying the
|
||||
# escaped text must not have leaked a bare newline that splits it in two.
|
||||
text_lines = [ln for ln in result.splitlines() if "Quote" in ln]
|
||||
assert len(text_lines) == 1
|
||||
assert text_lines[0].rstrip().endswith(" .")
|
||||
|
||||
|
||||
def test_rdfxml_escapes_markup(exporter):
|
||||
"""RDF/XML character data must escape &, <, > so names cannot inject markup (#1113)."""
|
||||
result = _export(exporter, 'Acme <script>&"x"', "rdfxml")
|
||||
assert "<script>" not in result
|
||||
assert "<script>" in result
|
||||
assert "&" in result
|
||||
# The document must still parse as well-formed XML.
|
||||
import xml.dom.minidom
|
||||
|
||||
xml.dom.minidom.parseString(result)
|
||||
|
||||
|
||||
def test_turtle_output_is_parseable_with_special_name(exporter):
|
||||
"""A name full of metacharacters must still yield parseable Turtle (#1113)."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
result = _export(exporter, 'Tricky "quote" \\ and <angle> & amp', "turtle")
|
||||
graph = rdflib.Graph()
|
||||
# Should not raise a parser error.
|
||||
graph.parse(data=result, format="turtle")
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
"""Regression tests for the application-id / internal-id mismatch (#1136).
|
||||
|
||||
`GraphStore.add_edges` reads application-level string ids from
|
||||
``source_id``/``target_id`` while backends such as Neo4j match relationships
|
||||
on their internal integer ids (``id(n)``). Every edge therefore failed with
|
||||
"nodes not found" and a graph persisted with all nodes and zero
|
||||
relationships.
|
||||
|
||||
The fix keeps the application-id -> internal-id map that ``add_nodes``
|
||||
already receives from the backend (instead of discarding it) and resolves
|
||||
known string ids in ``create_relationship``. These tests pin that contract
|
||||
with a fake manager — no live database needed.
|
||||
"""
|
||||
|
||||
import unittest
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from semantica.graph_store.graph_store import GraphStore
|
||||
|
||||
|
||||
class FakeNodeManager:
|
||||
"""Mimics NodeManager against a backend that mints internal integer ids."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._next_internal_id = 100
|
||||
self.created: List[Dict[str, Any]] = []
|
||||
|
||||
def create_batch(
|
||||
self, nodes: List[Dict[str, Any]], **options: Any
|
||||
) -> List[Dict[str, Any]]:
|
||||
created: List[Dict[str, Any]] = []
|
||||
for node in nodes:
|
||||
internal_id = self._next_internal_id
|
||||
self._next_internal_id += 1
|
||||
record = {
|
||||
"id": internal_id,
|
||||
"labels": node.get("labels", []),
|
||||
"properties": dict(node.get("properties", {})),
|
||||
}
|
||||
created.append(record)
|
||||
self.created.append(record)
|
||||
return created
|
||||
|
||||
def create(
|
||||
self, labels: List[str], properties: Dict[str, Any], **options: Any
|
||||
) -> Dict[str, Any]:
|
||||
return self.create_batch(
|
||||
[{"labels": labels, "properties": properties}], **options
|
||||
)[0]
|
||||
|
||||
|
||||
class FakeRelationshipManager:
|
||||
"""Records the resolved ids create_relationship hands to the backend."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.calls: List[Tuple[Any, Any, str]] = []
|
||||
|
||||
def create(
|
||||
self,
|
||||
start_node_id: Any,
|
||||
end_node_id: Any,
|
||||
rel_type: str,
|
||||
properties: Optional[Dict[str, Any]] = None,
|
||||
**options: Any,
|
||||
) -> Dict[str, Any]:
|
||||
self.calls.append((start_node_id, end_node_id, rel_type))
|
||||
return {"start": start_node_id, "end": end_node_id, "type": rel_type}
|
||||
|
||||
|
||||
class FakeManager:
|
||||
def __init__(self) -> None:
|
||||
self.nodes = FakeNodeManager()
|
||||
self.relationships = FakeRelationshipManager()
|
||||
|
||||
|
||||
def make_store() -> Tuple[GraphStore, FakeManager]:
|
||||
"""Build a GraphStore around fakes, skipping backend initialization."""
|
||||
store = GraphStore.__new__(GraphStore)
|
||||
store.logger = None # type: ignore[assignment]
|
||||
store.progress_tracker = None # type: ignore[assignment]
|
||||
store.backend = "fake"
|
||||
store.config = {}
|
||||
store._app_node_id_map = {}
|
||||
manager = FakeManager()
|
||||
store._store_backend = None
|
||||
store._manager = manager # type: ignore[assignment]
|
||||
return store, manager
|
||||
|
||||
|
||||
class AppIdResolutionTests(unittest.TestCase):
|
||||
def test_add_edges_resolves_application_ids_to_internal_ids(self) -> None:
|
||||
store, manager = make_store()
|
||||
|
||||
node_count = store.add_nodes(
|
||||
[
|
||||
{"id": "e1", "type": "Person", "properties": {}},
|
||||
{"id": "e2", "type": "Organization", "properties": {}},
|
||||
]
|
||||
)
|
||||
self.assertEqual(node_count, 2)
|
||||
|
||||
edge_count = store.add_edges(
|
||||
[{"source_id": "e1", "target_id": "e2", "type": "knows"}]
|
||||
)
|
||||
|
||||
self.assertEqual(edge_count, 1)
|
||||
self.assertEqual(len(manager.relationships.calls), 1)
|
||||
start, end, rel_type = manager.relationships.calls[0]
|
||||
self.assertEqual(start, 100, "source app id must resolve to the internal id")
|
||||
self.assertEqual(end, 101, "target app id must resolve to the internal id")
|
||||
self.assertEqual(rel_type, "knows")
|
||||
|
||||
def test_build_from_entities_and_relationships_creates_the_edge(self) -> None:
|
||||
"""The exact shape of the #1136 reproduction, against fakes."""
|
||||
store, manager = make_store()
|
||||
|
||||
stats = store.build_from_entities_and_relationships(
|
||||
[
|
||||
{"id": "e1", "type": "Person", "text": "Alice"},
|
||||
{"id": "e2", "type": "Organization", "text": "Acme"},
|
||||
],
|
||||
[{"source_id": "e1", "target_id": "e2", "type": "knows"}],
|
||||
)
|
||||
|
||||
self.assertEqual(stats["statistics"]["node_count"], 2)
|
||||
self.assertEqual(
|
||||
stats["statistics"]["edge_count"], 1, "the edge must be created, not dropped"
|
||||
)
|
||||
self.assertEqual(manager.relationships.calls, [(100, 101, "knows")])
|
||||
|
||||
def test_unknown_ids_pass_through_unchanged(self) -> None:
|
||||
"""Ids the map does not know keep the pre-fix pass-through behavior."""
|
||||
store, manager = make_store()
|
||||
|
||||
store.create_relationship(7, 8, "RELATES_TO")
|
||||
|
||||
self.assertEqual(manager.relationships.calls, [(7, 8, "RELATES_TO")])
|
||||
|
||||
def test_create_node_also_populates_the_map(self) -> None:
|
||||
store, manager = make_store()
|
||||
|
||||
store.create_node(["Person"], {"id": "app-1", "name": "Alice"})
|
||||
store.add_edges([{"source_id": "app-1", "target_id": 42, "type": "knows"}])
|
||||
|
||||
self.assertEqual(manager.relationships.calls, [(100, 42, "knows")])
|
||||
|
||||
def test_integer_application_ids_never_collide_with_internal_ids(self) -> None:
|
||||
# Qodo review: an int application id must not be recorded/resolved,
|
||||
# or a caller passing that same int as an internal id would be
|
||||
# silently remapped to a different node.
|
||||
store, manager = make_store()
|
||||
|
||||
store.add_nodes(
|
||||
[
|
||||
{"id": 100, "type": "Person", "properties": {}},
|
||||
{"id": "str-app", "type": "Person", "properties": {}},
|
||||
]
|
||||
)
|
||||
# Internal id 100 legitimately targets the FIRST node; the integer
|
||||
# app id of that same value must not redirect it to the second.
|
||||
store.create_relationship(100, 101, "RELATES_TO")
|
||||
|
||||
self.assertEqual(manager.relationships.calls, [(100, 101, "RELATES_TO")])
|
||||
self.assertNotIn(100, store._app_node_id_map)
|
||||
|
||||
def test_nodes_without_application_id_do_not_pollute_the_map(self) -> None:
|
||||
store, manager = make_store()
|
||||
|
||||
store.add_nodes([{"labels": ["Person"], "properties": {"name": "Anon"}}])
|
||||
store.create_relationship("not-mapped", 5, "RELATES_TO")
|
||||
|
||||
self.assertEqual(manager.relationships.calls, [("not-mapped", 5, "RELATES_TO")])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -7,6 +7,7 @@ Tests integration between all KG components and algorithms.
|
||||
import pytest
|
||||
import networkx as nx
|
||||
import numpy as np
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, List, Any, Tuple
|
||||
import time
|
||||
import json
|
||||
@@ -14,7 +15,6 @@ import json
|
||||
from semantica.kg import (
|
||||
GraphBuilderWithProvenance,
|
||||
AlgorithmTrackerWithProvenance,
|
||||
NodeEmbedder,
|
||||
SimilarityCalculator,
|
||||
PathFinder,
|
||||
LinkPredictor,
|
||||
@@ -24,6 +24,40 @@ from semantica.kg import (
|
||||
)
|
||||
|
||||
|
||||
def _stored(owner, entity_id):
|
||||
"""Read a provenance record back. Fails if tracking only minted an ID."""
|
||||
record = owner._prov_manager.get_provenance(entity_id)
|
||||
assert record is not None, (
|
||||
f"no stored provenance for {entity_id!r} — an ID was generated without a write"
|
||||
)
|
||||
assert record.get("entity_id") == entity_id
|
||||
return record
|
||||
|
||||
|
||||
def _assert_utc_iso(value, field="timestamp"):
|
||||
assert value, f"missing {field}"
|
||||
parsed = datetime.fromisoformat(value)
|
||||
assert parsed.tzinfo is not None, (
|
||||
f"{field}={value!r} is naive (datetime.utcnow leftover)"
|
||||
)
|
||||
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
|
||||
return parsed
|
||||
|
||||
|
||||
def _assert_tracked(owner, entity_id, source, **metadata):
|
||||
record = _stored(owner, entity_id)
|
||||
assert record["source_document"] == source
|
||||
actual = record.get("metadata") or {}
|
||||
for key, expected in metadata.items():
|
||||
assert actual.get(key) == expected, (
|
||||
f"{entity_id} metadata[{key!r}]={actual.get(key)!r}, expected {expected!r}"
|
||||
)
|
||||
_assert_utc_iso(record["timestamp"], "timestamp")
|
||||
if record.get("last_updated"):
|
||||
_assert_utc_iso(record["last_updated"], "last_updated")
|
||||
return record
|
||||
|
||||
|
||||
class TestComprehensiveIntegration:
|
||||
"""Comprehensive integration tests for KG module."""
|
||||
|
||||
@@ -157,8 +191,6 @@ class TestComprehensiveIntegration:
|
||||
# Initialize all components
|
||||
builder = GraphBuilderWithProvenance(provenance=True)
|
||||
tracker = AlgorithmTrackerWithProvenance(provenance=True)
|
||||
embedder = NodeEmbedder()
|
||||
embedder.enable_provenance = True
|
||||
sim_calc = SimilarityCalculator()
|
||||
path_finder = PathFinder()
|
||||
link_predictor = LinkPredictor()
|
||||
@@ -351,14 +383,34 @@ class TestComprehensiveIntegration:
|
||||
source='comprehensive_integration_test'
|
||||
)
|
||||
|
||||
# Verify all phases completed successfully
|
||||
expected = {
|
||||
'construction': 'graph_construction',
|
||||
'centrality': 'centrality_calculation',
|
||||
'connectivity': 'connectivity_analysis',
|
||||
'community': 'community_detection',
|
||||
'similarity': 'similarity_calculation',
|
||||
'link_prediction': 'link_prediction',
|
||||
'path_analysis': 'path_analysis',
|
||||
'cross_layer': 'cross_layer_analysis',
|
||||
}
|
||||
assert len(execution_ids) == 8
|
||||
for phase, exec_id in execution_ids.items():
|
||||
assert exec_id is not None
|
||||
assert len(exec_id) > 10
|
||||
|
||||
print(f"Full pipeline integration completed: {pipeline_id}")
|
||||
return pipeline_id
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
exec_id,
|
||||
source=pipeline_id,
|
||||
entity_type=expected[phase],
|
||||
)
|
||||
summary_record = _assert_tracked(
|
||||
tracker,
|
||||
summary_id,
|
||||
source='comprehensive_integration_test',
|
||||
entity_type='pipeline_summary',
|
||||
pipeline_id=pipeline_id,
|
||||
phases_count=8,
|
||||
)
|
||||
assert summary_id.startswith('pipeline_summary_')
|
||||
assert summary_record['metadata']['input_data_size'] == len(complex_graph_data)
|
||||
|
||||
def test_multi_layer_network_analysis(self, multi_layer_network, realistic_embeddings):
|
||||
"""Test multi-layer network analysis."""
|
||||
@@ -382,39 +434,39 @@ class TestComprehensiveIntegration:
|
||||
|
||||
# Centrality analysis
|
||||
if graph.number_of_nodes() > 0:
|
||||
try:
|
||||
degree_cent = centrality_calc.calculate_degree_centrality(graph_dict)
|
||||
layer_results[f"{layer_name}_centrality"] = degree_cent
|
||||
|
||||
# Track with provenance
|
||||
cent_id = tracker.track_layer_analysis(
|
||||
layer_name=layer_name,
|
||||
graph=graph,
|
||||
analysis_type='centrality',
|
||||
results=degree_cent,
|
||||
source=multi_layer_id
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Centrality analysis failed for {layer_name}: {e}")
|
||||
degree_cent = centrality_calc.calculate_degree_centrality(graph_dict)
|
||||
layer_results[f"{layer_name}_centrality"] = degree_cent
|
||||
cent_id = tracker.track_centrality_calculation(
|
||||
graph=graph,
|
||||
centrality_scores=degree_cent['centrality'],
|
||||
method='degree',
|
||||
source=multi_layer_id
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
cent_id,
|
||||
source=multi_layer_id,
|
||||
method='degree',
|
||||
scores_count=len(degree_cent['centrality']),
|
||||
)
|
||||
|
||||
# Community detection
|
||||
if graph.number_of_edges() > 0:
|
||||
try:
|
||||
communities = community_detector.detect_communities(graph_dict, method='label_propagation')
|
||||
layer_results[f"{layer_name}_communities"] = communities
|
||||
|
||||
# Track with provenance
|
||||
comm_id = tracker.track_layer_analysis(
|
||||
layer_name=layer_name,
|
||||
graph=graph,
|
||||
analysis_type='communities',
|
||||
results=communities,
|
||||
source=multi_layer_id
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Community detection failed for {layer_name}: {e}")
|
||||
communities = community_detector.detect_communities(graph_dict, method='label_propagation')
|
||||
layer_results[f"{layer_name}_communities"] = communities
|
||||
comm_id = tracker.track_community_detection(
|
||||
graph=graph,
|
||||
communities=communities['communities'],
|
||||
method='label_propagation',
|
||||
source=multi_layer_id
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
comm_id,
|
||||
source=multi_layer_id,
|
||||
method='label_propagation',
|
||||
communities_count=len(communities['communities']),
|
||||
)
|
||||
|
||||
# Cross-layer similarity analysis
|
||||
print("Cross-layer similarity analysis")
|
||||
@@ -433,39 +485,43 @@ class TestComprehensiveIntegration:
|
||||
similarity_score = len(common_nodes) / max(len(graph1.nodes()), len(graph2.nodes()))
|
||||
layer_similarities[f"{layer1_name}_{layer2_name}"] = similarity_score
|
||||
|
||||
# Track cross-layer analysis
|
||||
cross_layer_id = tracker.track_cross_layer_analysis(
|
||||
multi_layer_network=multi_layer_network,
|
||||
layer_similarities=layer_similarities,
|
||||
graph_data=multi_layer_network,
|
||||
cross_layer_results=layer_similarities,
|
||||
source='multi_layer_test'
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
cross_layer_id,
|
||||
source='multi_layer_test',
|
||||
entity_type='cross_layer_analysis',
|
||||
layers_count=len(layer_similarities),
|
||||
)
|
||||
|
||||
# Embedding-based entity similarity
|
||||
print("Embedding-based entity similarity")
|
||||
|
||||
entity_similarities = sim_calc.pairwise_similarity(realistic_embeddings)
|
||||
|
||||
# Track embedding analysis
|
||||
embed_id = tracker.track_embedding_analysis(
|
||||
embeddings=realistic_embeddings,
|
||||
similarities=entity_similarities,
|
||||
analysis_results=entity_similarities,
|
||||
source='multi_layer_test'
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
embed_id,
|
||||
source='multi_layer_test',
|
||||
entity_type='embedding_analysis',
|
||||
embeddings_count=len(realistic_embeddings),
|
||||
)
|
||||
|
||||
# Verify results
|
||||
assert len(layer_results) > 0
|
||||
assert len(layer_similarities) > 0
|
||||
assert len(entity_similarities) > 0
|
||||
|
||||
print(f"Multi-layer analysis completed")
|
||||
print(f"Layers analyzed: {list(multi_layer_network.keys())}")
|
||||
print(f"Layer similarities: {list(layer_similarities.keys())}")
|
||||
|
||||
return multi_layer_id
|
||||
|
||||
def test_error_handling_and_recovery(self):
|
||||
"""Test error handling and recovery mechanisms."""
|
||||
tracker = AlgorithmTrackerWithProvenance(provenance=True)
|
||||
centrality_calc = CentralityCalculator()
|
||||
|
||||
# Test with invalid graph data
|
||||
invalid_graph = {
|
||||
@@ -473,46 +529,24 @@ class TestComprehensiveIntegration:
|
||||
'edges': []
|
||||
}
|
||||
|
||||
# Should handle gracefully
|
||||
try:
|
||||
centrality_calc = CentralityCalculator()
|
||||
result = centrality_calc.calculate_degree_centrality(invalid_graph)
|
||||
# Should return empty result or handle gracefully
|
||||
assert isinstance(result, dict)
|
||||
except Exception as e:
|
||||
# Should be a controlled exception
|
||||
assert isinstance(e, (ValueError, RuntimeError))
|
||||
result = centrality_calc.calculate_degree_centrality(invalid_graph)
|
||||
assert isinstance(result, dict)
|
||||
|
||||
# Test with invalid embeddings
|
||||
invalid_embeddings = {
|
||||
'node1': [1, 2], # Different dimensions
|
||||
'node2': [1, 2, 3, 4] # Different dimensions
|
||||
}
|
||||
|
||||
try:
|
||||
sim_calc = SimilarityCalculator()
|
||||
result = sim_calc.batch_similarity(
|
||||
embeddings=invalid_embeddings,
|
||||
query_embedding=[1, 2, 3, 4],
|
||||
method='cosine'
|
||||
)
|
||||
# Should handle dimension mismatch
|
||||
except Exception as e:
|
||||
# Should handle gracefully
|
||||
assert isinstance(e, ValueError)
|
||||
|
||||
# Test provenance tracking with invalid data
|
||||
try:
|
||||
result = tracker.track_embedding_computation(
|
||||
graph=None, # Invalid graph
|
||||
algorithm='test',
|
||||
embeddings={},
|
||||
parameters={}
|
||||
)
|
||||
# Should either return None or handle gracefully
|
||||
except Exception as e:
|
||||
# Should be a controlled exception
|
||||
assert isinstance(e, (ValueError, TypeError))
|
||||
# Provenance tracking with a None graph still writes a record.
|
||||
result = tracker.track_embedding_computation(
|
||||
graph=None,
|
||||
algorithm='test',
|
||||
embeddings={},
|
||||
parameters={},
|
||||
source='integration_error_recovery'
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
result,
|
||||
source='integration_error_recovery',
|
||||
algorithm='test',
|
||||
input_data_type='NoneType',
|
||||
)
|
||||
|
||||
# Test graceful degradation when provenance is disabled
|
||||
tracker_no_prov = AlgorithmTrackerWithProvenance(provenance=False)
|
||||
@@ -526,8 +560,7 @@ class TestComprehensiveIntegration:
|
||||
|
||||
# Should return None when provenance is disabled
|
||||
assert result is None
|
||||
|
||||
print("Error handling and recovery test completed")
|
||||
assert tracker_no_prov._prov_manager is None
|
||||
|
||||
def test_performance_benchmarks(self, realistic_embeddings):
|
||||
"""Test performance benchmarks with realistic data."""
|
||||
|
||||
@@ -7,13 +7,13 @@ Tests complete provenance tracking workflows across multiple algorithms.
|
||||
import pytest
|
||||
import networkx as nx
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, List, Any
|
||||
import uuid
|
||||
|
||||
from semantica.kg import (
|
||||
GraphBuilderWithProvenance,
|
||||
AlgorithmTrackerWithProvenance,
|
||||
NodeEmbedder,
|
||||
SimilarityCalculator,
|
||||
LinkPredictor,
|
||||
CentralityCalculator,
|
||||
@@ -21,6 +21,40 @@ from semantica.kg import (
|
||||
)
|
||||
|
||||
|
||||
def _stored(owner, entity_id):
|
||||
"""Read a provenance record back. Fails if tracking only minted an ID."""
|
||||
record = owner._prov_manager.get_provenance(entity_id)
|
||||
assert record is not None, (
|
||||
f"no stored provenance for {entity_id!r} — an ID was generated without a write"
|
||||
)
|
||||
assert record.get("entity_id") == entity_id
|
||||
return record
|
||||
|
||||
|
||||
def _assert_utc_iso(value, field="timestamp"):
|
||||
assert value, f"missing {field}"
|
||||
parsed = datetime.fromisoformat(value)
|
||||
assert parsed.tzinfo is not None, (
|
||||
f"{field}={value!r} is naive (datetime.utcnow leftover)"
|
||||
)
|
||||
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
|
||||
return parsed
|
||||
|
||||
|
||||
def _assert_tracked(owner, entity_id, source, **metadata):
|
||||
record = _stored(owner, entity_id)
|
||||
assert record["source_document"] == source
|
||||
actual = record.get("metadata") or {}
|
||||
for key, expected in metadata.items():
|
||||
assert actual.get(key) == expected, (
|
||||
f"{entity_id} metadata[{key!r}]={actual.get(key)!r}, expected {expected!r}"
|
||||
)
|
||||
_assert_utc_iso(record["timestamp"], "timestamp")
|
||||
if record.get("last_updated"):
|
||||
_assert_utc_iso(record["last_updated"], "last_updated")
|
||||
return record
|
||||
|
||||
|
||||
class TestProvenanceWorkflows:
|
||||
"""Test provenance tracking workflows."""
|
||||
|
||||
@@ -116,10 +150,25 @@ class TestProvenanceWorkflows:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert construction_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
construction_id,
|
||||
source=workflow_id,
|
||||
entity_type="graph_construction",
|
||||
entities_count=8,
|
||||
relationships_count=12,
|
||||
)
|
||||
assert construction_id.startswith('graph_construction_')
|
||||
|
||||
for entity in graph_result['entities']:
|
||||
built = _stored(builder, entity['id'])
|
||||
assert built['metadata']['operation'] == 'build_entity'
|
||||
assert built['metadata']['entity_type'] == entity['type']
|
||||
_assert_utc_iso(built['activity_started_at_time'], 'activity_started_at_time')
|
||||
_assert_utc_iso(built['activity_ended_at_time'], 'activity_ended_at_time')
|
||||
|
||||
# Step 3: Track entity processing
|
||||
processed = []
|
||||
for entity in graph_result['entities']:
|
||||
entity_id = tracker.track_entity_processing(
|
||||
entity_id=entity['id'],
|
||||
@@ -127,9 +176,19 @@ class TestProvenanceWorkflows:
|
||||
entity_data=entity,
|
||||
source=workflow_id
|
||||
)
|
||||
assert entity_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
entity_id,
|
||||
source=workflow_id,
|
||||
processed_entity_id=entity['id'],
|
||||
processed_entity_type=entity['type'],
|
||||
)
|
||||
processed.append(entity_id)
|
||||
assert len(processed) == 8
|
||||
assert len(set(processed)) == 8
|
||||
|
||||
# Step 4: Track relationship processing
|
||||
rel_ids = []
|
||||
for relationship in graph_result['relationships']:
|
||||
rel_id = tracker.track_relationship_processing(
|
||||
relationship_id=f"{relationship['source']}-{relationship['target']}",
|
||||
@@ -137,10 +196,15 @@ class TestProvenanceWorkflows:
|
||||
relationship_data=relationship,
|
||||
source=workflow_id
|
||||
)
|
||||
assert rel_id is not None
|
||||
|
||||
print(f"Graph construction workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
rel_id,
|
||||
source=workflow_id,
|
||||
processed_relationship_type=relationship['type'],
|
||||
)
|
||||
rel_ids.append(rel_id)
|
||||
assert len(rel_ids) == 12
|
||||
assert len(set(rel_ids)) == 12
|
||||
|
||||
def test_embedding_workflow(self, workflow_graph, workflow_embeddings):
|
||||
"""Test complete embedding workflow with provenance."""
|
||||
@@ -176,8 +240,20 @@ class TestProvenanceWorkflows:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert embed_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
embed_id,
|
||||
source=workflow_id,
|
||||
algorithm='node2vec',
|
||||
node_count=6,
|
||||
embedding_dimension=4,
|
||||
)
|
||||
assert embed_id.startswith('embedding_')
|
||||
for node_id, vector in computed_embeddings.items():
|
||||
node_record = _stored(tracker, f"embedding_{node_id}")
|
||||
assert node_record['metadata']['node_id'] == node_id
|
||||
assert node_record['metadata']['execution_id'] == embed_id
|
||||
assert node_record['metadata']['embedding_dimension'] == len(vector)
|
||||
|
||||
# Step 2: Track embedding quality metrics
|
||||
quality_metrics = {
|
||||
@@ -196,10 +272,13 @@ class TestProvenanceWorkflows:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert quality_id is not None
|
||||
|
||||
print(f"Embedding workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
quality_id,
|
||||
source=workflow_id,
|
||||
algorithm='node2vec_quality_check',
|
||||
)
|
||||
assert quality_id != embed_id
|
||||
|
||||
def test_similarity_analysis_workflow(self, workflow_embeddings):
|
||||
"""Test complete similarity analysis workflow with provenance."""
|
||||
@@ -229,10 +308,18 @@ class TestProvenanceWorkflows:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert sim_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
sim_id,
|
||||
source=workflow_id,
|
||||
method='cosine',
|
||||
similarities_count=len(similarities),
|
||||
)
|
||||
assert sim_id.startswith('similarity_')
|
||||
assert len(similarities) == 3
|
||||
|
||||
# Step 2: Track individual similarity results
|
||||
result_ids = []
|
||||
for node_id, similarity_score in similarities.items():
|
||||
result_id = tracker.track_similarity_result(
|
||||
node_id=node_id,
|
||||
@@ -241,7 +328,17 @@ class TestProvenanceWorkflows:
|
||||
execution_id=sim_id,
|
||||
source=workflow_id
|
||||
)
|
||||
assert result_id is not None
|
||||
record = _assert_tracked(
|
||||
tracker,
|
||||
result_id,
|
||||
source=workflow_id,
|
||||
node_id=node_id,
|
||||
method='cosine',
|
||||
execution_id=sim_id,
|
||||
)
|
||||
assert record['metadata']['similarity_score'] == similarity_score
|
||||
result_ids.append(result_id)
|
||||
assert len(result_ids) == len(similarities)
|
||||
|
||||
# Step 3: Track similarity threshold analysis
|
||||
threshold = 0.7
|
||||
@@ -254,10 +351,14 @@ class TestProvenanceWorkflows:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert threshold_id is not None
|
||||
|
||||
print(f"Similarity analysis workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
threshold_id,
|
||||
source=workflow_id,
|
||||
threshold=threshold,
|
||||
execution_id=sim_id,
|
||||
)
|
||||
assert _stored(tracker, threshold_id)['metadata']['high_similarity_count'] == len(high_similarity)
|
||||
|
||||
def test_link_prediction_workflow(self, workflow_graph):
|
||||
"""Test complete link prediction workflow with provenance."""
|
||||
@@ -287,7 +388,13 @@ class TestProvenanceWorkflows:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert pred_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
pred_id,
|
||||
source=workflow_id,
|
||||
method=method,
|
||||
predictions_count=len(predictions),
|
||||
)
|
||||
assert pred_id.startswith('link_prediction_')
|
||||
|
||||
# Step 2: Track individual predictions
|
||||
@@ -300,10 +407,17 @@ class TestProvenanceWorkflows:
|
||||
execution_id=pred_id,
|
||||
source=workflow_id
|
||||
)
|
||||
assert result_id is not None
|
||||
|
||||
print(f"Link prediction workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
record = _assert_tracked(
|
||||
tracker,
|
||||
result_id,
|
||||
source=workflow_id,
|
||||
source_node=source,
|
||||
target_node=target,
|
||||
method=method,
|
||||
execution_id=pred_id,
|
||||
)
|
||||
assert record['metadata']['prediction_score'] == score
|
||||
assert len(methods) == 3
|
||||
|
||||
def test_centrality_analysis_workflow(self, workflow_graph):
|
||||
"""Test complete centrality analysis workflow with provenance."""
|
||||
@@ -326,54 +440,42 @@ class TestProvenanceWorkflows:
|
||||
('eigenvector', centrality_calc.calculate_eigenvector_centrality)
|
||||
]
|
||||
|
||||
tracked_methods = []
|
||||
for method_name, method_func in centrality_methods:
|
||||
try:
|
||||
start_time = time.time()
|
||||
result = method_func(graph_dict)
|
||||
calculation_time = time.time() - start_time
|
||||
|
||||
cent_id = tracker.track_centrality_calculation(
|
||||
graph=workflow_graph,
|
||||
centrality_scores=result['centrality'],
|
||||
method=method_name,
|
||||
parameters={},
|
||||
calculation_time=calculation_time,
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert cent_id is not None
|
||||
assert cent_id.startswith('centrality_')
|
||||
|
||||
# Step 2: Track individual centrality scores
|
||||
for node_id, score in result['centrality'].items():
|
||||
score_id = tracker.track_centrality_score(
|
||||
node_id=node_id,
|
||||
centrality_score=score,
|
||||
method=method_name,
|
||||
execution_id=cent_id,
|
||||
source=workflow_id
|
||||
)
|
||||
assert score_id is not None
|
||||
|
||||
# Step 3: Track centrality ranking analysis
|
||||
rankings = result['rankings']
|
||||
top_nodes = rankings[:3] # Top 3 nodes
|
||||
|
||||
ranking_id = tracker.track_centrality_ranking(
|
||||
execution_id=cent_id,
|
||||
rankings=rankings,
|
||||
top_nodes=top_nodes,
|
||||
method=method_name,
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert ranking_id is not None
|
||||
|
||||
except Exception as e:
|
||||
print(f"Warning: {method_name} centrality failed: {e}")
|
||||
|
||||
print(f"Centrality analysis workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
except Exception:
|
||||
# Algorithm failure is allowed for optional methods, not for degree.
|
||||
if method_name == 'degree':
|
||||
raise
|
||||
continue
|
||||
|
||||
calculation_time = 0.0
|
||||
cent_id = tracker.track_centrality_calculation(
|
||||
graph=workflow_graph,
|
||||
centrality_scores=result['centrality'],
|
||||
method=method_name,
|
||||
parameters={},
|
||||
calculation_time=calculation_time,
|
||||
source=workflow_id
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
cent_id,
|
||||
source=workflow_id,
|
||||
method=method_name,
|
||||
scores_count=len(result['centrality']),
|
||||
)
|
||||
assert cent_id.startswith('centrality_')
|
||||
for node_id, score in result['centrality'].items():
|
||||
score_record = _stored(tracker, f"centrality_{node_id}_{cent_id}")
|
||||
assert score_record['metadata']['node_id'] == node_id
|
||||
assert score_record['metadata']['method'] == method_name
|
||||
assert score_record['metadata']['centrality_score'] == score
|
||||
tracked_methods.append(method_name)
|
||||
|
||||
assert 'degree' in tracked_methods
|
||||
assert len(tracked_methods) >= 1
|
||||
|
||||
def test_community_detection_workflow(self, workflow_graph):
|
||||
"""Test complete community detection workflow with provenance."""
|
||||
@@ -391,56 +493,39 @@ class TestProvenanceWorkflows:
|
||||
# Step 1: Track community detection
|
||||
methods = ['label_propagation', 'louvain']
|
||||
|
||||
tracked_methods = []
|
||||
for method in methods:
|
||||
try:
|
||||
start_time = time.time()
|
||||
result = community_detector.detect_communities(graph_dict, method=method)
|
||||
detection_time = time.time() - start_time
|
||||
|
||||
comm_id = tracker.track_community_detection(
|
||||
graph=workflow_graph,
|
||||
communities=result['communities'],
|
||||
method=method,
|
||||
parameters={},
|
||||
detection_time=detection_time,
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert comm_id is not None
|
||||
assert comm_id.startswith('community_')
|
||||
|
||||
# Step 2: Track individual communities
|
||||
for i, community in enumerate(result['communities']):
|
||||
comm_result_id = tracker.track_community_result(
|
||||
community_id=i,
|
||||
nodes=community,
|
||||
method=method,
|
||||
execution_id=comm_id,
|
||||
source=workflow_id
|
||||
)
|
||||
assert comm_result_id is not None
|
||||
|
||||
# Step 3: Track community quality metrics
|
||||
quality_metrics = {
|
||||
'modularity': 0.3,
|
||||
'num_communities': len(result['communities']),
|
||||
'avg_community_size': len(result['communities']) / len(result['communities']) if result['communities'] else 0
|
||||
}
|
||||
|
||||
quality_id = tracker.track_community_quality(
|
||||
execution_id=comm_id,
|
||||
metrics=quality_metrics,
|
||||
method=method,
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert quality_id is not None
|
||||
|
||||
except Exception as e:
|
||||
print(f"Warning: {method} community detection failed: {e}")
|
||||
|
||||
print(f"Community detection workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
except Exception:
|
||||
if method == 'label_propagation':
|
||||
raise
|
||||
continue
|
||||
|
||||
comm_id = tracker.track_community_detection(
|
||||
graph=workflow_graph,
|
||||
communities=result['communities'],
|
||||
method=method,
|
||||
parameters={},
|
||||
detection_time=0.0,
|
||||
source=workflow_id
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
comm_id,
|
||||
source=workflow_id,
|
||||
method=method,
|
||||
communities_count=len(result['communities']),
|
||||
)
|
||||
assert comm_id.startswith('community_')
|
||||
for i, community in enumerate(result['communities']):
|
||||
comm_record = _stored(tracker, f"community_{comm_id}_{i}")
|
||||
assert comm_record['metadata']['community_id'] == i
|
||||
assert comm_record['metadata']['method'] == method
|
||||
assert comm_record['metadata']['nodes'] == community
|
||||
tracked_methods.append(method)
|
||||
|
||||
assert 'label_propagation' in tracked_methods
|
||||
|
||||
def test_comprehensive_provenance_workflow(self, workflow_data, workflow_graph, workflow_embeddings):
|
||||
"""Test comprehensive provenance workflow combining all algorithms."""
|
||||
@@ -555,29 +640,37 @@ class TestProvenanceWorkflows:
|
||||
source='comprehensive_test'
|
||||
)
|
||||
|
||||
# Verify all execution IDs
|
||||
# Verify all execution IDs were stored with the expected payload
|
||||
assert len(execution_ids) == 6
|
||||
expected_types = {
|
||||
'construction': ('graph_construction_', 'graph_construction'),
|
||||
'embedding': ('embedding_', 'embedding_computation'),
|
||||
'similarity': ('similarity_', 'similarity_calculation'),
|
||||
'link_prediction': ('link_prediction_', 'link_prediction'),
|
||||
'centrality': ('centrality_', 'centrality_calculation'),
|
||||
'community_detection': ('community_', 'community_detection'),
|
||||
}
|
||||
for phase, exec_id in execution_ids.items():
|
||||
assert exec_id is not None
|
||||
assert len(exec_id) > 10
|
||||
|
||||
# Verify all IDs are unique
|
||||
prefix, entity_type = expected_types[phase]
|
||||
assert exec_id.startswith(prefix)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
exec_id,
|
||||
source=master_workflow_id,
|
||||
entity_type=entity_type,
|
||||
)
|
||||
|
||||
summary_record = _assert_tracked(
|
||||
tracker,
|
||||
summary_id,
|
||||
source='comprehensive_test',
|
||||
entity_type='workflow_summary',
|
||||
)
|
||||
assert summary_id.startswith('workflow_summary_')
|
||||
assert summary_record['metadata']['master_workflow_id'] == master_workflow_id
|
||||
|
||||
all_ids = list(execution_ids.values()) + [summary_id]
|
||||
assert len(set(all_ids)) == len(all_ids)
|
||||
|
||||
# Verify ID prefixes
|
||||
assert execution_ids['construction'].startswith('graph_construction_')
|
||||
assert execution_ids['embedding'].startswith('embedding_')
|
||||
assert execution_ids['similarity'].startswith('similarity_')
|
||||
assert execution_ids['link_prediction'].startswith('link_prediction_')
|
||||
assert execution_ids['centrality'].startswith('centrality_')
|
||||
assert execution_ids['community_detection'].startswith('community_')
|
||||
assert summary_id.startswith('workflow_summary_')
|
||||
|
||||
print(f"Comprehensive provenance workflow completed: {master_workflow_id}")
|
||||
print(f"Execution IDs: {list(execution_ids.keys())}")
|
||||
|
||||
return master_workflow_id
|
||||
|
||||
def test_provenance_data_integrity(self, workflow_graph, workflow_embeddings):
|
||||
"""Test provenance data integrity and consistency."""
|
||||
@@ -618,24 +711,24 @@ class TestProvenanceWorkflows:
|
||||
)
|
||||
operations.append(('link_prediction', link_id))
|
||||
|
||||
# Verify data integrity
|
||||
expected = {
|
||||
'embedding': ('embedding_', 'embedding_computation', embed_id),
|
||||
'similarity': ('similarity_', 'similarity_calculation', sim_id),
|
||||
'link_prediction': ('link_prediction_', 'link_prediction', link_id),
|
||||
}
|
||||
for op_type, op_id in operations:
|
||||
assert op_id is not None
|
||||
assert len(op_id) > 10
|
||||
|
||||
# Verify ID format consistency
|
||||
if op_type == 'embedding':
|
||||
assert op_id.startswith('embedding_')
|
||||
elif op_type == 'similarity':
|
||||
assert op_id.startswith('similarity_')
|
||||
elif op_type == 'link_prediction':
|
||||
assert op_id.startswith('link_prediction_')
|
||||
|
||||
# Verify workflow consistency
|
||||
prefix, entity_type, expected_id = expected[op_type]
|
||||
assert op_id == expected_id
|
||||
assert op_id.startswith(prefix)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
op_id,
|
||||
source=workflow_id,
|
||||
entity_type=entity_type,
|
||||
)
|
||||
|
||||
workflow_ids = [op_id for _, op_id in operations]
|
||||
assert len(set(workflow_ids)) == len(workflow_ids) # All unique
|
||||
|
||||
print(f"Provenance data integrity test completed: {workflow_id}")
|
||||
assert len(set(workflow_ids)) == len(workflow_ids)
|
||||
|
||||
def test_provenance_error_recovery(self):
|
||||
"""Test provenance system error recovery."""
|
||||
@@ -650,24 +743,27 @@ class TestProvenanceWorkflows:
|
||||
)
|
||||
|
||||
assert result is None # Should return None when provenance is disabled
|
||||
assert tracker_no_prov._prov_manager is None
|
||||
|
||||
# Test error handling during tracking
|
||||
# Tracking still records an execution when the graph is None — that is
|
||||
# current production behavior, so assert the write rather than
|
||||
# "either returns or raises".
|
||||
tracker = AlgorithmTrackerWithProvenance(provenance=True)
|
||||
|
||||
# This should not raise exceptions even with invalid data
|
||||
try:
|
||||
result = tracker.track_embedding_computation(
|
||||
graph=None, # Invalid graph
|
||||
algorithm='test',
|
||||
embeddings={},
|
||||
parameters={}
|
||||
)
|
||||
# Should either return None or handle gracefully
|
||||
except Exception as e:
|
||||
# If it raises, it should be a controlled exception
|
||||
assert isinstance(e, (ValueError, TypeError))
|
||||
|
||||
print("Provenance error recovery test completed")
|
||||
result = tracker.track_embedding_computation(
|
||||
graph=None,
|
||||
algorithm='test',
|
||||
embeddings={},
|
||||
parameters={},
|
||||
source='error_recovery'
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
result,
|
||||
source='error_recovery',
|
||||
algorithm='test',
|
||||
input_data_type='NoneType',
|
||||
node_count=0,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -7,13 +7,13 @@ Tests provenance tracking workflows using only available methods.
|
||||
import pytest
|
||||
import networkx as nx
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, List, Any
|
||||
import uuid
|
||||
|
||||
from semantica.kg import (
|
||||
GraphBuilderWithProvenance,
|
||||
AlgorithmTrackerWithProvenance,
|
||||
NodeEmbedder,
|
||||
SimilarityCalculator,
|
||||
LinkPredictor,
|
||||
CentralityCalculator,
|
||||
@@ -21,6 +21,40 @@ from semantica.kg import (
|
||||
)
|
||||
|
||||
|
||||
def _stored(owner, entity_id):
|
||||
"""Read a provenance record back. Fails if tracking only minted an ID."""
|
||||
record = owner._prov_manager.get_provenance(entity_id)
|
||||
assert record is not None, (
|
||||
f"no stored provenance for {entity_id!r} — an ID was generated without a write"
|
||||
)
|
||||
assert record.get("entity_id") == entity_id
|
||||
return record
|
||||
|
||||
|
||||
def _assert_utc_iso(value, field="timestamp"):
|
||||
assert value, f"missing {field}"
|
||||
parsed = datetime.fromisoformat(value)
|
||||
assert parsed.tzinfo is not None, (
|
||||
f"{field}={value!r} is naive (datetime.utcnow leftover)"
|
||||
)
|
||||
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
|
||||
return parsed
|
||||
|
||||
|
||||
def _assert_tracked(owner, entity_id, source, **metadata):
|
||||
record = _stored(owner, entity_id)
|
||||
assert record["source_document"] == source
|
||||
actual = record.get("metadata") or {}
|
||||
for key, expected in metadata.items():
|
||||
assert actual.get(key) == expected, (
|
||||
f"{entity_id} metadata[{key!r}]={actual.get(key)!r}, expected {expected!r}"
|
||||
)
|
||||
_assert_utc_iso(record["timestamp"], "timestamp")
|
||||
if record.get("last_updated"):
|
||||
_assert_utc_iso(record["last_updated"], "last_updated")
|
||||
return record
|
||||
|
||||
|
||||
class TestProvenanceWorkflowsSimple:
|
||||
"""Test provenance tracking workflows with available methods."""
|
||||
|
||||
@@ -93,11 +127,19 @@ class TestProvenanceWorkflowsSimple:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert embed_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
embed_id,
|
||||
source=workflow_id,
|
||||
algorithm='node2vec',
|
||||
node_count=6,
|
||||
embedding_dimension=4,
|
||||
)
|
||||
assert embed_id.startswith('embedding_')
|
||||
|
||||
print(f"Simple embedding workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
for node_id in workflow_embeddings:
|
||||
node_record = _stored(tracker, f"embedding_{node_id}")
|
||||
assert node_record['metadata']['execution_id'] == embed_id
|
||||
assert node_record['metadata']['node_id'] == node_id
|
||||
|
||||
def test_similarity_workflow_simple(self, workflow_embeddings):
|
||||
"""Test simple similarity workflow with provenance."""
|
||||
@@ -124,11 +166,15 @@ class TestProvenanceWorkflowsSimple:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert sim_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
sim_id,
|
||||
source=workflow_id,
|
||||
method='cosine',
|
||||
similarities_count=len(similarities),
|
||||
)
|
||||
assert sim_id.startswith('similarity_')
|
||||
|
||||
print(f"Simple similarity workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
assert len(similarities) == 3
|
||||
|
||||
def test_link_prediction_workflow_simple(self, workflow_graph):
|
||||
"""Test simple link prediction workflow with provenance."""
|
||||
@@ -153,11 +199,15 @@ class TestProvenanceWorkflowsSimple:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert link_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
link_id,
|
||||
source=workflow_id,
|
||||
method='preferential_attachment',
|
||||
predictions_count=len(predictions),
|
||||
)
|
||||
assert link_id.startswith('link_prediction_')
|
||||
|
||||
print(f"Simple link prediction workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
assert len(predictions) >= 1
|
||||
|
||||
def test_centrality_workflow_simple(self, workflow_graph):
|
||||
"""Test simple centrality workflow with provenance."""
|
||||
@@ -184,11 +234,15 @@ class TestProvenanceWorkflowsSimple:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert cent_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
cent_id,
|
||||
source=workflow_id,
|
||||
method='degree',
|
||||
scores_count=len(degree_cent['centrality']),
|
||||
)
|
||||
assert cent_id.startswith('centrality_')
|
||||
|
||||
print(f"Simple centrality workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
assert set(degree_cent['centrality']) == set(workflow_graph.nodes())
|
||||
|
||||
def test_community_detection_workflow_simple(self, workflow_graph):
|
||||
"""Test simple community detection workflow with provenance."""
|
||||
@@ -215,11 +269,15 @@ class TestProvenanceWorkflowsSimple:
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert comm_id is not None
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
comm_id,
|
||||
source=workflow_id,
|
||||
method='label_propagation',
|
||||
communities_count=len(communities['communities']),
|
||||
)
|
||||
assert comm_id.startswith('community_')
|
||||
|
||||
print(f"Simple community detection workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
assert len(communities['communities']) >= 1
|
||||
|
||||
def test_graph_construction_workflow_simple(self, workflow_data):
|
||||
"""Test simple graph construction workflow with provenance."""
|
||||
@@ -237,23 +295,27 @@ class TestProvenanceWorkflowsSimple:
|
||||
assert len(graph_result['entities']) == 4
|
||||
assert len(graph_result['relationships']) == 3
|
||||
|
||||
# Track graph construction using embedding computation method as proxy
|
||||
construction_id = tracker.track_embedding_computation(
|
||||
graph={'nodes': list(graph_result['entities']), 'edges': list(graph_result['relationships'])},
|
||||
algorithm='graph_construction',
|
||||
embeddings={'graph_size': len(graph_result['entities'])},
|
||||
parameters={
|
||||
'entities_count': len(graph_result['entities']),
|
||||
'relationships_count': len(graph_result['relationships'])
|
||||
},
|
||||
construction_id = tracker.track_graph_construction(
|
||||
input_data=workflow_data,
|
||||
output_graph=graph_result,
|
||||
entities_count=len(graph_result['entities']),
|
||||
relationships_count=len(graph_result['relationships']),
|
||||
construction_time=0.0,
|
||||
source=workflow_id
|
||||
)
|
||||
|
||||
assert construction_id is not None
|
||||
assert construction_id.startswith('embedding_') # Using embedding method as proxy
|
||||
|
||||
print(f"Simple graph construction workflow completed: {workflow_id}")
|
||||
return workflow_id
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
construction_id,
|
||||
source=workflow_id,
|
||||
entity_type='graph_construction',
|
||||
entities_count=4,
|
||||
relationships_count=3,
|
||||
)
|
||||
assert construction_id.startswith('graph_construction_')
|
||||
for entity in graph_result['entities']:
|
||||
built = _stored(builder, entity['id'])
|
||||
assert built['metadata']['operation'] == 'build_entity'
|
||||
_assert_utc_iso(built['activity_started_at_time'], 'activity_started_at_time')
|
||||
|
||||
def test_comprehensive_workflow_simple(self, workflow_data, workflow_graph, workflow_embeddings):
|
||||
"""Test comprehensive workflow with all available methods."""
|
||||
@@ -271,11 +333,12 @@ class TestProvenanceWorkflowsSimple:
|
||||
# Phase 1: Graph Construction
|
||||
graph_result = builder.build_single_source(workflow_data)
|
||||
|
||||
construction_id = tracker.track_embedding_computation(
|
||||
graph={'nodes': list(graph_result['entities']), 'edges': list(graph_result['relationships'])},
|
||||
algorithm='graph_construction',
|
||||
embeddings={'graph_size': len(graph_result['entities'])},
|
||||
parameters={'entities_count': len(graph_result['entities'])},
|
||||
construction_id = tracker.track_graph_construction(
|
||||
input_data=workflow_data,
|
||||
output_graph=graph_result,
|
||||
entities_count=len(graph_result['entities']),
|
||||
relationships_count=len(graph_result['relationships']),
|
||||
construction_time=0.0,
|
||||
source=master_workflow_id
|
||||
)
|
||||
execution_ids['construction'] = construction_id
|
||||
@@ -351,27 +414,25 @@ class TestProvenanceWorkflowsSimple:
|
||||
)
|
||||
execution_ids['community_detection'] = comm_id
|
||||
|
||||
# Verify all execution IDs
|
||||
expected = {
|
||||
'construction': ('graph_construction_', 'graph_construction'),
|
||||
'embedding': ('embedding_', 'embedding_computation'),
|
||||
'similarity': ('similarity_', 'similarity_calculation'),
|
||||
'link_prediction': ('link_prediction_', 'link_prediction'),
|
||||
'centrality': ('centrality_', 'centrality_calculation'),
|
||||
'community_detection': ('community_', 'community_detection'),
|
||||
}
|
||||
assert len(execution_ids) == 6
|
||||
for phase, exec_id in execution_ids.items():
|
||||
assert exec_id is not None
|
||||
assert len(exec_id) > 10
|
||||
|
||||
# Verify all IDs are unique
|
||||
all_ids = list(execution_ids.values())
|
||||
assert len(set(all_ids)) == len(all_ids)
|
||||
|
||||
# Verify ID prefixes
|
||||
assert execution_ids['embedding'].startswith('embedding_')
|
||||
assert execution_ids['similarity'].startswith('similarity_')
|
||||
assert execution_ids['link_prediction'].startswith('link_prediction_')
|
||||
assert execution_ids['centrality'].startswith('centrality_')
|
||||
assert execution_ids['community_detection'].startswith('community_')
|
||||
|
||||
print(f"Comprehensive workflow completed: {master_workflow_id}")
|
||||
print(f"Execution IDs: {list(execution_ids.keys())}")
|
||||
|
||||
return master_workflow_id
|
||||
prefix, entity_type = expected[phase]
|
||||
assert exec_id.startswith(prefix)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
exec_id,
|
||||
source=master_workflow_id,
|
||||
entity_type=entity_type,
|
||||
)
|
||||
assert len(set(execution_ids.values())) == len(execution_ids)
|
||||
|
||||
def test_provenance_data_integrity_simple(self, workflow_graph, workflow_embeddings):
|
||||
"""Test provenance data integrity with available methods."""
|
||||
@@ -412,24 +473,19 @@ class TestProvenanceWorkflowsSimple:
|
||||
)
|
||||
operations.append(('link_prediction', link_id))
|
||||
|
||||
# Verify data integrity
|
||||
expected = {
|
||||
'embedding': 'embedding_computation',
|
||||
'similarity': 'similarity_calculation',
|
||||
'link_prediction': 'link_prediction',
|
||||
}
|
||||
for op_type, op_id in operations:
|
||||
assert op_id is not None
|
||||
assert len(op_id) > 10
|
||||
|
||||
# Verify ID format consistency
|
||||
if op_type == 'embedding':
|
||||
assert op_id.startswith('embedding_')
|
||||
elif op_type == 'similarity':
|
||||
assert op_id.startswith('similarity_')
|
||||
elif op_type == 'link_prediction':
|
||||
assert op_id.startswith('link_prediction_')
|
||||
|
||||
# Verify workflow consistency
|
||||
workflow_ids = [op_id for _, op_id in operations]
|
||||
assert len(set(workflow_ids)) == len(workflow_ids) # All unique
|
||||
|
||||
print(f"Provenance data integrity test completed: {workflow_id}")
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
op_id,
|
||||
source=workflow_id,
|
||||
entity_type=expected[op_type],
|
||||
)
|
||||
assert len(set(op_id for _, op_id in operations)) == 3
|
||||
|
||||
def test_provenance_error_recovery_simple(self):
|
||||
"""Test provenance system error recovery."""
|
||||
@@ -443,25 +499,25 @@ class TestProvenanceWorkflowsSimple:
|
||||
parameters={}
|
||||
)
|
||||
|
||||
assert result is None # Should return None when provenance is disabled
|
||||
|
||||
# Test error handling during tracking
|
||||
assert result is None
|
||||
assert tracker_no_prov._prov_manager is None
|
||||
|
||||
tracker = AlgorithmTrackerWithProvenance(provenance=True)
|
||||
|
||||
# This should not raise exceptions even with invalid data
|
||||
try:
|
||||
result = tracker.track_embedding_computation(
|
||||
graph=None, # Invalid graph
|
||||
algorithm='test',
|
||||
embeddings={},
|
||||
parameters={}
|
||||
)
|
||||
# Should either return None or handle gracefully
|
||||
except Exception as e:
|
||||
# If it raises, it should be a controlled exception
|
||||
assert isinstance(e, (ValueError, TypeError))
|
||||
|
||||
print("Provenance error recovery test completed")
|
||||
result = tracker.track_embedding_computation(
|
||||
graph=None,
|
||||
algorithm='test',
|
||||
embeddings={},
|
||||
parameters={},
|
||||
source='error_recovery'
|
||||
)
|
||||
_assert_tracked(
|
||||
tracker,
|
||||
result,
|
||||
source='error_recovery',
|
||||
algorithm='test',
|
||||
input_data_type='NoneType',
|
||||
node_count=0,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
from semantica.ontology.class_inferrer import ClassInferrer
|
||||
from semantica.ontology.ontology_generator import OntologyGenerator
|
||||
from semantica.ontology.property_generator import PropertyGenerator
|
||||
|
||||
|
||||
def _entities():
|
||||
return [
|
||||
{
|
||||
"id": "e1",
|
||||
"type": "software engineer",
|
||||
"name": "Alice",
|
||||
"email": "alice@example.org",
|
||||
},
|
||||
{
|
||||
"id": "e2",
|
||||
"type": "software engineer",
|
||||
"name": "Bob",
|
||||
"email": "bob@example.org",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def test_property_generator_matches_normalized_class_names():
|
||||
entities = _entities()
|
||||
classes = ClassInferrer().infer_classes(entities)
|
||||
|
||||
properties = PropertyGenerator().infer_properties(entities, [], classes)
|
||||
|
||||
email = next(prop for prop in properties if prop["name"] == "email")
|
||||
assert email["domain"] == ["SoftwareEngineer"]
|
||||
assert email["range"] == "xsd:string"
|
||||
|
||||
|
||||
def test_ontology_pipeline_emits_data_properties_for_normalized_types():
|
||||
ontology = OntologyGenerator().generate_ontology(
|
||||
{"entities": _entities(), "relationships": []}
|
||||
)
|
||||
|
||||
email = next(prop for prop in ontology["properties"] if prop["name"] == "email")
|
||||
assert email["domain"] == ["SoftwareEngineer"]
|
||||
assert email["range"] == "xsd:string"
|
||||
@@ -4,6 +4,7 @@ import tempfile
|
||||
import os
|
||||
import json
|
||||
import csv
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -137,29 +138,32 @@ class TestParseComprehensive(unittest.TestCase):
|
||||
|
||||
# --- Document Parser Tests ---
|
||||
|
||||
@patch('semantica.parse.pdf_parser.pdfplumber')
|
||||
def test_pdf_parser(self, mock_pdfplumber):
|
||||
def test_pdf_parser(self):
|
||||
parser = PDFParser()
|
||||
# pdfplumber is imported inside PDFParser.parse, so inject a fake
|
||||
# module into sys.modules instead of patching a module attribute
|
||||
mock_pdfplumber = MagicMock()
|
||||
mock_pdf = MagicMock()
|
||||
mock_page = MagicMock()
|
||||
mock_page.extract_text.return_value = "Page text"
|
||||
mock_pdf.pages = [mock_page]
|
||||
# Ensure metadata is a dict, not a property object if that's an issue
|
||||
mock_pdf.metadata = {"Title": "Test PDF"}
|
||||
|
||||
|
||||
# Setup the context manager
|
||||
mock_context_manager = MagicMock()
|
||||
mock_context_manager.__enter__.return_value = mock_pdf
|
||||
mock_context_manager.__exit__.return_value = None
|
||||
mock_pdfplumber.open.return_value = mock_context_manager
|
||||
|
||||
|
||||
# We don't need a real file if we mock open, but the parser likely checks file existence
|
||||
with tempfile.NamedTemporaryFile(mode='wb', delete=False, suffix='.pdf') as tmp:
|
||||
tmp.write(b"dummy pdf content")
|
||||
tmp_path = tmp.name
|
||||
|
||||
|
||||
try:
|
||||
result = parser.parse(tmp_path)
|
||||
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
|
||||
result = parser.parse(tmp_path)
|
||||
# Returns dict with full_text
|
||||
self.assertIn("Page text", result["full_text"])
|
||||
self.assertEqual(result["metadata"].get("title"), "Test PDF")
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
"""
|
||||
Tests for scanned-PDF (empty text layer) detection in PDFParser.
|
||||
|
||||
Covers:
|
||||
- Warning when all parsed pages lack a text layer (scanned PDFs)
|
||||
- No warning when text is present or text extraction is disabled
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from semantica.parse.pdf_parser import PDFParser
|
||||
|
||||
|
||||
class TestScannedPdfWarning(unittest.TestCase):
|
||||
"""PDFParser warns when a PDF has pages but no extractable text."""
|
||||
|
||||
def setUp(self):
|
||||
self.mock_logger = MagicMock()
|
||||
logger_patcher = patch(
|
||||
"semantica.parse.pdf_parser.get_logger", return_value=self.mock_logger
|
||||
)
|
||||
tracker_patcher = patch("semantica.parse.pdf_parser.get_progress_tracker")
|
||||
logger_patcher.start()
|
||||
tracker_patcher.start()
|
||||
self.addCleanup(logger_patcher.stop)
|
||||
self.addCleanup(tracker_patcher.stop)
|
||||
|
||||
def _make_pdf_path(self):
|
||||
with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=".pdf") as tmp:
|
||||
tmp.write(b"dummy pdf content")
|
||||
return tmp.name
|
||||
|
||||
def _mock_pdfplumber(self, page_text):
|
||||
"""Build a fake pdfplumber module whose single page yields page_text."""
|
||||
mock_page = MagicMock()
|
||||
mock_page.extract_text.return_value = page_text
|
||||
mock_page.extract_tables.return_value = []
|
||||
mock_page.width = 612
|
||||
mock_page.height = 792
|
||||
|
||||
mock_pdf = MagicMock()
|
||||
mock_pdf.pages = [mock_page]
|
||||
mock_pdf.metadata = {}
|
||||
|
||||
mock_pdfplumber = MagicMock()
|
||||
context_manager = MagicMock()
|
||||
context_manager.__enter__.return_value = mock_pdf
|
||||
mock_pdfplumber.open.return_value = context_manager
|
||||
return mock_pdfplumber
|
||||
|
||||
def test_warns_when_no_text_layer(self):
|
||||
mock_pdfplumber = self._mock_pdfplumber(page_text=None) # scanned page
|
||||
parser = PDFParser()
|
||||
path = self._make_pdf_path()
|
||||
try:
|
||||
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
|
||||
result = parser.parse(path)
|
||||
self.assertEqual(result["full_text"], "")
|
||||
self.mock_logger.warning.assert_called_once()
|
||||
message = self.mock_logger.warning.call_args[0][0]
|
||||
self.assertIn("scanned", message)
|
||||
self.assertIn("enable_ocr", message)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_no_warning_when_text_present(self):
|
||||
mock_pdfplumber = self._mock_pdfplumber(page_text="Real digital text")
|
||||
parser = PDFParser()
|
||||
path = self._make_pdf_path()
|
||||
try:
|
||||
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
|
||||
result = parser.parse(path)
|
||||
self.assertIn("Real digital text", result["full_text"])
|
||||
self.mock_logger.warning.assert_not_called()
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_no_warning_when_extract_text_disabled(self):
|
||||
mock_pdfplumber = self._mock_pdfplumber(page_text=None)
|
||||
parser = PDFParser()
|
||||
path = self._make_pdf_path()
|
||||
try:
|
||||
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
|
||||
result = parser.parse(path, extract_text=False, extract_tables=False)
|
||||
self.assertEqual(result["full_text"], "")
|
||||
self.mock_logger.warning.assert_not_called()
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_warns_when_pages_have_only_whitespace(self):
|
||||
"""Pages that return only whitespace count as no extractable text."""
|
||||
mock_pdfplumber = self._mock_pdfplumber(page_text=" \n\t ")
|
||||
parser = PDFParser()
|
||||
path = self._make_pdf_path()
|
||||
try:
|
||||
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
|
||||
parser.parse(path)
|
||||
self.mock_logger.warning.assert_called_once()
|
||||
message = self.mock_logger.warning.call_args[0][0]
|
||||
self.assertIn("scanned", message)
|
||||
self.assertIn("enable_ocr", message)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_no_warning_for_mixed_pdf(self):
|
||||
"""A PDF where at least one page has real text must not warn."""
|
||||
# Build a two-page PDF: page 1 has text, page 2 is image-only
|
||||
mock_page_text = MagicMock()
|
||||
mock_page_text.extract_text.return_value = "Actual content"
|
||||
mock_page_text.extract_tables.return_value = []
|
||||
mock_page_text.width = 612
|
||||
mock_page_text.height = 792
|
||||
|
||||
mock_page_image = MagicMock()
|
||||
mock_page_image.extract_text.return_value = None
|
||||
mock_page_image.extract_tables.return_value = []
|
||||
mock_page_image.width = 612
|
||||
mock_page_image.height = 792
|
||||
|
||||
mock_pdf = MagicMock()
|
||||
mock_pdf.pages = [mock_page_text, mock_page_image]
|
||||
mock_pdf.metadata = {}
|
||||
|
||||
mock_pdfplumber = MagicMock()
|
||||
context_manager = MagicMock()
|
||||
context_manager.__enter__.return_value = mock_pdf
|
||||
mock_pdfplumber.open.return_value = context_manager
|
||||
|
||||
parser = PDFParser()
|
||||
path = self._make_pdf_path()
|
||||
try:
|
||||
with patch.dict(sys.modules, {"pdfplumber": mock_pdfplumber}):
|
||||
result = parser.parse(path)
|
||||
self.assertIn("Actual content", result["full_text"])
|
||||
self.mock_logger.warning.assert_not_called()
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -66,6 +66,93 @@ class TestPipelineModule(unittest.TestCase):
|
||||
self.assertEqual(result.output, 12) # (5 + 1) * 2 = 12
|
||||
self.assertEqual(pipeline.steps[0].status, StepStatus.COMPLETED)
|
||||
|
||||
def test_registered_step_handler_executes(self):
|
||||
"""A handler registered by step type should execute."""
|
||||
def increment(data):
|
||||
return data + 1
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.register_step_handler("math", increment)
|
||||
builder.add_step("increment", "math")
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
builder.build("registered"), data=1
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(result.output, 2)
|
||||
|
||||
def test_explicit_handler_does_not_receive_control_fields(self):
|
||||
"""Builder-only fields should not be passed to strict handlers."""
|
||||
def source(data):
|
||||
return data
|
||||
|
||||
def increment(data, amount):
|
||||
return data + amount
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("source", "source", handler=source)
|
||||
step = builder.add_step(
|
||||
"increment",
|
||||
"math",
|
||||
handler=increment,
|
||||
dependencies=["source"],
|
||||
amount=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
builder.build("strict"), data=1
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(result.output, 3)
|
||||
self.assertEqual(step.config, {"amount": 2})
|
||||
|
||||
def test_explicit_handler_overrides_registered_handler(self):
|
||||
"""An explicit step handler should take precedence over the registry."""
|
||||
builder = PipelineBuilder()
|
||||
builder.register_step_handler("math", lambda data: data + 100)
|
||||
builder.add_step("increment", "math", handler=lambda data: data + 1)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
builder.build("override"), data=1
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(result.output, 2)
|
||||
|
||||
def test_step_without_handler_passes_input_through(self):
|
||||
"""A step with no explicit or registered handler should be a no-op."""
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("passthrough", "unregistered")
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
builder.build("handlerless"), data={"value": 1}
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(result.output, {"value": 1})
|
||||
|
||||
def test_falsy_explicit_handler_is_invoked(self):
|
||||
"""A handler whose __bool__ is False must still be dispatched."""
|
||||
|
||||
class FalseyHandler:
|
||||
def __bool__(self):
|
||||
return False
|
||||
|
||||
def __call__(self, data):
|
||||
return {"explicit": data}
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("step1", "mytype", handler=FalseyHandler())
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
builder.build("falsy"), data=1
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(result.output, {"explicit": 1})
|
||||
|
||||
def test_execution_engine_failure(self):
|
||||
"""Test pipeline failure handling."""
|
||||
def failing_handler(data, **kwargs):
|
||||
@@ -112,8 +199,8 @@ class TestPipelineModule(unittest.TestCase):
|
||||
|
||||
_ = semantica.pipeline
|
||||
|
||||
from semantica.pipeline import PipelineBuilder, PipelineValidator
|
||||
from semantica.deduplication import DuplicateDetector
|
||||
from semantica.pipeline import PipelineBuilder, PipelineValidator
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("step1", "dummy")
|
||||
|
||||
@@ -6,8 +6,9 @@ chunk tracking, source tracking, and lineage tracing.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import warnings
|
||||
from unittest.mock import patch
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from semantica.provenance import ProvenanceManager, SourceReference, ProvenanceEntry
|
||||
from semantica.provenance.storage import InMemoryStorage, SQLiteStorage
|
||||
|
||||
@@ -733,8 +734,8 @@ class TestProvenanceManager:
|
||||
entity_type="entity",
|
||||
activity_id="test",
|
||||
source_document="doc_1",
|
||||
first_seen=datetime.utcnow().isoformat(),
|
||||
last_updated=datetime.utcnow().isoformat(),
|
||||
first_seen=datetime.now(timezone.utc).isoformat(),
|
||||
last_updated=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
with patch.object(prov_mgr.storage, "store", side_effect=RuntimeError("storage error")), \
|
||||
patch.object(prov_mgr.logger, "error") as mock_log_error:
|
||||
@@ -1672,3 +1673,195 @@ class TestActivityTimingAcrossWrappers:
|
||||
assert entry["activity_ended_at_time"] is not None
|
||||
|
||||
|
||||
class TestTimezoneAwareUtcTimestamps:
|
||||
"""Issue #946 — timezone-aware UTC stamps without datetime.utcnow()."""
|
||||
|
||||
def test_track_entity_stamps_timezone_aware_utc(self):
|
||||
before = datetime.now(timezone.utc) - timedelta(seconds=1)
|
||||
with warnings.catch_warnings(record=True) as caught:
|
||||
warnings.simplefilter("always", DeprecationWarning)
|
||||
prov_mgr = ProvenanceManager()
|
||||
entry = prov_mgr.track_entity("utc_entity", source="doc_1")
|
||||
after = datetime.now(timezone.utc) + timedelta(seconds=1)
|
||||
|
||||
utcnow_warnings = [
|
||||
w
|
||||
for w in caught
|
||||
if issubclass(w.category, DeprecationWarning)
|
||||
and "utcnow" in str(w.message).lower()
|
||||
]
|
||||
assert utcnow_warnings == []
|
||||
|
||||
for field in ("timestamp", "first_seen", "last_updated"):
|
||||
value = getattr(entry, field)
|
||||
parsed = datetime.fromisoformat(value)
|
||||
assert parsed.tzinfo is not None, f"{field}={value!r} is naive"
|
||||
assert parsed.utcoffset() == timedelta(0), f"{field}={value!r} is not UTC"
|
||||
assert before <= parsed <= after
|
||||
|
||||
stored = prov_mgr.get_provenance("utc_entity")
|
||||
assert stored is not None
|
||||
assert stored["source_document"] == "doc_1"
|
||||
assert stored["last_updated"] == entry.last_updated
|
||||
|
||||
def test_invalidate_stamps_timezone_aware_utc(self):
|
||||
prov_mgr = ProvenanceManager()
|
||||
prov_mgr.track_entity("utc_invalidate", source="doc_1")
|
||||
result = prov_mgr.invalidate(
|
||||
"utc_invalidate", agent_id="reviewer", reason="test"
|
||||
)
|
||||
parsed = datetime.fromisoformat(result.invalidated_at_time)
|
||||
assert parsed.tzinfo is not None
|
||||
assert parsed.utcoffset() == timedelta(0)
|
||||
|
||||
def test_graph_builder_activity_times_are_timezone_aware_utc(self):
|
||||
from semantica.kg.kg_provenance import GraphBuilderWithProvenance
|
||||
|
||||
builder = GraphBuilderWithProvenance(provenance=True, agent_id="builder_svc")
|
||||
result = builder.build_single_source({
|
||||
"entities": [{"id": "person1", "type": "Person", "name": "Ada"}],
|
||||
"relationships": [],
|
||||
})
|
||||
assert result["entities"][0]["id"] == "person1"
|
||||
|
||||
person = builder._prov_manager.get_provenance("person1")
|
||||
assert person is not None
|
||||
assert person["metadata"]["operation"] == "build_entity"
|
||||
for field in ("activity_started_at_time", "activity_ended_at_time"):
|
||||
parsed = datetime.fromisoformat(person[field])
|
||||
assert parsed.tzinfo is not None, (
|
||||
f"{field}={person[field]!r} is naive"
|
||||
)
|
||||
assert parsed.utcoffset() == timedelta(0)
|
||||
started = datetime.fromisoformat(person["activity_started_at_time"])
|
||||
ended = datetime.fromisoformat(person["activity_ended_at_time"])
|
||||
assert started <= ended
|
||||
|
||||
class TestMixedFormatTimestampComparisons:
|
||||
"""Issue #946 review — naive and offset-bearing stamps must compare as instants.
|
||||
|
||||
Records written before the timezone-aware change carry a naive UTC stamp
|
||||
(``2026-08-19T22:35:38.501697``); records written after carry ``+00:00``.
|
||||
Comparing the two as raw strings puts the offset-bearing form above a naive
|
||||
bound at the identical instant, so the record falls outside a range that
|
||||
should contain it. These tests pin the boundary in both directions.
|
||||
"""
|
||||
|
||||
NAIVE = "2026-08-19T12:00:00.500000"
|
||||
AWARE = "2026-08-19T12:00:00.500000+00:00"
|
||||
|
||||
def _manager_with(self, *timestamps):
|
||||
prov_mgr = ProvenanceManager()
|
||||
for index, stamp in enumerate(timestamps):
|
||||
prov_mgr.storage.store(
|
||||
ProvenanceEntry(
|
||||
entity_id=f"entity_{index}",
|
||||
entity_type="entity",
|
||||
activity_id="test",
|
||||
source_document="doc_1",
|
||||
timestamp=stamp,
|
||||
)
|
||||
)
|
||||
return prov_mgr
|
||||
|
||||
def test_raw_string_compare_would_exclude_the_boundary_record(self):
|
||||
"""Guards the premise: the two forms are not string-comparable."""
|
||||
assert not (self.AWARE <= self.NAIVE)
|
||||
assert datetime.fromisoformat(self.AWARE) == datetime.fromisoformat(
|
||||
self.NAIVE
|
||||
).replace(tzinfo=timezone.utc)
|
||||
|
||||
def test_query_range_with_naive_bounds_includes_aware_record(self):
|
||||
prov_mgr = self._manager_with(self.AWARE)
|
||||
results = prov_mgr.query_recorded_between(
|
||||
"2026-08-19T12:00:00.500000", "2026-08-19T12:00:00.500000"
|
||||
)
|
||||
assert [r["entity_id"] for r in results] == ["entity_0"]
|
||||
|
||||
def test_query_range_with_aware_bounds_includes_naive_record(self):
|
||||
prov_mgr = self._manager_with(self.NAIVE)
|
||||
results = prov_mgr.query_recorded_between(
|
||||
"2026-08-19T12:00:00.500000+00:00", "2026-08-19T12:00:00.500000+00:00"
|
||||
)
|
||||
assert [r["entity_id"] for r in results] == ["entity_0"]
|
||||
|
||||
def test_query_range_returns_both_formats_from_a_mixed_store(self):
|
||||
prov_mgr = self._manager_with(self.NAIVE, self.AWARE)
|
||||
results = prov_mgr.query_recorded_between(
|
||||
"2026-08-19T11:00:00", "2026-08-19T13:00:00+00:00"
|
||||
)
|
||||
assert {r["entity_id"] for r in results} == {"entity_0", "entity_1"}
|
||||
|
||||
def test_query_range_sorts_mixed_formats_chronologically(self):
|
||||
prov_mgr = self._manager_with(
|
||||
"2026-08-19T12:00:02+00:00", # entity_0, latest
|
||||
"2026-08-19T12:00:00", # entity_1, earliest
|
||||
"2026-08-19T12:00:01+00:00", # entity_2, middle
|
||||
)
|
||||
results = prov_mgr.query_recorded_between(
|
||||
"2026-08-19T11:00:00", "2026-08-19T13:00:00"
|
||||
)
|
||||
assert [r["entity_id"] for r in results] == [
|
||||
"entity_1",
|
||||
"entity_2",
|
||||
"entity_0",
|
||||
]
|
||||
|
||||
def test_query_range_accepts_trailing_z(self):
|
||||
prov_mgr = self._manager_with(self.NAIVE)
|
||||
results = prov_mgr.query_recorded_between(
|
||||
"2026-08-19T11:00:00Z", "2026-08-19T13:00:00Z"
|
||||
)
|
||||
assert len(results) == 1
|
||||
|
||||
def test_query_range_skips_unparseable_timestamp(self):
|
||||
prov_mgr = self._manager_with("not-a-timestamp", self.AWARE)
|
||||
results = prov_mgr.query_recorded_between(
|
||||
"2026-08-19T11:00:00", "2026-08-19T13:00:00"
|
||||
)
|
||||
assert [r["entity_id"] for r in results] == ["entity_1"]
|
||||
|
||||
def test_audit_log_since_naive_bound_includes_aware_record(self):
|
||||
prov_mgr = self._manager_with(self.AWARE)
|
||||
entries = prov_mgr.audit_log(
|
||||
since="2026-08-19T12:00:00.500000", format="json"
|
||||
)
|
||||
assert [e["entity_id"] for e in entries] == ["entity_0"]
|
||||
|
||||
def test_audit_log_since_aware_bound_includes_naive_record(self):
|
||||
prov_mgr = self._manager_with(self.NAIVE)
|
||||
entries = prov_mgr.audit_log(
|
||||
since="2026-08-19T12:00:00.500000+00:00", format="json"
|
||||
)
|
||||
assert [e["entity_id"] for e in entries] == ["entity_0"]
|
||||
|
||||
def test_audit_log_excludes_records_before_since_across_formats(self):
|
||||
prov_mgr = self._manager_with(
|
||||
"2026-08-19T11:59:59", # entity_0, before
|
||||
"2026-08-19T12:00:01+00:00", # entity_1, after
|
||||
)
|
||||
entries = prov_mgr.audit_log(since="2026-08-19T12:00:00+00:00", format="json")
|
||||
assert [e["entity_id"] for e in entries] == ["entity_1"]
|
||||
|
||||
def test_audit_log_sorts_mixed_formats_chronologically(self):
|
||||
prov_mgr = self._manager_with(
|
||||
"2026-08-19T12:00:02+00:00",
|
||||
"2026-08-19T12:00:00",
|
||||
"2026-08-19T12:00:01+00:00",
|
||||
)
|
||||
entries = prov_mgr.audit_log(format="json")
|
||||
assert [e["entity_id"] for e in entries] == [
|
||||
"entity_1",
|
||||
"entity_2",
|
||||
"entity_0",
|
||||
]
|
||||
|
||||
def test_audit_log_unparseable_timestamp_sorts_first(self):
|
||||
prov_mgr = self._manager_with("not-a-timestamp", "2026-08-19T12:00:00+00:00")
|
||||
entries = prov_mgr.audit_log(format="json")
|
||||
assert [e["entity_id"] for e in entries] == ["entity_0", "entity_1"]
|
||||
|
||||
def test_audit_log_since_excludes_unparseable_timestamp(self):
|
||||
prov_mgr = self._manager_with("not-a-timestamp", "2026-08-19T12:00:00+00:00")
|
||||
entries = prov_mgr.audit_log(since="2026-08-19T11:00:00", format="json")
|
||||
assert [e["entity_id"] for e in entries] == ["entity_1"]
|
||||
|
||||
@@ -0,0 +1,531 @@
|
||||
"""Tests for rule-driven actions (production-rule behaviour) on the Reasoner.
|
||||
|
||||
Covers the L1 Action layer (Assert/Retract/Call/Emit), provenance logging of
|
||||
fired actions (L2), and backward compatibility with the legacy Rule.handler
|
||||
callback.
|
||||
"""
|
||||
|
||||
import unittest
|
||||
from collections import UserDict
|
||||
|
||||
from semantica.reasoning import (
|
||||
AssertAction,
|
||||
CallAction,
|
||||
EmitEventAction,
|
||||
Fact,
|
||||
Match,
|
||||
Reasoner,
|
||||
ReteEngine,
|
||||
RetractAction,
|
||||
)
|
||||
|
||||
|
||||
class TestRuleActions(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.reasoner = Reasoner()
|
||||
|
||||
def _add_person_parent_facts(self):
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
self.reasoner.add_fact("Parent(John, Jane)")
|
||||
|
||||
def test_assert_action_fires_and_substitutes_bindings(self):
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [AssertAction("Adult(?x)")]
|
||||
self._add_person_parent_facts()
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
# Action-asserted fact uses the match bindings (?x -> John).
|
||||
self.assertIn("Adult(John)", self.reasoner.facts)
|
||||
|
||||
def test_retract_action_removes_fact(self):
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [RetractAction("Person(?x)")]
|
||||
self._add_person_parent_facts()
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertNotIn("Person(John)", self.reasoner.facts)
|
||||
|
||||
def test_call_action_invoked_with_bindings(self):
|
||||
seen = {}
|
||||
|
||||
def record(bindings, reasoner):
|
||||
seen.update(bindings)
|
||||
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [CallAction(record, name="record")]
|
||||
self._add_person_parent_facts()
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(seen.get("x"), "John")
|
||||
self.assertEqual(seen.get("y"), "Jane")
|
||||
|
||||
def test_emit_event_action_delivers_to_sink(self):
|
||||
events = []
|
||||
self.reasoner.on_event(lambda name, payload: events.append((name, payload)))
|
||||
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [EmitEventAction("child_derived:?y")]
|
||||
self._add_person_parent_facts()
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(len(events), 1)
|
||||
name, payload = events[0]
|
||||
self.assertEqual(name, "child_derived:Jane")
|
||||
self.assertEqual(payload["bindings"]["x"], "John")
|
||||
|
||||
def test_assert_action_write_back_to_knowledge_graph(self):
|
||||
class FakeKG:
|
||||
def __init__(self):
|
||||
self.added = []
|
||||
|
||||
def add_fact(self, fact):
|
||||
self.added.append(fact)
|
||||
|
||||
kg = FakeKG()
|
||||
reasoner = Reasoner(knowledge_graph=kg)
|
||||
rule = reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [AssertAction("Adult(?x)", write_back=True)]
|
||||
reasoner.add_fact("Person(John)")
|
||||
reasoner.add_fact("Parent(John, Jane)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
|
||||
self.assertIn("Adult(John)", kg.added)
|
||||
|
||||
def test_provenance_logs_fired_actions(self):
|
||||
reasoner = Reasoner(provenance=True)
|
||||
rule = reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [AssertAction("Adult(?x)")]
|
||||
reasoner.add_fact("Person(John)")
|
||||
reasoner.add_fact("Parent(John, Jane)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(len(reasoner.action_log), 1)
|
||||
entry = reasoner.action_log[0]
|
||||
self.assertEqual(entry["action"], "AssertAction")
|
||||
self.assertEqual(entry["rule_id"], rule.rule_id)
|
||||
self.assertEqual(entry["bindings"]["x"], "John")
|
||||
self.assertIn("Adult(John)", entry["description"])
|
||||
|
||||
def test_repeated_forward_chain_fires_same_activation_once(self):
|
||||
calls = []
|
||||
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [
|
||||
CallAction(lambda bindings, reasoner: calls.append(dict(bindings)))
|
||||
]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(calls, [{"x": "John"}])
|
||||
|
||||
def test_repeated_forward_chain_records_provenance_once(self):
|
||||
reasoner = Reasoner(provenance=True)
|
||||
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [AssertAction("Verified(?x)")]
|
||||
reasoner.add_fact("Person(John)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(len(reasoner.action_log), 1)
|
||||
|
||||
def test_new_binding_creates_a_new_activation(self):
|
||||
calls = []
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [
|
||||
CallAction(lambda bindings, reasoner: calls.append(bindings["x"]))
|
||||
]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
self.reasoner.add_fact("Person(Jane)")
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertCountEqual(calls, ["John", "Jane"])
|
||||
|
||||
def test_reset_action_history_allows_deliberate_replay(self):
|
||||
calls = []
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("called"))]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
self.reasoner.reset_action_history()
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(calls, ["called", "called"])
|
||||
|
||||
def test_clear_resets_action_history(self):
|
||||
calls = []
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("called"))]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.reasoner.clear()
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("called"))]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(calls, ["called", "called"])
|
||||
|
||||
def test_failed_action_is_not_retried_without_explicit_reset(self):
|
||||
attempts = []
|
||||
|
||||
def fail(bindings, reasoner):
|
||||
attempts.append(bindings["x"])
|
||||
raise RuntimeError("boom")
|
||||
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [CallAction(fail)]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
self.reasoner.forward_chain()
|
||||
self.assertEqual(attempts, ["John"])
|
||||
|
||||
self.reasoner.reset_action_history()
|
||||
self.reasoner.forward_chain()
|
||||
self.assertEqual(attempts, ["John", "John"])
|
||||
|
||||
def test_replacing_actions_in_place_requires_explicit_history_reset(self):
|
||||
calls = []
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("first"))]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
rule.actions = [CallAction(lambda bindings, reasoner: calls.append("second"))]
|
||||
self.reasoner.forward_chain()
|
||||
self.assertEqual(calls, ["first"])
|
||||
|
||||
self.reasoner.reset_action_history()
|
||||
self.reasoner.forward_chain()
|
||||
self.assertEqual(calls, ["first", "second"])
|
||||
|
||||
def test_activation_is_recorded_before_reentrant_action_execution(self):
|
||||
calls = []
|
||||
|
||||
def reenter(bindings, reasoner):
|
||||
calls.append(bindings["x"])
|
||||
if len(calls) == 1:
|
||||
reasoner.forward_chain()
|
||||
|
||||
rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [CallAction(reenter)]
|
||||
self.reasoner.add_fact("Person(John)")
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(calls, ["John"])
|
||||
|
||||
def test_no_provenance_log_when_disabled(self):
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [AssertAction("Adult(?x)")]
|
||||
self._add_person_parent_facts()
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(self.reasoner.action_log, [])
|
||||
|
||||
def test_legacy_handler_still_invoked(self):
|
||||
calls = []
|
||||
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.handler = lambda bindings, reasoner: calls.append(bindings)
|
||||
self._add_person_parent_facts()
|
||||
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertEqual(len(calls), 1)
|
||||
self.assertEqual(calls[0]["x"], "John")
|
||||
|
||||
def test_action_error_does_not_break_chain(self):
|
||||
def boom(bindings, reasoner):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
rule = self.reasoner.add_rule(
|
||||
"IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
|
||||
)
|
||||
rule.actions = [CallAction(boom, name="boom"), AssertAction("Adult(?x)")]
|
||||
self._add_person_parent_facts()
|
||||
|
||||
# A failing action is logged but must not abort the pass; the later
|
||||
# action still runs and the conclusion is still derived.
|
||||
self.reasoner.forward_chain()
|
||||
|
||||
self.assertIn("Adult(John)", self.reasoner.facts)
|
||||
self.assertIn("Child(Jane, John)", self.reasoner.facts)
|
||||
|
||||
|
||||
class TestRuleActionRegressions(unittest.TestCase):
|
||||
"""Regression coverage for the qodo-flagged bugs on PR #1096."""
|
||||
|
||||
def test_variable_substitution_no_prefix_collision(self):
|
||||
# bug7: naive str.replace of "?x" would also corrupt "?xy". A
|
||||
# token-aware substitution must bind ?x and ?xy independently.
|
||||
reasoner = Reasoner()
|
||||
rule = reasoner.add_rule("IF Pair(?x, ?xy) THEN Linked(?x, ?xy)")
|
||||
rule.actions = [AssertAction("Tag(?x, ?xy)")]
|
||||
reasoner.add_fact("Pair(John, Johny)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
|
||||
self.assertIn("Tag(John, Johny)", reasoner.facts)
|
||||
|
||||
def test_assert_write_back_to_canonical_knowledge_graph(self):
|
||||
# bug1: a KG exposing only entities/relationships (no add_fact) must
|
||||
# still receive the asserted fact via canonical translation.
|
||||
from semantica.kg.knowledge_graph import KnowledgeGraph
|
||||
|
||||
kg = KnowledgeGraph()
|
||||
reasoner = Reasoner(knowledge_graph=kg)
|
||||
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [AssertAction("Adult(?x)", write_back=True)]
|
||||
reasoner.add_fact("Person(John)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
|
||||
# A single-argument fact lands as an entity node.
|
||||
self.assertTrue(any("John" in str(e) for e in kg.entities))
|
||||
|
||||
def test_write_back_unsupported_target_raises(self):
|
||||
# bug1: an unsupported write-back target must fail loudly, not silently.
|
||||
from semantica.reasoning.reasoner import _write_fact_to_graph
|
||||
|
||||
with self.assertRaises(ValueError):
|
||||
_write_fact_to_graph(object(), "Adult(John)")
|
||||
|
||||
def test_provenance_entry_has_timestamp(self):
|
||||
# bug3: action_log entries must be structured with a timestamp.
|
||||
reasoner = Reasoner(provenance=True)
|
||||
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [AssertAction("Adult(?x)")]
|
||||
reasoner.add_fact("Person(John)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
|
||||
entry = reasoner.action_log[0]
|
||||
self.assertIn("timestamp", entry)
|
||||
self.assertTrue(entry["timestamp"])
|
||||
|
||||
def test_action_fires_even_when_conclusion_already_known(self):
|
||||
# bug4: previously an activation whose conclusion was already known
|
||||
# skipped firing its actions. Now it must still fire exactly once.
|
||||
reasoner = Reasoner()
|
||||
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [AssertAction("Verified(?x)")]
|
||||
reasoner.add_fact("Person(John)")
|
||||
# Conclusion already present before the pass runs.
|
||||
reasoner.add_fact("Adult(John)")
|
||||
|
||||
reasoner.forward_chain()
|
||||
|
||||
self.assertIn("Verified(John)", reasoner.facts)
|
||||
|
||||
def test_retract_self_conclusion_terminates(self):
|
||||
# bug5: a RetractAction removing its own premise previously re-fired
|
||||
# every pass up to max_iterations. It must fire once and terminate.
|
||||
reasoner = Reasoner()
|
||||
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.actions = [RetractAction("Person(?x)")]
|
||||
reasoner.add_fact("Person(John)")
|
||||
|
||||
# Should return promptly without exhausting iterations.
|
||||
reasoner.forward_chain()
|
||||
|
||||
self.assertNotIn("Person(John)", reasoner.facts)
|
||||
|
||||
def test_infer_with_results_preserves_confidence(self):
|
||||
# bug9: confidence must survive to the InferenceResult objects.
|
||||
reasoner = Reasoner()
|
||||
rule = reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
rule.confidence = 0.8
|
||||
|
||||
results = reasoner.infer_with_results(["Person(John)"])
|
||||
|
||||
self.assertTrue(results)
|
||||
self.assertTrue(all(0.0 <= r.confidence <= 1.0 for r in results))
|
||||
self.assertAlmostEqual(
|
||||
min(r.confidence for r in results), 0.8, places=6
|
||||
)
|
||||
|
||||
|
||||
class TestReteActionExecution(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.calls = []
|
||||
self.reasoner = Reasoner()
|
||||
self.rule = self.reasoner.add_rule("IF Person(?x) THEN Adult(?x)")
|
||||
self.rule.actions = [
|
||||
CallAction(
|
||||
lambda bindings, reasoner: self.calls.append(dict(bindings))
|
||||
)
|
||||
]
|
||||
self.match = Match(
|
||||
rule=self.rule,
|
||||
facts=[Fact("person-1", "Person", ["John"])],
|
||||
bindings={"x": "John"},
|
||||
)
|
||||
self.engine = ReteEngine(reasoner=self.reasoner)
|
||||
|
||||
def test_rete_repeated_execute_matches_fires_activation_once(self):
|
||||
first_results = self.engine.execute_matches([self.match])
|
||||
second_results = self.engine.execute_matches([self.match])
|
||||
|
||||
self.assertEqual(first_results, ["Adult(?x)"])
|
||||
self.assertEqual(second_results, ["Adult(?x)"])
|
||||
self.assertEqual(self.calls, [{"x": "John"}])
|
||||
|
||||
def test_rete_duplicate_match_preserves_results_but_fires_once(self):
|
||||
results = self.engine.execute_matches([self.match, self.match])
|
||||
|
||||
self.assertEqual(results, ["Adult(?x)", "Adult(?x)"])
|
||||
self.assertEqual(self.calls, [{"x": "John"}])
|
||||
|
||||
def test_rete_distinct_fact_ids_create_distinct_activations(self):
|
||||
other_match = Match(
|
||||
rule=self.rule,
|
||||
facts=[Fact("person-2", "Person", ["John"])],
|
||||
bindings={"x": "John"},
|
||||
)
|
||||
|
||||
self.engine.execute_matches([self.match])
|
||||
self.engine.execute_matches([other_match])
|
||||
|
||||
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
|
||||
|
||||
def test_rete_equivalent_nested_bindings_share_an_activation(self):
|
||||
first_match = Match(
|
||||
rule=self.rule,
|
||||
facts=self.match.facts,
|
||||
bindings={"x": {"a": 1, "b": 2}},
|
||||
)
|
||||
reordered_match = Match(
|
||||
rule=self.rule,
|
||||
facts=self.match.facts,
|
||||
bindings={"x": {"b": 2, "a": 1}},
|
||||
)
|
||||
|
||||
self.engine.execute_matches([first_match])
|
||||
self.engine.execute_matches([reordered_match])
|
||||
|
||||
self.assertEqual(len(self.calls), 1)
|
||||
|
||||
def test_rete_structured_fact_identity_avoids_separator_collisions(self):
|
||||
first_match = Match(
|
||||
rule=self.rule,
|
||||
facts=[Fact("a", "b:C", [])],
|
||||
bindings={"x": "John"},
|
||||
)
|
||||
colliding_text_match = Match(
|
||||
rule=self.rule,
|
||||
facts=[Fact("a:b", "C", [])],
|
||||
bindings={"x": "John"},
|
||||
)
|
||||
|
||||
self.engine.execute_matches([first_match])
|
||||
self.engine.execute_matches([colliding_text_match])
|
||||
|
||||
self.assertEqual(len(self.calls), 2)
|
||||
|
||||
def test_rete_cyclic_binding_preserves_results_and_deduplicates_actions(self):
|
||||
cyclic_value = []
|
||||
cyclic_value.append(cyclic_value)
|
||||
cyclic_match = Match(
|
||||
rule=self.rule,
|
||||
facts=self.match.facts,
|
||||
bindings={"x": cyclic_value},
|
||||
)
|
||||
|
||||
first_results = self.engine.execute_matches([cyclic_match])
|
||||
second_results = self.engine.execute_matches([cyclic_match])
|
||||
|
||||
self.assertEqual(first_results, ["Adult(?x)"])
|
||||
self.assertEqual(second_results, ["Adult(?x)"])
|
||||
self.assertEqual(len(self.calls), 1)
|
||||
|
||||
def test_rete_equivalent_mapping_implementations_share_an_activation(self):
|
||||
first_match = Match(
|
||||
rule=self.rule,
|
||||
facts=self.match.facts,
|
||||
bindings={"x": UserDict({"a": 1, "b": 2})},
|
||||
)
|
||||
reordered_match = Match(
|
||||
rule=self.rule,
|
||||
facts=self.match.facts,
|
||||
bindings={"x": UserDict({"b": 2, "a": 1})},
|
||||
)
|
||||
|
||||
self.engine.execute_matches([first_match])
|
||||
self.engine.execute_matches([reordered_match])
|
||||
|
||||
self.assertEqual(len(self.calls), 1)
|
||||
|
||||
def test_rete_key_error_does_not_suppress_conclusion(self):
|
||||
class UnrepresentableValue:
|
||||
def __repr__(self):
|
||||
raise RuntimeError("cannot represent")
|
||||
|
||||
invalid_match = Match(
|
||||
rule=self.rule,
|
||||
facts=self.match.facts,
|
||||
bindings={"x": UnrepresentableValue()},
|
||||
)
|
||||
|
||||
results = self.engine.execute_matches([invalid_match])
|
||||
|
||||
self.assertEqual(results, ["Adult(?x)"])
|
||||
self.assertEqual(self.calls, [])
|
||||
|
||||
def test_rete_reset_action_history_allows_deliberate_replay(self):
|
||||
self.engine.execute_matches([self.match])
|
||||
|
||||
self.engine.reset_action_history()
|
||||
self.engine.execute_matches([self.match])
|
||||
|
||||
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
|
||||
|
||||
def test_rete_reset_allows_action_replay(self):
|
||||
self.engine.execute_matches([self.match])
|
||||
|
||||
self.engine.reset()
|
||||
self.engine.execute_matches([self.match])
|
||||
|
||||
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
|
||||
|
||||
def test_rete_build_network_allows_action_replay(self):
|
||||
self.engine.execute_matches([self.match])
|
||||
|
||||
self.engine.build_network([self.rule])
|
||||
self.engine.execute_matches([self.match])
|
||||
|
||||
self.assertEqual(self.calls, [{"x": "John"}, {"x": "John"}])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -216,14 +216,13 @@ class TestNERExtractorSpacyModelCache:
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
e1 = NERExtractor(method="ml")
|
||||
e2 = NERExtractor(method="ml")
|
||||
e3 = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
NERExtractor(method="ml")
|
||||
NERExtractor(method="ml")
|
||||
NERExtractor(method="ml", model="en_core_web_sm")
|
||||
|
||||
assert len(calls) == 1, (
|
||||
"repeated NERExtractor constructions should not reload the model"
|
||||
)
|
||||
assert e1.nlp is e2.nlp is e3.nlp
|
||||
|
||||
def test_ner_extractor_and_split_callers_share_one_cached_model(self, monkeypatch):
|
||||
"""NERExtractor, SemanticChunker, and split_by_sentences must all use
|
||||
@@ -248,20 +247,23 @@ class TestNERExtractorSpacyModelCache:
|
||||
def test_ner_extractor_distinct_model_names_load_separately(self, monkeypatch):
|
||||
"""Different model names must produce separate cache entries."""
|
||||
calls = []
|
||||
loaded = {}
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append(name)
|
||||
return _nlp_mock()
|
||||
nlp = _nlp_mock()
|
||||
loaded[name] = nlp
|
||||
return nlp
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
sm = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
lg = NERExtractor(method="ml", model="en_core_web_lg")
|
||||
sm_again = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
NERExtractor(method="ml", model="en_core_web_sm")
|
||||
NERExtractor(method="ml", model="en_core_web_lg")
|
||||
NERExtractor(method="ml", model="en_core_web_sm")
|
||||
|
||||
assert calls == ["en_core_web_sm", "en_core_web_lg"]
|
||||
assert sm.nlp is sm_again.nlp
|
||||
assert sm.nlp is not lg.nlp
|
||||
# Same model name -> same cached Language object; different names -> different objects.
|
||||
assert loaded["en_core_web_sm"] is not loaded["en_core_web_lg"]
|
||||
|
||||
def test_ner_extractor_failed_load_not_cached_and_retried(self, monkeypatch):
|
||||
"""A missing model must not poison the cache. A subsequent construction
|
||||
@@ -274,31 +276,33 @@ class TestNERExtractorSpacyModelCache:
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
|
||||
|
||||
# Construction with missing model: nlp must remain None, no crash
|
||||
extractor1 = NERExtractor(method="ml")
|
||||
assert extractor1.nlp is None
|
||||
# Construction with missing model: must not raise
|
||||
NERExtractor(method="ml")
|
||||
assert len(attempts) == 1, "one load attempt expected for the missing model"
|
||||
|
||||
# Second construction: must retry (cache must not hold the failure)
|
||||
extractor2 = NERExtractor(method="ml")
|
||||
assert extractor2.nlp is None
|
||||
NERExtractor(method="ml")
|
||||
assert len(attempts) == 2, "a failed load must not be cached"
|
||||
|
||||
# Now install a working model and verify recovery
|
||||
loaded_models = {}
|
||||
|
||||
def working_load(name, **_kwargs):
|
||||
attempts.append(name)
|
||||
return _nlp_mock()
|
||||
nlp = _nlp_mock()
|
||||
loaded_models[name] = nlp
|
||||
return nlp
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
|
||||
|
||||
extractor3 = NERExtractor(method="ml")
|
||||
extractor4 = NERExtractor(method="ml")
|
||||
NERExtractor(method="ml")
|
||||
NERExtractor(method="ml")
|
||||
|
||||
assert extractor3.nlp is not None
|
||||
assert extractor3.nlp is extractor4.nlp
|
||||
assert len(attempts) == 3, (
|
||||
"exactly one successful load expected after the model becomes available"
|
||||
)
|
||||
# Confirm the recovered model is cached and shared across callers.
|
||||
assert se_methods.load_spacy_model("en_core_web_sm") is loaded_models["en_core_web_sm"]
|
||||
|
||||
def test_ner_extractor_non_ml_method_does_not_load_model(self, monkeypatch):
|
||||
"""NERExtractor with a non-ml method must not touch the spaCy cache."""
|
||||
|
||||
@@ -33,10 +33,24 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
# Mock LLM provider
|
||||
mock_provider = MagicMock()
|
||||
mock_provider.is_available.return_value = True
|
||||
mock_provider.generate_structured.return_value = [
|
||||
{"text": "Apple Inc.", "label": "ORG", "start": 0, "end": 10, "confidence": 0.95},
|
||||
{"text": "Steve Jobs", "label": "PERSON", "start": 26, "end": 36, "confidence": 0.98}
|
||||
# The LLM path uses generate_typed (Pydantic schema validation), not
|
||||
# generate_structured. Build a typed response object whose .entities
|
||||
# carries simple namespace-like items.
|
||||
def _entity(text, label, start, end, confidence):
|
||||
item = MagicMock()
|
||||
item.text = text
|
||||
item.label = label
|
||||
item.start = start
|
||||
item.end = end
|
||||
item.confidence = confidence
|
||||
return item
|
||||
|
||||
typed_response = MagicMock()
|
||||
typed_response.entities = [
|
||||
_entity("Apple Inc.", "ORG", 0, 10, 0.95),
|
||||
_entity("Steve Jobs", "PERSON", 26, 36, 0.98),
|
||||
]
|
||||
mock_provider.generate_typed.return_value = typed_response
|
||||
mock_create_provider.return_value = mock_provider
|
||||
|
||||
# Initialize extractor with LLM method
|
||||
@@ -56,7 +70,7 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
self.assertEqual(len(entities), 2)
|
||||
self.assertEqual(entities[0].text, "Apple Inc.")
|
||||
self.assertEqual(entities[0].label, "ORG")
|
||||
self.assertEqual(entities[0].metadata["extraction_method"], "llm")
|
||||
self.assertEqual(entities[0].metadata["extraction_method"], "llm_typed")
|
||||
self.assertEqual(entities[0].metadata["model"], "gpt-4")
|
||||
|
||||
@patch('semantica.semantic_extract.methods.spacy')
|
||||
@@ -115,7 +129,6 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
with patch('semantica.semantic_extract.ner_extractor.SPACY_AVAILABLE', True):
|
||||
extractor = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
|
||||
self.assertIsNone(extractor.nlp)
|
||||
self.assertFalse(extractor._ml_runtime_usable)
|
||||
|
||||
@patch('semantica.semantic_extract.methods.get_entity_method')
|
||||
@@ -183,7 +196,7 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
|
||||
self.assertTrue(len(entities) >= 2)
|
||||
texts = [e.text for e in entities]
|
||||
self.assertIn("Apple Inc", texts) # Regex pattern does not capture the trailing dot
|
||||
self.assertIn("Apple Inc.", texts) # The ORG pattern includes the trailing dot via (?:\.|\b)
|
||||
# Actually methods.py regex: r"\b([A-Z][a-zA-Z]+(?:\s+[A-Z][a-zA-Z]+)*\s+(?:Inc|Corp|LLC|Ltd|Company))\b"
|
||||
# "Apple Inc." -> "Apple Inc" (dot is outside \b if not matched?)
|
||||
# Let's check the result strictly
|
||||
|
||||
+37
-10
@@ -31,21 +31,48 @@ class TestHelpers(unittest.TestCase):
|
||||
dict2 = {"b": {"d": 3}, "e": 4}
|
||||
merged = helpers.merge_dicts(dict1, dict2, deep=True)
|
||||
self.assertEqual(merged, {"a": 1, "b": {"c": 2, "d": 3}, "e": 4})
|
||||
|
||||
def test_flatten_dict(self):
|
||||
data = {"a": {"b": 1, "c": 2}}
|
||||
"""Basic nested flattening with multiple sibling keys."""
|
||||
data = {"a": {"b": 1, "c": 2}, "d": 3}
|
||||
result = helpers.flatten_dict(data)
|
||||
self.assertEqual(result, {"a.b": 1, "a.c": 2})
|
||||
self.assertEqual(result, {"a.b": 1, "a.c": 2, "d": 3})
|
||||
|
||||
def test_flatten_dict_deeply_nested(self):
|
||||
"""Deeply nested structure is fully flattened."""
|
||||
self.assertEqual(
|
||||
helpers.flatten_dict({"a": {"b": {"c": 1}}}),
|
||||
{"a.b.c": 1},
|
||||
)
|
||||
|
||||
def test_flatten_dict_custom_separator(self):
|
||||
"""Custom separator is used in generated keys."""
|
||||
self.assertEqual(
|
||||
helpers.flatten_dict({"a": {"b": 1}}, sep="__"),
|
||||
{"a__b": 1},
|
||||
)
|
||||
|
||||
def test_flatten_dict_empty(self):
|
||||
"""Empty input returns empty output."""
|
||||
self.assertEqual(helpers.flatten_dict({}), {})
|
||||
|
||||
def test_flatten_dict_key_collision(self):
|
||||
data = {
|
||||
"a.b": 1,
|
||||
"a": {
|
||||
"b": 2
|
||||
}
|
||||
}
|
||||
"""#1010 regression: a top-level key containing the separator must not
|
||||
silently overwrite a value produced from a nested dict when both resolve
|
||||
to the same flattened key. Before the fix, {'a.b': 1, 'a': {'b': 2}}
|
||||
silently dropped one value; now it raises ValueError."""
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
helpers.flatten_dict({"a.b": 1, "a": {"b": 2}})
|
||||
self.assertIn("Key collision", str(ctx.exception))
|
||||
self.assertIn("a.b", str(ctx.exception))
|
||||
|
||||
with self.assertRaises(ValueError):
|
||||
helpers.flatten_dict(data)
|
||||
def test_flatten_dict_no_false_positive(self):
|
||||
"""Similar-looking keys that produce distinct flattened keys must not
|
||||
trigger the collision guard."""
|
||||
self.assertEqual(
|
||||
helpers.flatten_dict({"a.b": 1, "a": {"c": 2}}),
|
||||
{"a.b": 1, "a.c": 2},
|
||||
)
|
||||
|
||||
def test_safe_import_returns_module_and_flag(self):
|
||||
module, available = helpers.safe_import("json")
|
||||
|
||||
Reference in New Issue
Block a user