feat: add Amazon Redshift ingestor (#1587 )

* feat: add Redshift ingestor * fix(redshift): resolve 5 code-review issues in RedshiftIngestor Fix #1 — restore autocommit on caller-owned connections All four methods (ingest_table, ingest_query, get_table_schema, list_tables) now capture the connection's prior autocommit value before enabling it for read-only ingestion and restore it in the finally block when the connection was already open. Transient connections are still closed normally. Fix #2 — batch fetching: process rows per-batch, not after accumulating ingest_query's batch path now converts and extends data[] directly inside the fetch loop so raw tuples from each batch are eligible for GC before the next fetch. The documentation is updated to accurately state that batch_size controls driver round-trip size, not total memory. Fix #3 — schema env-var now honoured RedshiftIngestor.__init__ changes schema default from 'public' to None so the existing os.getenv('REDSHIFT_SCHEMA', 'public') fallback in the body is reached when no explicit argument is supplied. Fix #4 — db-redshift included in db-all aggregate pyproject.toml db-all now references db-redshift alongside the other database extras. Fix #5 — duplicate column labels no longer silently drop values _disambiguate_columns() appends _1, _2, ... suffixes to repeated cursor labels before dict(zip) so every value is retained. A new _make_row_dicts() helper encapsulates the zip+disambiguate pattern used by both _fetch_all and ingest_query. Tests: 22 new targeted tests added covering all five fixes. --------- Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan> Co-authored-by: Kaif <98801504+KaifAhmad1@users.noreply.github.com>
docs: rewrite remaining bare relative links that 404 on the live site (#1580 )
2026-09-12 04:01:35 +00:00 · 2026-09-11 18:43:14 +05:30 · 2026-09-11 17:40:52 +05:30 · 2026-09-11 17:06:18 +05:30 · 2026-09-11 16:55:06 +05:30 · 2026-09-11 15:37:15 +05:30
71 changed files with 6448 additions and 275 deletions
@@ -120,6 +120,7 @@ jobs:
          print('semantica', semantica.__version__, 'core installed and importable')
          "
          pytest -q tests/test_issue_1513_slim_core.py
+          pytest -q tests/test_docs_check.py
      - name: Install Explorer backend test dependencies
        run: |
          # Run the deterministic backend path before the all-extras CI
@@ -34,7 +34,7 @@ jobs:
      # meaningful state carried over from a failed attempt.
      - name: Initialize CodeQL (attempt 1)
        id: codeql-init-1
-        uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/init@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        continue-on-error: true
        with:
          languages: python
@@ -44,7 +44,7 @@ jobs:
      - name: Initialize CodeQL (attempt 2)
        id: codeql-init-2
        if: steps.codeql-init-1.outcome == 'failure'
-        uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/init@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        continue-on-error: true
        with:
          languages: python
@@ -54,17 +54,17 @@ jobs:
      - name: Initialize CodeQL (attempt 3)
        id: codeql-init-3
        if: steps.codeql-init-2.outcome == 'failure'
-        uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/init@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        with:
          languages: python
          queries: security-and-quality
          config-file: .github/codeql/codeql-config.yml

      - name: Autobuild
-        uses: github/codeql-action/autobuild@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/autobuild@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4

      - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/analyze@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        with:
          category: "/language:python"
          upload: false
@@ -74,7 +74,7 @@ jobs:
        # Uploads results only when Default Setup is not active.
        # If Default Setup is still enabled, this step skips gracefully
        # instead of failing the workflow with HTTP 409.
-        uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        with:
          sarif_file: ${{ steps.codeql.outputs.sarif-output }}
          category: "/language:python"
@@ -61,7 +61,7 @@ jobs:

      - name: Upload Trivy SARIF
        if: always()
-        uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        with:
          sarif_file: trivy-results.sarif
          category: trivy-container
@@ -59,7 +59,7 @@ jobs:
        # avoiding the guardian.cmd/checkov exit-code bug in the MSDO wrapper.
        tools: eslint,templateanalyzer,terrascan
    - name: Upload results to Security tab
-      uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+      uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
      with:
        sarif_file: ${{ steps.msdo.outputs.sarifFile }}

@@ -100,7 +100,7 @@ jobs:
      run: python .github/scripts/filter_checkov_skipped.py reports/results_json.json reports/results_sarif.sarif reports/checkov.sarif

    - name: Upload Checkov results to Security tab
-      uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+      uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
      if: always()
      with:
        sarif_file: reports/checkov.sarif
@@ -40,6 +40,6 @@ jobs:
          retention-days: 5

      - name: Upload to code-scanning
-        uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
+        uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
        with:
          sarif_file: results.sarif
@@ -42,6 +42,8 @@ pip install semantica

 </div>

+[English](https://readme-i18n.com/semantica-agi/semantica?lang=en) · [Deutsch](https://readme-i18n.com/semantica-agi/semantica?lang=de) · [Français](https://readme-i18n.com/semantica-agi/semantica?lang=fr) · [Español](https://readme-i18n.com/semantica-agi/semantica?lang=es) · [Italiano](https://readme-i18n.com/semantica-agi/semantica?lang=it) · [Português](https://readme-i18n.com/semantica-agi/semantica?lang=pt) · [العربية](https://readme-i18n.com/semantica-agi/semantica?lang=ar) · [اردو](https://readme-i18n.com/semantica-agi/semantica?lang=ur) · [हिन्दी](https://readme-i18n.com/semantica-agi/semantica?lang=hi) · [中文](https://readme-i18n.com/semantica-agi/semantica?lang=zh) · [日本語](https://readme-i18n.com/semantica-agi/semantica?lang=ja) · [한국어](https://readme-i18n.com/semantica-agi/semantica?lang=ko)
+
 ---

 <div align="center">
@@ -62,16 +64,21 @@ pip install semantica

 ---

-Most AI agents run on embeddings, not meaning: similarity scores with no structure, no relationships, and no way to explain why a result came back. Semantica is the semantic/context layer underneath your LLM, vector store, and agent framework: a deterministic infrastructure layer (no LLM required for graph construction, reasoning, or provenance) that turns fragmented enterprise data into a structured, queryable Context Graph and knowledge graph, governed by ontologies and controlled vocabularies (OWL, SHACL, SKOS) so the meaning of your data is explicit, not just its embedding. Decision provenance and audit trails fall out of that structure as a property, not the product itself; in domains a regulator can question, that same structure just happens to double as a straight answer to "why."
+Most AI agents run on embeddings, not meaning: similarity scores with no structure, no relationships, and no way to explain why a result came back.

-> ⚠️ **System-level explainability, not foundation-model explainability.** Semantica does not expose or reconstruct what happens *inside* the LLM — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. Semantica explains what's *outside* the model: the context and data fed in, the decision produced, its provenance, relevant relationships, applied policies, and the full execution trail.
+Semantica is the semantic/context layer underneath your LLM, vector store, and agent framework: deterministic infrastructure (no LLM required for graph construction, reasoning, or provenance; where an LLM is used, it's optional and vendor-neutral, every major provider supported, OpenAI, Anthropic, Gemini, and more, via `semantica.llms`) that turns fragmented enterprise data into a structured, queryable Context Graph and knowledge graph that carries the business context, not just the data structure. Ontologies and controlled vocabularies (OWL, SHACL, SKOS) make what an entity *means* to your business, its definitions, relationships, and rules, as explicit as the data itself, not just its embedding.
+
+Decision provenance and audit trails aren't the product. They fall out of that structure for free, and in domains a regulator can question, the same structure that makes your agent smarter also gives you a straight answer to "why."
+
+> [!NOTE]
+> **System-level explainability, not foundation-model explainability.** Semantica doesn't expose or reconstruct what happens *inside* the LLM: its internal reasoning stays opaque, like it does for any external system. Semantica explains what's *outside* the model: the context fed in, the decision produced, its provenance, relevant relationships, applied policies, and the full execution trail.

 **Who it's for:**

 - **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context, not just a vector index
- **Data platform teams on Databricks or Snowflake** turning tables already in Unity Catalog or a warehouse into a governed, lineage-tracked knowledge graph, without exporting to a third-party SaaS
+- **Enterprise data teams on Databricks, Snowflake, or SAP** turning tables already in the lakehouse or warehouse into a governed, lineage-tracked knowledge graph, without exporting to a third-party SaaS
 - **Compliance, risk, and audit teams** who need a straight answer to "why did the AI do that?" in a format a regulator accepts
- **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box or send their data to someone else's SaaS to get one
+- **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box or hand their data to someone else's SaaS to get one
 - **Platform and infra engineers** who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend
 - **Data and knowledge engineers** building a KG from messy, multi-source data, where conflicting facts get flagged and duplicates get merged, not silently overwritten

@@ -83,15 +90,15 @@ Most AI agents run on embeddings, not meaning: similarity scores with no structu

 - **Context Graphs:** A structured, queryable graph of everything your agent knows, decides, and reasons about
 - **Decision Intelligence:** Every decision is a first-class object: traceable, searchable by precedent, and causally linked
- **AI Governance & Ontology:** SHACL constraints, conflict detection, compliance rules, OWL generation, and SKOS vocabulary management with a visual editor
- **Full Auditability:** W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF
- **Deterministic Reasoning:** Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes
- **Knowledge Pipeline:** Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout
- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection), Snowflake (warehouse/database/schema, key-pair and OAuth auth), and SAP OData (Business Partners, Sales Orders, OAuth2/Basic auth), so data already living in your lakehouse or warehouse becomes graph nodes with provenance, not another export/import hop
+- **AI Governance & Ontology:** SHACL constraints, conflict detection, compliance rules, OWL generation, and SKOS vocabularies, all with a visual editor
+- **Full Auditability:** W3C PROV-O provenance on every fact, exportable to JSON, CSV, or RDF
+- **Deterministic Reasoning:** Forward chaining, Rete network, Datalog, and SPARQL, with fully explainable paths, not black boxes
+- **Knowledge Pipeline:** Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and graph construction, with semantic dedup and provenance-preserving merges built in
+- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake), Snowflake, and SAP OData, so data already in your lakehouse or warehouse becomes graph nodes with provenance, no export/import hop
 - **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
+- **Polyglot Graph Storage:** RDF (Oxigraph, Blazegraph, Jena, RDF4J) and Labeled Property Graphs (Neo4j, FalkorDB, AGE, Neptune), plus vector stores, all swappable without touching your code
 - **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
- **Drop-in Integrations:** Native Agno, CrewAI, and LangChain support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
+- **Drop-in Integrations:** Agno, CrewAI, and LangChain support, a full MCP server, a CLI, a REST API, and plugins across major editors

 ---

@@ -1480,16 +1487,6 @@ app = create_app(session=GraphSession(graph), agent_memory=memory)
 The Memories workspace is shown only when `agent_memory` is provided. Apply
 updates the supplied runtime object; it does not add disk persistence.

-## What's New in v0.7.0
-
-**Slim core dependencies: lightweight base install with granular optional extras** — `pip install semantica` now installs only 22 essential core dependencies, moving heavy packages into dedicated optional extras:
- **Dramatically lighter and faster installation**: Core installation no longer pulls heavy machine learning or visualization packages by default.
- **Granular extras**: Install only what your workload requires (`documents`, `embeddings-local`, `models-huggingface`, `nlp-spacy`, `viz`, `media`, `vectorstore-faiss`, `graph-embeddings`, `ingest-git`).
- **Full backward compatibility**: `pip install "semantica[all]"` preserves the full bundled suite, while `semantica<0.7.0` remains a permanent escape hatch.
- **Lazy parser construction & graceful fallbacks**: Document parsers can be constructed without extras and only raise actionable error hints upon calling `.parse()`; `XMLParser` automatically falls back to Python's standard library `xml.etree`.
-
---
-
 ## What's New in v0.6.8

 **Every release from here on is cryptographically signed** — the build now runs SLSA build-provenance attestation plus Sigstore signing, and `.sigstore.json` bundles ship alongside the wheel/sdist on every GitHub Release, closing the OpenSSF Scorecard Signed-Releases gap. Beyond that, this is a large fix-and-hardening release plus a batch of vector-store and LLM-provider additions:
@@ -1507,22 +1504,6 @@ Also fixes 35 correctness bugs (Python 3.9 install breakage, FAISS save/load met

 ---

-## Built for High-Stakes Domains
-
-Semantica is designed for environments where AI outputs must be explainable, auditable, and defensible, and where the data itself can't leave your infrastructure. Self-hostable with zero vendor lock-in, it's built as much for organizations handling confidential or classified data as for regulated industries chasing an audit trail:
-
- **Finance:** Loan underwriting audit trails, fraud detection, AML compliance, regulatory risk knowledge graphs
- **Healthcare:** Clinical decision support, drug interaction graphs, and patient safety audit trails
- **Legal:** Evidence-backed research, contract analysis, case law reasoning, and privilege tracking
- **Government & Defense:** Policy decision records, classified information governance, and regulatory reporting, fully self-hosted with no data leaving your perimeter
- **Law Enforcement:** Case linkage, evidence provenance chains, and investigative knowledge graphs that hold up under legal scrutiny
- **Cybersecurity:** Threat attribution, incident response timelines, and IOC provenance tracking
- **Autonomous Systems:** Decision logs, safety validation, and explainable AI for certification
-
-> ⚠️ **This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits what the AI system did, not the LLM's private internal reasoning.
-
---
-
 ## Installation

 ```bash
@@ -1530,7 +1511,7 @@ pip install semantica             # lightweight core (22 essential dependencies)
 pip install "semantica[all]"      # full bundled behavior with all extras
 ```

-> **Note for upgrades from <0.7.0**: In Semantica 0.7.0+, heavy machine learning, NLP, visualization, and document dependencies were moved into optional extras to make core installation significantly lighter and faster. If you want the previous bundled installation, install with `pip install "semantica[all]"` or pin `semantica<0.7.0`.
+> **Note:** Heavy machine learning, NLP, visualization, and document dependencies live in optional extras to keep core installation lightweight and fast. If you want the previous bundled installation, install with `pip install "semantica[all]"`.

 ```bash
 # Granular Extras
@@ -107,7 +107,8 @@
              "integrations/docling",
              "integrations/snowflake",
              "integrations/databricks",
-              "integrations/salesforce"
+              "integrations/salesforce",
+              "integrations/redshift"
            ]
          },
          {
@@ -5,7 +5,7 @@ icon: "circle-question"
 ---

 <Info>
-  Use **Ctrl+F** / **Cmd+F** to search this page. Common jumps: [Installation](#installation) · [Data & Features](#data--features) · [Troubleshooting](#troubleshooting)
+  Use **Ctrl+F** / **Cmd+F** to search this page. Common jumps: [Installation](#installation) · [Data & Features](#data-&-features) · [Troubleshooting](#troubleshooting)
 </Info>

 ## Quick Answers
@@ -661,7 +661,7 @@ print("Total memories: {}".format(s.get("total_items", 0)))
 - [Decision Intelligence](/guides/decision-intelligence) — Recording decisions as graph nodes with causal chains and policy gating.
 - [Multi-Agent Systems](/guides/multi-agent) — Coordinating multiple agents through a shared `AgentContext` and save/load handoffs.
 - [LLM Integrations](/guides/llm-integrations) — Configuring the LLM provider passed to `query_with_reasoning()`.
- [Deduplication Guide](deduplication) — Full reference for `DuplicateDetector`, `EntityMerger`, similarity methods, and cluster strategies.
- [Ontology Management](ontology) — Generate and validate OWL ontologies from the knowledge graph; export to Turtle, OWL/XML, JSON-LD.
+- [Deduplication Guide](/guides/deduplication) — Full reference for `DuplicateDetector`, `EntityMerger`, similarity methods, and cluster strategies.
+- [Ontology Management](/guides/ontology) — Generate and validate OWL ontologies from the knowledge graph; export to Turtle, OWL/XML, JSON-LD.
 - [Context Module Reference](../reference/context) — Full API: `AgentContext`, `AgentMemory`, `MemoryItem`, `ContextRetriever`.
 - [Vector Store Reference](../reference/vector_store) — FAISS, Qdrant, pgvector, Pinecone backends.
@@ -497,7 +497,7 @@ print("Model v1.1 verified and approved for production.")
 ## Related Guides

 - [Context Graphs](/guides/context-graphs) — `ContextGraph.to_dict()` feeds `create_snapshot()`
- [Ontology Management](ontology) — pair ontology versioning with graph versioning for a complete schema + data audit trail
+- [Ontology Management](/guides/ontology) — pair ontology versioning with graph versioning for a complete schema + data audit trail
 - [SHACL Validation](/guides/shacl-validation) — validate graph data at each version gate before snapshotting
- [Provenance](provenance) — combine change management with W3C PROV-O lineage for a full audit trail
- [Visualization](visualization) — `TemporalVisualizer.visualize_snapshot_comparison()` and `visualize_metrics_evolution()` render version diffs as interactive charts
+- [Provenance](/guides/provenance) — combine change management with W3C PROV-O lineage for a full audit trail
+- [Visualization](/guides/visualization) — `TemporalVisualizer.visualize_snapshot_comparison()` and `visualize_metrics_evolution()` render version diffs as interactive charts
@@ -65,7 +65,7 @@ flowchart TD
    G --> H[SHACL Validation]
 ```

-1. **Deduplication** — Merge duplicate nodes so each entity has exactly one canonical record. Conflict resolution operates on a single canonical entity; you must identify it before comparing what different sources say about it. See [Deduplication](deduplication).
+1. **Deduplication** — Merge duplicate nodes so each entity has exactly one canonical record. Conflict resolution operates on a single canonical entity; you must identify it before comparing what different sources say about it. See [Deduplication](/guides/deduplication).
 2. **Conflict Detection** — Call `detect_entity_conflicts()` to surface all property disagreements at once, or `detect_value_conflicts()` to target a specific property.
 3. **Resolution** — For each conflict, apply a strategy (`CREDIBILITY_WEIGHTED`, `MOST_RECENT`, `VOTING`, etc.) or route it for expert review (`EXPERT_REVIEW`).
 4. **Persist Canonical Values** — Write resolved values back to your canonical entities or graph store. See [Persisting resolved values](#persisting-resolved-values).
@@ -696,8 +696,8 @@ Calling `set_resolution_rule()` for every entity-property pair just to apply the

 ## Related Guides

- [Deduplication](deduplication) — remove duplicate nodes before running conflict detection
- [Provenance](provenance) — track which source each resolved value came from, and verify the audit trail cryptographically
+- [Deduplication](/guides/deduplication) — remove duplicate nodes before running conflict detection
+- [Provenance](/guides/provenance) — track which source each resolved value came from, and verify the audit trail cryptographically
 - [SHACL Validation](/guides/shacl-validation) — enforce structural constraints after conflicts are resolved
 - [Change Management](/guides/change-management) — snapshot the graph before and after conflict resolution runs
- [Ontology Management](ontology) — align entity types to a shared vocabulary to reduce type conflicts at the schema level
+- [Ontology Management](/guides/ontology) — align entity types to a shared vocabulary to reduce type conflicts at the schema level
@@ -489,7 +489,7 @@ context2.load("agent_state/")

 ## Common Pitfalls

-**Duplicate entities.** Adding "APT-29", "APT29", and "Cozy Bear" as separate nodes fragments the graph when they should be one entity. Use consistent naming conventions upfront, or use `detect_duplicates()` and `EntityMerger` from the [Deduplication](deduplication) guide to merge them after ingestion.
+**Duplicate entities.** Adding "APT-29", "APT29", and "Cozy Bear" as separate nodes fragments the graph when they should be one entity. Use consistent naming conventions upfront, or use `detect_duplicates()` and `EntityMerger` from the [Deduplication](/guides/deduplication) guide to merge them after ingestion.

 **Inconsistent naming conventions.** Mixing "ThreatActor", "threat_actor", and "Threat-Actor" as node types breaks queries that filter by type. Pick one convention and enforce it across all data sources.

@@ -706,8 +706,8 @@ for n in stress_reach:

 - [Graph Analytics](/guides/graph-analytics) — centrality rankings, community detection, node embeddings, and link prediction on a populated `ContextGraph`
 - [Decision Intelligence](/guides/decision-intelligence) — recording decisions as typed nodes, causal chain analysis, precedent search, and policy enforcement
- [Ingest](ingest) — loading data from PDFs, APIs, databases, STIX bundles, and RSS feeds into the graph
- [Deduplication](deduplication) — detecting and merging near-duplicate nodes before insertion to prevent graph fragmentation
- [Reasoning](reasoning) — temporal interval algebra (Allen relations), forward/backward chaining, and SPARQL over the knowledge graph
- [Ontology Management](ontology) — deriving formal OWL ontologies from `graph.to_dict()` for downstream reasoning engines
+- [Ingest](/guides/ingest) — loading data from PDFs, APIs, databases, STIX bundles, and RSS feeds into the graph
+- [Deduplication](/guides/deduplication) — detecting and merging near-duplicate nodes before insertion to prevent graph fragmentation
+- [Reasoning](/guides/reasoning) — temporal interval algebra (Allen relations), forward/backward chaining, and SPARQL over the knowledge graph
+- [Ontology Management](/guides/ontology) — deriving formal OWL ontologies from `graph.to_dict()` for downstream reasoning engines
 - [Context Module Reference](../reference/context) — full API for `AgentContext`, `ContextGraph`, `ContextNode`, `ContextEdge`
@@ -664,6 +664,6 @@ results = context.find_precedents("APT29 infrastructure attribution", limit=5)

 - [Context Graphs](/guides/context-graphs) — how `ContextGraph` stores decision nodes and causal edges
 - [Distance Intelligence](/guides/distance-intelligence) — `trace_decision_causality()` annotates causal chains with confidence decay and distance bands
- [Provenance](provenance) — W3C PROV-O audit trail that wraps decision records in standards-compliant provenance
+- [Provenance](/guides/provenance) — W3C PROV-O audit trail that wraps decision records in standards-compliant provenance
 - [MCP Server](/guides/mcp-server) — expose decision recording and precedent search to LLM agents via the `record_decision` and `find_precedents` tools
 - [Change Management](/guides/change-management) — checkpoint decision state with `flush_checkpoint()` for versioned snapshots
@@ -611,8 +611,8 @@ The similarity threshold controls sensitivity. Start at 0.7 and examine false po

 ## Related Guides

- [Ingest Anything](ingest) — multi-source ingestion creates the duplicates this module resolves
+- [Ingest Anything](/guides/ingest) — multi-source ingestion creates the duplicates this module resolves
 - [Context Graphs](/guides/context-graphs) — store deduplicated entities directly in the knowledge graph
 - [Conflict Resolution](/guides/conflict-resolution) — after merging, reconcile disagreeing property values on the canonical entity
- [Provenance](provenance) — track merge lineage so every canonical entity traces back to its original sources
- [Pipeline](pipeline) — chain ingest, deduplicate, and store as a `PipelineBuilder` workflow
+- [Provenance](/guides/provenance) — track merge lineage so every canonical entity traces back to its original sources
+- [Pipeline](/guides/pipeline) — chain ingest, deduplicate, and store as a `PipelineBuilder` workflow
@@ -561,4 +561,4 @@ for chain in chains:
 - [Graph Analytics](/guides/graph-analytics) — centrality, community detection, Node2Vec embeddings, link prediction
 - [Agent Memory](/guides/agent-memory) — proximity-blended retrieval (`proximity_weight`) integrates distance intelligence into memory search
 - [Decision Intelligence](/guides/decision-intelligence) — `trace_decision_causality()` for causal chains with distance annotations
- [Reasoning & Rules](reasoning) — `TemporalReasoningEngine` for Allen interval algebra over time-bounded graph nodes
+- [Reasoning & Rules](/guides/reasoning) — `TemporalReasoningEngine` for Allen interval algebra over time-bounded graph nodes
@@ -444,7 +444,7 @@ For semantic reasoning and ontology work, OWL/XML is the format — it is the on
 ## Related Guides

 - [Context Graphs](/guides/context-graphs) — the `ContextGraph` object whose `to_dict()` feeds all exports
- [Ontology Management](ontology) — export OWL ontologies generated from your graph
- [Reasoning & Rules](reasoning) — reasoning results can be exported as RDF triples
+- [Ontology Management](/guides/ontology) — export OWL ontologies generated from your graph
+- [Reasoning & Rules](/guides/reasoning) — reasoning results can be exported as RDF triples
 - [Change Management](/guides/change-management) — snapshot a graph before exporting to prove the export was made from a verified state
- [Pipeline](pipeline) — chain ingest, extract, and export in a single `PipelineBuilder`
+- [Pipeline](/guides/pipeline) — chain ingest, extract, and export in a single `PipelineBuilder`
@@ -539,6 +539,6 @@ print(f"\n{len(result['communities'])} exposure clusters  "
 ## Related Guides

 - [Context Graphs](/guides/context-graphs) — building and querying the underlying `ContextGraph`
- [Visualization](visualization) — render centrality rankings and community clusters as interactive dashboards
+- [Visualization](/guides/visualization) — render centrality rankings and community clusters as interactive dashboards
 - [Decision Intelligence](/guides/decision-intelligence) — link prediction and structural similarity applied to decision nodes
 - [GraphRAG](/guides/graphrag) — using analytics results to ground LLM generation in the most contextually relevant subgraph
@@ -950,9 +950,9 @@ print(f"Compliance graph: {graph.stats()['node_count']} nodes, "

 ## Related Guides

- [Pipeline](pipeline) — chain ingest steps with `PipelineBuilder` for automated, retryable, parallelised workflows
+- [Pipeline](/guides/pipeline) — chain ingest steps with `PipelineBuilder` for automated, retryable, parallelised workflows
 - [Context Graphs](/guides/context-graphs) — storing and querying the entities you ingest as a typed property graph
 - [Semantic Extraction](/guides/semantic-extraction) — NER, relation extraction, and triplet extraction from ingested text
- [Provenance](provenance) — tracking the origin document, confidence score, and ingestion timestamp for every extracted entity
+- [Provenance](/guides/provenance) — tracking the origin document, confidence score, and ingestion timestamp for every extracted entity
 - [Databricks Integration](../integrations/databricks) — Unity Catalog setup, PAT/OAuth M2M authentication, and lineage introspection
 - [Snowflake Integration](../integrations/snowflake) — warehouse setup and password/key-pair/OAuth authentication
@@ -342,8 +342,8 @@ The result is a fully auditable credit decision trail with precedent links, read

 ## Related Guides

- [Reasoning & Rules](reasoning) — the engine behind the `run_reasoning` tool
+- [Reasoning & Rules](/guides/reasoning) — the engine behind the `run_reasoning` tool
 - [Decision Intelligence](/guides/decision-intelligence) — how decisions are stored as causal graph nodes
 - [Context Graphs](/guides/context-graphs) — the graph that `add_entity` and `add_relationship` write to
- [Export & Serialization](export) — all export formats available via `export_graph`
- [Ontology Management](ontology) — generate OWL ontologies from the graph built via MCP
+- [Export & Serialization](/guides/export) — all export formats available via `export_graph`
+- [Ontology Management](/guides/ontology) — generate OWL ontologies from the graph built via MCP
@@ -504,7 +504,7 @@ else:
 ## Related Guides

 - [SHACL Validation](/guides/shacl-validation) — generate W3C SHACL constraint shapes from your ontology and validate live graph data against them
- [Reasoning & Rules](reasoning) — apply forward/backward-chaining rules over your ontology to derive new facts
- [Export & Serialization](export) — export graphs to RDF, GraphML, CSV, and Neo4j Cypher
+- [Reasoning & Rules](/guides/reasoning) — apply forward/backward-chaining rules over your ontology to derive new facts
+- [Export & Serialization](/guides/export) — export graphs to RDF, GraphML, CSV, and Neo4j Cypher
 - [Semantic Extraction](/guides/semantic-extraction) — extract entities and relationships that feed ontology generation
 - [Context Graphs](/guides/context-graphs) — the knowledge graph that ontology generation reads from
@@ -718,7 +718,7 @@ print(f"Compliance delta update: {result.output}")

 ## Related Guides

- [Ingest](ingest) — all source types for the ingest step: PDFs, APIs, databases, RSS feeds, STIX directories, and streams
+- [Ingest](/guides/ingest) — all source types for the ingest step: PDFs, APIs, databases, RSS feeds, STIX directories, and streams
 - [Semantic Extraction](/guides/semantic-extraction) — NER, relation extraction, triplet extraction, and event detection for the extract step
 - [Context Graphs](/guides/context-graphs) — building and querying the `ContextGraph` that the store step populates
- [Provenance](provenance) — tracking the origin document, confidence score, and pipeline run ID for every extracted entity
+- [Provenance](/guides/provenance) — tracking the origin document, confidence score, and pipeline run ID for every extracted entity
@@ -663,8 +663,8 @@ print("Policy updated to v2.4.0")
 ## Related Guides

 - [Decision Intelligence](/guides/decision-intelligence) — `record_decision()`, causal chains, and precedent search — the decisions that `check_compliance()` evaluates
- [Reasoning & Rules](reasoning) — complement policy rules with formal inference for logical conflict detection
+- [Reasoning & Rules](/guides/reasoning) — complement policy rules with formal inference for logical conflict detection
 - [SHACL Validation](/guides/shacl-validation) — enforce structural constraints on policy nodes themselves
 - [Change Management](/guides/change-management) — version-snapshot the policy graph alongside the knowledge graph
- [Provenance](provenance) — W3C PROV-O lineage for every policy decision and exception
+- [Provenance](/guides/provenance) — W3C PROV-O lineage for every policy decision and exception
 - [MCP Server](/guides/mcp-server) — expose `record_decision` and `find_precedents` as MCP tools for AI agents
@@ -661,5 +661,5 @@ Note: the banking example above passes `agent_id="credit_data_service_v2"` to `t

 - [Semantic Extraction](/guides/semantic-extraction) — the NER and relation extraction pipeline that auto-generates provenance entries for every extracted entity
 - [Conflict Resolution](/guides/conflict-resolution) — provenance property sources feed directly into conflict detection; every resolved value is traceable to its source
- [Deduplication](deduplication) — merge operations are recorded in merge history; pair with provenance for a complete lineage from source to canonical entity
+- [Deduplication](/guides/deduplication) — merge operations are recorded in merge history; pair with provenance for a complete lineage from source to canonical entity
 - [Provenance Reference](../reference/provenance) — full storage backend API, `InMemoryStorage`, `SQLiteStorage`, and `ProvenanceEntry` schema
@@ -840,7 +840,7 @@ if proof:

 - [Semantic Extraction](/guides/semantic-extraction) — extract the entities and relationships that populate the graph facts you reason over
 - [GraphRAG](/guides/graphrag) — retrieve graph-grounded context for LLM responses
- [Ontology Management](ontology) — generate OWL ontologies to give your rules formal semantics
+- [Ontology Management](/guides/ontology) — generate OWL ontologies to give your rules formal semantics
 - [Decision Intelligence](/guides/decision-intelligence) — record and trace inferred decisions through the full causal chain
 - [Context Graphs](/guides/context-graphs) — the knowledge graph that reasoning operates over
 - [MCP Server](/guides/mcp-server) — expose `run_reasoning` as a tool for Claude and other agents
@@ -71,7 +71,7 @@ This pipeline transforms documents like "APT29 deployed HAMMERTOSS malware targe
 `semantica.semantic_extract` turns unstructured text into structured graph-ready output: it identifies named entities, extracts relationships between them, detects time-anchored events, resolves coreferences, and serialises everything as RDF triplets. Use it to populate a `ContextGraph` from raw documents — intelligence reports, clinical notes, regulatory filings, or any free-text corpus.

 <Info>
-  Extracted entities and relationships feed into `ContextGraph` via `AgentContext.store()`. For how they are attributed back to source documents, see the [Provenance Guide](provenance). For how the populated graph is queried and traversed, see [Context Graphs](/guides/context-graphs).
+  Extracted entities and relationships feed into `ContextGraph` via `AgentContext.store()`. For how they are attributed back to source documents, see the [Provenance Guide](/guides/provenance). For how the populated graph is queried and traversed, see [Context Graphs](/guides/context-graphs).
 </Info>

 ## Step 1 — Named Entity Recognition: who and what is in the text
@@ -668,9 +668,9 @@ The fallback behaviour is automatic: if the primary method returns an empty list

 ## Related Guides

- [Provenance Guide](provenance) — track every extracted entity and chunk back to its source document
+- [Provenance Guide](/guides/provenance) — track every extracted entity and chunk back to its source document
 - [Agent Memory Guide](/guides/agent-memory) — store extracted knowledge as searchable agent memories with graph enrichment
 - [Context Graphs Guide](/guides/context-graphs) — how extracted entities populate `ContextGraph` nodes and edges
 - [GraphRAG Guide](/guides/graphrag) — retrieve facts from the populated graph to ground LLM responses
- [Reasoning Guide](reasoning) — derive new facts, run SPARQL queries, and apply inference rules over the extracted graph
+- [Reasoning Guide](/guides/reasoning) — derive new facts, run SPARQL queries, and apply inference rules over the extracted graph
 - [Semantic Extract Reference](../reference/semantic_extract) — full API for all extractor classes, providers, and validators
@@ -753,8 +753,8 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:

 ## Related Guides

- [Ontology Management](ontology) — generate the OWL ontology that SHACL shapes are derived from
- [Reasoning & Rules](reasoning) — complement SHACL structural constraints with logical inference rules
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `run_shacl_validation` input
+- [Ontology Management](/guides/ontology) — generate the OWL ontology that SHACL shapes are derived from
+- [Reasoning & Rules](/guides/reasoning) — complement SHACL structural constraints with logical inference rules
+- [Export & Serialization](/guides/export) — serialize graph data to Turtle/RDF/XML for `run_shacl_validation` input
 - [Conflict Resolution](/guides/conflict-resolution) — detect and resolve data conflicts before SHACL validation
 - [Change Management](/guides/change-management) — version-gate SHACL shapes alongside ontology versions
@@ -615,7 +615,7 @@ fig.write_html("out.html")   # manual export
 ## Related Guides

 - [Context Graphs](/guides/context-graphs) — `graph.to_dict()` is the primary input for `KGVisualizer`
- [Ontology Management](ontology) — `OntologyVisualizer` renders ontologies produced by `OntologyGenerator`
+- [Ontology Management](/guides/ontology) — `OntologyVisualizer` renders ontologies produced by `OntologyGenerator`
 - [Change Management](/guides/change-management) — `TemporalVersionManager` snapshots feed `visualize_metrics_evolution()` and `visualize_snapshot_comparison()`
 - [Graph Analytics](/guides/graph-analytics) — centrality scores, community dicts, and connectivity results that feed the `AnalyticsVisualizer`
- [Export & Serialization](export) — export the same graph to GraphML, GEXF, or DOT for Gephi and Graphviz
+- [Export & Serialization](/guides/export) — export the same graph to GraphML, GEXF, or DOT for Gephi and Graphviz
@@ -0,0 +1,342 @@
+---
+title: "Amazon Redshift Integration"
+description: "Ingest structured data from Amazon Redshift tables and queries into Semantica's KG pipeline."
+icon: "database"
+---
+
+> Extract data from Amazon Redshift into Semantica with password/native or IAM-role authentication, using the PostgreSQL-compatible wire protocol.
+
+
+## Installation
+
+```bash
+# Install with Redshift support
+pip install "semantica[db-redshift]"
+
+# Or install the connector separately
+pip install redshift-connector>=2.0.0
+```
+
+`redshift-connector` is an optional dependency. A plain `pip install semantica` never pulls it in, and `import semantica.ingest` never loads it eagerly — the SDK is imported only when you first use `RedshiftConnector` or `RedshiftIngestor`.
+
+<Note>
+This connector uses the Redshift database wire protocol for read ingestion. COPY, UNLOAD, S3 integration, Spectrum external tables, and the Redshift Data API are outside the current scope of this integration.
+</Note>
+
+
+## Basic Usage
+
+```python
+from semantica.ingest import RedshiftIngestor
+import os
+
+ingestor = RedshiftIngestor(
+    host=os.getenv("REDSHIFT_HOST"),
+    database=os.getenv("REDSHIFT_DATABASE"),
+    user=os.getenv("REDSHIFT_USER"),
+    password=os.getenv("REDSHIFT_PASSWORD"),
+)
+
+data = ingestor.ingest_table("customers")
+print(f"Retrieved {data.row_count} rows — columns: {data.columns}")
+```
+
+<Tip>
+Use environment variables (or a `.env` file with `python-dotenv`) to keep credentials out of source code. `RedshiftIngestor()` with no arguments reads from `REDSHIFT_*` environment variables automatically.
+</Tip>
+
+
+## Authentication
+
+<Tabs>
+  <Tab title="Password / Native">
+    The standard Redshift database username and password:
+
+    ```python
+    import os
+    from semantica.ingest import RedshiftIngestor
+
+    ingestor = RedshiftIngestor(
+        host=os.getenv("REDSHIFT_HOST"),       # e.g. cluster.abc.us-east-1.redshift.amazonaws.com
+        database=os.getenv("REDSHIFT_DATABASE"),
+        user=os.getenv("REDSHIFT_USER"),
+        password=os.getenv("REDSHIFT_PASSWORD"),
+        port=5439,    # default; omit to use the default
+        ssl=True,     # default
+    )
+    ```
+
+    Required environment variables:
+
+    ```bash
+    export REDSHIFT_HOST="cluster.abc.us-east-1.redshift.amazonaws.com"
+    export REDSHIFT_DATABASE="dev"
+    export REDSHIFT_USER="awsuser"
+    export REDSHIFT_PASSWORD="your-password"
+    ```
+  </Tab>
+  <Tab title="IAM Role (Recommended for AWS)">
+    Use `iam=True` to obtain temporary credentials via
+    `GetClusterCredentials`. The connector delegates credential resolution
+    entirely to `redshift-connector` / boto3 — Semantica never calls AWS
+    APIs directly.
+
+    **Profile-based** (reads `~/.aws/credentials`):
+
+    ```python
+    import os
+    from semantica.ingest import RedshiftIngestor
+
+    ingestor = RedshiftIngestor(
+        host=os.getenv("REDSHIFT_HOST"),
+        database=os.getenv("REDSHIFT_DATABASE"),
+        iam=True,
+        db_user=os.getenv("REDSHIFT_DB_USER"),
+        cluster_identifier=os.getenv("REDSHIFT_CLUSTER_IDENTIFIER"),
+        profile="default",   # AWS credentials-file profile
+    )
+    ```
+
+    **Explicit AWS credentials** (e.g. for CI/CD or IAM roles with short-lived keys):
+
+    ```python
+    ingestor = RedshiftIngestor(
+        host=os.getenv("REDSHIFT_HOST"),
+        database=os.getenv("REDSHIFT_DATABASE"),
+        iam=True,
+        db_user=os.getenv("REDSHIFT_DB_USER"),
+        cluster_identifier=os.getenv("REDSHIFT_CLUSTER_IDENTIFIER"),
+        region=os.getenv("REDSHIFT_REGION"),
+        access_key_id=os.getenv("REDSHIFT_ACCESS_KEY_ID"),
+        secret_access_key=os.getenv("REDSHIFT_SECRET_ACCESS_KEY"),
+        session_token=os.getenv("REDSHIFT_SESSION_TOKEN"),  # only for temporary creds
+    )
+    ```
+
+    When neither `profile` nor explicit keys are supplied, `redshift-connector`
+    falls back to the standard AWS credential chain: `AWS_*` environment
+    variables, instance-profile metadata, etc.
+
+    Required for IAM mode: `host`, `database`, `db_user`, `cluster_identifier`.
+    The `region` parameter is optional when it can be inferred from the
+    credential chain or the cluster endpoint.
+  </Tab>
+</Tabs>
+
+
+## Environment Variables
+
+All constructor parameters have `REDSHIFT_*` environment-variable fallbacks.
+Explicit constructor values always take precedence.
+
+| Variable | Parameter | Default |
+|---|---|---|
+| `REDSHIFT_HOST` | `host` | — |
+| `REDSHIFT_DATABASE` | `database` | — |
+| `REDSHIFT_USER` | `user` | — |
+| `REDSHIFT_PASSWORD` | `password` | — |
+| `REDSHIFT_PORT` | `port` | `5439` |
+| `REDSHIFT_SCHEMA` | `schema` | `"public"` |
+| `REDSHIFT_DB_USER` | `db_user` | — |
+| `REDSHIFT_CLUSTER_IDENTIFIER` | `cluster_identifier` | — |
+| `REDSHIFT_REGION` | `region` | — |
+| `REDSHIFT_PROFILE` | `profile` | — |
+| `REDSHIFT_ACCESS_KEY_ID` | `access_key_id` | — |
+| `REDSHIFT_SECRET_ACCESS_KEY` | `secret_access_key` | — |
+| `REDSHIFT_SESSION_TOKEN` | `session_token` | — |
+
+
+## Querying
+
+### Ingest a table
+
+```python
+data = ingestor.ingest_table("orders")
+print(f"{data.row_count} rows, columns: {data.columns}")
+```
+
+### Schema, filters, and pagination
+
+```python
+data = ingestor.ingest_table(
+    "orders",
+    schema="sales",          # defaults to the ingestor's schema attribute ("public")
+    database="analytics",    # defaults to the connector's database
+    where="status = 'shipped' AND total > 100",
+    order_by="created_at DESC",
+    limit=5000,
+    offset=0,
+)
+```
+
+<Warning>
+`where` and `order_by` accept raw SQL fragments and must be trusted, operator-controlled input. Do not pass raw end-user strings here. They are validated against a blocklist that rejects statement separators, `UNION`, DML/DDL keywords, and time-based injection patterns, but this is not a full parser.
+</Warning>
+
+### Custom SQL
+
+```python
+data = ingestor.ingest_query("""
+    SELECT customer_id, SUM(total) AS lifetime_value
+    FROM sales.orders
+    WHERE status = 'completed'
+    GROUP BY customer_id
+    ORDER BY lifetime_value DESC
+    LIMIT 1000
+""")
+print(f"{data.row_count} rows")
+```
+
+### Parameterized queries
+
+Use `%s` placeholders (DB-API 2.0 `format` paramstyle, which is the default for `redshift-connector`):
+
+```python
+data = ingestor.ingest_query(
+    "SELECT id, name FROM users WHERE region = %s AND active = %s",
+    params=("us-east-1", True),
+)
+```
+
+### Batch fetching for large result sets
+
+Use `batch_size` to control the driver fetch size — rows are fetched from
+the server in chunks of that size rather than all at once, and each chunk is
+converted immediately before the next is requested:
+
+```python
+data = ingestor.ingest_query(
+    "SELECT * FROM large_events_table",
+    batch_size=10000,
+)
+```
+
+`batch_size` controls how many rows the driver reads from Redshift per
+round-trip. The returned `RedshiftData.data` list still contains all matching
+rows; use `LIMIT`/`OFFSET` in the query itself if you need to cap the total
+result size.
+
+
+## Schema Discovery
+
+### List base tables in a schema
+
+```python
+tables = ingestor.list_tables(schema="public")
+print(tables)  # ["customers", "orders", "products", ...]
+```
+
+Views are excluded; only base tables are returned.
+
+### Inspect column metadata
+
+```python
+schema = ingestor.get_table_schema("customers", schema="public")
+
+for col in schema["columns"]:
+    print(f"{col['name']}: {col['type']} (nullable={col['nullable']})")
+
+print("Primary keys:", schema["primary_keys"])
+```
+
+Each column dict contains:
+
+| Key | Type | Description |
+|---|---|---|
+| `name` | `str` | Column name |
+| `type` | `str` | Redshift data type (e.g. `"integer"`, `"character varying"`) |
+| `nullable` | `bool` | Whether the column accepts `NULL` |
+
+`primary_keys` is a list of column-name strings (empty list when no primary key is defined).
+
+
+## Export as Semantica Documents
+
+Convert ingested rows to the document format that `GraphBuilder` consumes:
+
+```python
+documents = ingestor.export_as_documents(
+    data,
+    id_field="customer_id",        # column used as document ID; defaults to "id"
+    text_fields=["name", "notes"], # columns joined as document text; omit to auto-select
+)
+
+print(f"Created {len(documents)} documents")
+# Each document:
+# {
+#   "id": "12345",
+#   "text": "Alice Acme customer notes here",
+#   "metadata": {
+#     "source": "redshift",
+#     "table": "customers",
+#     "database": "analytics",
+#     "schema": "public",
+#     "row_data": { ... full cleaned row ... }
+#   }
+# }
+```
+
+**ID resolution**: `str(row.get(id_field, row_index))` — the integer row index is used as a deterministic fallback when the `id_field` column is absent.
+
+**Text when `text_fields` is provided**: each non-`None` field value is converted to a string and joined with a single space.
+
+**Text when `text_fields=None`**: only columns whose values are already `str` type are joined. Integer, float, boolean, and `None` values are excluded, matching the Snowflake and Databricks connector behavior.
+
+Pass the documents directly to `GraphBuilder`:
+
+```python
+from semantica.kg import GraphBuilder
+
+builder = GraphBuilder()
+graph = builder.build(documents)
+print(f"Entities: {graph['metadata']['num_entities']}")
+```
+
+
+## Context Manager
+
+Prefer the context manager for jobs that run multiple queries — it opens one connection on entry and closes it on exit, so every call inside the `with` block reuses the same authenticated session:
+
+```python
+with RedshiftIngestor(
+    host=os.getenv("REDSHIFT_HOST"),
+    database=os.getenv("REDSHIFT_DATABASE"),
+    user=os.getenv("REDSHIFT_USER"),
+    password=os.getenv("REDSHIFT_PASSWORD"),
+) as ingestor:
+    customers = ingestor.ingest_table("customers", limit=50000)
+    orders    = ingestor.ingest_table("orders",    limit=50000)
+    schema    = ingestor.get_table_schema("customers")
+    tables    = ingestor.list_tables()
+# Connection closed automatically on exit, even if an exception is raised.
+```
+
+Standalone calls (without `with`) open and close a transient connection per call.
+
+
+## Connection Test
+
+```python
+from semantica.ingest import RedshiftConnector
+
+connector = RedshiftConnector(
+    host="cluster.abc.us-east-1.redshift.amazonaws.com",
+    database="dev",
+    user="awsuser",
+    password="your-password",
+)
+if connector.test_connection():
+    print("Connection OK")
+else:
+    print("Connection failed — check host, credentials, and network access")
+```
+
+
+## See Also
+
+- [Ingest Module](../reference/ingest) — Full `RedshiftIngestor` reference and all other ingestors.
+- [Snowflake Integration](/integrations/snowflake) — SQL data warehouse connector with similar table/query ingestion.
+- [Databricks Integration](/integrations/databricks) — Delta Lake / Unity Catalog connector.
+- [Pipeline](../reference/pipeline) — Use Redshift ingestion as a pipeline step.
+- [Installation](../installation) — All optional dependency extras.
+- [Knowledge Graph](../reference/kg) — Build a knowledge graph from ingested Redshift data.
@@ -350,7 +350,7 @@ for record in history:
  </Accordion>
 </AccordionGroup>

- [Provenance](provenance) — W3C PROV-O lineage tracking.
+- [Provenance](/reference/provenance) — W3C PROV-O lineage tracking.
 - [Knowledge Graph](/reference/kg) — The graph being versioned.
- [Export](export) — Export versioned snapshots.
+- [Export](/reference/export) — Export versioned snapshots.
 - [Conflicts](/reference/conflicts) — Detect conflicts introduced between versions.
@@ -317,7 +317,7 @@ chain = tracker.get_traceability_chain("apple_inc")
 </Warning>

 <Tip>
-  **Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain.
+  **Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](/reference/provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain.
 </Tip>

 ## ConflictAnalyzer
@@ -450,7 +450,7 @@ class InvestigationStep:
  </Accordion>
 </AccordionGroup>

- [Deduplication](deduplication) — Resolve duplicate entities before conflict detection.
- [Ontology](ontology) — Logical conflicts use SHACL shapes and ontology axioms.
- [Provenance](provenance) — Track which source each conflicting fact came from.
+- [Deduplication](/reference/deduplication) — Resolve duplicate entities before conflict detection.
+- [Ontology](/reference/ontology) — Logical conflicts use SHACL shapes and ontology axioms.
+- [Provenance](/reference/provenance) — Track which source each conflicting fact came from.
 - [Knowledge Graph](/reference/kg) — The graph being checked for conflicts.
@@ -226,7 +226,7 @@ result = build_knowledge_base(sources=["doc.pdf"], method="fast")
  Use `Semantica` and `LifecycleManager` only when building a long-running application (e.g. a FastAPI service) that needs ordered startup, health checks, and graceful shutdown. For scripts and notebooks, use individual modules directly.
 </Tip>

- [Pipeline](pipeline) — Pipeline execution and step orchestration.
+- [Pipeline](/reference/pipeline) — Pipeline execution and step orchestration.
 - [Utils](/reference/utils) — Shared utilities used by Core internally.
 - [Getting Started](../getting-started) — Learn the basics before using Core.
 - [LLMs](/reference/llms) — Configure LLM providers via ConfigManager.
@@ -440,4 +440,4 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name")
 - [Conflicts](/reference/conflicts) — Detect value conflicts between non-duplicate entities.
 - [Knowledge Graph](/reference/kg) — GraphBuilder uses deduplication during construction.
 - [Normalize](/reference/normalize) — Normalize entity names before deduplication.
- [Provenance](provenance) — Track merged entity lineage.
+- [Provenance](/reference/provenance) — Track merged entity lineage.
@@ -609,5 +609,5 @@ The Knowledge Explorer embeds Distance Intelligence directly in the browser dash

 - [Context Module](/reference/context) — `ContextGraph.get_neighbors()` and proximity-blended retrieval.
 - [Knowledge Graph Module](/reference/kg) — `NodeEmbedder`, `SimilarityCalculator`, and graph analytics.
- [Visualization](visualization) — Programmatic distance heatmaps and ego-mode graph renders.
+- [Visualization](/reference/visualization) — Programmatic distance heatmaps and ego-mode graph renders.
 - [Explorer](/reference/explorer) — Knowledge Explorer with built-in Distance Intelligence dashboard.
@@ -404,6 +404,6 @@ Semantic neighborhood requires node embeddings stored in node properties (keys `
 Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot before shutting down.

 - [Context](/reference/context) — Build and save the ContextGraph that Explorer loads.
- [Ontology](ontology) — Programmatic ontology management and SHACL generation.
- [Visualization](visualization) — Programmatic graph rendering without the Explorer server.
- [Export](export) — Export to RDF, Parquet, and other formats without launching a server.
+- [Ontology](/reference/ontology) — Programmatic ontology management and SHACL generation.
+- [Visualization](/reference/visualization) — Programmatic graph rendering without the Explorer server.
+- [Export](/reference/export) — Export to RDF, Parquet, and other formats without launching a server.
@@ -395,6 +395,6 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p
 </Tip>

 - [Triplet Store](/reference/triplet_store) — Store RDF exports in a SPARQL-queryable backend.
- [Ontology](ontology) — Export OWL ontologies.
- [Provenance](provenance) — Include provenance metadata in RDF exports.
- [Pipeline](pipeline) — Add export as a final pipeline step.
+- [Ontology](/reference/ontology) — Export OWL ontologies.
+- [Provenance](/reference/provenance) — Include provenance metadata in RDF exports.
+- [Pipeline](/reference/pipeline) — Add export as a final pipeline step.
@@ -505,5 +505,5 @@ stats = store.get_stats()

 - [KG Module](/reference/kg) — Build the graph before persisting it.
 - [Triplet Store](/reference/triplet_store) — RDF triple store for semantic web and SPARQL queries.
- [Visualization](visualization) — Visualize graphs stored in any backend.
+- [Visualization](/reference/visualization) — Visualize graphs stored in any backend.
 - [Context](/reference/context) — AgentContext uses GraphStore for memory retrieval.
@@ -647,7 +647,7 @@ result = ingest_file("source_path", method="my_format")
 ```

 - [Parse](/reference/parse) — Parse raw sources into structured text and tables.
- [Pipeline](pipeline) — Orchestrate ingest as the first pipeline step.
+- [Pipeline](/reference/pipeline) — Orchestrate ingest as the first pipeline step.
 - [Snowflake Integration](../integrations/snowflake) — Snowflake-specific setup and authentication guide.
 - [Databricks Integration](../integrations/databricks) — Databricks Unity Catalog setup, authentication, and lineage guide.
- [Provenance](provenance) — Track lineage from ingest through to inference.
+- [Provenance](/reference/provenance) — Track lineage from ingest through to inference.
@@ -477,7 +477,7 @@ kg:

 - [Graph Store](/reference/graph_store) — Persist graphs in Neo4j, FalkorDB, or Apache AGE.
 - [Semantic Extract](/reference/semantic_extract) — Source of entities and relationships fed to GraphBuilder.
- [Visualization](visualization) — Visualize knowledge graphs interactively.
+- [Visualization](/reference/visualization) — Visualize knowledge graphs interactively.
 - [Conflicts](/reference/conflicts) — Conflict detection and resolution.

 ### Cookbooks
@@ -27,7 +27,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
 | `HuggingFaceLLM` | Local HuggingFace Transformers | None (local) |

 <Tip>
-  **Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, Cohere, and 90+ others** are all available via `LiteLLM` using their model-string prefix. See the [LiteLLM section](#litellm-100-providers) below.
+  **Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, Cohere, and 90+ others** are all available via `LiteLLM` using their model-string prefix. See the [LiteLLM section](#litellm-100+-providers) below.
 </Tip>

 ## What You Get
@@ -441,5 +441,5 @@ extractor = NERExtractor(

 - [Semantic Extract](/reference/semantic_extract) — Use LLMs for NER and relation extraction.
 - [Agno Integration](../integrations/agno) — LLM providers in Agno multi-agent teams.
- [Reasoning](reasoning) — LLM-backed deductive and abductive reasoning.
+- [Reasoning](/reference/reasoning) — LLM-backed deductive and abductive reasoning.
 - [Context](/reference/context) — GraphRAG uses LLMs for reasoning over knowledge graphs.
@@ -495,5 +495,5 @@ The MCP server exposes three readable resources:

 - [Context](/reference/context) — The ContextGraph that the MCP server operates on.
 - [Semantic Extract](/reference/semantic_extract) — NER and relation extraction powering the MCP tools.
- [Reasoning](reasoning) — Forward-chaining engine behind run_reasoning.
+- [Reasoning](/reference/reasoning) — Forward-chaining engine behind run_reasoning.
 - [Agno Integration](../integrations/agno) — Use Semantica inside Agno multi-agent teams.
@@ -586,5 +586,5 @@ normalized = normalize_text("Apple Inc.", method="expand_suffixes")

 - [Parse](/reference/parse) — Parse documents before normalization.
 - [Split](/reference/split) — Chunk normalized text for embedding.
- [Deduplication](deduplication) — Resolve duplicate entities after normalization.
- [Pipeline](pipeline) — Include normalization as a named pipeline step.
+- [Deduplication](/reference/deduplication) — Resolve duplicate entities after normalization.
+- [Pipeline](/reference/pipeline) — Include normalization as a named pipeline step.
@@ -316,7 +316,7 @@ ontology_data = ingest_ontology("schema.jsonld")  # JSON-LD
  Ontology versioning (`VersionManager`, `OntologyVersion`) has moved to `semantica.change_management`. Import from there: `from semantica.change_management import VersionManager`.
 </Note>

- [Reasoning](reasoning) — Apply inference rules over ontology axioms.
+- [Reasoning](/reference/reasoning) — Apply inference rules over ontology axioms.
 - [Knowledge Graph](/reference/kg) — The graph being modeled by the ontology.
- [Export](export) — Export ontologies as RDF, OWL, or JSON-LD.
+- [Export](/reference/export) — Export ontologies as RDF, OWL, or JSON-LD.
 - [Conflicts](/reference/conflicts) — Detect ontology constraint violations.
@@ -297,7 +297,7 @@ for source in sources:
  Docling is an optional dependency. If `docling` is not installed, `DoclingParser` raises an `ImportError` with installation instructions: `pip install docling`. `DocumentParser` is always available and requires no extras.
 </Note>

- [Ingest](ingest) — Load files before parsing.
+- [Ingest](/reference/ingest) — Load files before parsing.
 - [Split](/reference/split) — Chunk parsed text for embedding and extraction.
 - [Docling Integration](../integrations/docling) — Full Docling integration setup guide.
 - [Semantic Extract](/reference/semantic_extract) — Extract entities and relations from parsed text.
@@ -588,7 +588,7 @@ StepStatus.SKIPPED    # Skipped due to FailureHandler "skip" strategy
  </Accordion>
 </AccordionGroup>

- [Ingest](ingest) — First step in most pipelines.
+- [Ingest](/reference/ingest) — First step in most pipelines.
 - [Semantic Extract](/reference/semantic_extract) — Core extraction step.
 - [Knowledge Graph](/reference/kg) — Graph construction step.
- [Export](export) — Final output step.
+- [Export](/reference/export) — Final output step.
@@ -523,6 +523,6 @@ Provenance tracking in Semantica produces the following audit artifacts:
 </Note>

 - [Change Management](/reference/change_management) — Version control and snapshot audit trails.
- [Ingest](ingest) — Provenance begins at the ingestion stage.
- [Export](export) — Include provenance metadata in RDF exports.
+- [Ingest](/reference/ingest) — Provenance begins at the ingestion stage.
+- [Export](/reference/export) — Include provenance metadata in RDF exports.
 - [Context](/reference/context) — Decision provenance via AgentContext.
@@ -31,7 +31,7 @@ icon: "microchip"

 ## Which Engine Should I Use?

- [Reasoner](#reasoner-forwardbackward-chaining) — IF/THEN rules, forward and backward chaining. **Start here**: covers 90% of use cases. No query language required.
+- [Reasoner](#reasoner-forward/backward-chaining) — IF/THEN rules, forward and backward chaining. **Start here**: covers 90% of use cases. No query language required.
 - [GraphReasoner](#graphreasoner) — Natural language queries over a knowledge graph via LLM. No SPARQL or rules: just ask a question.
 - [DatalogReasoner](#datalogreasoner) — Recursive Horn clause rules with guaranteed termination. Use for complex multi-hop transitive rules.
 - [ReteEngine](#reteengine) — Rete pattern matching for high-frequency inference. Use when you need to match many facts against many rules simultaneously.
@@ -483,6 +483,6 @@ step.confidence     # float
 </Warning>

 - [Knowledge Graph](/reference/kg) — The knowledge graph being reasoned over.
- [Ontology](ontology) — Ontology axioms and SHACL constraints for logical reasoning.
+- [Ontology](/reference/ontology) — Ontology axioms and SHACL constraints for logical reasoning.
 - [Triplet Store](/reference/triplet_store) — RDF backend for SPARQL-based reasoning.
 - [Context](/reference/context) — Reasoning integrated into agent decision intelligence.
@@ -321,7 +321,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first
  **Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment.
 </Tip>

- [Ingest](ingest) — Load unstructured data alongside seed data.
+- [Ingest](/reference/ingest) — Load unstructured data alongside seed data.
 - [Knowledge Graph](/reference/kg) — The target graph that seed data populates.
- [Deduplication](deduplication) — Handle duplicates during seed-extracted merge.
- [Pipeline](pipeline) — Incorporate seed loading as a named pipeline step.
+- [Deduplication](/reference/deduplication) — Handle duplicates during seed-extracted merge.
+- [Pipeline](/reference/pipeline) — Incorporate seed loading as a named pipeline step.
@@ -448,4 +448,4 @@ triplets      = trip.extract(text)
 - [LLM Providers](/reference/llms) — Configure which LLM is used for extraction.
 - [Knowledge Graph](/reference/kg) — Build graphs from extracted entities and relationships.
 - [Parse Module](/reference/parse) — Parse documents before extraction.
- [Deduplication](deduplication) — Resolve duplicate entities after extraction.
+- [Deduplication](/reference/deduplication) — Resolve duplicate entities after extraction.
@@ -371,9 +371,9 @@ for chunk in chunks:
    print(f"  {len(entities)} entities in chunk starting at {chunk.start_index}")
 ```

-For the full pipeline orchestration API, see the [Pipeline reference](pipeline).
+For the full pipeline orchestration API, see the [Pipeline reference](/reference/pipeline).

 - [Parse](/reference/parse) — Parse documents before chunking: produces sections and metadata.
 - [Embeddings](/reference/embeddings) — Embed chunks for vector search and semantic chunking.
 - [Semantic Extract](/reference/semantic_extract) — Extract entities and relations from individual chunks.
- [Pipeline](pipeline) — Integrate splitting as a named pipeline step.
+- [Pipeline](/reference/pipeline) — Integrate splitting as a named pipeline step.
@@ -876,8 +876,8 @@ kg:

 - [Knowledge Graph Module](/reference/kg) — Core graph construction, `GraphBuilder`, analytics.
 - [Context Module](/reference/context) — Decision temporal windows and `find_active_nodes()`.
- [Provenance](provenance) — W3C PROV-O lineage stamped alongside temporal metadata.
- [Export](export) — OWL, Turtle, JSON-LD, and Parquet export with temporal annotations.
+- [Provenance](/reference/provenance) — W3C PROV-O lineage stamped alongside temporal metadata.
+- [Export](/reference/export) — OWL, Turtle, JSON-LD, and Parquet export with temporal annotations.

 - [Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb) — Temporal reasoning and Allen algebra · Advanced
 - [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb) — Including temporal decision windows · Intermediate
@@ -561,7 +561,7 @@ for row in result.bindings:
    print(row)
 ```

- [Export](export) — Export knowledge graphs to RDF formats.
- [Ontology](ontology) — Load OWL ontologies and store as RDF triples.
- [Reasoning](reasoning) — SPARQL-based property chain inference.
+- [Export](/reference/export) — Export knowledge graphs to RDF formats.
+- [Ontology](/reference/ontology) — Load OWL ontologies and store as RDF triples.
+- [Reasoning](/reference/reasoning) — SPARQL-based property chain inference.
 - [Graph Store](/reference/graph_store) — Property graph alternative for Cypher queries.
@@ -223,4 +223,4 @@ config = read_json_file("config.json")
 ```

 - [Core](/reference/core) — Framework orchestration that uses Utils internally.
- [Pipeline](pipeline) — Uses ProgressTracker for per-step tracking.
+- [Pipeline](/reference/pipeline) — Uses ProgressTracker for per-step tracking.
@@ -591,4 +591,4 @@ store.create_index(index_type="pq", metric="L2", m=8)
 - [Embeddings](/reference/embeddings) — Generate the vectors stored here.
 - [Context](/reference/context) — AgentContext uses VectorStore for memory retrieval.
 - [Split](/reference/split) — Chunk documents before embedding and storing.
- [Ingest](ingest) — Ingest documents before embedding and storing.
+- [Ingest](/reference/ingest) — Ingest documents before embedding and storing.
@@ -291,6 +291,6 @@ semantica-explorer --graph my_graph.json
 See the [Explorer reference](/reference/explorer) for the full feature set and REST API.

 - [Knowledge Graph](/reference/kg) — The graph being visualized.
- [Ontology](ontology) — Visualize ontology class structure.
+- [Ontology](/reference/ontology) — Visualize ontology class structure.
 - [Embeddings](/reference/embeddings) — Generate the embeddings visualized here.
 - [Explorer](/reference/explorer) — Full interactive Knowledge Explorer UI.
@@ -236,6 +236,8 @@ def _() -> list[str]:
            cwd=DOCS,
            capture_output=True,
            text=True,
+            encoding="utf-8",
+            errors="replace",
            timeout=600,
        )
        # Clean up zip regardless of outcome
@@ -158,6 +158,7 @@ db-arrow = [
  "pyarrow>=14.0.0,<24.0.0; python_version < '3.10'"
 ]
 db-salesforce = ["simple-salesforce>=1.12.0"]
+db-redshift = ["redshift-connector>=2.0.0"]
 ingest-parquet = [
  "pyarrow>=24.0.0; python_version >= '3.10'",
  "pyarrow>=14.0.0,<24.0.0; python_version < '3.10'"
@@ -170,7 +171,7 @@ ingest-sap = ["requests>=2.28.0"]
 ingest-git = ["GitPython>=3.1.58"]

 db-all = [
-  "semantica[db-snowflake,db-databricks,db-salesforce,db-arrow]"
+  "semantica[db-snowflake,db-databricks,db-salesforce,db-redshift,db-arrow]"
 ]

 # ---- Embedding / Models ----
@@ -2737,15 +2737,15 @@ librt==0.15.0 \
    --hash=sha256:fc1ed11c4ad0b91af24def2050f2840ea4567828e3dd058fbe608d982f6e5465 \
    --hash=sha256:febb1ce6cac545a54e6b769982824e955a700fdd9fbf3a08a3d82c990968b57d
    # via mypy
-litellm==1.99.0 \
-    --hash=sha256:1c45097e426fed2ae7fbd38b5404c3addeb203d0e1148c0a59848aabd5fe83c6 \
-    --hash=sha256:5617804e838499bce8fecb41ad9bc984b7977361e557666fed0fef0c4623ce62 \
-    --hash=sha256:594bf4b6ff6b79c6aa3c3b78c0e939d4afd12687e076ba3fb608d38a5aa7f9c6 \
-    --hash=sha256:71109c323164b4b6776ff259876523e6e883a465aa1413dd51a1bda8e92efc5f \
-    --hash=sha256:a43e8716da8beed04480e91b4233ff2f1ab1fedd84cad332dbc526b54a9229ca \
-    --hash=sha256:e2b383070656fdbec4bc44602edaaed2a21e99ceee4ea0a4650c8cb381e67b59 \
-    --hash=sha256:e42f94731665b68e263481efd79f7629e9b97eed7e10c57dc37a890eca058227 \
-    --hash=sha256:e461b7ce53af990e5287cf7ae30d82c956b56dcce886f63ef39a76e678f82a3b
+litellm==1.100.0 \
+    --hash=sha256:098a413e398e2220734cf9c8dd75fb34a38b65b64090619c2869af1fdeaf4ae5 \
+    --hash=sha256:0b7ec93013e18535481cd811b776ee95c6b957b3a9fb44dc53f7c459e7e60e38 \
+    --hash=sha256:0f87fae695edbca27e5cf970bea52fa405fa9910fdf7c29c963d6413db767299 \
+    --hash=sha256:8224c8eed9cab3319a88e6665d1275ad8faf21d353b1b22223a6d6115a302ea2 \
+    --hash=sha256:a07370d116905485e9ac99679bac991a0a6c81e13c99ca3f007128fbdf2b0082 \
+    --hash=sha256:c6f2f56808d05d8d2a7766129d958101eafb1a47e30ba5ddcfa900cdbc50af67 \
+    --hash=sha256:e3787fb7ad1f20aebdde7686a85061880bba6550c9d62bfa1cf8df089fe7899b \
+    --hash=sha256:ece94e817a453a5b3a9517c03547c428d501cea719edb728c1b260e53f78ea35
    # via semantica (pyproject.toml)
 llvmlite==0.49.0 \
    --hash=sha256:00f16db782f4a13c78c5804aedc434e46794a77e89999a168f9401106270e50a \
@@ -152,9 +152,9 @@ class MemoryItem:
        """Reconstruct a MemoryItem from a serialised dict."""
        raw_ts = data.get("timestamp")
        try:
-            ts = datetime.fromisoformat(raw_ts) if raw_ts else datetime.utcnow()
+            ts = datetime.fromisoformat(raw_ts) if raw_ts else datetime.now(timezone.utc)
        except (ValueError, TypeError):
-            ts = datetime.utcnow()
+            ts = datetime.now(timezone.utc)
        return cls(
            content=data.get("content", ""),
            timestamp=ts,
@@ -355,7 +355,14 @@ class AgentMemory:

        try:
            memory_id = options.get("memory_id") or self._generate_memory_id()
-            timestamp = options.get("timestamp") or datetime.now()
+            # Aware UTC, deliberately: _timestamp_comparison_key interprets a
+            # naive stamp as LOCAL time, so a naive local "now" here and a
+            # naive utcnow() elsewhere differ by the host's UTC offset — on a
+            # UTC+8 host that pushed freshly added memories 8 hours outside
+            # every start_date/end_date window. One aware producer removes
+            # the ambiguity; legacy naive stamps keep their local-time
+            # meaning through the comparison key.
+            timestamp = options.get("timestamp") or datetime.now(timezone.utc)

            if memory_id in self.memory_items:
                replacement_options = dict(options)
@@ -1073,7 +1080,7 @@ class AgentMemory:
        else:
            days = 30

-        cutoff_date = datetime.now() - timedelta(days=days)
+        cutoff_date = datetime.now(timezone.utc) - timedelta(days=days)

        # Delete old items
        memory_ids_to_delete = []
@@ -247,6 +247,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
    "SalesforceIngestor": (".salesforce_ingestor", "SalesforceIngestor"),
    "SalesforceData": (".salesforce_ingestor", "SalesforceData"),
    "SalesforceConnector": (".salesforce_ingestor", "SalesforceConnector"),
+    # Redshift ingestion
+    "RedshiftIngestor": (".redshift_ingestor", "RedshiftIngestor"),
+    "RedshiftData": (".redshift_ingestor", "RedshiftData"),
+    "RedshiftConnector": (".redshift_ingestor", "RedshiftConnector"),
 }

 _OPTIONAL_DEPENDENCY_MESSAGES = {
@@ -289,6 +293,10 @@ _OPTIONAL_DEPENDENCY_MESSAGES = {
        "Salesforce ingestion requires optional dependency 'simple-salesforce'. "
        "Install it with: pip install 'semantica[db-salesforce]'"
    ),
+    ".redshift_ingestor": (
+        "Redshift ingestion requires optional dependency 'redshift-connector'. "
+        "Install it with: pip install 'semantica[db-redshift]'"
+    ),
 }


@@ -307,7 +315,14 @@ def __getattr__(name: str) -> Any:
            missing_name is None
            or any(
                pkg in missing_name
-                for pkg in ("git", "bs4", "pyarrow", "simple_salesforce", "lxml")
+                for pkg in (
+                    "git",
+                    "bs4",
+                    "pyarrow",
+                    "simple_salesforce",
+                    "lxml",
+                    "redshift_connector",
+                )
            )
        ):
            raise ImportError(message) from exc
@@ -349,6 +364,15 @@ def __getattr__(name: str) -> Any:
            if message:
                raise ImportError(message)

+    if module_name == ".redshift_ingestor" and name in {
+        "RedshiftIngestor",
+        "RedshiftConnector",
+    }:
+        if not getattr(module, "REDSHIFT_AVAILABLE", True):
+            message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+            if message:
+                raise ImportError(message)
+
    value = getattr(module, attr_name)
    globals()[name] = value
    return value
@@ -439,6 +463,10 @@ __all__ = [
    "SalesforceIngestor",
    "SalesforceData",
    "SalesforceConnector",
+    # Redshift ingestion
+    "RedshiftIngestor",
+    "RedshiftData",
+    "RedshiftConnector",
    # Registry and Methods
    "MethodRegistry",
    "method_registry",
@@ -491,8 +491,8 @@ class DoclingParser:
                                elif hasattr(item, 'export_to_markdown'):
                                    try:
                                        page_text_parts.append(item.export_to_markdown(doc=doc))
-                                    except:
-                                        pass
+                                    except Exception:
+                                        pass  # export failure degrades to skipping this text item
                                
                                # Extract tables on this page
                                from docling_core.types.doc import TableItem
@@ -549,7 +549,7 @@ class DoclingParser:
                            "tables": [],
                            "images": [],
                        })
-                    except:
+                    except Exception:
                        pages.append({
                            "page_number": 1,
                            "text": "",
@@ -572,7 +572,7 @@ class DoclingParser:
                    "tables": [],
                    "images": [],
                })
-            except:
+            except Exception:
                pages.append({
                    "page_number": 1,
                    "text": "",
@@ -109,7 +109,9 @@ License: MIT
 import re
 import difflib
 import threading
+from collections import OrderedDict
 from concurrent.futures import ThreadPoolExecutor, as_completed
+from contextlib import contextmanager
 from typing import Any, Dict, List, Optional, Tuple, Union

 from ..utils.exceptions import ProcessingError
@@ -199,32 +201,122 @@ _embedder_cache = None
 # spacy.load() on every call. Entries record the spacy module they were loaded
 # from: tests patch `methods.spacy` with a mock, and an entry produced by a
 # different module object must not be handed back to a later caller.
-_spacy_model_cache: Dict[str, Tuple[Any, Any]] = {}
+# Bounded: each spaCy Language costs hundreds of MB, and a long-running
+# service that varies the `model` option would otherwise pin every model it
+# ever touched for the life of the process. 4 covers lg/md/sm + one custom.
+MAX_SPACY_MODELS_CACHED = 4
+_spacy_model_cache: "OrderedDict[str, Tuple[Any, Any, threading.Lock]]" = OrderedDict()
 _spacy_model_cache_lock = threading.Lock()


+def _load_spacy_entry(name: str) -> Tuple[Any, Any, threading.Lock]:
+    """Return the cache entry for ``name``: ``(spacy_module, nlp, call_lock)``.
+
+    The entire lookup-plus-update is performed under ``_spacy_model_cache_lock``
+    so that the LRU ``move_to_end`` and eviction are always consistent.
+    ``spacy.load`` is called inside the lock; that is acceptable here because
+    model loads are rare (at most ``MAX_SPACY_MODELS_CACHED`` per process) and
+    the simpler design eliminates the TOCTOU window that a lockless fast-path
+    would introduce.
+
+    Raises whatever ``spacy.load`` raises (``OSError`` for a missing model).
+    """
+    if spacy is None:
+        raise ImportError(
+            "spaCy is not installed. Install with: pip install 'semantica[nlp-spacy]'"
+        )
+    with _spacy_model_cache_lock:
+        cached = _spacy_model_cache.get(name)
+        if cached is not None and cached[0] is spacy:
+            _spacy_model_cache.move_to_end(name)
+            return cached
+        nlp = spacy.load(name)
+        entry: Tuple[Any, Any, threading.Lock] = (spacy, nlp, threading.Lock())
+        _spacy_model_cache[name] = entry
+        while len(_spacy_model_cache) > MAX_SPACY_MODELS_CACHED:
+            _spacy_model_cache.popitem(last=False)
+        return entry
+
+
 def load_spacy_model(name: str):
    """Load a spaCy model once per process, keyed by model name.

    Raises whatever ``spacy.load`` raises (``OSError`` for a missing model), so
    callers keep their existing fallback behavior.
+
+    .. warning::
+        The returned ``Language`` object is shared process-wide.  Calling it
+        from multiple threads concurrently is not safe.  Use
+        ``spacy_pipeline_guard`` (or ``run_spacy_text``) to serialize calls.
    """
-    if spacy is None:
-        raise ImportError(
-            "spaCy is not installed. Install with: pip install 'semantica[nlp-spacy]'"
+    _spacy_module, nlp, _call_lock = _load_spacy_entry(name)
+    return nlp
+
+
+@contextmanager
+def spacy_pipeline_guard(name: str):
+    """Yield the cached ``Language`` for ``name`` under its per-model call lock.
+
+    Concurrent calls on the *same* model serialize; calls on *different* models
+    run in parallel.  Raises whatever ``spacy.load`` raises (``OSError`` for a
+    missing model).
+    """
+    _spacy_module, nlp, call_lock = _load_spacy_entry(name)
+    with call_lock:
+        yield nlp
+
+
+def run_spacy_text(
+    name: str,
+    text: str,
+    *,
+    fallback: Optional[str] = None,
+    log_label: str = "spaCy",
+):
+    """Call ``nlp(text)`` under the per-model guard; try ``fallback`` on ``OSError``.
+
+    Returns the ``Doc``, or ``None`` when no usable model is available — the
+    caller's cue to take its own pattern fallback.  All ``nlp(text)`` calls go
+    through ``spacy_pipeline_guard``, serializing concurrent thread access for
+    the same model while letting different models run in parallel.
+
+    Fallback is only attempted when the *primary* model is missing (``OSError``).
+    A pipeline error (non-OSError) returns ``None`` immediately without trying
+    the fallback, because the fallback would likely hit the same error.
+    """
+    try:
+        with spacy_pipeline_guard(name) as nlp:
+            return nlp(text)
+    except OSError:
+        if not fallback:
+            logger.warning("%s model '%s' not found", log_label, name)
+            return None
+        logger.warning(
+            "%s model '%s' not found, trying fallback '%s'",
+            log_label, name, fallback,
        )
+    except Exception:
+        logger.warning(
+            "%s model '%s' raised an unexpected error; skipping.",
+            log_label, name, exc_info=True,
+        )
+        return None

-    cached = _spacy_model_cache.get(name)
-    if cached is not None and cached[0] is spacy:
-        return cached[1]
-
-    with _spacy_model_cache_lock:
-        cached = _spacy_model_cache.get(name)
-        if cached is not None and cached[0] is spacy:
-            return cached[1]
-        nlp = spacy.load(name)
-        _spacy_model_cache[name] = (spacy, nlp)
-        return nlp
+    # Reached only when primary raised OSError and fallback is set.
+    try:
+        with spacy_pipeline_guard(fallback) as nlp:
+            return nlp(text)
+    except OSError:
+        logger.warning(
+            "%s fallback model '%s' not found either",
+            log_label, fallback,
+        )
+    except Exception:
+        logger.warning(
+            "%s fallback model '%s' raised an unexpected error; skipping.",
+            log_label, fallback, exc_info=True,
+        )
+    return None


 def clear_spacy_model_cache() -> None:
@@ -282,8 +374,8 @@ def get_nlp_model():
        try:
            _nlp_cache = spacy.load("en_core_web_sm", disable=["parser", "ner", "lemmatizer"])
            return _nlp_cache
-        except:
-            pass
+        except Exception:
+            pass  # spacy.load raises OSError for a missing model
            
    except Exception as e:
        logger.warning(f"Failed to load spaCy model for similarity: {e}")
@@ -765,32 +857,11 @@ def extract_entities_ml(
        logger.warning("spaCy not available, falling back to pattern extraction")
        return extract_entities_pattern(text, **kwargs)

-    try:
-        nlp = load_spacy_model(model)
-    except OSError:
-        logger.warning(f"spaCy model {model} not found, using en_core_web_sm")
-        try:
-            nlp = load_spacy_model("en_core_web_sm")
-        except OSError:
-            logger.warning(
-                "spaCy model not available, falling back to pattern extraction"
-            )
-            return extract_entities_pattern(text, **kwargs)
-        except Exception as exc:
-            logger.warning(
-                "spaCy fallback triggered because the default model failed to initialize. Falling back to pattern extraction.",
-                exc_info=True,
-            )
-            return extract_entities_pattern(text, **kwargs)
-    except Exception as exc:
-        logger.warning(
-            "spaCy model %s failed to initialize, falling back to pattern extraction.",
-            model,
-            exc_info=True,
-        )
+    doc = run_spacy_text(
+        model, text, fallback="en_core_web_sm", log_label="spaCy NER"
+    )
+    if doc is None:
        return extract_entities_pattern(text, **kwargs)
-
-    doc = nlp(text)
    entities = []

    for ent in doc.ents:
@@ -1444,34 +1515,30 @@ def extract_relations_similarity(
        return extract_relations_cooccurrence(text, entities, **kwargs)

    relations = []
-    
-    # Try to load spaCy model with vectors
-    nlp = None
+
+    # Resolve which model to use for vectors (prefer larger models), then
+    # invoke it only under its guard: spaCy Languages are shared process-wide
+    # and not safe to call from concurrent threads.
+    chosen_model = None
    if SPACY_AVAILABLE:
-        try:
-            # Prefer larger models for vectors
-            for model_name in ["en_core_web_lg", "en_core_web_md", "en_core_web_sm"]:
-                if spacy.util.is_package(model_name):
-                    nlp = load_spacy_model(model_name)
-                    break
-            if not nlp:
-                 # Try loading what we have
-                 try:
-                     nlp = load_spacy_model("en_core_web_sm") 
-                 except:
-                     pass
-        except Exception:
-            pass
+        for model_name in ["en_core_web_lg", "en_core_web_md", "en_core_web_sm"]:
+            if spacy.util.is_package(model_name):
+                chosen_model = model_name
+                break

    # Pre-compute relation type vectors if possible
    relation_vectors = {}
    has_vectors = False
-    if nlp:
-        # Check if model has vectors
-        if nlp.vocab.vectors.shape[0] > 0:
-            has_vectors = True
-            for rt in relation_types:
-                relation_vectors[rt] = nlp(rt)
+    if chosen_model:
+        try:
+            with spacy_pipeline_guard(chosen_model) as guarded_nlp:
+                # Check if model has vectors
+                if guarded_nlp.vocab.vectors.shape[0] > 0:
+                    has_vectors = True
+                    for rt in relation_types:
+                        relation_vectors[rt] = guarded_nlp(rt)
+        except Exception:
+            pass
    
    for entity1 in entities:
        for entity2 in entities:
@@ -1501,10 +1568,12 @@ def extract_relations_similarity(
            best_type = None
            best_score = 0.0

-            if has_vectors and relation_vectors:
-                # Vector similarity
-                doc = nlp(between_text)
-                if doc.vector_norm:
+            if has_vectors and relation_vectors and chosen_model:
+                # Vector similarity — each call re-enters the model's guard
+                doc = run_spacy_text(
+                    chosen_model, between_text, log_label="spaCy similarity"
+                )
+                if doc is not None and doc.vector_norm:
                    for rt, vec in relation_vectors.items():
                        if vec.vector_norm:
                            sim = doc.similarity(vec)
@@ -1556,13 +1625,9 @@ def extract_relations_dependency(
        logger.warning("spaCy not available, falling back to pattern extraction")
        return extract_relations_pattern(text, entities, **kwargs)

-    try:
-        nlp = load_spacy_model(model)
-    except OSError:
-        logger.warning(f"spaCy model {model} not found")
+    doc = run_spacy_text(model, text, log_label="spaCy dependency")
+    if doc is None:
        return extract_relations_pattern(text, entities, **kwargs)
-
-    doc = nlp(text)
    relations = []
    
    # Map tokens to entities
@@ -514,9 +514,9 @@ class FAISSIndexBuilder:
        return FAISSIndex(index, self.dimension, index_type)

    def train_index(self, index: FAISSIndex, training_vectors: np.ndarray):
-        """Train index on sample vectors."""
-        if not isinstance(index.index, faiss.IndexIVFFlat):
-            return  # Only IVF indices need training
+        """Train IVF and PQ indexes on sample vectors."""
+        if not isinstance(index.index, (faiss.IndexIVFFlat, faiss.IndexPQ)):
+            return

        index.index.train(training_vectors.astype(np.float32))

@@ -15,27 +15,57 @@ log = logging.getLogger("semantica.mcp.tools.extraction")
 def _clear_cache() -> None:
    try:
        from semantica.semantic_extract.cache import _result_cache
+
        _result_cache.clear()
    except Exception:
        log.debug("Could not clear semantic_extract cache; continuing", exc_info=True)


+def _relation_endpoint(value: Any) -> Any:
+    """Return a stable textual value for a relation endpoint."""
+    if value is None or isinstance(value, str):
+        return value
+    return getattr(value, "text", str(value))
+
+
+def _serialize_relation(relation: Any) -> dict[str, Any]:
+    return {
+        "source": _relation_endpoint(getattr(relation, "subject", None)),
+        "type": getattr(relation, "predicate", None),
+        "target": _relation_endpoint(getattr(relation, "object", None)),
+        "confidence": getattr(relation, "confidence", None),
+    }
+
+
+def _serialize_coreference(chain: Any) -> dict[str, Any]:
+    representative = getattr(chain, "representative", None)
+    return {
+        "representative": getattr(representative, "text", ""),
+        "mentions": [
+            getattr(mention, "text", "") for mention in getattr(chain, "mentions", [])
+        ],
+        "entity_type": getattr(chain, "entity_type", None),
+    }
+
+
 def handle_extract_entities(args: dict) -> dict:
    """Extract named entities from text using Semantica NER."""
-    text = args.get("text", "").strip()
-    if not text:
+    text = args.get("text", "")
+    if not text.strip():
        return {"error": "text is required", "entities": []}
    _clear_cache()
    try:
        from semantica.semantic_extract import NamedEntityRecognizer
-        entities = NamedEntityRecognizer().extract(text) or []
+
+        entities = NamedEntityRecognizer().extract_entities(text) or []
        return {
            "entities": [
                {
-                    "label": getattr(e, "label", str(e)),
-                    "type": getattr(e, "type", None),
-                    "start": getattr(e, "start", None),
-                    "end": getattr(e, "end", None),
+                    "text": getattr(e, "text", ""),
+                    "label": getattr(e, "label", ""),
+                    "type": getattr(e, "label", ""),
+                    "start": getattr(e, "start_char", getattr(e, "start", None)),
+                    "end": getattr(e, "end_char", getattr(e, "end", None)),
                    "confidence": getattr(e, "confidence", None),
                }
                for e in entities
@@ -49,25 +79,22 @@ def handle_extract_entities(args: dict) -> dict:

 def handle_extract_relations(args: dict) -> dict:
    """Extract relations and triplets from text."""
-    text = args.get("text", "").strip()
-    if not text:
+    text = args.get("text", "")
+    if not text.strip():
        return {"error": "text is required", "relations": [], "triplets": []}
    _clear_cache()
    try:
-        from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripletExtractor
-        entities = NamedEntityRecognizer().extract(text) or []
+        from semantica.semantic_extract import (
+            NamedEntityRecognizer,
+            RelationExtractor,
+            TripletExtractor,
+        )
+
+        entities = NamedEntityRecognizer().extract_entities(text) or []
        relations = RelationExtractor().extract(text, entities) or []
        triplets = TripletExtractor().extract(text) or []
        return {
-            "relations": [
-                {
-                    "source": getattr(r, "source", None),
-                    "type": getattr(r, "type", None),
-                    "target": getattr(r, "target", None),
-                    "confidence": getattr(r, "confidence", None),
-                }
-                for r in relations
-            ],
+            "relations": [_serialize_relation(r) for r in relations],
            "triplets": [
                {
                    "subject": getattr(t, "subject", None),
@@ -86,8 +113,8 @@ def handle_extract_relations(args: dict) -> dict:

 def handle_extract_all(args: dict) -> dict:
    """Run the full extraction pipeline: NER + relations + events + triplets."""
-    text = args.get("text", "").strip()
-    if not text:
+    text = args.get("text", "")
+    if not text.strip():
        return {"error": "text is required"}
    include_events = args.get("include_events", True)
    include_triplets = args.get("include_triplets", True)
@@ -102,40 +129,48 @@ def handle_extract_all(args: dict) -> dict:
            TripletExtractor,
        )

-        entities = NamedEntityRecognizer().extract(text) or []
+        entities = NamedEntityRecognizer().extract_entities(text) or []
        result["entities"] = [
-            {"label": getattr(e, "label", str(e)), "type": getattr(e, "type", None)}
+            {
+                "text": getattr(e, "text", ""),
+                "label": getattr(e, "label", ""),
+                "type": getattr(e, "label", ""),
+            }
            for e in entities
        ]

-        resolved = CoreferenceResolver().resolve(text)
-        relations = RelationExtractor().extract(resolved, entities) or []
-        result["relations"] = [
-            {"source": getattr(r, "source", None),
-             "type": getattr(r, "type", None),
-             "target": getattr(r, "target", None)}
-            for r in relations
+        coreferences = CoreferenceResolver().resolve(text, entities=entities) or []
+        result["coreferences"] = [
+            _serialize_coreference(chain) for chain in coreferences
        ]

+        relations = RelationExtractor().extract(text, entities) or []
+        result["relations"] = [_serialize_relation(r) for r in relations]
+
        if include_events:
            events = EventDetector().extract(text) or []
            result["events"] = [
-                {"type": getattr(ev, "type", None),
-                 "trigger": getattr(ev, "trigger", str(ev))}
+                {
+                    "type": getattr(ev, "event_type", getattr(ev, "type", None)),
+                    "trigger": getattr(ev, "text", getattr(ev, "trigger", str(ev))),
+                }
                for ev in events
            ]

        if include_triplets:
-            triplets = TripletExtractor().extract(resolved) or []
+            triplets = TripletExtractor().extract(text) or []
            result["triplets"] = [
-                {"subject": getattr(t, "subject", None),
-                 "predicate": getattr(t, "predicate", None),
-                 "object": getattr(t, "object", None)}
+                {
+                    "subject": getattr(t, "subject", None),
+                    "predicate": getattr(t, "predicate", None),
+                    "object": getattr(t, "object", None),
+                }
                for t in triplets
            ]

        result["summary"] = {
            "entities": len(result.get("entities", [])),
+            "coreferences": len(result.get("coreferences", [])),
            "relations": len(result.get("relations", [])),
            "events": len(result.get("events", [])),
            "triplets": len(result.get("triplets", [])),
@@ -0,0 +1,123 @@
+"""Timestamp semantics for AgentMemory (#found-by-audit).
+
+MemoryItem timestamps had two producers with different naive conventions:
+``store()`` defaulted to ``datetime.now()`` (naive LOCAL) while ``from_dict``
+fell back to ``datetime.utcnow()`` (naive UTC). ``_timestamp_comparison_key``
+interprets every naive stamp as LOCAL time, so on any host off UTC the two
+producers disagree by the host's offset — a UTC+8 host pushed freshly added
+memories 8 hours outside every ``start_date``/``end_date`` window, and
+``cleanup_old_memories`` aged them wrong by the same amount.
+
+The contract pinned here:
+
+* every NEW stamp the module produces is timezone-aware UTC;
+* legacy naive stamps keep their documented local-time meaning through the
+  comparison key (it must not flip to interpreting naive as UTC);
+* aware stamps compare correctly against naive and aware boundaries alike,
+  on hosts in any timezone.
+"""
+
+import sys
+import os
+import unittest
+from datetime import datetime, timedelta, timezone
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+
+from semantica.context.agent_memory import AgentMemory, MemoryItem
+
+
+def host_offset() -> timedelta:
+    return datetime.now().astimezone().utcoffset() or timedelta(0)
+
+
+class TestAwareUtcProducers(unittest.TestCase):
+    def test_store_default_timestamp_is_aware_utc(self):
+        memory = AgentMemory()
+        memory.store("recall me")
+
+        item = next(iter(memory.memory_items.values()))
+        self.assertIsNotNone(item.timestamp.tzinfo, "store() must not produce naive stamps")
+        self.assertEqual(item.timestamp.utcoffset(), timedelta(0), "store() must stamp UTC, not local")
+
+    def test_from_dict_missing_timestamp_is_aware_utc(self):
+        item = MemoryItem.from_dict({"content": "reconstructed"})
+        self.assertIsNotNone(item.timestamp.tzinfo)
+        self.assertEqual(item.timestamp.utcoffset(), timedelta(0))
+
+    def test_round_trip_preserves_the_instant(self):
+        memory = AgentMemory()
+        memory.store("persisted")
+        original = next(iter(memory.memory_items.values()))
+
+        restored = MemoryItem.from_dict(original.to_dict())
+
+        self.assertEqual(
+            restored.timestamp,
+            original.timestamp,
+            "isoformat() must round-trip the exact instant for the comparison key to stay truthful",
+        )
+
+
+class TestComparisonKeySemantics(unittest.TestCase):
+    def test_naive_stamps_keep_their_local_time_meaning(self):
+        # _timestamp_comparison_key documents naive-as-local. If it silently
+        # flips to naive-as-UTC, every legacy persisted stamp shifts by the
+        # host offset — the same class of drift this file exists for, in the
+        # other direction.
+        naive = datetime(2026, 1, 1, 12, 0, 0)
+        self.assertEqual(
+            AgentMemory._timestamp_comparison_key(naive),
+            naive.astimezone(timezone.utc),
+        )
+
+    def test_aware_utc_stamp_is_not_shifted(self):
+        aware = datetime(2026, 1, 1, 12, 0, 0, tzinfo=timezone.utc)
+        self.assertEqual(
+            AgentMemory._timestamp_comparison_key(aware),
+            aware,
+        )
+
+
+class TestDateRangeFiltering(unittest.TestCase):
+    def test_freshly_added_memory_is_inside_a_utc_window_around_now(self):
+        # The regression, end to end: a memory added "now" must land inside a
+        # start_date/end_date window expressed in UTC around now. With the
+        # naive-local producer this fails by exactly the host offset on every
+        # non-UTC host.
+        memory = AgentMemory()
+        memory.store("window probe")
+        now = datetime.now(timezone.utc)
+
+        results = memory.retrieve(
+            "probe",
+            max_results=10,
+            start_date=(now - timedelta(minutes=5)).isoformat(),
+            end_date=(now + timedelta(minutes=5)).isoformat(),
+        )
+
+        self.assertTrue(
+            any(r.get("content") == "window probe" for r in results),
+            f"memory added now must be within the UTC window (host offset {host_offset()})",
+        )
+
+    def test_explicit_utc_boundary_excludes_older_stamps(self):
+        memory = AgentMemory()
+        old = datetime(2020, 1, 1, tzinfo=timezone.utc)
+        memory.store("old memory", timestamp=old)
+        boundary = datetime(2025, 1, 1, tzinfo=timezone.utc)
+
+        results = memory.retrieve(
+            "memory",
+            max_results=10,
+            start_date=boundary.isoformat(),
+        )
+
+        self.assertFalse(
+            any(r.get("content") == "old memory" for r in results),
+            "a 2020 stamp must not match a window starting in 2025, whatever the host timezone",
+        )
+
+
+if __name__ == "__main__":
+    unittest.main()
@@ -272,3 +272,78 @@ else:
    assert "ConfigurationError" in result.stdout
    assert "Salesforce ingestion" in result.stdout
    assert "simple-salesforce" in result.stdout
+
+
+def test_redshift_package_imports_without_sdk() -> None:
+    """``import semantica.ingest`` must not eagerly pull in redshift_connector."""
+    result = _run_python_with_blocked_modules(
+        """
+from semantica.ingest import FileIngestor, ingest_file
+print(FileIngestor.__name__, callable(ingest_file))
+""",
+        ("redshift_connector",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "FileIngestor True" in result.stdout
+
+
+def test_redshift_data_importable_without_sdk() -> None:
+    """``RedshiftData`` is a plain dataclass — no SDK needed to import it."""
+    result = _run_python_with_blocked_modules(
+        """
+from semantica.ingest import RedshiftData
+print(RedshiftData.__name__)
+""",
+        ("redshift_connector",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "RedshiftData" in result.stdout
+
+
+def test_redshift_ingestor_probe_fails_without_sdk() -> None:
+    """``from semantica.ingest import RedshiftIngestor`` must raise ``ImportError``
+    when ``redshift_connector`` is absent."""
+    result = _run_python_with_blocked_modules(
+        """
+try:
+    from semantica.ingest import RedshiftIngestor
+    has_redshift = True
+except ImportError:
+    has_redshift = False
+
+assert not has_redshift, (
+    "Expected RedshiftIngestor import to fail without redshift-connector"
+)
+print("RedshiftIngestor probe passed")
+""",
+        ("redshift_connector",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "RedshiftIngestor probe passed" in result.stdout
+
+
+def test_redshift_connector_reports_missing_dep_with_install_hint() -> None:
+    """Constructing ``RedshiftConnector`` without the SDK must raise ``ImportError``
+    with a message that names the ``db-redshift`` extra."""
+    result = _run_python_with_blocked_modules(
+        """
+from semantica.ingest.redshift_ingestor import RedshiftConnector
+
+try:
+    RedshiftConnector()
+except ImportError as exc:
+    print(type(exc).__name__, exc)
+else:
+    raise SystemExit(
+        "expected RedshiftConnector() to fail without redshift-connector"
+    )
+""",
+        ("redshift_connector",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "ImportError" in result.stdout
+    assert "db-redshift" in result.stdout
@@ -0,0 +1,83 @@
+"""Regression tests for the standalone documentation checker."""
+
+import runpy
+import subprocess
+import sys
+from pathlib import Path
+
+
+def test_mintlify_export_uses_utf8_lossy_capture(monkeypatch):
+    """Mintlify output remains available regardless of the host code page.
+
+    Verifies that:
+    - subprocess.run is called with encoding="utf-8" and errors="replace"
+    - The Windows EPERM noise filter fires correctly when output is captured
+      (i.e. result.stdout is non-empty and "EPERM" reaches the filter)
+    """
+    calls = []
+
+    def fake_run(command, **kwargs):
+        calls.append((command, kwargs))
+        return subprocess.CompletedProcess(
+            command,
+            1,
+            stdout="cleanup EPERM",
+            stderr="",
+        )
+
+    monkeypatch.setattr(subprocess, "run", fake_run)
+    monkeypatch.setattr(sys, "platform", "win32")
+
+    # docs_check.py calls sys.exit(1) when any check fails.  If the EPERM
+    # noise filter cannot see stdout (because it was None / lost), the
+    # Mintlify check reports a failure and sys.exit(1) is raised here.
+    runpy.run_path(
+        str(Path(__file__).parents[1] / "docs_check.py"),
+        run_name="__main__",
+    )
+
+    assert calls, "the Mintlify export check did not invoke subprocess.run"
+    _, kwargs = calls[-1]
+    assert kwargs["encoding"] == "utf-8", (
+        "subprocess.run must use explicit UTF-8 encoding; "
+        "without it the Windows locale codec silently discards Mintlify output"
+    )
+    assert kwargs["errors"] == "replace", (
+        "errors='replace' is required so invalid bytes produce U+FFFD "
+        "instead of raising UnicodeDecodeError and losing the entire stream"
+    )
+
+
+def test_utf8_bytes_survive_capture():
+    """Bytes valid in UTF-8 but invalid in Windows cp1252 must not be lost.
+
+    This is the minimal reproduction from issue #1578: the three bytes
+    0xe2 0xa0 0x8f encode U+2800 (BRAILLE PATTERN BLANK) in UTF-8 but are
+    not representable in cp1252.  Without encoding="utf-8" the subprocess
+    reader thread raises UnicodeDecodeError and result.stdout becomes None,
+    making combined == "" and the EPERM/EBUSY noise filter unreachable.
+    """
+    code = (
+        "import sys; "
+        "sys.stdout.buffer.write(bytes([0xe2, 0xa0, 0x8f])); "
+        "sys.stdout.flush()"
+    )
+    result = subprocess.run(
+        [sys.executable, "-c", code],
+        capture_output=True,
+        text=True,
+        encoding="utf-8",
+        errors="replace",
+    )
+    assert result.returncode == 0
+    assert result.stdout is not None, (
+        "stdout must not be None; a UnicodeDecodeError in the reader thread "
+        "would have discarded the entire captured stream"
+    )
+    # The three bytes decode to a single Unicode character (U+280F); with
+    # errors="replace" any truly undecodable byte would become U+FFFD instead.
+    # Either way the stream is preserved rather than lost — that is the fix.
+    assert len(result.stdout.strip()) == 1, (
+        "expected exactly one decoded character; "
+        "an empty result means the stream was silently lost"
+    )
@@ -0,0 +1,162 @@
+"""Regression tests for the packaged MCP extraction handlers."""
+
+from types import SimpleNamespace
+from unittest.mock import patch
+
+from semantica.semantic_extract import (
+    CoreferenceResolver,
+    Event,
+    EventDetector,
+    NamedEntityRecognizer,
+    RelationExtractor,
+    TripletExtractor,
+)
+from semantica.semantic_extract.coreference_resolver import CoreferenceChain, Mention
+from semantica.semantic_extract.types import Entity, Relation, Triplet
+from semantica_mcp.mcp.server import call_tool
+
+
+def test_extract_entities_preserves_input_text_and_serializes_entity_fields():
+    """Entity offsets must remain relative to the original, untrimmed payload."""
+    text = "  Alice works at Acme Corp."
+    extracted = Entity("Acme Corp", "ORG", 17, 26, confidence=0.7)
+
+    with patch.object(
+        NamedEntityRecognizer, "extract_entities", return_value=[extracted]
+    ) as extract_entities:
+        result = call_tool("extract_entities", {"text": text})
+
+    extract_entities.assert_called_once_with(text)
+    assert result == {
+        "entities": [
+            {
+                "text": "Acme Corp",
+                "label": "ORG",
+                "type": "ORG",
+                "start": 17,
+                "end": 26,
+                "confidence": 0.7,
+            }
+        ],
+        "count": 1,
+    }
+
+
+def test_extract_entities_uses_consistent_defaults_for_missing_label():
+    extracted = SimpleNamespace(text="unknown", start_char=0, end_char=7)
+
+    with patch.object(
+        NamedEntityRecognizer, "extract_entities", return_value=[extracted]
+    ):
+        result = call_tool("extract_entities", {"text": "unknown"})
+
+    assert result["entities"][0]["label"] == ""
+    assert result["entities"][0]["type"] == ""
+
+
+def test_extract_relations_serializes_relation_fields_and_preserves_text():
+    text = "  Alice founded Acme Corp."
+    alice = Entity("Alice", "PERSON", 2, 7, confidence=0.9)
+    acme = Entity("Acme Corp", "ORG", 16, 25, confidence=0.8)
+    relation = Relation(alice, "founded", acme, confidence=0.75)
+    triplet = Triplet("Alice", "founded", "Acme Corp", confidence=0.7)
+
+    with (
+        patch.object(
+            NamedEntityRecognizer, "extract_entities", return_value=[alice, acme]
+        ) as extract_entities,
+        patch.object(
+            RelationExtractor, "extract", return_value=[relation]
+        ) as extract_relations,
+        patch.object(
+            TripletExtractor, "extract", return_value=[triplet]
+        ) as extract_triplets,
+    ):
+        result = call_tool("extract_relations", {"text": text})
+
+    extract_entities.assert_called_once_with(text)
+    extract_relations.assert_called_once_with(text, [alice, acme])
+    extract_triplets.assert_called_once_with(text)
+    assert result == {
+        "relations": [
+            {
+                "source": "Alice",
+                "type": "founded",
+                "target": "Acme Corp",
+                "confidence": 0.75,
+            }
+        ],
+        "triplets": [
+            {"subject": "Alice", "predicate": "founded", "object": "Acme Corp"}
+        ],
+        "relation_count": 1,
+        "triplet_count": 1,
+    }
+
+
+def test_extract_all_keeps_coreferences_separate_from_downstream_text():
+    text = "  Alice founded Acme Corp. She leads it."
+    alice = Entity("Alice", "PERSON", 2, 7, confidence=0.9)
+    acme = Entity("Acme Corp", "ORG", 16, 25, confidence=0.8)
+    representative = Mention("Alice", 2, 7, "entity", entity_id="alice")
+    pronoun = Mention("She", 27, 30, "pronoun", entity_id="alice")
+    chain = CoreferenceChain(
+        mentions=[representative, pronoun],
+        representative=representative,
+        entity_type="PERSON",
+    )
+    relation = Relation(alice, "founded", acme, confidence=0.75)
+    triplet = Triplet("Alice", "founded", "Acme Corp", confidence=0.7)
+    event = Event("founded", "FOUNDING", 8, 15, confidence=0.85)
+
+    with (
+        patch.object(
+            NamedEntityRecognizer, "extract_entities", return_value=[alice, acme]
+        ) as extract_entities,
+        patch.object(CoreferenceResolver, "resolve", return_value=[chain]) as resolve,
+        patch.object(
+            RelationExtractor, "extract", return_value=[relation]
+        ) as extract_relations,
+        patch.object(EventDetector, "extract", return_value=[event]),
+        patch.object(
+            TripletExtractor, "extract", return_value=[triplet]
+        ) as extract_triplets,
+    ):
+        result = call_tool("extract_all", {"text": text})
+
+    extract_entities.assert_called_once_with(text)
+    resolve.assert_called_once_with(text, entities=[alice, acme])
+    extract_relations.assert_called_once_with(text, [alice, acme])
+    extract_triplets.assert_called_once_with(text)
+    assert result == {
+        "entities": [
+            {"text": "Alice", "label": "PERSON", "type": "PERSON"},
+            {"text": "Acme Corp", "label": "ORG", "type": "ORG"},
+        ],
+        "coreferences": [
+            {
+                "representative": "Alice",
+                "mentions": ["Alice", "She"],
+                "entity_type": "PERSON",
+            }
+        ],
+        "relations": [
+            {
+                "source": "Alice",
+                "type": "founded",
+                "target": "Acme Corp",
+                "confidence": 0.75,
+            }
+        ],
+        "events": [{"type": "FOUNDING", "trigger": "founded"}],
+        "triplets": [
+            {"subject": "Alice", "predicate": "founded", "object": "Acme Corp"}
+        ],
+        "summary": {
+            "entities": 2,
+            "coreferences": 1,
+            "relations": 1,
+            "events": 1,
+            "triplets": 1,
+        },
+    }
@@ -0,0 +1,197 @@
+"""Bounded, call-serialized spaCy model cache — regression tests.
+
+Background
+----------
+The cache used to retain every model it ever loaded for the life of the
+process (each spaCy Language is hundreds of MB), and it handed the SAME
+Language object to every caller while batch extraction fans out over a
+ThreadPoolExecutor — spaCy pipelines are not safe to invoke from concurrent
+threads.
+
+This file tests:
+  - LRU eviction at MAX_SPACY_MODELS_CACHED
+  - Per-model call serialization
+  - Parallelism across different models
+  - The single-lock design's correctness under concurrent load
+  - The spacy-not-installed guard
+"""
+
+import sys
+import os
+import threading
+import unittest
+from unittest.mock import MagicMock, patch
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+
+from semantica.semantic_extract import methods
+from semantica.semantic_extract.methods import (
+    MAX_SPACY_MODELS_CACHED,
+    clear_spacy_model_cache,
+    run_spacy_text,
+    spacy_pipeline_guard,
+    _spacy_model_cache,
+)
+
+
+def make_spacy_mock():
+    """Return a minimal spaCy mock whose .load() returns a distinct nlp per name."""
+    mock = MagicMock()
+    mock.load = MagicMock(side_effect=lambda name: MagicMock(name=f"nlp-{name}"))
+    return mock
+
+
+class TestBoundedCache(unittest.TestCase):
+    def setUp(self):
+        clear_spacy_model_cache()
+
+    def tearDown(self):
+        clear_spacy_model_cache()
+
+    def test_cache_evicts_beyond_the_bound(self):
+        spacy = make_spacy_mock()
+        with patch.object(methods, "spacy", spacy):
+            for i in range(MAX_SPACY_MODELS_CACHED + 2):
+                with spacy_pipeline_guard(f"model_{i}"):
+                    pass
+        self.assertLessEqual(
+            len(_spacy_model_cache), MAX_SPACY_MODELS_CACHED,
+            f"cache must stay bounded at {MAX_SPACY_MODELS_CACHED} entries",
+        )
+        self.assertNotIn("model_0", _spacy_model_cache, "oldest entry must be evicted first")
+
+    def test_recently_used_entries_survive(self):
+        """LRU: touching model_0 before overflow keeps it alive; model_1 is evicted."""
+        spacy = make_spacy_mock()
+        with patch.object(methods, "spacy", spacy):
+            for i in range(MAX_SPACY_MODELS_CACHED):
+                with spacy_pipeline_guard(f"model_{i}"):
+                    pass
+            # Re-touch the oldest; overflow should evict model_1 (now the true LRU).
+            with spacy_pipeline_guard("model_0"):
+                pass
+            with spacy_pipeline_guard("model_new"):
+                pass
+        self.assertIn("model_0", _spacy_model_cache)
+        self.assertNotIn("model_1", _spacy_model_cache)
+
+    def test_bound_never_exceeded_under_concurrent_load(self):
+        """The single-lock design must not allow concurrent inserts to temporarily
+        break the cache bound — something the old lockless fast-path could do.
+
+        The patch is applied in the outer (test) thread so that all worker threads
+        share one stable mock for the entire test. Patching per-thread would race:
+        a thread exiting its patch context could restore the real spacy module while
+        another thread is mid-load, causing sporadic OSError from the real spacy.
+        """
+        spacy = make_spacy_mock()
+        errors = []
+
+        def load_model(i):
+            try:
+                with spacy_pipeline_guard(f"concurrent_model_{i}"):
+                    # While holding the call lock, snapshot cache size.
+                    size = len(_spacy_model_cache)
+                    if size > MAX_SPACY_MODELS_CACHED:
+                        errors.append(
+                            f"cache size {size} exceeded bound at model_{i}"
+                        )
+            except Exception as exc:
+                errors.append(str(exc))
+
+        n_threads = MAX_SPACY_MODELS_CACHED * 3
+        with patch.object(methods, "spacy", spacy):
+            threads = [
+                threading.Thread(target=load_model, args=(i,))
+                for i in range(n_threads)
+            ]
+            for t in threads:
+                t.start()
+            for t in threads:
+                t.join()
+
+        self.assertEqual(errors, [], f"cache bound violated under concurrent load: {errors}")
+        self.assertLessEqual(len(_spacy_model_cache), MAX_SPACY_MODELS_CACHED)
+
+    def test_spacy_not_installed_raises_import_error(self):
+        """_load_spacy_entry must raise ImportError if spacy is None, not AttributeError."""
+        with patch.object(methods, "spacy", None):
+            with self.assertRaises(ImportError):
+                with spacy_pipeline_guard("any_model"):
+                    pass
+
+
+class TestPerModelSerialization(unittest.TestCase):
+    def setUp(self):
+        clear_spacy_model_cache()
+
+    def tearDown(self):
+        clear_spacy_model_cache()
+
+    def test_concurrent_calls_on_one_model_never_overlap(self):
+        """The per-model call lock must prevent concurrent nlp(text) on the same Language."""
+        overlaps = []
+        active = []
+        state_lock = threading.Lock()
+
+        def slow_nlp(text):
+            with state_lock:
+                active.append(text)
+                if len(active) > 1:
+                    overlaps.append(list(active))
+            import time
+            time.sleep(0.05)
+            with state_lock:
+                active.pop()
+            return MagicMock(name=f"doc-{text}")
+
+        spacy = MagicMock()
+        spacy.load = MagicMock(return_value=MagicMock(side_effect=slow_nlp))
+        with patch.object(methods, "spacy", spacy):
+            threads = [
+                threading.Thread(target=lambda i=i: run_spacy_text("m", f"t{i}"))
+                for i in range(4)
+            ]
+            for t in threads:
+                t.start()
+            for t in threads:
+                t.join()
+
+        self.assertEqual(
+            overlaps, [],
+            "concurrent calls on the same model must be serialized by the per-model lock",
+        )
+
+    def test_different_models_run_in_parallel(self):
+        """Two different models must be callable simultaneously (per-model, not global lock)."""
+        entered = threading.Barrier(2, timeout=5)
+
+        def blocking_nlp(text):
+            # Both threads must reach this point simultaneously to clear the barrier.
+            # If a single global lock were used, the second thread would be blocked
+            # waiting for the first to finish, and the barrier would time out.
+            entered.wait()
+            return MagicMock()
+
+        spacy = MagicMock()
+        spacy.load = MagicMock(return_value=MagicMock(side_effect=blocking_nlp))
+        with patch.object(methods, "spacy", spacy):
+            threads = [
+                threading.Thread(target=lambda n=n: run_spacy_text(n, "t"))
+                for n in ("model_a", "model_b")
+            ]
+            for t in threads:
+                t.start()
+            for t in threads:
+                t.join(timeout=10)
+                if t.is_alive():
+                    for alive in threads:
+                        alive.join(timeout=1)
+                    self.fail(
+                        "Barrier timed out — different models must run concurrently, "
+                        "not serialize on a single global lock"
+                    )
+
+
+if __name__ == "__main__":
+    unittest.main()
@@ -9,6 +9,7 @@ import pytest
 from semantica.utils.exceptions import ProcessingError
 from semantica.vector_store.faiss_store import (
    FAISSIndex,
+    FAISSIndexBuilder,
    FAISSStore,
    _metadata_path,
 )
@@ -583,3 +584,27 @@ def test_loading_index_without_meta_json_warns(tmp_path):

    assert loaded.vector_ids == []
    assert loaded.metadata == {}
+
+
+def test_builder_trains_pq_index():
+    faiss = pytest.importorskip("faiss")
+    vectors = np.random.default_rng(42).random((256, 4), dtype=np.float32)
+    builder = FAISSIndexBuilder(dimension=4)
+    index = builder.build_index("pq", m=2, bits=2)
+
+    assert not index.index.is_trained
+
+    previous_threads = faiss.omp_get_max_threads()
+    faiss.omp_set_num_threads(1)
+    try:
+        builder.train_index(index, vectors)
+        assert index.index.is_trained
+
+        index.add_vectors(vectors[:2], ids=["a", "b"])
+        assert index.index.ntotal == 2
+
+        distances, indices = index.search(vectors[:1], k=1)
+        assert distances.shape == (1, 1)
+        assert int(indices[0, 0]) in (0, 1)
+    finally:
+        faiss.omp_set_num_threads(previous_threads)
Author	SHA1	Message	Date
Sameer Kadam Sameer Kadam Kaif	05aec1975d	feat: add Amazon Redshift ingestor (#1587 ) * feat: add Redshift ingestor * fix(redshift): resolve 5 code-review issues in RedshiftIngestor Fix #1 — restore autocommit on caller-owned connections All four methods (ingest_table, ingest_query, get_table_schema, list_tables) now capture the connection's prior autocommit value before enabling it for read-only ingestion and restore it in the finally block when the connection was already open. Transient connections are still closed normally. Fix #2 — batch fetching: process rows per-batch, not after accumulating ingest_query's batch path now converts and extends data[] directly inside the fetch loop so raw tuples from each batch are eligible for GC before the next fetch. The documentation is updated to accurately state that batch_size controls driver round-trip size, not total memory. Fix #3 — schema env-var now honoured RedshiftIngestor.__init__ changes schema default from 'public' to None so the existing os.getenv('REDSHIFT_SCHEMA', 'public') fallback in the body is reached when no explicit argument is supplied. Fix #4 — db-redshift included in db-all aggregate pyproject.toml db-all now references db-redshift alongside the other database extras. Fix #5 — duplicate column labels no longer silently drop values _disambiguate_columns() appends _1, _2, ... suffixes to repeated cursor labels before dict(zip) so every value is retained. A new _make_row_dicts() helper encapsulates the zip+disambiguate pattern used by both _fetch_all and ingest_query. Tests: 22 new targeted tests added covering all five fixes. --------- Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan> Co-authored-by: Kaif <98801504+KaifAhmad1@users.noreply.github.com>	2026-09-11 18:43:14 +05:30
csy20andSameer Kadam	bea9b2aa48	docs: rewrite remaining bare relative links that 404 on the live site (#1580 ) #1407 skipped 89 Next Steps links whose targets exist in both guides/ and reference/. Rewrite them as /guides/<name> or /reference/<name> from the source section so Mintlify+GitHub Pages trailing-slash redirects no longer 404. Also retarget three in-page heading anchors to Mintlify's rendered slugs (data-&-features, reasoner-forward/backward-chaining, litellm-100+-providers). Closes #1566 Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>	2026-09-11 17:40:52 +05:30
Kaif	6148cb6f8c	Merge pull request #1563 from gyroscope1110/fix/plugin-skills-stale-api-references fix(plugins): repair broken API references in five skills	2026-09-11 17:06:18 +05:30
Kaif	202acc779e	Merge branch 'main' into fix/plugin-skills-stale-api-references	2026-09-11 16:55:06 +05:30
dependabot[bot]	888c2c4e34	deps(deps): bump litellm from 1.99.0 to 1.100.0 (#1588 ) Bumps [litellm](https://github.com/BerriAI/litellm) from 1.99.0 to 1.100.0. - [Release notes](https://github.com/BerriAI/litellm/releases) - [Commits](https://github.com/BerriAI/litellm/compare/v1.99.0...v1.100.0) --- updated-dependencies: - dependency-name: litellm dependency-version: 1.100.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-09-11 15:37:15 +05:30
Wu ShuwenandSameer Kadam	f36a264571	fix(docs-check): preserve Mintlify output on Windows (#1579 ) * fix(docs-check): preserve Mintlify output on Windows * test(docs-check): add Windows regression coverage * fix(docs-check): preserve Mintlify output on Windows --------- Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>	2026-09-11 10:09:40 +05:30
Sameer Kadam	76515280a4	docs: add multilingual README links (#1567 ) Add links for multilingual README using zdoc service	2026-09-11 00:40:52 +05:00
Zheng Feng	9905e4321d	fix(mcp): repair extraction handlers and pipeline serialization (#1535 ) Fix runtime crashes and serialization defects across the packaged MCP extraction handlers in semantica_mcp/mcp/tools/extraction.py. * fix(mcp): use public named entity extraction API - Replace non-existent NamedEntityRecognizer.extract() with extract_entities() - Preserve original input text to prevent entity character offset coordinate drift - Fixes #1533 * fix(mcp): correct extraction pipeline serialization - Map Relation fields (subject, predicate, object) to MCP keys (source, type, target) - Retain coreference chains in result["coreferences"] instead of passing to text extractors - Fixes #1534 * fix(mcp): serialize event fields correctly - Map Event dataclass fields (event_type, text) to MCP keys (type, trigger) - Add regression tests covering entity offsets, relations, coreferences, and events	2026-09-10 20:43:13 +05:00
Mohd Kaif	161748f0b0	Merge pull request #1564 from Evanwang-3/codex/fix-faiss-pq-training fix(vector-store): train PQ indexes in FAISSIndexBuilder	2026-09-10 16:22:53 +05:30
Mohd Kaif	731814d2b3	Merge branch 'main' into codex/fix-faiss-pq-training	2026-09-10 16:06:56 +05:30
yzxcj797 Sameer Kadam Sameer Kadam	6afe90e4fb	fix(context): stamp AgentMemory in aware UTC, not mixed naive conventions (#1073 ) * fix(context): stamp AgentMemory in aware UTC, not mixed naive conventions MemoryItem timestamps had two producers with different naive conventions: store() defaulted to datetime.now() (naive LOCAL) while from_dict fell back to datetime.utcnow() (naive UTC). _timestamp_comparison_key interprets every naive stamp as LOCAL time, so on any host off UTC the two producers disagree by the host's offset — a UTC+8 host pushed freshly stored memories 8 hours outside every start_date/end_date window, and cleanup_old_memories aged them wrong by the same amount. All three producers now stamp datetime.now(timezone.utc): - store() default and the from_dict fallbacks stop introducing naive values; - the cleanup cutoff compares against an aware instant instead of a naive local one; - legacy naive stamps keep their documented local-time meaning through the comparison key, which is deliberately unchanged and now pinned by test. Seven regression tests cover the producer contract, the naive-as-local comparison semantics (so the key cannot silently flip to naive-as-UTC), the isoformat round-trip, and end-to-end date-range filtering with UTC boundaries on hosts in any timezone. * test(context): rename add() references to store() in timestamp test --------- Co-authored-by: Sameer Kadam <sskadam6305@gmail.com> Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>	2026-09-10 15:16:35 +05:30
xudong	33f8719c80	fix(vector-store): train PQ indexes in FAISSIndexBuilder	2026-09-10 16:45:58 +08:00
Mohd Kaif	3bb0d28112	docs(readme): note LLM use is optional and vendor-neutral (#1561 ) Make explicit in the opening pitch that where Semantica does use an LLM, it's optional and works with any major provider via semantica.llms, not tied to one vendor.	2026-09-10 13:59:45 +05:30
yzxcj797 Sameer Kadam Sameer Kadam	1914dd508c	fix: replace bare except with except Exception in 3 sites (#1069 ) * fix: bare except -> except Exception in docling_parser * fix: bare except -> except Exception in methods.py * Address review: bound the spaCy model cache, serialize calls per model Two findings from the Qodo review: - _spacy_model_cache retained every successfully loaded model for the process lifetime — each spaCy Language costs hundreds of MB, so a service varying the model option grew without bound. Now an LRU capped at MAX_SPACY_MODELS_CACHED (4: lg/md/sm + one custom), evicting least-recently-used on insert. - The cached Language was handed to every caller while batch extraction fans out over a ThreadPoolExecutor; spaCy pipelines are not safe to invoke from concurrent threads. Calls now go through per-model locks: spacy_pipeline_guard() yields the pipeline under its own lock (different models still run in parallel), and run_spacy_text() wraps the load/fallback/call chain the entity and relation extractors used to open-code. load_spacy_model() keeps its signature but documents that its result must only be called under the guard. Four regression tests: the bound holds, LRU evicts oldest-not-recently- used, concurrent calls on one model never overlap, and two models can be inside calls at the same time (the lock is per-model, not global). * fix: bare except, bounded LRU spaCy cache, per-model call lock (#1069) Problems fixed -------------- - Replace all 5 bare 'except:' clauses with 'except Exception:' in methods.py (get_nlp_model, extract_relations_similarity fallback) and docling_parser.py (3 sites, including 2 missed by the original PR). - Replace the unbounded dict cache with an OrderedDict LRU bounded at MAX_SPACY_MODELS_CACHED=4. Each spaCy Language is 100-700 MB; the old design pinned every model name ever used for the lifetime of the process. - Fix a TOCTOU race in _load_spacy_entry: the lockless fast-path could call move_to_end() on an entry that had just been evicted by another thread (KeyError). Rewrite as a single critical section: the cache lookup, LRU update, model load, insertion, and eviction all happen under one lock. spacy.load() inside the lock is acceptable because model loads are rare (at most MAX_SPACY_MODELS_CACHED per process). - Add a per-model threading.Lock stored in each cache entry. All calls to nlp(text) go through spacy_pipeline_guard(), which acquires the entry's lock before yielding the Language. This serializes concurrent calls to the same model (spaCy pipelines are not thread-safe) while allowing different models to run in parallel. - Add run_spacy_text() helper to DRY up load-with-fallback-and-logging used in extract_entities_ml, extract_relations_dependency, and extract_relations_similarity. Behavior changes vs main ------------------------ - Pipeline errors during nlp(text) (non-OSError) now log a warning and fall back to pattern extraction, instead of propagating to the caller. This is strictly safer for a service context. - extract_relations_similarity drops an unreachable bare-except path that tried en_core_web_sm when is_package() returned False for all three model names. The is_package() guard is unchanged. - Log messages for model-not-found events now carry a function-scoped label (spaCy NER, spaCy dependency, spaCy similarity) for easier triage in production logs. Tests ----- - Keep and improve four existing cache/lock tests. - Add test_bound_never_exceeded_under_concurrent_load: verifies the single-lock design never exceeds the bound under concurrent pressure (would have caught the old TOCTOU). - Add test_spacy_not_installed_raises_import_error: verifies that spacy=None produces a clear ImportError, not an AttributeError. --------- Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan> Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>	2026-09-10 13:53:55 +05:30
Mohd Kaif	375ffc8522	docs(readme): remove unshipped 0.7.0 references and duplicate content (#1559 ) The v0.7.0 "What's New" section and the note pinning to <0.7.0 named a version that hasn't been tagged or published to PyPI yet (latest release remains v0.6.8); reword the installation note without asserting an unshipped version boundary. Also drop the "Built for High-Stakes Domains" section, which duplicated the top-of-file positioning, the Regulated enterprises bullet, and the explainability [!NOTE] callout.	2026-09-10 13:46:25 +05:30
Mohd Kaif	365d473743	Merge pull request #1558 from semantica-agi/readme-positioning-refresh docs(readme): tighten intro copy and enterprise data platform framing	2026-09-10 13:29:24 +05:30
KaifAhmad1andClaude Sonnet 5	e40ba9b254	docs(readme): tighten intro copy and clarify enterprise data platform support Break the dense opening paragraph into scannable lines, lead with the semantic/context layer framing, name Databricks/Snowflake/SAP together under "enterprise data teams" instead of a two-platform subset, and switch the explainability caveat to GitHub's native [!NOTE] alert. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-10 13:23:57 +05:30