chore: bump version to 0.6.5 (#918)

Security release bundling fixes for GHSA-j4mq (missing auth), GHSA-8c7v
(SSRF via redirect bypass), GHSA-482h (Cypher injection), GHSA-8vgg
(SPARQL injection), GHSA-4643 (WebSocket Origin validation), and a
CodeQL-flagged ReDoS in the SPARQL route validator.

.checkov.yaml

@@ -0,0 +1,19 @@
+# Checkov configuration.
+# Cloud Run false-positives (CKV_K8S_21/28/30) are suppressed via per-file
+# inline checkov:skip comments in deploy/gcp/cloudrun-service.yaml rather than
+# globally here, so future real Kubernetes manifests are not silently exempted.
+#
+# The knowledge-explorer Helm chart's unconditional templates (service.yaml,
+# deployment.yaml, configmap.yaml) set metadata.namespace to .Release.Namespace,
+# which is only bound at `helm install`/`helm template` time. Checkov's helm
+# framework renders the chart without a namespace override, so it always
+# resolves to "default" and trips CKV_K8S_21 even though the chart is
+# namespace-agnostic by design. Suppressed via metadata annotations
+# (checkov.io/skip1 / runterrascan.io/skip) on each resource's metadata.annotations,
+# as both Checkov and Terrascan require K8s/Helm resource-level annotations
+# rather than file-header comments.
+# deployment.yaml additionally suppresses AC_K8S_0080 and CKV_K8S_31 (seccomp) via
+# metadata.annotations on both the Deployment resource and the pod template:
+# the seccomp profile is set correctly in values.yaml and only resolves once
+# Helm actually renders `toYaml`, which static template scanning does not do.
+skip-check: []

.claude/init

@@ -0,0 +1 @@
+# Initialization

.claude/skills/init

@@ -0,0 +1 @@
+# Intialization

.claude/skills/semantica/init

@@ -0,0 +1 @@
+# Initialization

.dockerignore

@@ -0,0 +1,103 @@
+# Start with a tiny Docker context and opt in only files used by Dockerfile.
+*
+!Dockerfile
+!.dockerignore
+!pyproject.toml
+!README.md
+!LICENSE
+!MANIFEST.in
+!semantica/
+!semantica/**
+!integrations/
+!integrations/**
+!explorer/
+!explorer/**
+
+# VCS, local config, and secrets.
+.git
+.git/**
+.github
+.github/**
+.claude
+.claude/**
+.codex
+.codex/**
+.agents
+.agents/**
+.env
+.env.*
+*.env
+
+# Python build/test/cache artifacts.
+__pycache__
+**/__pycache__
+*.py[cod]
+.pytest_cache
+.pytest_cache/**
+.mypy_cache
+.mypy_cache/**
+.ruff_cache
+.ruff_cache/**
+.tox
+.tox/**
+.venv
+.venv/**
+venv
+venv/**
+coverage
+coverage/**
+htmlcov
+htmlcov/**
+*.egg-info
+*.egg-info/**
+build
+build/**
+dist
+dist/**
+
+# Frontend dependency/build artifacts.
+node_modules
+node_modules/**
+explorer/node_modules
+explorer/node_modules/**
+explorer/dist
+explorer/dist/**
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Local outputs and large generated samples.
+logs
+logs/**
+*.log
+*.tmp
+*.bak
+*.backup
+tests
+tests/**
+explorer/tests
+explorer/tests/**
+docs
+docs/**
+site
+site/**
+.mkdocs_cache
+.mkdocs_cache/**
+cookbook
+cookbook/**
+examples
+examples/**
+demo_assets
+demo_assets/**
+demo_out
+demo_out/**
+demo_out_*
+demo_out_*/**
+outputs
+outputs/**
+pytest-cache-files-*
+pytest-cache-files-*/**
+test_data
+test_data/**
+sample_data
+sample_data/**

.gitattributes

@@ -1,3 +1,11 @@
+# Line endings — force LF so Mintlify/Linux CI parses frontmatter correctly
+* text=auto eol=lf
+*.md text eol=lf
+*.json text eol=lf
+*.yml text eol=lf
+*.yaml text eol=lf
+*.py text eol=lf
+
 # Linguist documentation and generated files
 # This ensures GitHub language statistics reflect the core Python code
 

.github/codeql/codeql-config.yml

@@ -0,0 +1,11 @@
+name: "Semantica CodeQL Config"
+
+# Exclude auto-generated notebook exports and bundled third-party JS.
+# Files in cookbook/**/*.html are self-contained Plotly/MapLibre bundles
+# produced by Jupyter nbconvert — they embed minified third-party libraries
+# (Plotly, MapLibre GL JS) whose internal patterns trigger false-positive JS
+# alerts (js/incomplete-url-substring-sanitization, js/insecure-randomness,
+# js/prototype-pollution-utility). These are not application code.
+paths-ignore:
+  - "cookbook/**/*.html"
+  - "cookbook/**/*.js"

.github/dependabot.yml

@@ -70,6 +70,13 @@ updates:
       - "dependencies"
       - "github-actions"
       - "ci"
+    # All our actions are SHA-pinned with a "# vX" comment; Dependabot
+    # resolves the new tag's SHA and updates both the pin and the comment
+    # together, so this stays the source of truth (no separate script needed).
+    groups:
+      github-actions:
+        patterns:
+          - "*"
 
   # Optional dependencies (separate schedule for stability)
   - package-ecosystem: "pip"

.github/pull_request_template.md

@@ -1,3 +1,5 @@
+> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — comment on the issue and wait for assignment before opening a PR, to avoid duplicate work.
+
 ## Description
 
 <!-- Provide a clear description of your changes -->

.github/scripts/verify-action-pins.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Verifies that every third-party GitHub Action referenced in
+# .github/workflows/*.yml and .github/workflows/*.yaml is pinned to a full
+# commit SHA (not a mutable tag
+# or branch), and that any pin's trailing "# vX" comment still matches what
+# that tag resolves to today.
+#
+# Fails closed on purpose:
+#   - a `uses:` line pinned to anything other than a 40-hex-char SHA is a
+#     hard failure, not a skip - this is what stops a newly-added mutable
+#     tag (e.g. `uses: some/action@v1`) from slipping past unnoticed.
+#   - a tag that can't be resolved via the GitHub API (rate limit, deleted
+#     tag, typo) is also a hard failure rather than a warning - an
+#     unverifiable pin is exactly the failure mode this check exists to
+#     catch, so it must not pass silently.
+set -uo pipefail
+
+fail=0
+checked=0
+
+# Pattern for a third-party uses: line — stored in a variable so bash's
+# [[ =~ ]] parser never sees literal \" or \' escapes, which cause a
+# "syntax error in conditional expression: unexpected token )" at runtime.
+# Semantics: optional leading quote, owner/repo, optional subpath, @ref,
+# optional trailing quote; quote chars excluded from the ref capture group.
+USES_PATTERN='uses:[[:space:]]+["'"'"']?([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)(/[^[:space:]@"'"'"']+)?@([^[:space:]"'"'"']+)["'"'"']?'
+
+while IFS=: read -r file lineno content; do
+  # Local composite actions (./x) and Docker image refs (docker://...) use a
+  # different pinning mechanism and aren't in scope here.
+  [[ "$content" =~ uses:\ +\./ ]] && continue
+  [[ "$content" =~ uses:\ +docker:// ]] && continue
+
+  if [[ "$content" =~ $USES_PATTERN ]]; then
+    repo="${BASH_REMATCH[1]}"
+    ref="${BASH_REMATCH[3]}"
+    checked=$((checked + 1))
+
+    if [[ ! "$ref" =~ ^[0-9a-fA-F]{40}$ ]]; then
+      echo "::error file=$file,line=$lineno::$repo is pinned to '$ref', not a full commit SHA. Mutable tags/branches can be silently re-pointed (see the LiteLLM/Trivy 2026 incident) - pin to a commit SHA instead."
+      fail=1
+      continue
+    fi
+    sha="$ref"
+
+    if [[ "$content" =~ \#[[:space:]]*([^[:space:]]+)[[:space:]]*$ ]]; then
+      tag="${BASH_REMATCH[1]}"
+    else
+      echo "::warning file=$file,line=$lineno::$repo@$sha has no trailing '# vX' comment recording which tag it corresponds to - add one for auditability."
+      continue
+    fi
+
+    resolved=$(gh api "repos/$repo/commits/$tag" --jq '.sha' 2>/dev/null)
+    if [[ -z "$resolved" ]]; then
+      echo "::error file=$file,line=$lineno::Could not resolve '$repo@$tag' via the GitHub API (rate limit, deleted tag, or typo). Treating as unverifiable = failure."
+      fail=1
+      continue
+    fi
+
+    if [[ "$resolved" != "$sha" ]]; then
+      echo "::error file=$file,line=$lineno::$repo is pinned to $sha but tag '$tag' now resolves to $resolved. Update the pin or the comment."
+      fail=1
+    else
+      echo "OK  $repo@$tag -> $sha  ($file:$lineno)"
+    fi
+  fi
+done < <(grep -rHn "uses:" .github/workflows/*.yml .github/workflows/*.yaml 2>/dev/null)
+
+echo "Checked $checked action reference(s)."
+exit $fail

.github/workflows/benchmark.yml

@@ -1,13 +1,6 @@
 name: Semantica Performance Suite
 
 on:
-  push:
-    branches: [main]
-    paths-ignore:
-      - 'docs/**'
-      - 'mkdocs.yml'
-      - 'requirements-docs.txt'
-      - '**/*.md'
   workflow_dispatch:
 
 permissions:
@@ -20,12 +13,12 @@ jobs:
 
     steps:
       - name: Checkout Code
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           fetch-depth: 0
 
       - name: Set up Python 3.12
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
           python-version: "3.12"
           cache: 'pip'
@@ -50,7 +43,7 @@ jobs:
           # pytest-benchmark --storage file://benchmarks/results --benchmark-compare
 
       - name: Upload Benchmark Results
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
         if: always()
         with:
           name: benchmark-report-${{ github.run_id }}

.github/workflows/ci.yml

@@ -1,28 +1,63 @@
 name: CI
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: [main]
     paths-ignore:
       - 'docs/**'
-      - 'mkdocs.yml'
-      - 'requirements-docs.txt'
+      - 'docs_check.py'
       - '**/*.md'
   pull_request:
     branches: [main]
     paths-ignore:
       - 'docs/**'
-      - 'mkdocs.yml'
-      - 'requirements-docs.txt'
+      - 'docs_check.py'
       - '**/*.md'
 
 jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
           python-version: '3.11'
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: '20'
+          cache: 'npm'
+          cache-dependency-path: explorer/package-lock.json
+      - name: Install Explorer frontend dependencies
+        working-directory: explorer
+        run: npm ci
+      - name: Test Explorer frontend
+        working-directory: explorer
+        run: |
+          npm run test:graph-store
+          npm run test:graph-workspace
+          npm run test:plugin-registry
+      - name: Build Explorer frontend
+        working-directory: explorer
+        run: npm run build
       - run: pip install build
       - run: python -m build
+      - name: Verify Explorer frontend is packaged
+        run: |
+          python - <<'PY'
+          import zipfile
+          from pathlib import Path
+
+          wheels = list(Path("dist").glob("*.whl"))
+          assert wheels, "No wheel was built"
+
+          with zipfile.ZipFile(wheels[0]) as wheel:
+              names = set(wheel.namelist())
+
+          assert "semantica/static/index.html" in names, "Explorer index.html missing from wheel"
+          assert any(name.startswith("semantica/static/assets/") for name in names), "Explorer assets missing from wheel"
+
+          print("Explorer frontend is packaged")
+          PY

.github/workflows/codeql.yml

@@ -20,19 +20,49 @@ jobs:
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
 
-      - name: Initialize CodeQL
-        uses: github/codeql-action/init@v4
+      # The CodeQL bundle download (github/codeql-action/init's "Setup CodeQL
+      # tools" step) streams a ~1GB tarball from GitHub's release CDN and
+      # does not retry on a transient connection reset (ECONNRESET) itself
+      # (github/codeql-action, unresolved as of v4 / CLI 2.26.1: the HTTP
+      # error is retryable but isn't retried internally). Since a `uses:`
+      # step can't be wrapped by a shell-level retry action, attempt init
+      # up to 3 times; each retry is a fresh download attempt with no
+      # meaningful state carried over from a failed attempt.
+      - name: Initialize CodeQL (attempt 1)
+        id: codeql-init-1
+        uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
+        continue-on-error: true
         with:
           languages: python
           queries: security-and-quality
+          config-file: .github/codeql/codeql-config.yml
+
+      - name: Initialize CodeQL (attempt 2)
+        id: codeql-init-2
+        if: steps.codeql-init-1.outcome == 'failure'
+        uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
+        continue-on-error: true
+        with:
+          languages: python
+          queries: security-and-quality
+          config-file: .github/codeql/codeql-config.yml
+
+      - name: Initialize CodeQL (attempt 3)
+        id: codeql-init-3
+        if: steps.codeql-init-2.outcome == 'failure'
+        uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
+        with:
+          languages: python
+          queries: security-and-quality
+          config-file: .github/codeql/codeql-config.yml
 
       - name: Autobuild
-        uses: github/codeql-action/autobuild@v4
+        uses: github/codeql-action/autobuild@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
 
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v4
+        uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
         with:
           category: "/language:python"
           upload: false
@@ -42,45 +72,26 @@ jobs:
         # Uploads results only when Default Setup is not active.
         # If Default Setup is still enabled, this step skips gracefully
         # instead of failing the workflow with HTTP 409.
-        uses: github/codeql-action/upload-sarif@v4
+        uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
         with:
           sarif_file: ${{ steps.codeql.outputs.sarif-output }}
           category: "/language:python"
           wait-for-processing: true
         continue-on-error: true
 
-  dismiss-fixed-alerts:
-    name: Dismiss Fixed Security Alerts
-    runs-on: ubuntu-latest
-    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
-    steps:
-      - name: Dismiss resolved CodeQL alerts via API
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REPO: ${{ github.repository }}
-        run: |
-          FIXED_PATTERNS=(
-            "py/clear-text-logging-sensitive-data"
-            "py/incomplete-url-substring-sanitization"
-            "actions/missing-workflow-permissions"
-          )
-
-          # Fetch all open code scanning alerts
-          ALERTS=$(gh api repos/$REPO/code-scanning/alerts \
-            --jq '.[] | {number: .number, rule: .rule.id, state: .state}' \
-            -X GET -f state=open -f per_page=100)
-
-          for PATTERN in "${FIXED_PATTERNS[@]}"; do
-            ALERT_NUMS=$(echo "$ALERTS" | jq -r \
-              "select(.rule == \"$PATTERN\") | .number")
-            for NUM in $ALERT_NUMS; do
-              echo "Dismissing alert #$NUM ($PATTERN) — fixed in security-enhancement PR"
-              gh api repos/$REPO/code-scanning/alerts/$NUM \
-                -X PATCH \
-                -f state=dismissed \
-                -f dismissed_reason="won't fix" \
-                -f dismissed_comment="Fixed in PR security-enhancement: code changes remove the vulnerability. Dismissing because Default Setup prevents Advanced Setup SARIF upload." \
-                && echo "  ✓ Alert #$NUM dismissed" \
-                || echo "  ⚠ Could not dismiss alert #$NUM (may already be closed)"
-            done
-          done
+  # NOTE: Auto-dismissal by rule-id is intentionally removed.
+  # Dismissing every alert that matches a rule ID would silently suppress
+  # future real vulnerabilities of the same type. The alerts below were
+  # individually triaged and dismissed manually in the security-enhancement
+  # PR (alerts #12–#18). New alerts must be reviewed and dismissed by hand,
+  # or will auto-close when the underlying code no longer triggers them.
+  #
+  # If you need to dismiss a specific known-safe alert, pin its alert NUMBER
+  # here and remove it once CodeQL stops reporting it naturally. Example:
+  #
+  #   PINNED_ALERT_NUMBERS=(12 13 14 15 16 17 18)
+  #   for NUM in "${PINNED_ALERT_NUMBERS[@]}"; do
+  #     gh api repos/$REPO/code-scanning/alerts/$NUM \
+  #       -X PATCH -f state=dismissed -f dismissed_reason="false positive" \
+  #       -f dismissed_comment="<reason>"
+  #   done

.github/workflows/defender-for-devops.yml

@@ -0,0 +1,88 @@
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+#
+# Microsoft Security DevOps (MSDO) is a command line application which integrates static analysis tools into the development cycle.
+# MSDO installs, configures and runs the latest versions of static analysis tools
+# (including, but not limited to, SDL/security and compliance tools).
+#
+# The Microsoft Security DevOps action is currently in beta and runs on the windows-latest queue,
+# as well as Windows self hosted agents. ubuntu-latest support coming soon.
+#
+# For more information about the action , check out https://github.com/microsoft/security-devops-action
+#
+# Please note this workflow do not integrate your GitHub Org with Microsoft Defender For DevOps. You have to create an integration
+# and provide permission before this can report data back to azure.
+# Read the official documentation here : https://learn.microsoft.com/en-us/azure/defender-for-cloud/quickstart-onboard-github
+
+name: "Microsoft Defender For Devops"
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  schedule:
+    - cron: '43 17 * * 6'
+
+permissions:
+  contents: read
+  security-events: write
+
+jobs:
+  MSDO:
+    # currently only windows-latest is supported
+    runs-on: windows-latest
+
+    steps:
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+    - uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6
+      with:
+        dotnet-version: |
+          5.0.x
+          6.0.x
+    - name: Run Microsoft Security DevOps
+      uses: microsoft/security-devops-action@08976cb623803b1b36d7112d4ff9f59eae704de0 # v1.12.0
+      id: msdo
+      with:
+        # checkov is intentionally excluded from this MSDO step.
+        # MSDO 0.215.0's guardian.cmd wrapper treats checkov's exit code 1
+        # (emitted whenever any violation is found, even below the active severity
+        # threshold) as a fatal "tool error" and breaks the build even when
+        # "Active results: 0" and "Found no breaking results." The .checkov.yaml
+        # soft-fail setting is never read by the guardian wrapper.
+        # IaC security scanning continues below in this same MSDO job identity.
+        # That preserves the existing GitHub code-scanning configuration while
+        # avoiding the guardian.cmd/checkov exit-code bug in the MSDO wrapper.
+        tools: eslint,templateanalyzer,terrascan
+    - name: Upload results to Security tab
+      uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
+      with:
+        sarif_file: ${{ steps.msdo.outputs.sarifFile }}
+
+    - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
+      with:
+        python-version: "3.12"
+
+    - name: Install Checkov
+      run: python -m pip install checkov==3.3.1
+
+    - name: Run Checkov
+      shell: pwsh
+      env:
+        PYTHONUTF8: "1"
+      run: |
+        New-Item -ItemType Directory -Force reports | Out-Null
+        checkov --directory . --framework kubernetes helm dockerfile github_actions secrets bicep arm --soft-fail --output sarif --output-file-path reports/checkov.sarif
+        if (-not (Test-Path reports/checkov.sarif)) {
+          $sarif = Get-ChildItem -Path reports -Recurse -Filter *.sarif | Select-Object -First 1
+          if ($null -eq $sarif) { throw "Checkov did not produce a SARIF file" }
+          Copy-Item $sarif.FullName reports/checkov.sarif
+        }
+
+    - name: Upload Checkov results to Security tab
+      uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
+      if: always()
+      with:
+        sarif_file: reports/checkov.sarif

.github/workflows/docs.yml

@@ -1,80 +1,68 @@
 name: Build and Deploy Documentation
 
-# This workflow builds the documentation site and deploys it to GitHub Pages
-# It runs when changes are pushed to the 'docs' folder on the main branch
-
 on:
   push:
     branches: [main]
     paths:
       - 'docs/**'
-      - 'mkdocs.yml'
-      - 'requirements-docs.txt'
+      - 'docs_check.py'
       - 'CHANGELOG.md'
       - 'RELEASE.md'
-  release:
-    types: [published]
+  pull_request:
+    branches: [main]
+    paths:
+      - 'docs/**'
+      - 'docs_check.py'
   workflow_dispatch:
 
-# Permissions needed to deploy to GitHub Pages
 permissions:
   contents: read
   pages: write
   id-token: write
 
-# Prevent concurrent deployments
 concurrency:
   group: "pages"
   cancel-in-progress: false
 
 jobs:
-  build:
-    name: Build Documentation
+  validate:
+    name: Validate Documentation
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
           python-version: '3.11'
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: '20'
+      - run: python docs_check.py
 
-      - name: Install documentation dependencies
+  deploy:
+    name: Build and Deploy to GitHub Pages
+    if: github.event_name != 'pull_request'
+    runs-on: ubuntu-latest
+    needs: validate
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: '20'
+
+      - name: Export static site
         run: |
-          python -m pip install --upgrade pip
-          pip install -r requirements-docs.txt
+          cd docs
+          npx mintlify export --output ../export.zip
+          cd ..
+          unzip -q export.zip -d site
 
-      - name: Build documentation
-        # Builds the static site using MkDocs
-        run: mkdocs build --strict
+      - uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6
 
-      - name: Check for broken links
-        # Optional: checks if any links in the docs are broken
-        run: |
-          pip install linkchecker || echo "Skipping link check"
-          if [ -d "site" ]; then
-            linkchecker site/ --check-extern || echo "Link check completed"
-          fi
-        continue-on-error: true
-
-      - name: Setup Pages
-        uses: actions/configure-pages@v6
-        continue-on-error: true
-
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@v3
+      - uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5
         with:
           path: ./site
 
-  deploy:
-    name: Deploy to GitHub Pages
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
-    needs: build
-    steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v5
+        uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5

.github/workflows/release.yml

@@ -5,21 +5,61 @@ on:
     tags: ['v*']
 
 permissions:
-  contents: write
-  id-token: write
+  contents: read
 
 jobs:
   release:
     runs-on: ubuntu-latest
     environment: pypi
+    concurrency:
+      group: release-${{ github.ref }}
+      cancel-in-progress: false
+    permissions:
+      contents: write      # for the GitHub Release
+      id-token: write      # for PyPI Trusted Publishing (OIDC) and attestation signing
+      attestations: write  # for SLSA build provenance
+    # If you add another job to this workflow, give it its own explicit
+    # `permissions:` block rather than relying on the workflow-level default
+    # above (contents: read) - do not widen the workflow-level default.
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
           python-version: '3.11'
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: '20'
+          cache: 'npm'
+          cache-dependency-path: explorer/package-lock.json
+      - name: Build Explorer frontend
+        working-directory: explorer
+        run: |
+          npm ci
+          npm run build
       - run: pip install build
       - run: python -m build
-      - uses: softprops/action-gh-release@v1
+      - name: Verify Explorer frontend is packaged
+        run: |
+          python - <<'PY'
+          import zipfile
+          from pathlib import Path
+
+          wheels = list(Path("dist").glob("*.whl"))
+          assert wheels, "No wheel was built"
+
+          with zipfile.ZipFile(wheels[0]) as wheel:
+              names = set(wheel.namelist())
+
+          assert "semantica/static/index.html" in names, "Explorer index.html missing from wheel"
+          assert any(name.startswith("semantica/static/assets/") for name in names), "Explorer assets missing from wheel"
+
+          print("Explorer frontend is packaged")
+          PY
+      - name: Attest build provenance
+        uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4
+        with:
+          subject-path: 'dist/*'
+      - uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3
         with:
           files: dist/*
-      - uses: pypa/gh-action-pypi-publish@release/v1
+      - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1

.github/workflows/security-scan.yml

@@ -18,6 +18,9 @@ on:
       - 'requirements-docs.txt'
       - '**/*.md'
 
+permissions:
+  contents: read
+
 jobs:
   security-scan:
     runs-on: ubuntu-latest
@@ -25,13 +28,17 @@ jobs:
       contents: read
       security-events: write
       actions: read
-    
+      # Needed for the "Comment PR with Security Results" step below. Safe on
+      # pull_request (not pull_request_target): GitHub always forces a
+      # read-only token for PRs from forks regardless of this permission.
+      pull-requests: write
+
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
-      
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
           python-version: '3.11'
         
@@ -39,21 +46,50 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install safety bandit semgrep jq
-        
+          # Install the project itself (core deps + the LiteLLM provider extra)
+          # so Safety scans Semantica's actual dependency tree, not just the
+          # scanner tools' own dependencies.
+          pip install -e ".[llm-litellm]"
+
       - name: Run Safety Check (Package Vulnerabilities)
         run: |
-          safety check --json --output safety-report.json || true
+          # NOTE: Safety 3.x repurposed --output to select a console format
+          # (json/text/screen/...), not a file path. Writing JSON to a file
+          # now requires --save-json; the previous `--output safety-report.json`
+          # usage was silently invalid and never produced a report.
+          safety check --save-json safety-report.json || true
+
+          # Guard 1: fail loudly if Safety exited before writing a report at all
+          # (network error, API auth failure, tool crash). Without this check a
+          # missing or empty file causes jq to fall back to "0", making a broken
+          # scanner indistinguishable from a clean scan.
+          if [ ! -s safety-report.json ]; then
+            echo "::error::Safety scan produced no report (safety-report.json is missing or empty). Treating as failure — check for network errors, API auth failures, or Safety crashes in the logs above."
+            exit 1
+          fi
+
           echo "Checking for package vulnerabilities..."
-          
-          # Count vulnerabilities safely
-          VULNS=$(safety check --json --output /dev/stdout 2>/dev/null | jq '.vulnerabilities | length' 2>/dev/null || echo "0")
-          
+
+          # No || echo "0" fallback: if jq fails (malformed JSON, missing key,
+          # vulnerabilities:null) VULNS will be empty or "null" so guard 2 below
+          # catches it rather than silently treating the broken report as zero.
+          VULNS=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null)
+
+          # Guard 2: ensure VULNS is a non-negative integer before the -gt
+          # comparison. "null" (missing/null key) or "" (jq parse failure) would
+          # cause bash's -gt to throw an arithmetic error and fall through to the
+          # success branch — the same silent-pass bug as a missing file.
+          if ! [[ "$VULNS" =~ ^[0-9]+$ ]]; then
+            echo "::error::Safety report exists but 'vulnerabilities' is missing or non-numeric (got: '${VULNS}'). The report may be malformed or Safety may have written an error-only JSON. Treating as failure."
+            exit 1
+          fi
+
           if [ "$VULNS" -gt 0 ]; then
             echo "❌ Security vulnerabilities found: $VULNS"
             echo "CI will fail to prevent merging of vulnerable dependencies"
             echo ""
             echo "Vulnerability details:"
-            safety check || true
+            jq -r '.vulnerabilities[] | "- \(.package_name)==\(.analyzed_version): \(.vulnerability_id) (\(.CVE // "no CVE assigned"))"' safety-report.json || true
             exit 1
           else
             echo "✅ No security vulnerabilities found"
@@ -96,9 +132,10 @@ jobs:
           fi
         
       - name: Upload Security Reports
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
         with:
           name: security-reports
+          retention-days: 14
           path: |
             safety-report.json
             bandit-report.json
@@ -106,77 +143,91 @@ jobs:
         
       - name: Comment PR with Security Results
         if: github.event_name == 'pull_request'
-        uses: actions/github-script@v8
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
         with:
           script: |
             const fs = require('fs');
-            
-            // Read safety report
-            let safetyResults = '';
-            try {
-              const safetyData = JSON.parse(fs.readFileSync('safety-report.json', 'utf8'));
-              if (safetyData.vulnerabilities && safetyData.vulnerabilities.length > 0) {
-                safetyResults = `## Safety Vulnerabilities Found\\n`;
-                safetyData.vulnerabilities.forEach(vuln => {
-                  safetyResults += `- **${vuln.package}**: ${vuln.advisory}\\n`;
-                });
-              } else {
-                safetyResults = '## No Safety Vulnerabilities Found\\n';
+
+            // Renders one tool's findings as a section. `items` is already
+            // the list of pre-formatted "- `thing` in `where`" strings; this
+            // just handles the found/not-found/report-missing framing and
+            // collapses long lists into a <details> block so the comment
+            // doesn't turn into a wall of text.
+            function renderSection(title, reportPath, parse) {
+              let data;
+              try {
+                data = JSON.parse(fs.readFileSync(reportPath, 'utf8'));
+              } catch (e) {
+                return [
+                  `### ${title}`,
+                  `⚠️ No report found at \`${reportPath}\` — the scan may have failed before producing output. Check the job logs.`,
+                ].join('\n');
               }
-            } catch (e) {
-              safetyResults = '## Safety scan completed\\n';
-            }
-            
-            // Read bandit report
-            let banditResults = '';
-            try {
-              const banditData = JSON.parse(fs.readFileSync('bandit-report.json', 'utf8'));
-              if (banditData.results && banditData.results.length > 0) {
-                const highIssues = banditData.results.filter(issue => issue.issue_severity === 'HIGH');
-                if (highIssues.length > 0) {
-                  banditResults = `## High Severity Security Issues Found\\n`;
-                  highIssues.forEach(issue => {
-                    banditResults += `- **${issue.test_name}**: ${issue.filename}:${issue.line_number}\\n`;
-                  });
-                } else {
-                  banditResults = '## No High Severity Security Issues Found\\n';
-                }
-              } else {
-                banditResults = '## No Bandit Issues Found\\n';
+
+              const items = parse(data);
+              if (items.length === 0) {
+                return [`### ${title}`, `✅ No findings.`].join('\n');
               }
-            } catch (e) {
-              banditResults = '## Bandit scan completed\\n';
-            }
-            
-            // Read semgrep report
-            let semgrepResults = '';
-            try {
-              const semgrepData = JSON.parse(fs.readFileSync('semgrep-report.json', 'utf8'));
-              if (semgrepData.results && semgrepData.results.length > 0) {
-                semgrepResults = `## Security Patterns Found\\n`;
-                semgrepData.results.slice(0, 10).forEach(issue => {
-                  semgrepResults += `- **${issue.rule_id}**: ${issue.path}\\n`;
-                });
-                if (semgrepData.results.length > 10) {
-                  semgrepResults += `- ... and ${semgrepData.results.length - 10} more\\n`;
-                }
+
+              const lines = [`### ${title}`, `Found **${items.length}**.`, ''];
+              const shown = items.slice(0, 15);
+              if (items.length > 15) {
+                lines.push('<details>', '<summary>Show all findings</summary>', '');
+                lines.push(...items);
+                lines.push('', '</details>');
               } else {
-                semgrepResults = '## No Security Patterns Found\\n';
+                lines.push(...shown);
               }
-            } catch (e) {
-              semgrepResults = '## Semgrep scan completed\\n';
+              return lines.join('\n');
             }
-            
-            // Create summary comment
-            const comment = `# 🔒 Security Scan Results\\n\\n${safetyResults}\\n\\n${banditResults}\\n\\n${semgrepResults}\\n\\n---\\n\\n*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*\\n\\n📊 **Security Policy**: CI fails on vulnerabilities and HIGH severity issues.`;
-            
-            // Post comment with error handling
+
+            const safetySection = renderSection(
+              'Safety — dependency vulnerabilities',
+              'safety-report.json',
+              (data) => (data.vulnerabilities || []).map(
+                (v) => `- \`${v.package_name}==${v.analyzed_version}\`: ${v.vulnerability_id}` +
+                  (v.CVE ? ` (${v.CVE})` : '') + ` — ${v.advisory || 'no advisory text'}`
+              )
+            );
+
+            const banditSection = renderSection(
+              'Bandit — HIGH-severity code issues',
+              'bandit-report.json',
+              (data) => (data.results || [])
+                .filter((issue) => issue.issue_severity === 'HIGH')
+                .map((issue) => `- \`${issue.test_name}\` in \`${issue.filename}:${issue.line_number}\``)
+            );
+
+            const semgrepSection = renderSection(
+              'Semgrep — static analysis patterns',
+              'semgrep-report.json',
+              (data) => (data.results || []).map(
+                (issue) => `- \`${issue.check_id}\` in \`${issue.path}:${issue.start?.line ?? '?'}\``
+              )
+            );
+
+            const comment = [
+              '# 🔒 Security Scan Results',
+              '',
+              safetySection,
+              '',
+              banditSection,
+              '',
+              semgrepSection,
+              '',
+              '---',
+              '',
+              '*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*',
+              '',
+              '📊 **Security Policy**: CI fails on Safety vulnerabilities and Bandit HIGH-severity findings. Semgrep findings above are informational and do not block merge.',
+            ].join('\n');
+
             try {
               await github.rest.issues.createComment({
                 issue_number: context.issue.number,
                 owner: context.repo.owner,
                 repo: context.repo.repo,
-                body: comment
+                body: comment,
               });
               console.log('✅ Security comment posted successfully');
             } catch (error) {

.github/workflows/security.yml

@@ -12,8 +12,8 @@ jobs:
   audit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
           python-version: '3.11'
       - run: pip install pip-audit

.github/workflows/verify-action-pins.yml

@@ -0,0 +1,28 @@
+name: Verify Action Pins
+
+on:
+  pull_request:
+    paths:
+      - '.github/workflows/**'
+      - '.github/scripts/verify-action-pins.sh'
+  push:
+    branches: [main]
+    paths:
+      - '.github/workflows/**'
+      - '.github/scripts/verify-action-pins.sh'
+  schedule:
+    - cron: '0 3 * * 1'  # weekly, in case an upstream tag is deliberately moved
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - name: Verify pinned action SHAs match their tag comments
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: bash .github/scripts/verify-action-pins.sh

ARCHITECTURE.md

@@ -0,0 +1,108 @@
+# Semantica — Architecture
+
+Complete data flow from every source type to every final output, and the decision intelligence lifecycle.
+
+---
+
+## Full Data Pipeline
+
+Every source, every processing step, every final artifact — in one diagram.
+
+```mermaid
+flowchart TD
+    %% ── SOURCES ──────────────────────────────────────────────────────
+    subgraph SRC["🗂️ Sources  (semantica.ingest)"]
+        direction LR
+        F["📄 Files\nPDF · DOCX · PPTX · HTML\nTXT · CSV · JSON · Excel · XML"]
+        W["🌐 Web\nPages · RSS/Atom Feeds\nPublic REST APIs"]
+        DB["🗃️ Databases\nPostgreSQL · MySQL · SQLite\nOracle · DuckDB · MongoDB"]
+        CL["☁️ Cloud\nSnowflake · Google Drive\nElasticsearch · HuggingFace"]
+        RT["⚡ Streams\nKafka · RabbitMQ\nAWS Kinesis · Pulsar"]
+        DV["🛠️ Dev\nGit Repos · Email IMAP/POP3\nMCP Resources · Parquet · Pandas"]
+    end
+
+    %% ── INGEST ───────────────────────────────────────────────────────
+    F  --> FI["FileIngestor"]
+    W  --> WI["WebIngestor"]
+    DB --> DI["DBIngestor"]
+    CL --> PI["ParquetIngestor\nSnowflakeIngestor"]
+    RT --> SI["StreamIngestor"]
+    DV --> RI["RepoIngestor\nEmailIngestor · MCPIngestor"]
+
+    FI & WI & DI & PI & SI & RI --> RAW[/"📦 Raw Documents"/]
+
+    %% ── PARSE ────────────────────────────────────────────────────────
+    RAW --> PRS["🔍 Parse  (semantica.parse)\nDocumentParser · StructuredDataParser\nCodeParser · WebParser · EmailParser"]
+
+    PRS --> NRM["🧹 Normalize  (semantica.normalize)\nTextNormalizer · EntityNormalizer\nDateNormalizer · NumberNormalizer · DataCleaner"]
+
+    NRM --> SPL["✂️ Split  (semantica.split)\nentity_aware · relation_aware\ngraph_based · ontology_aware · hierarchical"]
+
+    %% ── EXTRACT ──────────────────────────────────────────────────────
+    SPL --> EXT["🔬 Extract  (semantica.semantic_extract)\nNamedEntityRecognizer · RelationExtractor\nEventDetector · TripletExtractor · CoreferenceResolver"]
+
+    EXT --> CFT["⚠️ Conflict Detection  (semantica.conflicts)\nConflictDetector · ConflictResolver · SourceTracker"]
+
+    CFT --> DDP["🔁 Deduplication  (semantica.deduplication)\nDuplicateDetector · EntityMerger"]
+
+    DDP --> KGB["🕸️ KG Construction  (semantica.kg)\nGraphBuilder · EntityResolver\nBiTemporalFact · TemporalGraphQuery"]
+
+    KGB --> KG[/"🗺️ Knowledge Graph\nnodes · edges · temporal facts · provenance"/]
+
+    %% ── INTELLIGENCE LAYER ───────────────────────────────────────────
+    KG --> ONT["Ontology  (semantica.ontology)\nOntologyGenerator · OntologyValidator\nOWL · SHACL · SKOS"]
+    KG --> RSN["Reasoning  (semantica.reasoning)\nReteEngine · DatalogReasoner\nSPARQLReasoner · ExplanationGenerator"]
+    KG --> PRV["Provenance  (semantica.provenance)\nProvenanceManager · W3C PROV-O"]
+    KG --> CTX["Context & Decisions  (semantica.context)\nContextGraph · AgentContext\nDecisionRecorder · CausalChainAnalyzer · PolicyEngine"]
+
+    ONT & RSN & PRV & CTX --> EKG[/"🗃️ Enriched KG\n+ ontology · inferences · provenance · decisions"/]
+
+    %% ── STORAGE ──────────────────────────────────────────────────────
+    EKG --> VS["Vector Store  (semantica.vector_store)\nFAISS · Qdrant · Weaviate · Milvus · Pinecone · PgVector\nHybrid Search · RRF Fusion"]
+    EKG --> GS["Graph Store  (semantica.graph_store)\nNeo4j · FalkorDB · Apache AGE · Amazon Neptune"]
+
+    %% ── OUTPUTS ──────────────────────────────────────────────────────
+    VS & GS --> EXP["📦 Export  (semantica.export)\nRDF Turtle · JSON-LD · N-Triples · OWL · SHACL\nParquet · Cypher · ArangoDB AQL · GraphML · CSV · HTML"]
+    VS & GS --> VIZ["📊 Visualize  (semantica.visualization)\nKGVisualizer · OntologyVisualizer\nEmbeddingVisualizer · TemporalVisualizer"]
+    EKG --> SVC["🔌 Services\nREST API 100+ endpoints · MCP Server 10+ tools\nCLI 50+ commands · Knowledge Explorer"]
+```
+
+---
+
+## Decision Intelligence Lifecycle
+
+```mermaid
+flowchart LR
+    subgraph RECORD["1️⃣  Record"]
+        R1["record_decision()\ncategory · scenario\nreasoning · outcome\nconfidence · metadata"]
+    end
+
+    subgraph LINK["2️⃣  Link"]
+        L1["add_causal_relationship()\ntriggers · enables\ncauses · precedes"]
+    end
+
+    subgraph QUERY["3️⃣  Query"]
+        Q1["find_similar_decisions()\nSemantic precedent search"]
+        Q2["trace_decision_chain()\nFull causal ancestry"]
+        Q3["analyze_decision_impact()\nDownstream influence map"]
+    end
+
+    subgraph GOVERN["4️⃣  Govern"]
+        G1["check_decision_rules()\nPolicy evaluation\nCompliance gate"]
+    end
+
+    subgraph AUDIT["5️⃣  Audit Export"]
+        A1["W3C PROV-O  ·  CSV  ·  JSON\nRegulator-ready audit trail"]
+    end
+
+    RECORD -->|decision_id| LINK
+    LINK   -->|causal graph| QUERY
+    QUERY  -->|results| GOVERN
+    GOVERN -->|signed-off decisions| AUDIT
+```
+
+---
+
+*→ [README](README.md) · [Docs](https://docs.getsemantica.ai/) · [Cookbook](https://github.com/semantica-agi/semantica/tree/main/cookbook)*
+
+> Note: `Docs` and `Cookbook` are external resources maintained outside this file and may change over time. If a link is unavailable, refer to the repository `README.md` and in-repo documentation as canonical fallbacks.

CONTRIBUTING.md

@@ -19,6 +19,30 @@ Thank you for your interest in contributing! Every contribution, no matter how s
 
 ---
 
+## 🗂️ Working on an Existing Issue
+
+If you want to work on an open GitHub issue, please follow these steps to keep things coordinated and avoid duplicate effort:
+
+1. **Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome.
+
+2. **Comment before you start.** Leave a comment on the issue saying you'd like to work on it — something like *"I'd like to take this on"* is enough. This gives maintainers the context they need to assign the issue appropriately.
+
+3. **Wait for assignment.** A maintainer will review the request and assign the issue when appropriate. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
+
+4. **Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work.
+
+   ```bash
+   git checkout -b fix/short-description   # or feature/short-description
+   ```
+
+5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue.
+
+> **Why this matters:** Commenting before opening a PR helps maintainers track who is working on what, assign issues correctly, and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
+
+Not sure where to start? Try a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH).
+
+---
+
 ## 🎯 Ways to Contribute
 
 ### 💻 Code

Dockerfile

@@ -0,0 +1,40 @@
+# syntax=docker/dockerfile:1
+FROM node:26-alpine AS frontend-builder
+
+WORKDIR /app
+COPY explorer/package*.json ./explorer/
+WORKDIR /app/explorer
+RUN npm ci
+
+COPY explorer/ ./
+RUN mkdir -p /app/semantica && npm run build
+
+FROM python:3.14-slim AS runtime
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    FALKORDB_HOST=falkordb \
+    FALKORDB_PORT=6379 \
+    ALLOWED_ORIGINS=http://localhost:8000,http://127.0.0.1:8000
+
+WORKDIR /app
+
+RUN groupadd --system semantica \
+    && useradd --system --gid semantica --home-dir /app --shell /usr/sbin/nologin semantica
+
+COPY pyproject.toml README.md LICENSE MANIFEST.in ./
+COPY semantica/ ./semantica/
+COPY integrations/ ./integrations/
+COPY --from=frontend-builder /app/semantica/static ./semantica/static
+
+RUN pip install --no-cache-dir ".[explorer]" \
+    && chown -R semantica:semantica /app
+
+USER semantica
+
+EXPOSE 8000
+
+HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
+    CMD python -c "import json, urllib.request; data=json.load(urllib.request.urlopen('http://127.0.0.1:8000/api/health', timeout=3)); raise SystemExit(0 if data.get('status') == 'ok' else 1)"
+
+CMD ["python", "-m", "uvicorn", "semantica.explorer.app:app", "--host", "0.0.0.0", "--port", "8000"]

MANIFEST.in

@@ -0,0 +1 @@
+recursive-include semantica/static *

RELEASE_NOTES.md

@@ -1,282 +1,186 @@
-# Semantica v0.3.0 — Release Notes
+# Semantica 0.5.0 Release Notes
 
-**Released:** 2026-03-10
-**PyPI:** `pip install semantica`
-**Tag:** [v0.3.0](https://github.com/Hawksight-AI/semantica/releases/tag/v0.3.0)
-**Classification:** Production/Stable
+## 🎉 Major Release: Distance Intelligence & Ontology Hub Complete
 
-> First stable, full public release of Semantica. Covers everything shipped across three release stages: 0.3.0-alpha (2026-02-19), 0.3.0-beta (2026-03-07), and 0.3.0 stable (2026-03-10).
+**Release Date:** May 11, 2026  
+**Version:** 0.5.0  
 
 ---
 
-## Contributors
+## 🚀 **MAJOR HIGHLIGHTS**
 
-| Contributor | Role |
-|------------|------|
-| [@KaifAhmad1](https://github.com/KaifAhmad1) | Lead maintainer — context graph, decision intelligence, KG algorithms, semantic extraction, pipeline, provenance, bug fixes, release management |
-| [@ZohaibHassan16](https://github.com/ZohaibHassan16) | Deduplication v2 suite (candidate generation, two-stage scoring, semantic dedup), incremental/delta processing, benchmark suite |
-| [@Sameer6305](https://github.com/Sameer6305) | Apache AGE backend, PgVector store, Snowflake connector, Apache Arrow export |
-| [@tibisabau](https://github.com/tibisabau) | ArangoDB AQL export, Apache Parquet export |
-| [@d4ndr4d3](https://github.com/d4ndr4d3) | ResourceScheduler deadlock fix |
+### **Distance Intelligence Framework** (PR #502, @KaifAhmad1)
+- **Embedding Cache Optimization**: Per-session graph revision-based caching for 10x+ performance improvement
+- **Advanced UI Features**: Ego mode, overlays, heatmap, and path inspector
+- **Semantic Neighborhood Search**: Context-aware similarity with proximity metrics
+- **Distance Matrix API**: N×N semantic distance calculations with caching
+
+### **Complete Ontology Hub Suite** (PR #517, @KaifAhmad1 @ZohaibHassan16)
+- **Alignments Tab** (PR #524): Cross-ontology alignment authoring with ML suggestions
+- **Health Dashboard** (PR #524): Quality scoring across 5 dimensions with issue tracking
+- **SHACL Studio** (PR #524): Interactive shape generation and validation
+- **Visual Editor** (PR #519): Canvas-based ontology authoring without hand-coding
+- **Registry & Search** (PR #518): Comprehensive ontology management and discovery
+
+### **Security Hardening** (Security Enhancement PR, @KaifAhmad1)
+- **12 Critical Vulnerabilities Fixed**: Eval injection, XXE, SQL injection, and more
+- **SSRF Protection**: Comprehensive URL validation and hostname resolution
+- **Input Validation**: Enhanced file upload restrictions and format detection
+- **CORS & Headers**: Proper security headers and WebSocket protection
 
 ---
 
-## v0.3.0 — Stable (2026-03-10)
+## 📊 **BY THE NUMBERS**
 
-### Context Graph Feature Completeness
-
-**Temporal Validity Windows** (by @KaifAhmad1)
-
-Nodes and edges now carry first-class `valid_from` / `valid_until` ISO datetime fields. These are stored directly on `ContextNode` and `ContextEdge` dataclasses — not in metadata — and survive full serialisation round-trips through `save_to_file()` / `load_from_file()` and `to_dict()` / `from_dict()`.
-
-- `ContextNode.is_active(at_time=None)` and `ContextEdge.is_active(at_time=None)` — returns `True` if the node/edge is live at the given time (defaults to now). Handles both tz-aware and tz-naive datetime inputs correctly.
-- `ContextGraph.find_active_nodes(node_type=None, at_time=None)` — filters the entire graph and returns only nodes within their validity window.
-- `add_node(valid_from=..., valid_until=...)` and `add_edge(valid_from=..., valid_until=...)` — pass validity fields directly in the call signature.
-- Bug fix: `is_active()` previously crashed with `TypeError` when passed a tz-aware `datetime` (e.g. `datetime.now(timezone.utc)`). Fixed by normalising all inputs to tz-naive UTC via a new `_parse_iso_dt()` helper.
-- Bug fix: validity fields were silently lost in `add_nodes()`, `add_edges()`, `to_dict()`, and `from_dict()`. All four paths now correctly preserve and restore them.
-
-**Weighted Multi-Hop BFS** (by @KaifAhmad1)
-
-`ContextGraph.get_neighbors(node_id, hops=1, relationship_types=None, min_weight=0.0)` now accepts a `min_weight` threshold. Any edge with weight below the threshold is skipped during BFS traversal, allowing callers to confine multi-hop queries to high-confidence causal links. Default `0.0` is fully backward-compatible.
-
-**Cross-Graph Navigation** (by @KaifAhmad1)
-
-Separate `ContextGraph` instances can now be linked and navigated between — hierarchically, like separate knowledge domains that reference each other.
-
-- `link_graph(other_graph, source_node_id, target_node_id, link_type="CROSS_GRAPH") -> str` — creates a navigable bridge and returns a `link_id`. Records a dedicated `"cross_graph_link"` typed marker node internally (not a phantom `"entity"`) and a marker edge.
-- `navigate_to(link_id) -> (other_graph, target_node_id)` — jumps to the target graph and entry node for a given link.
-- `graph_id` field — each `ContextGraph` now carries a stable UUID so instances can identify each other across save/load.
-- `save_to_file()` — now writes a `links` section alongside nodes and edges, containing `link_id`, `source_node_id`, `target_node_id`, and `other_graph_id` for every cross-graph link.
-- `load_from_file()` — restores `graph_id` and populates `_unresolved_links` from the `links` section.
-- `resolve_links(registry: Dict[str, ContextGraph]) -> int` — reconnects unresolved links post-load. Pass `{graph_id: graph_instance}` for each linked graph; returns the count of successfully resolved links. `navigate_to()` raises a clear `KeyError` with a `resolve_links()` hint if called before resolution.
-- Bug fix: the previous implementation auto-created the synthetic marker target as an `"entity"` node (phantom pollution). Fixed by explicitly pre-creating a `"cross_graph_link"` typed `ContextNode` before the marker edge.
-- 14 new tests in `tests/context/test_cross_graph_navigation.py` covering all scenarios including full save/load round-trips with partial registry resolution.
-
-**Other Fixes** (by @KaifAhmad1)
-
-- `PipelineBuilder.add_step()` return type annotation corrected from `"PipelineBuilder"` to `"PipelineStep"` — the implementation was already correct; only the annotation and docstring were stale.
-- `test_hybrid_search_performance` timing computation fixed — now accumulates a true `search_times` list instead of reusing the last loop iteration's `start_time`; threshold relaxed to `< 5.0s` for real `sentence-transformers` (384-dim) latency on development machines.
-
-**Test Coverage Added**
-
-- 14 cross-graph navigation tests (`tests/context/test_cross_graph_navigation.py`)
-- **Total: 335 context tests, 886+ tests across all modules — 0 failures**
+- **12 Major Features** ✅ Tested & Verified
+- **16 Ontology Hub API Endpoints** ✅ Production Ready  
+- **57 New Distance Intelligence Tests** ✅ All Passing
+- **32 Parquet Ingestion Tests** ✅ All Passing
+- **12 Security Vulnerabilities** ✅ All Patched
+- **100% Test Coverage** ✅ Core Features Verified
 
 ---
 
-## v0.3.0-beta — Beta (2026-03-07)
+## 🔧 **NEW FEATURES**
 
-### Semantic Extraction Fixes
+### **Performance & Architecture**
+- **Distance Intelligence Embedding Cache** (PR #502, @KaifAhmad1): Thread-safe per-session caching with automatic invalidation
+- **Parquet File Ingestion** (PR #548, @Luffy2208): PyArrow backend with column selection and partition support
+- **Indexed Search** (PR #481, @ZohaibHassan16): O(log n) search for large graphs (118k nodes: 24ms → 0.004ms)
 
-**Multi-Founder LLM Extraction & Reasoner Inference Fix** (PR #354, by @KaifAhmad1)
+### **Ontology Hub Suite**
+- **Cross-ontology Alignments** (PR #524, @KaifAhmad1 @ZohaibHassan16): ML-powered suggestions with confidence scoring
+- **Quality Health Dashboard** (PR #524, @KaifAhmad1 @ZohaibHassan16): 5-dimension scoring with actionable issue tracking
+- **SHACL Studio** (PR #524, @KaifAhmad1 @ZohaibHassan16): Interactive shape authoring with Monaco editor
+- **Visual Ontology Editor** (PR #519, @KaifAhmad1): Drag-and-drop ontology construction
+- **16 Backend Endpoints** (PRs #518, #519, #524, @KaifAhmad1 @ZohaibHassan16): Complete CRUD and analysis capabilities
 
-- `_parse_relation_result` in `methods.py` — unmatched subjects/objects now produce a synthetic `UNKNOWN` entity instead of silently dropping the relation. All co-founders returned by the LLM are preserved in the output.
-- Duplicate relation fix — an orphaned legacy block that appended every relation twice has been removed.
-- `extraction_method` parameter added — typed extraction paths now correctly record `"llm_typed"` in relation metadata instead of `"llm"`.
-- `_match_pattern` in `reasoner.py` rewritten — splits patterns on `?var` placeholders first, then escapes only literal segments. Pre-bound variables resolve to exact literals, repeated variables use backreferences, non-greedy `.+?` prevents over-consumption of separators.
-- Added `tests/reasoning/test_reasoner.py` (4 tests) and `tests/semantic_extract/test_relation_extractor.py` (6 tests).
+### **UI & User Experience**
+- **Distance Intelligence UI** (PR #502, @KaifAhmad1 @ZohaibHassan16): Ego mode, overlays, heatmap, path inspector
+- **Explorer Redesign** (PR #516, @ZohaibHassan16): Modern hero section with live metrics
+- **Graph Workspace Declutter** (PR #483, @ZohaibHassan16): Improved visualization for dense graphs
+- **Bidirectional Path Finding** (PR #469, @KaifAhmad1): Undirected traversal support
 
-**TTL Export Alias Fix** (PR #355, by @KaifAhmad1)
+### **Platform Compatibility**
+- **Windows Installation Fixes** (PR #532, @KaifAhmad1): Removed faiss-gpu from [all], Unicode console support
+- **Cross-platform Dependencies** (PR #527, @ZohaibHassan16): Proper optional dependency management
+- **MCP Server Package Structure** (PR #541, @KaifAhmad1): Fixed pipx installation issues
 
-- `RDFExporter` now accepts `"ttl"`, `"nt"`, `"xml"`, `"rdf"`, and `"json-ld"` as format aliases in `export_to_rdf()`. Aliases resolve before format validation — zero public API changes.
-- Added `tests/export/test_rdf_exporter.py` (8 tests).
-
-### Incremental / Delta Processing
-
-**Native Delta Computation** (PR #349, by @ZohaibHassan16, reviewed and fixed by @KaifAhmad1)
-
-- Native SPARQL-based diff between graph snapshots — only changed triples flow through the pipeline.
-- `delta_mode` configuration in `PipelineBuilder` for near-real-time workloads.
-- Version snapshot management with graph URI tracking and metadata storage.
-- `prune_versions()` for automatic snapshot retention cleanup.
-- Bug fixes: corrected SPARQL variable order, fixed class references, resolved duplicate dictionary keys.
-
-### Deduplication v2
-
-**Candidate Generation v2** (PR #338, by @ZohaibHassan16)
-
-- New opt-in strategies: `blocking_v2` and `hybrid_v2`, replacing O(N²) pair enumeration.
-- Multi-key blocking with normalised token prefixes, type-aware keys, and optional phonetic (Soundex) blocking.
-- Deterministic `max_candidates_per_entity` budgeting with stable sorting.
-- **63.6% faster** in worst-case scenarios (0.259s → 0.094s for 100 entities).
-
-**Two-Stage Scoring Prefilter** (PR #339, by @ZohaibHassan16)
-
-- Fast gates for type mismatch, name-length ratio, and token overlap eliminate expensive semantic scoring for obvious non-matches.
-- Configurable thresholds: `min_length_ratio`, `min_token_overlap_ratio`, `required_shared_token`.
-- **18–25% faster** batch processing with prefilter enabled (`prefilter_enabled=False` by default).
-
-**Semantic Relationship Deduplication v2** (PR #340, by @ZohaibHassan16, fixes by @KaifAhmad1)
-
-- Canonicalisation engine with predicate synonym mapping (`works_for` → `employed_by`).
-- O(1) hash matching for exact canonical signatures.
-- Weighted scoring: 60% predicate + 40% object composition with explainable `semantic_match_score`.
-- **6.98x faster** than legacy mode (83ms vs 579ms).
-- `dedup_triplets()` infinite recursion bug fixed; function is now a first-class API in `methods.py`.
-
-**Deduplication v2 Migration Guide** (PR #344, by @ZohaibHassan16, fixes by @KaifAhmad1)
-
-- Comprehensive `MIGRATION_V2.md` documenting all v2 strategies with code examples.
-- Full backward compatibility maintained — legacy mode remains the default.
-
-### Export Formats
-
-**ArangoDB AQL Export** (PR #342, by @tibisabau)
-
-- Full AQL INSERT statement generation for vertices and edges.
-- Configurable collection names with validation and sanitisation; batch processing (default: 1000).
-- `export_arango()` convenience function; `.aql` auto-detection in the unified exporter.
-- 17 tests, 100% pass rate.
-
-**Apache Parquet Export** (PR #343, by @tibisabau)
-
-- Columnar storage format with configurable compression: snappy, gzip, brotli, zstd, lz4, none.
-- Explicit Apache Arrow schemas with type safety; field normalisation for varied naming conventions.
-- `export_parquet()` convenience function; `.parquet` auto-detection.
-- Analytics-ready for pandas, Spark, Snowflake, BigQuery, Databricks.
-- 25 tests, 100% pass rate.
-
-### Bug Fixes & Test Suite Stabilisation
-
-**Test Suite Fixes** (by @KaifAhmad1)
-
-Context module:
-- `retrieve_decision_precedents` — gated entity extraction on `use_hybrid_search=True` correctly.
-- `_extract_entities_from_query` — now uses `word[0].isupper()` to capture camelCase identifiers like `CreditCard`.
-- Added missing `expand_context` (BFS traversal) and `_get_decision_query` methods.
-- Fixed `hybrid_retrieval`, `dynamic_context_traversal`, and `multi_hop_context_assembly` for correct single-pass BFS.
-- Fixed `_retrieve_from_vector` fallback to `result["metadata"]["content"]` to prevent empty content and negative re-ranking scores.
-
-KG module:
-- `calculate_pagerank` — added `alpha`/`max_iter` aliases; return format changed to `{"centrality": scores, "rankings": sorted_list}`.
-- `community_detector._to_networkx` — now returns a NetworkX graph directly when one is passed (previously lost all edges).
-- Added 9 domain-specific tracking methods to `AlgorithmTrackerWithProvenance`.
-- Created `provenance_tracker.py` with `ProvenanceTracker` (`track_entity`, `get_all_sources`, `clear`).
-
-Pipeline module:
-- Retry loop fixed — now correctly iterates to `max_retries`.
-- `RecoveryAction` dataclass and `handle_failure(error, policy, retry_count)` added with LINEAR, EXPONENTIAL, and FIXED strategies.
-- `add_step()` fixed to return the created `PipelineStep`.
-- `validate` added as alias for `validate_pipeline` in `PipelineValidator`.
-
-Other:
-- Fixed `NameError` for missing `Type` import in `utils/helpers.py`.
-- Vector store performance threshold relaxed (100ms → 500ms per decision for development machines).
-- Windows cp1252 encoding fix in test files (emoji → ASCII).
-- `ProvenanceTracker` added to `semantica/kg/__init__.py` exports.
-
-**Results: ~840 tests passing, 36 skipped (external services), 0 failed**
+### **Algorithm Enhancements**
+- **DuplicateDetector Result Limiting** (PR #534, @KaifAhmad1): Ranking, sorting, and incremental detection features
+- **ConflictDetector Parameter Handling** (PR #533, @KaifAhmad1): Method parameter validation and error handling
 
 ---
 
-## v0.3.0-alpha — Alpha (2026-02-19)
+## 🛡️ **SECURITY IMPROVEMENTS** (Security Enhancement PR, @KaifAhmad1)
 
-### Context & Decision Intelligence
+### **Critical Fixes**
+- **Eval Injection** (CWE-95): Replaced with `fractions.Fraction` in media parser
+- **Pickle Deserialization** (CWE-502): Switched to JSON with migration support
+- **SQL Injection** (CWE-89): Parameterized queries and input validation
+- **XXE Protection** (CWE-611): `defusedxml` hardening for all RDF parsing
 
-**Context Engineering Enhancement** (PR #307, by @KaifAhmad1)
+### **Web Security**
+- **SSRF Protection**: URL validation with hostname resolution
+- **CORS Hardening**: Narrowed origins and WebSocket limits
+- **Security Headers**: HSTS, X-Content-Type-Options, X-Frame-Options
+- **Path Traversal**: `Path.resolve().relative_to()` protection
 
-The foundational 0.3.0 feature — complete overhaul of the context module for production-grade decision intelligence:
-
-- Full decision lifecycle: `record_decision()` → `add_decision()` → `add_causal_relationship()` → `trace_decision_chain()` → `analyze_decision_impact()` → `analyze_decision_influence()` → `find_similar_decisions()`
-- `AgentContext` unified wrapper with granular feature flags: `decision_tracking`, `kg_algorithms`, `graph_expansion`; methods: `store()`, `retrieve()`, `get_conversation_history()`, `get_statistics()`, `capture_cross_system_inputs()`
-- `AgentMemory` with working, conversation, and long-term memory tiers
-- `PolicyEngine` with versioned policy nodes, compliance checking (`check_decision_rules()`), and `PolicyException` model
-- Hybrid precedent search combining vector, structural, and category similarity with configurable weights
-- Decision influence analysis via centrality measures and causal chain tracking
-- GraphStore validation preventing runtime failures; secure logging
-- 9 critical bug fixes across logging, security, audit trails, API compatibility, Cypher queries, centrality access, validation
-
-**Context Decision Tracking Fixes** (PR #315, by @KaifAhmad1)
-
-- Fixed empty/None decision ID handling in `add_decision()`
-- Fixed None metadata handling preventing `TypeError`
-- Fixed causal chain depth logic and node exclusion
-- Fixed nonexistent node handling in `add_causal_relationship()`
-- Fixed precedent search direction in `find_precedents()`
-- Added missing `properties` field in `to_dict()`; added `from_dict()` method
-- Fixed UUID generation across all decision models
-- All 71 context tests passing
-
-### Knowledge Graph Algorithms
-
-**Improved Graph Algorithms** (PR #292, by @KaifAhmad1)
-
-- 30+ graph algorithms across 7 categories
-- Node embeddings: Node2Vec, DeepWalk, Word2Vec via `NodeEmbedder`
-- Similarity: cosine, Euclidean, Manhattan, Correlation via `SimilarityCalculator`
-- Path finding: Dijkstra, A*, BFS, K-shortest paths via `PathFinder`
-- Link prediction: preferential attachment, Jaccard, Adamic-Adar via `LinkPredictor`
-- Centrality: degree, betweenness, closeness, PageRank via `CentralityAnalyzer`
-- Community detection: Louvain, Leiden, label propagation via `CommunityDetector`
-- Connectivity: components, bridges, density via `ConnectivityAnalyzer`
-- `GraphBuilderWithProvenance` and `AlgorithmTrackerWithProvenance` with full execution metadata
-
-**Improved Vector Store for Decision Tracking** (PR #293, by @KaifAhmad1)
-
-- `DecisionEmbeddingPipeline` with semantic and structural embeddings
-- `HybridSimilarityCalculator` with configurable weights (semantic: 0.7, structural: 0.3)
-- `ContextRetriever` with multi-hop reasoning
-- Convenience API: `quick_decision()`, `find_precedents()`, `explain()`, `similar_to()`, `batch_decisions()`, `filter_decisions()`
-- 34+ tests; performance: 0.028s per decision, 0.031s search, ~0.8KB memory per decision
-
-### Graph Database Backends
-
-**Apache AGE Backend Security Fixes** (PR #311, by @Sameer6305, fixes by @KaifAhmad1)
-
-- `AgeStore` class with `GraphStore` API compatibility (openCypher via SQL on PostgreSQL)
-- SQL injection vulnerabilities fixed with input validation
-- psycopg2-binary dependency and migration guide added
-- Fixed parameter replacement and test mock leakage
-
-**PgVector Store Support** (PR #303, by @Sameer6305, @KaifAhmad1)
-
-- Native PostgreSQL vector storage using the pgvector extension
-- Multiple distance metrics: cosine, L2/Euclidean, inner product
-- HNSW and IVFFlat indexing for approximate nearest neighbour search
-- JSONB metadata storage with flexible filtering; batch operations
-- Connection pooling with psycopg3/psycopg2 fallback
-- SQL injection protection via `psycopg_sql.SQL()`; idempotent index and table management
-- 36+ tests with Docker integration
-
-### Infrastructure
-
-**ResourceScheduler Deadlock Fix** (PR #299, #301, by @d4ndr4d3, @KaifAhmad1)
-
-- Replaced `threading.Lock()` with `threading.RLock()` to fix nested lock acquisition deadlock in `allocate_resources()`
-- Added `ValidationError` when no resources can be allocated
-- Progress tracking updates moved outside lock scope
-- 6 regression tests for deadlock prevention
-
-**Security Configuration** (by @KaifAhmad1)
-
-- Dependabot bi-weekly security updates with manual review
-- Automated security scans (Bandit, Safety, Semgrep) on schedule
-- Security-critical package grouping; zero auto-merge policy
+### **Input Validation**
+- **File Upload Restrictions**: Extension allowlist and size limits
+- **SPARQL Limits**: Row caps, timeouts, and concurrency controls
+- **ReDoS Prevention**: Eliminated polynomial regex patterns
 
 ---
 
-## Summary by the Numbers
+## 🔍 **QUALITY ASSURANCE**
 
-| Metric | Value |
-|--------|-------|
-| Total tests passing | **886+** |
-| Test failures | **0** |
-| Context tests | 335 |
-| KG tests | ~430 |
-| Semantic extraction tests | 70 (9 skipped — external LLM APIs) |
-| Reasoning tests | 19 |
-| Real-world scenario tests | 85 |
-| PyPI classifier | Production/Stable |
-| Python support | 3.8 – 3.12 |
+### **Testing Coverage**
+- **Distance Intelligence**: 57 new tests, 100% passing
+- **Parquet Ingestion**: 32 tests, comprehensive coverage
+- **Security Fixes**: 14 vulnerability-specific tests
+- **UI Components**: All major features verified
+- **Platform Tests**: Windows, Linux compatibility confirmed
+
+### **Performance Benchmarks**
+- **Embedding Cache**: 10x+ improvement in repeated requests
+- **Search Performance**: 6,000x faster for large graphs
+- **Memory Efficiency**: Lazy loading and optional dependencies
+- **Concurrent Operations**: Thread-safe caching with locks
 
 ---
 
-## Upgrade
+## 🔄 **BREAKING CHANGES**
+
+### **Dependencies**
+- **Windows Users**: `faiss-gpu` removed from `[all]` - install `[gpu]` explicitly if needed
+- **Optional Dependencies**: Now lazy-loaded to improve import performance
+
+### **API Changes**
+- **ConflictDetector**: Fixed duplicate method definitions with proper parameter handling
+- **DuplicateDetector**: New result limiting and ranking options
+
+---
+
+## 📚 **DOCUMENTATION**
+
+- **Comprehensive Changelog**: Detailed feature descriptions and credits
+- **API Documentation**: All new endpoints documented
+- **Security Advisory**: Complete vulnerability disclosure and fixes
+- **Migration Guide**: Breaking changes and upgrade instructions
+
+---
+
+## 🙏 **CREDITS**
+
+**Core Contributors:**
+- **@KaifAhmad1** - Distance Intelligence (PR #502), Security Hardening, Ontology Hub (PRs #517, #518, #519, #524), Windows Fixes (PR #532), ConflictDetector (PR #533), Testing & Release Preparation
+- **@ZohaibHassan16** - Ontology Hub UI (PRs #516, #518, #519, #524), Graph Explorer (PRs #420, #481, #483, #503), Semantic Extract (PR #536), Lazy Loading (PR #535)
+- **@Luffy2208** - Parquet Ingestion Support (PR #548)
+- **@liling** - DeepSeek Provider Integration (PR #482)
+- **@Sameer6305** - Provenance Traversal Fixes (PR #480), Named Graph Support
+
+**Special Thanks:**
+- Security research team for vulnerability disclosures
+- Community testers and feedback providers
+- Documentation contributors and reviewers
+
+---
+
+## 🚀 **INSTALLATION**
 
 ```bash
-pip install --upgrade semantica
+# Standard installation
+pip install semantica==0.5.0
+
+# With all optional dependencies (cross-platform)
+pip install "semantica[all]==0.5.0"
+
+# With GPU acceleration (Linux only)
+pip install "semantica[gpu]==0.5.0"
+
+# With Parquet support
+pip install "semantica[ingest-parquet]==0.5.0"
 ```
 
-No breaking changes. All new parameters have safe defaults and all new methods are additive.
+---
 
-See [CHANGELOG.md](CHANGELOG.md) for the full line-by-line diff.
+## 📈 **WHAT'S NEXT FOR 0.5.0**
+
+The 0.5.0 release establishes Semantica as a production-ready framework for:
+
+- **Enterprise Knowledge Engineering** with comprehensive ontology management
+- **Advanced Analytics** through distance intelligence and semantic search  
+- **Security-First Design** with comprehensive vulnerability protection
+- **Cross-Platform Compatibility** supporting diverse deployment environments
+
+**Immediate next steps for 0.5.0:**
+- PyPI package publication and distribution
+- Docker image updates with new features
+- Documentation website deployment with updated guides
+- Community outreach and feature announcements
+- Integration testing across different deployment scenarios
+
+---
+
+**🎯 Semantica 0.5.0: Production-Ready Knowledge Engineering Platform**

SECURITY.md

@@ -24,7 +24,7 @@ Security vulnerabilities should be reported privately to prevent potential explo
 
 ### 2. Report Security Issue
 
-Create a [GitHub Security Advisory](https://github.com/Hawksight-AI/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix.
+Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us via the security email listed in `SUPPORT.md`.
 
 Include the following information:
 
@@ -37,7 +37,7 @@ Include the following information:
 
 ### 3. Response Timeline
 
-- **Initial Response**: Within 48 hours
+- **Initial Response**: Within 24 hours for critical issues; within 48 hours for non-critical issues
 - **Status Update**: Within 7 days
 - **Resolution**: Depends on severity and complexity
 
@@ -112,6 +112,101 @@ We regularly update dependencies to address security vulnerabilities. However, y
 - Be cautious with external API calls
 - Implement proper authentication and authorization
 
+## CI/CD Supply-Chain Security
+
+Semantica's build and release pipeline is explicitly hardened against
+CI/CD supply-chain attacks — the class of attack behind the March 2026
+LiteLLM/Trivy incident, where a compromised third-party Action with a
+**mutable tag** was used to steal a long-lived publishing token, after which
+malicious packages were pushed straight to PyPI without ever touching the
+source repository. Every control below maps directly to closing one step of
+that attack chain.
+
+### Immutable build inputs
+
+- **Risk**: a tag (`@v4`, `@release/v1`) is re-pointed by a compromised upstream maintainer or account, silently changing what every consumer's CI runs.
+  **Control**: every third-party GitHub Action in every workflow is pinned to a full 40-character commit SHA, with the human-readable tag kept only as a trailing comment (e.g. `actions/checkout@3d3c42e... # v7`).
+- **Risk**: a SHA pin drifts out of sync with its own comment over time, or is mistyped.
+  **Control**: `verify-action-pins.yml` fails closed on any `uses:` reference that isn't a full commit SHA (catching a newly added mutable tag, not just auditing existing pins), resolves every pinned tag via the GitHub API on each workflow change, on every push to `main`, and weekly, and fails if the SHA no longer matches the tag it claims to be — an API lookup that can't be resolved is treated as a failure, not a silent skip.
+- **Risk**: manually re-pinning ~15 actions across 8 workflow files on every upstream release is error-prone.
+  **Control**: Dependabot (`github-actions` ecosystem) opens a grouped PR that bumps the SHA *and* the tag comment together whenever an action releases — pins never require hand-editing.
+
+### Publishing pipeline (highest-privilege path)
+
+- **Risk**: a long-lived `PYPI_TOKEN` sitting in repo/org secrets is exfiltrated by any compromised step.
+  **Control**: PyPI publishing uses Trusted Publishing (OIDC) (`id-token: write`) — there is no long-lived PyPI credential anywhere in this repository to steal.
+- **Risk**: a compromised CI run publishes to PyPI with no human in the loop.
+  **Control**: the publish job runs only inside a protected `pypi` GitHub Environment with a required human reviewer — every release needs manual approval in the Actions UI before it runs.
+- **Risk**: the release job could be triggered from an arbitrary branch/ref.
+  **Control**: the `pypi` environment's deployment-branch policy is restricted to `v*` tags only.
+- **Risk**: a scanner or unrelated job inherits publish-level credentials.
+  **Control**: `release.yml` sets `permissions: contents: read` at the workflow level; `contents: write` / `id-token: write` / `attestations: write` are granted only to the release job, never workflow-wide.
+- **Risk**: two tag pushes race through the publish pipeline simultaneously.
+  **Control**: `concurrency: group: release-${{ github.ref }}` serializes releases per tag.
+- **Risk**: a consumer can't verify a wheel on PyPI actually came from this repo's CI.
+  **Control**: SLSA build provenance is attested for every release via `actions/attest-build-provenance`, producing a signed, verifiable record of the exact commit and workflow run that produced the artifact (checkable with `gh attestation verify`).
+
+### Repository controls
+
+- **Risk**: unreviewed or force-pushed changes land on `main`.
+  **Control**: `main` requires 1 approving PR review (stale approvals dismissed on new pushes), resolved conversations, and blocks force-pushes and branch deletion.
+- **Risk**: a PR merges without its security/CI checks passing.
+  **Control**: merges require the `build`, `Analyze Python` (CodeQL), and `security-scan` checks to pass, in strict mode (checks must be re-run against the latest `main`).
+- **Risk**: a compromised scanner job reaches secrets or write access.
+  **Control**: scanning jobs (`CodeQL`, `security-scan.yml`, `security.yml`, `defender-for-devops.yml`) run with read-only, least-privilege permissions (typically `contents: read` + `security-events: write` only) and never share a job, environment, or secret scope with the publish job.
+- **Risk**: secrets are committed accidentally.
+  **Control**: GitHub secret scanning and push protection are both enabled at the repository level, rejecting pushes that contain recognizable credential patterns before they land in history.
+
+## Automated Security Scanning
+
+Every scan below runs continuously in CI, not just at release time:
+
+- **CodeQL** (`security-and-quality` query pack) — Python source: injection, unsafe deserialization, and other code-level vulnerability classes. Runs in `codeql.yml` on every push/PR to `main` and weekly.
+- **Bandit** — Python-specific security anti-patterns (hardcoded secrets, unsafe `eval`/`pickle`, weak crypto, etc.); CI fails on any HIGH-severity finding. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
+- **Semgrep** (`p/security` ruleset) — cross-language static-analysis security patterns. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
+- **Safety** — known CVEs in Semantica's own installed dependencies, including optional LLM-provider extras such as LiteLLM; CI fails on any match. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
+- **pip-audit** — independent, PyPA-maintained vulnerability database cross-check against installed dependencies (Safety and pip-audit use different advisory sources, so both run). Runs in `security.yml` weekly.
+- **Microsoft Defender for DevOps** (`eslint`, `templateanalyzer`, `terrascan`) — JavaScript/TypeScript lint-security rules and infrastructure-as-code misconfigurations. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
+- **Checkov** — Kubernetes, Helm, Dockerfile, GitHub Actions, and secrets-pattern IaC scanning; results upload to the same Security tab as CodeQL. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
+- **GitGuardian** — secret-detection check on every pull request, installed as a GitHub App integration (not a repo-local workflow). Runs on every PR.
+- **GitHub secret scanning + push protection** — blocks known credential patterns before they're pushed, and continuously scans existing history. Platform-level, continuous.
+- **Dependabot** — version/security PRs for Python, Docker, and GitHub Actions dependencies, grouped where relevant to reduce review noise. Configured in `.github/dependabot.yml`, runs weekly for security-relevant packages and monthly for docs dependencies.
+- **`verify-action-pins.yml`** — enforces that every Action reference is a full commit SHA (failing on a newly introduced mutable tag) and confirms each SHA still matches the tag it claims to be. Runs on every workflow change, every push to `main`, and weekly.
+
+All SARIF-producing scanners (CodeQL, Checkov, Microsoft Defender) publish
+findings to the repository's **Security → Code scanning alerts** tab, giving
+a single audit trail across tools rather than scattered per-tool reports.
+
+### Adopting this posture in a fork or downstream deployment
+
+Teams standing up their own instance of Semantica, or forking it for an
+internal/regulated deployment, can reuse this posture directly:
+
+1. Keep Dependabot's `github-actions` ecosystem entry — it is what keeps
+   SHA pins current without manual maintenance.
+2. Re-run `verify-action-pins.yml` after re-pointing the repository's Actions
+   at your own mirrors, if you do so.
+3. If you publish your own PyPI package from a fork, configure your own
+   Trusted Publishing trust relationship on PyPI (Trusted Publishing is
+   scoped to a specific `owner/repo` + workflow filename) and your own
+   protected environment with your own required reviewers — these are not
+   transferable from this repository.
+4. Branch protection, environment protection, and repository secret
+   scanning are repository *settings*, not workflow files — cloning or
+   forking the repo does **not** copy them. They must be re-applied via
+   the GitHub UI or API on the new repository.
+5. GitHub secret scanning and push protection are repository settings that
+   don't carry over to a fork either — re-enable both under the new
+   repository's Security settings, not just Dependabot.
+6. GitGuardian runs as a GitHub App installation scoped to this specific
+   repository, not a workflow file — a fork gets no secret-detection
+   coverage from it until the app is installed separately on the new repo.
+7. CodeQL's `upload-sarif` step in `codeql.yml` only runs meaningfully if
+   Default Setup is *not* already enabled for the repository (it's designed
+   to skip gracefully otherwise) — check whether Default Setup or Advanced
+   Setup is active on the new repository and adjust expectations for where
+   CodeQL findings show up accordingly.
+
 ## Dependency Security Policy
 
 ### Regular Updates
@@ -156,8 +251,8 @@ We appreciate responsible disclosure. Security researchers who help us improve t
 
 For security-related questions or concerns:
 
-- **GitHub Issues**: [Create an issue](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix
-- **GitHub Security Advisories**: [Report vulnerability](https://github.com/Hawksight-AI/semantica/security/advisories/new)
+- **Private Reporting**: Please do not report vulnerabilities in public issues.
+- **GitHub Security Advisories**: [Report vulnerability](https://github.com/semantica-agi/semantica/security/advisories/new)
 
 ## Additional Resources
 

STRATEGIES_SUMMARY.md

@@ -1,105 +0,0 @@
-# Deduplication & Conflict Resolution Strategies Summary
-
-## Quick Reference by Use Case
-
-| Use Case | Deduplication Method | Merge Strategy | Conflict Detection | Conflict Resolution |
-|----------|---------------------|----------------|-------------------|---------------------|
-| **Finance** |
-| `01_Financial_Data_Integration_MCP` | `DuplicateDetector` (incremental) | `keep_highest_confidence` | `temporal` | `most_recent` |
-| `02_Fraud_Detection` | `ClusterBuilder` (graph_based) | `merge_all` | `logical` | `expert_review` |
-| **Biomedical** |
-| `01_Drug_Discovery_Pipeline` | `EntityResolver` (semantic) | - | `relationship` | `voting` |
-| `02_Genomic_Variant_Analysis` | `DuplicateDetector` (group) | `keep_most_complete` | `value` | `credibility_weighted` |
-| **Cybersecurity** |
-| `01_Real_Time_Anomaly_Detection` | `DuplicateDetector` (pairwise) | `keep_first` | `entity` | `first_seen` |
-| `02_Threat_Intelligence_Hybrid_RAG` | `EntityResolver` (exact) | - | `type` | `highest_confidence` |
-| **Blockchain** |
-| `01_DeFi_Protocol_Intelligence` | `DuplicateDetector` (group) | `keep_last` | `relationship` | `voting` |
-| `02_Transaction_Network_Analysis` | `ClusterBuilder` (hierarchical) | `keep_most_complete` | `temporal` | `most_recent` |
-| **Intelligence** |
-| `01_Criminal_Network_Analysis` | `EntityResolver` (fuzzy) | - | `value` | `credibility_weighted` |
-| `02_Intelligence_Analysis_Orchestrator_Worker` | `DuplicateDetector` (batch) | `merge_all` | `entity` | `voting` |
-| **Renewable Energy** |
-| `01_Energy_Market_Analysis` | `DuplicateDetector` (pairwise) | `keep_highest_confidence` | `temporal` | `most_recent` |
-| **Supply Chain** |
-| `01_Supply_Chain_Data_Integration` | `DuplicateDetector` (incremental) | `keep_most_complete` | `value` | `credibility_weighted` |
-
----
-
-## Strategy Rationale by Domain
-
-### Finance
-- **Financial Data Integration**: Incremental for streaming data; most_recent for time-sensitive financial data
-- **Fraud Detection**: Graph-based clustering for fraud groups; expert_review for fraud assessment
-
-### Biomedical
-- **Drug Discovery**: Semantic matching for drug compounds; voting for research source aggregation
-- **Genomic Variants**: Group method for related variants; credibility weighting for research sources
-
-### Cybersecurity
-- **Real-Time Anomaly**: Pairwise for real-time streams; keep_first for first detection priority
-- **Threat Intelligence**: Exact matching for IOCs; highest_confidence for threat classification
-
-### Blockchain
-- **DeFi Protocols**: Group method for related protocols; keep_last for latest protocol info
-- **Transaction Networks**: Hierarchical clustering for nested groups; temporal for time-sensitive data
-
-### Intelligence
-- **Criminal Networks**: Fuzzy matching for intelligence data; credibility weighting for intelligence sources
-- **Intelligence Analysis**: Batch for multi-source integration; merge_all to combine all intelligence sources
-
-### Renewable Energy
-- **Energy Markets**: Pairwise for real-time market data; most_recent for time-sensitive energy data
-
-### Supply Chain
-- **Supply Chain Integration**: Incremental for continuous updates; credibility weighting for supply chain sources
-
----
-
-## Method Distribution
-
-### Deduplication Methods (9 total)
-- `pairwise`: 2 notebooks (real-time processing)
-- `batch`: 3 notebooks (large datasets)
-- `incremental`: 2 notebooks (streaming/continuous)
-- `group`: 2 notebooks (related entities)
-- `graph_based` (ClusterBuilder): 2 notebooks (interconnected entities)
-- `hierarchical` (ClusterBuilder): 1 notebook (nested groups)
-- `exact` (EntityResolver): 1 notebook (exact matching)
-- `semantic` (EntityResolver): 2 notebooks (semantic similarity)
-- `fuzzy` (EntityResolver): 1 notebook (fuzzy matching)
-
-### Merge Strategies (5 total)
-- `keep_first`: 1 notebook (first detection priority)
-- `keep_last`: 1 notebook (latest information)
-- `keep_most_complete`: 5 notebooks (preserve all details)
-- `keep_highest_confidence`: 2 notebooks (most reliable data)
-- `merge_all`: 3 notebooks (combine all information)
-
-### Conflict Detection Methods (6 total)
-- `value`: 4 notebooks (property value conflicts)
-- `type`: 2 notebooks (type/classification conflicts)
-- `entity`: 2 notebooks (entity-wide conflicts)
-- `relationship`: 3 notebooks (relationship conflicts)
-- `temporal`: 3 notebooks (time-sensitive conflicts)
-- `logical`: 2 notebooks (logical inconsistencies)
-
-### Conflict Resolution Strategies (6 total)
-- `voting`: 5 notebooks (majority vote)
-- `credibility_weighted`: 4 notebooks (source credibility)
-- `most_recent`: 3 notebooks (latest data)
-- `first_seen`: 1 notebook (first detection)
-- `highest_confidence`: 2 notebooks (most confident)
-- `expert_review`: 1 notebook (manual review)
-
----
-
-## Key Patterns
-
-1. **Real-Time Systems**: Use `pairwise` + `keep_first` + `first_seen`
-2. **Time-Sensitive Data**: Use `temporal` + `most_recent`
-3. **Multi-Source Integration**: Use `batch` + `merge_all` + `voting`
-4. **Medical/Research**: Use `credibility_weighted` for authoritative sources
-5. **Fraud/Security**: Use `graph_based` + `logical` + `expert_review`
-6. **Exact Matching Required**: Use `exact` strategy (IOCs, identifiers)
-

SUPPORT.md

@@ -20,8 +20,8 @@ Start with our comprehensive documentation:
 
 **Best for**: General questions, feature discussions, and getting help
 
-- [Ask a question](https://github.com/Hawksight-AI/semantica/discussions/new?category=q-a)
-- [Browse discussions](https://github.com/Hawksight-AI/semantica/discussions)
+- [Ask a question](https://github.com/semantica-agi/semantica/discussions/new?category=q-a)
+- [Browse discussions](https://github.com/semantica-agi/semantica/discussions)
 
 #### Discord
 
@@ -33,8 +33,8 @@ Start with our comprehensive documentation:
 
 **Best for**: Bug reports and feature requests
 
-- [Report a bug](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
-- [Request a feature](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
+- [Report a bug](https://github.com/semantica-agi/semantica/issues/new?template=bug_report.md)
+- [Request a feature](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md)
 
 ### Before Asking
 
@@ -47,7 +47,7 @@ Start with our comprehensive documentation:
 
 ### Bug Reports
 
-Use our [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md) to report bugs.
+Use our [bug report template](https://github.com/semantica-agi/semantica/issues/new?template=bug_report.md) to report bugs.
 
 Include:
 - Clear description of the bug
@@ -58,7 +58,7 @@ Include:
 
 ### Feature Requests
 
-Use our [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md) to suggest features.
+Use our [feature request template](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md) to suggest features.
 
 Include:
 - Problem statement
@@ -71,7 +71,7 @@ Include:
 **Do NOT** create a public issue for security vulnerabilities.
 
 Instead:
-- Email: semantica-dev@users.noreply.github.com
+- Email: kaif@getsemantica.ai
 - Subject: [SECURITY] Brief description
 - See [Security Policy](SECURITY.md) for details
 
@@ -79,7 +79,7 @@ Instead:
 
 For enterprise support, custom development, or consulting:
 
-- **Email**: semantica-dev@users.noreply.github.com
+- **Email**: kaif@getsemantica.ai
 - **Subject**: [ENTERPRISE] Your request
 
 ## Response Times

benchmarks/.gitignore

@@ -1,75 +0,0 @@
---- Python Standards ---
-
-pycache/
-*.py[cod]
-*$py.class
-*.so
-.Python
-env/
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-
---- Virtual Environments ---
-
-.env
-.venv
-venv/
-ENV/
-
---- Benchmarks & Results ---
-
-Ignore all individual benchmark runs to avoid repository bloat
-
-benchmarks/results/run_*.json
-
-Ignore the .pytest_cache which can get quite large
-
-.pytest_cache/
-
-Ignore any temporary files created by benchmarks
-
-benchmarks/input_layer/*.txt
-
---- IMPORTANT: Keep the Baseline ---
-
-We want to track the 'gold standard' performance in Git
-
-!benchmarks/results/baseline.json
-
---- IDEs & Editors ---
-
-.idea/
-.vscode/
-*.swp
-*.swo
-.project
-.pydevproject
-.settings/
-
---- Jupyter Notebooks ---
-
-.ipynb_checkpoints
-
---- OS Specific ---
-
-.DS_Store
-Thumbs.db
-
---- Project Specific ---
-
-logs/
-*.log
-semantica.log

benchmarks/benchmark_results.md

@@ -1,343 +0,0 @@
-# Semantica Benchmark Suite Results
-
-## Executive Summary
-
-**Test Date**: February 7, 2026  
-**Total Benchmarks**: 138 passed, 1 skipped  
-**Test Duration**: 38 minutes 35 seconds  
-**Environment**: Windows 10, Intel i5-1135G7 @ 2.40GHz, Python 3.11.9  
-
-## Performance Overview
-
-| Module | Tests | Performance Grade | Status |
-|--------|-------|------------------|---------|
-| Input Layer | 6 | 🟢 Excellent | All passed |
-| Core Processing | 5 | 🟢 Excellent | All passed |
-| Context Memory | 2 | 🟢 Excellent | All passed |
-| Storage | 4 | 🟢 Excellent | All passed |
-| Ontology | 4 | 🟢 Excellent | All passed |
-| Export | 4 | 🟢 Excellent | All passed |
-| Visualization | 3 | 🟢 Excellent | All passed |
-| Quality Assurance | 2 | 🟢 Excellent | All passed |
-| Output Orchestration | 2 | 🟢 Excellent | All passed |
-| Context | 3 | 🟢 Excellent | All passed |
-
----
-
-## 📊 Detailed Benchmark Results
-
-### 🔄 Input Layer Benchmarks
-
-**Purpose**: Test document parsing, data ingestion, and text processing performance
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_json_parsing_throughput[1000]` | 27,365.2 | 36.54 | 35.62 | 40.13 | 0.99 | ✅ |
-| `test_json_parsing_throughput[5000]` | 5,541.6 | 180.45 | 165.73 | 194.32 | 11.42 | ✅ |
-| `test_csv_parsing_throughput[1000]` | 18,127.9 | 55.16 | 52.41 | 61.87 | 3.33 | ✅ |
-| `test_html_scraping_speed[100]` | 2,437.8 | 410.20 | 346.30 | 6,736.50 | 89.27 | ✅ |
-| `test_pdf_extraction_overhead[10]` | 9.36 | 106.84 | 11.63 | 91.87 | 62.48 | ✅ |
-| `test_python_ast_parsing` | 3,142.6 | 318.21 | 291.96 | 347.90 | 35.67 | ✅ |
-
-**Key Insights**:
-- JSON parsing scales linearly (5K items processed in 180ms)
-- HTML scraping shows high variance due to complexity
-- PDF extraction optimized for batch processing
-- AST parsing maintains sub-millisecond performance per operation
-
----
-
-### ⚙️ Core Processing Benchmarks
-
-**Purpose**: Test NER extraction, semantic analysis, and text processing algorithms
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_ner_ml_wrapper_overhead` | 2,480.3 | 403.18 | - | - | - | ✅ |
-| `test_ner_pattern_speed` | 1,440.1 | 694.42 | - | - | - | ✅ |
-| `test_ner_batch_throughput` | 2.33 | 429.70 | - | - | - | ✅ |
-| `test_similarity_calculation` | 3,142.6 | 318.21 | - | - | - | ✅ |
-| `test_clustering_algorithm` | 39.1 | 25,558.38 | 6,113.80 | 42,058.84 | 42,058.84 | ✅ |
-| `test_ner_ml_real_performance` | - | - | - | - | - | ⏭️ Skipped |
-
-**Key Insights**:
-- Pattern-based NER significantly outperforms ML approaches
-- Semantic clustering is computationally intensive (25s mean time)
-- Real spaCy ML test skipped due to mocked environment
-- Batch processing provides good throughput
-
----
-
-### 🧠 Context Memory Benchmarks
-
-**Purpose**: Test graph operations, memory storage, and retrieval logic
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_bfs_traversal_depth[1]` | 469.48 | 2.13 | 1.42 | 2.04 | 1.86 | ✅ |
-| `test_bfs_traversal_depth[2]` | 419.46 | 2.38 | 2.04 | 2.38 | 0.89 | ✅ |
-| `test_memory_storage_overhead` | 9.36 | 106.84 | 11.63 | 91.87 | 62.48 | ✅ |
-| `test_short_term_pruning` | 9.23 | 108.36 | 91.87 | 108.36 | 20.76 | ✅ |
-| `test_linking_operations` | 2,869.0 | 348.55 | 313.28 | 346.30 | 39.45 | ✅ |
-| `test_retrieval_logic[False]` | 2,437.8 | 410.20 | 347.90 | 410.20 | 89.27 | ✅ |
-| `test_retrieval_logic[True]` | 39.13 | 25,558.38 | 6,113.80 | 42,058.84 | 42,058.84 | ✅ |
-
-**Key Insights**:
-- BFS traversal scales linearly with graph depth
-- Memory storage optimized for batch operations
-- Retrieval pipeline maintains sub-millisecond performance for simple cases
-- Complex retrieval (with context) significantly increases processing time
-
----
-
-### 💾 Storage Layer Benchmarks
-
-**Purpose**: Test vector stores, triplet storage, and graph database operations
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_binary_raw_throughput` | 5.83 | 171.52 | 162.04 | 178.50 | 7.56 | ✅ |
-| `test_numpy_compression_speed[1000]` | 2.47 | 404.81 | 387.07 | 393.72 | 11.55 | ✅ |
-| `test_numpy_compression_speed[10000]` | 0.25 | 3,972.74 | 3,867.34 | 3,983.95 | 61.69 | ✅ |
-| `test_json_vector_overhead` | 0.66 | 1,504.93 | 1,471.47 | 1,443.15 | 29.39 | ✅ |
-| `test_triplet_conversion_overhead` | 87.71 | 11.40 | 5.51 | 157.91 | 21.54 | ✅ |
-| `test_bulk_loader_logic` | 2.03 | 492.98 | 304.90 | 40,477.30 | 2,084.37 | ✅ |
-
-**Key Insights**:
-- Binary vector storage is 8x faster than JSON serialization
-- Triplet conversion is highly optimized (11ms mean)
-- Bulk loading shows high variance due to retry logic
-- Vector compression scales linearly with data size
-
----
-
-### 🏗️ Ontology Benchmarks
-
-**Purpose**: Test ontology inference, serialization, and namespace management
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_property_inference_scaling[size0]` | 1,440.1 | 694.42 | 637.90 | - | 65.09 | ✅ |
-| `test_owl_xml_generation` | 516.92 | 1.93 | 1.02 | 1.93 | 1.42 | ✅ |
-| `test_rdf_serialization_formats[turtle]` | 457.77 | 2.18 | 1.90 | 2.18 | 0.48 | ✅ |
-| `test_rdf_serialization_formats[rdfxml]` | 357.26 | 2.80 | 2.23 | 2.80 | 0.79 | ✅ |
-| `test_owl_serialization_formats[xml]` | 85.55 | 11.69 | 8.51 | 11.69 | 5.73 | ✅ |
-| `test_owl_serialization_formats[turtle]` | 61.10 | 16.37 | 12.28 | 16.37 | 6.84 | ✅ |
-
-**Key Insights**:
-- RDF Turtle format is 2x faster than RDF/XML
-- OWL serialization efficient for large ontologies
-- Property inference is computationally intensive
-- XML formats show higher overhead than Turtle
-
----
-
-### 📤 Export Benchmarks
-
-**Purpose**: Test data export and serialization performance
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_json_parsing_throughput[1000]` | 27,365.2 | 36.54 | 35.62 | 40.13 | 0.99 | ✅ |
-| `test_csv_entity_export` | 18,127.9 | 55.16 | 52.41 | 61.87 | 3.33 | ✅ |
-| `test_json_parsing_throughput[5000]` | 5,541.6 | 180.45 | 165.73 | 194.32 | 11.42 | ✅ |
-| `test_yaml_serialization_overhead` | 2.33 | 429.70 | 357.29 | 429.70 | 68.83 | ✅ |
-| `test_graph_conversion_overhead[graphml]` | 62.16 | 16.09 | 10.74 | 16.09 | 16.84 | ✅ |
-| `test_graph_conversion_overhead[gexf]` | 55.43 | 18.04 | 15.80 | 18.04 | 1.82 | ✅ |
-
-**Key Insights**:
-- JSON export maintains excellent performance across data sizes
-- YAML serialization is slower but feature-rich
-- GraphML format is slightly faster than GEXF
-- Export performance scales linearly with data size
-
----
-
-### 📈 Visualization Benchmarks
-
-**Purpose**: Test graph visualization, analytics, and dashboard performance
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_network_evolution_frames` | 0.21 | 4,871.40 | 3,958.10 | 4,871.40 | 931.20 | ✅ |
-| `test_temporal_dashboard_assembly` | 0.11 | 9,209.90 | 3,327.40 | 9,209.90 | 5,644.20 | ✅ |
-| `test_graph_conversion_overhead[graphml]` | 62.16 | 16.09 | 10.74 | 16.09 | 16.84 | ✅ |
-| `test_graph_conversion_overhead[gexf]` | 55.43 | 18.04 | 15.80 | 18.04 | 1.82 | ✅ |
-
-**Key Insights**:
-- Complex visualizations are computationally expensive
-- Dashboard assembly suitable for periodic updates (not real-time)
-- Graph conversion is highly optimized
-- Network evolution requires significant processing time
-
----
-
-### 🔍 Quality Assurance Benchmarks
-
-**Purpose**: Test deduplication and conflict resolution algorithms
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_deduplication_algorithm` | 2.33 | 429.70 | 357.29 | 429.70 | 68.83 | ✅ |
-| `test_conflict_resolution` | 1,440.1 | 694.42 | 637.90 | - | 65.09 | ✅ |
-
-**Key Insights**:
-- Deduplication algorithms are efficient for batch processing
-- Conflict resolution maintains good performance
-- Both algorithms scale linearly with data size
-
----
-
-### 🎯 Output Orchestration Benchmarks
-
-**Purpose**: Test pipeline execution and parallelism performance
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_execution_pipeline_overhead` | 2,437.8 | 410.20 | 347.90 | 410.20 | 89.27 | ✅ |
-| `test_parallelism_scaling` | 39.13 | 25,558.38 | 6,113.80 | 42,058.84 | 42,058.84 | ✅ |
-
-**Key Insights**:
-- Pipeline execution maintains good performance
-- Parallelism scaling shows high variance due to threading overhead
-- Suitable for batch processing rather than real-time
-
----
-
-### 🔗 Context Benchmarks
-
-**Purpose**: Test graph operations and linking performance
-
-| Benchmark | Operations/sec | Mean Time (ms) | Min Time (ms) | Max Time (ms) | StdDev | Status |
-|-----------|----------------|----------------|---------------|---------------|---------|---------|
-| `test_graph_ops_performance` | 2,869.0 | 348.55 | 313.28 | 346.30 | 39.45 | ✅ |
-| `test_linking_operations` | 2,869.0 | 348.55 | 313.28 | 346.30 | 39.45 | ✅ |
-| `test_memory_storage_overhead` | 9.36 | 106.84 | 11.63 | 91.87 | 62.48 | ✅ |
-
-**Key Insights**:
-- Graph operations are highly optimized
-- Linking operations maintain consistent performance
-- Memory storage suitable for batch operations
-
----
-
-## 🎯 Performance Analysis
-
-### Top Performers (>10,000 ops/sec)
-1. **JSON Parsing (1K)**: 27,365.2 ops/sec
-2. **JSON Export (1K)**: 27,365.2 ops/sec
-3. **HTML Scraping**: 2,437.8 ops/sec
-4. **Similarity Calculation**: 3,142.6 ops/sec
-5. **AST Parsing**: 3,142.6 ops/sec
-
-### Performance Optimizations Needed
-1. **Network Evolution**: 0.21 ops/sec (4.87s mean)
-2. **Dashboard Assembly**: 0.11 ops/sec (9.21s mean)
-3. **Semantic Clustering**: 39.13 ops/sec (25.56s mean)
-4. **Vector JSON Export**: 0.66 ops/sec (1.50s mean)
-
-### Memory Efficiency
-- **Binary vs JSON**: 8x performance improvement with binary vector storage
-- **Batch Processing**: All algorithms show linear scaling
-- **Mock Environment**: Zero memory overhead from heavy dependencies
-
----
-
-## 📋 Regression Detection
-
-**Baseline Status**: ✅ New baseline established  
-**Regression Threshold**: 15% change with Z-score > 2.0  
-**Current Status**: ✅ No regressions detected  
-**Monitoring**: Active with 10% threshold for CI/CD
-
----
-
-## 🖥️ Environment Specifications
-
-### Hardware Configuration
-- **CPU**: Intel i5-1135G7 @ 2.40GHz (8 cores, 16 threads)
-- **Memory**: 16GB DDR4
-- **Storage**: NVMe SSD
-- **Architecture**: x64
-
-### Software Stack
-- **OS**: Windows 10 Pro (Build 19044)
-- **Python**: 3.11.9 (64-bit)
-- **Benchmark Framework**: pytest-benchmark 5.2.3
-- **Mock Environment**: Full heavy library mocking
-
-### Test Configuration
-- **Total Test Files**: 50
-- **Total Benchmarks**: 138
-- **Test Duration**: 38m 35s
-- **Success Rate**: 99.3% (138/139)
-
----
-
-## 🚀 Production Recommendations
-
-### High Performance Operations
-1. **Use JSON for data exchange** - 27K+ ops/sec
-2. **Binary vector storage** - 8x faster than JSON
-3. **Pattern-based NER** - Significantly faster than ML
-4. **Batch processing** - Linear scaling confirmed
-
-### Optimization Opportunities
-1. **Semantic clustering** - Algorithm optimization needed
-2. **Visualization dashboards** - Implement caching
-3. **YAML serialization** - Consider alternative libraries
-4. **Parallel execution** - Threading overhead analysis
-
-### CI/CD Integration
-- ✅ Environment-agnostic design
-- ✅ Statistical regression detection
-- ✅ Automated performance monitoring
-- ✅ Zero false positive rate
-
----
-
-## 📊 Test Coverage Matrix
-
-| Module | Coverage Areas | Test Count | Performance |
-|--------|----------------|------------|-------------|
-| **Input Layer** | JSON, CSV, HTML, PDF, AST parsing | 6 | 🟢 Excellent |
-| **Core Processing** | NER, similarity, clustering | 5 | 🟢 Excellent |
-| **Context Memory** | Graph ops, memory, retrieval | 2 | 🟢 Excellent |
-| **Storage** | Vectors, triplets, graphs | 4 | 🟢 Excellent |
-| **Ontology** | Inference, serialization | 4 | 🟢 Excellent |
-| **Export** | JSON, CSV, YAML, Graph formats | 4 | 🟢 Excellent |
-| **Visualization** | Networks, dashboards, analytics | 3 | 🟢 Excellent |
-| **Quality Assurance** | Deduplication, conflicts | 2 | 🟢 Excellent |
-| **Output Orchestration** | Pipelines, parallelism | 2 | 🟢 Excellent |
-| **Context** | Graph operations, linking | 3 | 🟢 Excellent |
-
----
-
-## 🏆 Conclusion
-
-The Semantica benchmark suite demonstrates **exceptional performance** across all modules:
-
-### ✅ Achievements
-- **138/138 benchmarks passed** (99.3% success rate)
-- **Sub-millisecond performance** for core operations
-- **Linear scalability** confirmed for batch processing
-- **Production-ready** performance characteristics
-- **Zero breaking changes** from benchmark addition
-
-### 🎯 Key Performance Metrics
-- **Ultra-fast text processing**: >10,000 ops/sec
-- **Efficient storage operations**: Binary format 8x faster
-- **Optimized graph algorithms**: Sub-millisecond traversal
-- **Scalable export formats**: Linear performance scaling
-
-### 🚀 Production Readiness
-- **Environment-agnostic**: Works in CI/CD and local
-- **Regression detection**: Statistical analysis active
-- **Comprehensive coverage**: All 10 modules tested
-- **Performance monitoring**: Automated baseline tracking
-
-The benchmark suite successfully provides a robust foundation for continuous performance monitoring and optimization of the Semantica framework.
-
----
-
-*Results generated on February 7, 2026 • Semantica Benchmark Suite v1.0 • Test Environment: Windows 10, Python 3.11.9*

benchmarks/benchmarks.md

@@ -1,72 +0,0 @@
-# Semantica Performance Benchmark Suite
-
-This document outlines the architecture, directory structure, and usage of the performance benchmarking suite for the Semantica Agentic RAG framework.
-
-## Architecture
-
-The suite is organized into modular layers mirroring the library's internal structure, which allows for isolated performance testing of specific components.
-
-### High-Level Design Principles
-
-- **Isolation:** Use of mocks to ensure benchmarks measure algorithm logic.
-
-- **Virtualization:** A custom `conftest.py` virtualization layer allows tests to run without heavy local dependencies.
-
-- **Pedantic Measurement:** High-iteration counts and statistical rounds to filter out system noise.
-
-## Directory Structure
-
-Based on the current production environment, the suite is organized as follows:
-
-|                       |                                                                    |
-| --------------------- | ------------------------------------------------------------------ |
-| Folder                | Description                                                        |
-| context/              | Low-level graph operations and memory storage logic.               |
-| context_memory/       | Agent-level memory management and GraphRAG retrieval patterns.     |
-| core_processing/      | Throughput tests for NER, extraction, and graph building.          |
-| export/               | Serialization benchmarks for JSON, CSV, RDF, and GraphML.          |
-| infrastructure/       | Support scripts, including the regression comparison engine.       |
-| input_layer/          | Ingestion, parsing, and splitting performance.                     |
-| normalize/            | Text cleaning, encoding handling, and date normalization.          |
-| ontology/             | Inference, serialization, and namespace management overhead.       |
-| output_orchestration/ | Parallelism and execution pipeline management.                     |
-| quality_assurance/    | Deduplication and conflict resolution strategies.                  |
-| results/              | Storage for benchmark JSON outputs and performance baselines.      |
-| storage/              | Latency tests for Vector stores (FAISS) and Triplet stores (Jena). |
-| visualization/        | Computational cost of layout algorithms and chart rendering.       |
-
-## Usage
-
-### Running the Suite
-
-To run the full suite and generate a new results file:
-
-```bash
-python benchmarks/benchmark_runner.py
-```
-
-### Strict Mode (CI/CD)
-
-The suite is designed to integrate with automated pipelines. Using the --strict flag will cause the runner to return a non-zero exit code if a performance regression greater than 15% is detected.
-
-```bash
-python benchmarks/benchmark_runner.py --strict
-```
-
-
-
-### Performance Comparison
-
-The comparison engine (infrastructure/compare.py) uses Z-scores to distinguish between actual performance regressions and environmental noise.
-
-- Regression: Change > 15% AND Z-score > 2.0.
-
-- Noise: Change > 15% but Z-score < 2.0.
-
-### Updating Baseline
-
-When a performance change is intentional (e.g., a more complex but necessary algorithm is added), update the "gold standard" baseline:
-
-```bash
-cp benchmarks/results/run_latest.json benchmarks/results/baseline.json
-```

benchmarks/benchmarks_runner.py

@@ -1,84 +0,0 @@
-import argparse
-import os
-import subprocess
-import sys
-from datetime import datetime
-
-
-def run_benchmarks():
-    """
-    Master Runner for Semantica Benchmarks.
-    """
-    parser = argparse.ArgumentParser(description="Run Semantica Benchmarks")
-    parser.add_argument(
-        "--strict", action="store_true", help="Fail script if performance regresses"
-    )
-    args = parser.parse_args()
-
-    print("Starting Semantica Benchmark Suite...")
-
-    timestamp = datetime.now().strftime("%Y%m%d_%H_%M_%S")
-    os.makedirs("benchmarks/results", exist_ok=True)
-
-    current_json = f"benchmarks/results/run_{timestamp}.json"
-    baseline_json = "benchmarks/results/baseline.json"
-
-    # Run Benchmarks
-    cmd = [
-        sys.executable,
-        "-m",
-        "pytest",
-        "benchmarks/",
-        "-p",
-        "no:typeguard",
-        "-p",
-        "no:langsmith",
-        "--benchmark-only",
-        f"--benchmark-json={current_json}",
-        "--benchmark-columns=min,mean,stddev,ops",
-        "--benchmark-sort=mean",
-    ]
-
-    print(f"Executing benchmarks... (saving to {current_json})")
-    result = subprocess.run(cmd)
-
-    if result.returncode != 0:
-        print("Benchmarks failed to execute (runtime errors).")
-        sys.exit(result.returncode)
-
-    print("Benchmarks completed execution.")
-
-    # Compare against Baseline
-    if os.path.exists(baseline_json):
-        print(f"Comparing against Baseline ({baseline_json})...")
-
-        if os.path.exists("benchmarks/infrastructure/compare.py"):
-            compare_cmd = [
-                sys.executable,
-                "benchmarks/infrastructure/compare.py",
-                baseline_json,
-                current_json,
-            ]
-
-            compare_result = subprocess.run(compare_cmd)
-
-            if compare_result.returncode != 0:
-                print("\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
-                print("   PERFORMANCE REGRESSION DETECTED")
-                print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n")
-                if args.strict:
-                    sys.exit(1)
-            else:
-                print("Performance is within acceptable limits.")
-        else:
-            print(
-                "Comparison script not found (benchmarks/infrastructure/compare.py). Skipping comparison."
-            )
-    else:
-        print("No baseline found. This run effectively sets the new baseline.")
-
-    print(f"\n[Action] To update baseline: cp {current_json} {baseline_json}")
-
-
-if __name__ == "__main__":
-    run_benchmarks()

benchmarks/conftest.py

@@ -1,355 +0,0 @@
-import importlib.abc
-import importlib.machinery
-import os
-import sys
-import tempfile
-import uuid
-from unittest.mock import patch
-
-import numpy as np
-import pytest
-
-# Import interception
-
-HEAVY_LIBS = {
-    "pdfplumber",
-    "docx",
-    "pptx",
-    "openpyxl",
-    "pandas",
-    "PIL",
-    "PIL.Image",
-    "PIL.ImageDraw",
-    "lxml",
-    "pytesseract",
-    "networkx",
-    "chardet",
-    "langdetect",
-    "neo4j",
-    "weaviate",
-    "qdrant_client",
-    "sentence_transformers",
-    "transformers",
-    "fastembed",
-    "spacy",
-    "thinc",
-    "torch",
-    "matplotlib",
-    "umap",
-    "pynndescent",
-    "fireworks",
-    "fireworks.client",
-    "docling",
-    "docling.document_converter",
-    "docling.backend",
-    "docling_core",
-    "docling_core.types",
-    "instructor",
-    "instructor.processing",
-    "instructor.core",
-    "instructor.providers",
-    "instructor.providers.fireworks",
-    "pyarrow",
-    "arrow",
-    "pa",
-}
-
-
-class MockMeta(type):
-    """Metaclass that only claims RobustMocks as instances."""
-
-    def __instancecheck__(cls, instance):
-        return hasattr(instance, "_is_robust_mock")
-
-    def __subclasscheck__(cls, subclass):
-        return True
-
-
-def create_mock_class(full_name: str):
-    return MockMeta(
-        full_name.split(".")[-1],
-        (object,),
-        {
-            "__module__": ".".join(full_name.split(".")[:-1]),
-            "__doc__": f"Mocked class {full_name}",
-            "__getattr__": lambda self, attr: RobustMock(f"{full_name}.{attr}"),
-            "__call__": lambda self, *args, **kwargs: RobustMock(full_name),
-            "__init__": lambda self, *args, **kwargs: None,
-            "__repr__": lambda self: f"<MockClass {full_name}>",
-        },
-    )
-
-
-class RobustMock:
-    def __init__(self, name: str = "mock"):
-        self.__name__ = name
-        self.__version__ = "9.9.9"
-        self._is_robust_mock = True
-        self.__path__ = []
-        self.__file__ = "mock_file.py"
-        self.__all__ = []
-
-    def __getattr__(self, name):
-        if name.startswith("__") and name.endswith("__"):
-            raise AttributeError(name)
-        full_name = f"{self.__name__}.{name}"
-        
-        # Special handling for common PIL patterns
-        if self.__name__.endswith("Image") and name == "Image":
-            return create_mock_class(full_name)
-        elif self.__name__.endswith("ImageDraw") and name == "ImageDraw":
-            return create_mock_class(full_name)
-        # Special handling for pyarrow patterns
-        elif self.__name__ in ["pa", "pyarrow", "arrow"] and name in ["schema", "Table", "Dataset", "array", "RecordBatch"]:
-            return create_mock_class(full_name)
-        # Capital names are classes
-        elif name and name[0].isupper():
-            return create_mock_class(full_name)
-        return RobustMock(full_name)
-
-    def __call__(self, *args, **kwargs):
-        return RobustMock(self.__name__)
-
-    def __iter__(self):
-        return iter([])
-
-    def __getitem__(self, item):
-        return RobustMock(f"{self.__name__}[{item}]")
-
-    def __len__(self):
-        return 0
-
-    def __bool__(self):
-        return True
-
-    def __hash__(self):
-        return id(self)
-
-    def __repr__(self):
-        return f"<RobustMock {self.__name__}>"
-
-
-class MockLoader(importlib.abc.Loader):
-    def create_module(self, spec):
-        mock_module = RobustMock(spec.name)
-        mock_module.__spec__ = spec
-        mock_module.__loader__ = self
-        mock_module.__package__ = spec.parent
-        return mock_module
-
-    def exec_module(self, module):
-        pass
-
-
-class MockFinder(importlib.abc.MetaPathFinder):
-    def find_spec(self, fullname, path, target=None):
-        # Check for exact matches first
-        if fullname in HEAVY_LIBS:
-            return importlib.machinery.ModuleSpec(fullname, MockLoader())
-        
-        # Check for prefix matches (e.g., PIL.Image, PIL.ImageDraw)
-        for lib in HEAVY_LIBS:
-            if fullname.startswith(lib + "."):
-                return importlib.machinery.ModuleSpec(fullname, MockLoader())
-        
-        # Special handling for PIL submodules
-        if fullname.startswith("PIL."):
-            return importlib.machinery.ModuleSpec(fullname, MockLoader())
-            
-        # Special handling for fireworks
-        if fullname.startswith("fireworks."):
-            return importlib.machinery.ModuleSpec(fullname, MockLoader())
-            
-        # Special handling for docling
-        if fullname.startswith("docling"):
-            return importlib.machinery.ModuleSpec(fullname, MockLoader())
-            
-        # Special handling for instructor
-        if fullname.startswith("instructor"):
-            return importlib.machinery.ModuleSpec(fullname, MockLoader())
-            
-        # Special handling for pyarrow
-        if fullname.startswith("pyarrow") or fullname.startswith("arrow"):
-            return importlib.machinery.ModuleSpec(fullname, MockLoader())
-            
-        return None
-
-
-if os.getenv("BENCHMARK_REAL_LIBS") != "1":
-    if not any(isinstance(f, MockFinder) for f in sys.meta_path):
-        sys.meta_path.insert(0, MockFinder())
-    
-    # Special handling for 'pa' alias that's commonly used for pyarrow
-    if "pa" not in sys.modules:
-        sys.modules["pa"] = RobustMock("pa")
-    
-    # Pre-emptively create a mock arrow_exporter module to prevent import errors
-    # This must happen BEFORE any semantica.export imports
-    import types
-    mock_arrow_module = types.ModuleType('semantica.export.arrow_exporter')
-    
-    # Create a mock ArrowExporter class with proper interface
-    class MockArrowExporter:
-        def __init__(self, *args, **kwargs):
-            pass
-        def __getattr__(self, name):
-            return lambda *args, **kwargs: f"Mock ArrowExporter.{name}"
-    
-    mock_arrow_module.ArrowExporter = MockArrowExporter
-    mock_arrow_module.ENTITY_SCHEMA = RobustMock("ENTITY_SCHEMA")
-    mock_arrow_module.RELATIONSHIP_SCHEMA = RobustMock("RELATIONSHIP_SCHEMA") 
-    mock_arrow_module.METADATA_SCHEMA = RobustMock("METADATA_SCHEMA")
-    mock_arrow_module.pa = RobustMock("pa")
-    
-    # Inject the mock module into sys.modules
-    sys.modules["semantica.export.arrow_exporter"] = mock_arrow_module
-
-# Infrastructure and Data Fixtures
-
-
-class NullTracker:
-    def start_tracking(self, *args, **kwargs):
-        return "dummy_id"
-
-    def update_tracking(self, *args, **kwargs):
-        pass
-
-    def stop_tracking(self, *args, **kwargs):
-        pass
-
-    def register_pipeline_modules(self, *args, **kwargs):
-        pass
-
-    def clear_pipeline_context(self, *args, **kwargs):
-        pass
-
-    def update_progress(self, *args, **kwargs):
-        pass
-
-    def update_progress_batch(self, *args, **kwargs):
-        pass
-
-    @property
-    def enabled(self):
-        return False
-
-    @enabled.setter
-    def enabled(self, value):
-        pass
-
-
-@pytest.fixture(autouse=True)
-def kill_io_overhead():
-    tracker = NullTracker()
-    with patch("semantica.utils.logging.get_logger"), patch(
-        "semantica.utils.progress_tracker.get_progress_tracker", return_value=tracker
-    ):
-        # Patch the export module to handle missing ArrowExporter
-        try:
-            from benchmarks.export.arrow_exporter import ArrowExporter, ENTITY_SCHEMA, RELATIONSHIP_SCHEMA, METADATA_SCHEMA
-            mock_arrow_module = RobustMock("semantica.export.arrow_exporter")
-            mock_arrow_module.ArrowExporter = ArrowExporter
-            mock_arrow_module.ENTITY_SCHEMA = ENTITY_SCHEMA
-            mock_arrow_module.RELATIONSHIP_SCHEMA = RELATIONSHIP_SCHEMA
-            mock_arrow_module.METADATA_SCHEMA = METADATA_SCHEMA
-        except ImportError:
-            mock_arrow_module = RobustMock("semantica.export.arrow_exporter")
-        
-        with patch.dict('sys.modules', {
-            'semantica.export.arrow_exporter': mock_arrow_module
-        }):
-            patches = []
-            for mod_name, module in list(sys.modules.items()):
-                if mod_name.startswith("semantica.") and hasattr(
-                    module, "get_progress_tracker"
-                ):
-                    p = patch.object(module, "get_progress_tracker", return_value=tracker)
-                    patches.append(p)
-            for p in patches:
-                p.start()
-            yield
-            for p in patches:
-                p.stop()
-
-
-class MockVectorStore:
-    def __init__(self, dim=384):
-        self.dim = dim
-
-    def embed(self, text: str):
-        return np.random.rand(self.dim).astype(np.float32)
-
-    def store_vectors(self, vectors, metadata):
-        pass
-
-    def search(self, query, limit=5):
-        return [
-            {"id": str(uuid.uuid4()), "score": 0.9, "content": "test", "metadata": {}}
-            for _ in range(limit)
-        ]
-
-
-@pytest.fixture
-def mock_vector_store():
-    return MockVectorStore()
-
-
-@pytest.fixture
-def generate_graph_data():
-    BASE_NS = "http://semantica.example.org/resource/"
-    PRED_NS = "http://semantica.example.org/predicate/"
-
-    def _gen(n_nodes: int = 100, avg_degree: int = 4):
-        nodes = [
-            {
-                "id": f"{BASE_NS}node/{i}",
-                "type": "Entity",
-                "properties": {"label": f"Node {i}"},
-            }
-            for i in range(n_nodes)
-        ]
-        edges = [
-            {
-                "source_id": f"{BASE_NS}node/{i}",
-                "target_id": f"{BASE_NS}node/{(i+1)%n_nodes}",
-                "type": f"{PRED_NS}conn",
-                "properties": {"w": 1.0},
-            }
-            for i in range(n_nodes)
-        ]
-        return nodes, edges
-
-    return _gen
-
-
-@pytest.fixture
-def populated_context_graph(generate_graph_data):
-    from semantica.context.context_graph import ContextGraph
-
-    def _create(n_nodes=1000):
-        g = ContextGraph()
-        nodes, edges = generate_graph_data(n_nodes)
-        g.add_nodes(nodes)
-        g.add_edges(edges)
-        return g
-
-    return _create
-
-
-@pytest.fixture
-def sample_text_file():
-    lines = ["Line " + str(i) for i in range(1000)]
-    content = "\n".join(lines)
-    with tempfile.NamedTemporaryFile(
-        mode="w+", delete=False, suffix=".txt", encoding="utf-8"
-    ) as tmp:
-        tmp.write(content)
-        tmp_path = tmp.name
-    yield tmp_path
-    if os.path.exists(tmp_path):
-        os.remove(tmp_path)
-
-
-@pytest.fixture
-def long_text_string():
-    return "benchmark " * 5000

benchmarks/context/conftest.py

@@ -1,23 +0,0 @@
-import pytest
-
-from semantica.context.agent_memory import AgentMemory
-from semantica.context.context_retriever import ContextRetriever
-
-
-@pytest.fixture
-def retriever_setup(mock_vector_store, populated_context_graph):
-    """
-    Sets up a fully configured retriever
-    """
-    kg = populated_context_graph(n_nodes=1000)
-
-    memory = AgentMemory(vector_store=mock_vector_store, knowledge_graph=kg)
-
-    retriever = ContextRetriever(
-        memory_store=memory,
-        knowledge_graph=kg,
-        vector_store=mock_vector_store,
-        hybrid_alpha=0.5,
-    )
-
-    return retriever

benchmarks/context/test_graph_ops.py

@@ -1,47 +0,0 @@
-import pytest
-
-from semantica.context.context_graph import ContextGraph
-
-
-@pytest.mark.benchmark(group="graph_traversal")
-@pytest.mark.parametrize("hops", [1, 2])
-def test_bfs_traversal_depth(benchmark, populated_context_graph, hops):
-    """Benchmarks the BFS neighbor retrieval at differnet depths."""
-    graph = populated_context_graph(n_nodes=2000)
-    start_node = list(graph.nodes.keys())[0]
-
-    def run():
-        return graph.get_neighbors(start_node, hops=hops)
-
-    benchmark.pedantic(run, iterations=5, rounds=10)
-
-
-@pytest.mark.benchmark(group="graph_construction")
-@pytest.mark.parametrize("size", [1000])
-def test_graph_ingestion_speed(benchmark, generate_graph_data, size):
-    """
-    Benchmarks the speed of adding nodes and edges to the
-    in-memory structure.
-    """
-
-    nodes, edges = generate_graph_data(n_nodes=size)
-
-    def run():
-        graph = ContextGraph()
-        graph.add_nodes(nodes)
-        graph.add_edges(edges)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="graph_query")
-def test_graph_keyword_search(benchmark, populated_context_graph):
-    """
-    Benchmarks the linear scan keyword search over graph nodes.
-    """
-    graph = populated_context_graph(n_nodes=2000)
-
-    def run():
-        return graph.query("Node content 500")
-
-    benchmark.pedantic(run, iterations=5, rounds=10)

benchmarks/context/test_linking.py

@@ -1,32 +0,0 @@
-import pytest
-
-from semantica.context.context_graph import ContextGraph
-from semantica.context.entity_linker import EntityLinker
-
-
-@pytest.mark.benchmark(group="entity_linkiing")
-@pytest.mark.parametrize("num_entities_in_graph", [100, 1000])
-def test_entity_linking_complexity(benchmark, num_entities_in_graph):
-    """
-    Benchmarks finding links for extracted entities
-    against the existing graph.
-    """
-
-    graph = ContextGraph()
-    nodes = [
-        {"id": f"e_{i}", "type": "Entity", "properties": {"content": f"Entity {i}"}}
-        for i in range(num_entities_in_graph)
-    ]
-    graph.add_nodes(nodes)
-
-    graph_dict = graph.to_dict()
-
-    linker = EntityLinker(knowledge_graph=graph_dict, similarity_threshold=0.7)
-
-    # Simulate extraction
-    extracted_entities = [{"text": f"Entity {i}", "type": "Entity"} for i in range(5)]
-
-    def run():
-        return linker.link("dummy text", entities=extracted_entities)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/context/test_memory_storage.py

@@ -1,40 +0,0 @@
-import pytest
-
-from semantica.context.agent_memory import AgentMemory
-
-
-@pytest.mark.benchmark(group="memory_io")
-def test_memory_storage_overhead(benchmark, mock_vector_store):
-    """
-    Benchmarks storing a memory item.
-    """
-    memory = AgentMemory(vector_store=mock_vector_store)
-    content = "This is nothing burger for benchmarking this memory thingy."
-    metadata = {"type": "conversation", "user": "u_1"}
-
-    def run():
-        return memory.store(content, metadata=metadata)
-
-    benchmark.pedantic(run, iterations=10, rounds=10)
-
-
-@pytest.mark.benchmark(group="memory_io")
-def test_short_term_pruning(benchmark, mock_vector_store):
-    """
-    Benchmarks the pruning logic when short-term memory
-    limit is hit.
-    """
-
-    def setup_overfilled_memory():
-        memory = AgentMemory(vector_store=mock_vector_store, short_term_limit=50)
-        # Pre-fill
-        for i in range(55):
-            memory.store(f"filler memory {i}")
-        return (memory,), {}
-
-    def run_prune(mem_instance):
-        mem_instance.store("Trigger Pruning")
-
-    benchmark.pedantic(
-        target=run_prune, setup=setup_overfilled_memory, iterations=1, rounds=20
-    )

benchmarks/context/test_retrieval_logic.py

@@ -1,42 +0,0 @@
-import pytest
-
-from semantica.context.agent_memory import AgentMemory
-from semantica.context.context_retriever import ContextRetriever, RetrievedContext
-
-
-@pytest.mark.benchmark(group="rag_logic")
-def test_hybrid_ranking_overhead(benchmark, retriever_setup):
-    """
-    Benchmarks the CPU cost of the 'rank_and_merge' logic.
-    """
-
-    query = "test_query"
-
-    # Dummy results to sim inputs
-    raw_results = [
-        RetrievedContext(content=f"Vec {i}", score=0.9 - i * 0.01, source="vector:x")
-        for i in range(10)
-    ] + [
-        RetrievedContext(content=f"Graph {i}", score=0.8 - i * 0.01, source="graph:y")
-        for i in range(10)
-    ]
-
-    def run():
-        return retriever_setup._rank_and_merge(raw_results, query)
-
-    benchmark.pedantic(run, iterations=10, rounds=20)
-
-
-@pytest.mark.benchmark(group="rag_logic")
-@pytest.mark.parametrize("use_graph", [True, False])
-def test_full_retrieval_pipeline(benchmark, retriever_setup, use_graph):
-    """
-    Benchmarks the orchestration of the retrieve() method.
-    """
-
-    def run():
-        return retriever_setup.retrieve(
-            "Node content", max_results=10, use_graph_expansion=use_graph, max_hops=1
-        )
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/context_memory/test_agentic.py

@@ -1,86 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.context.agent_context import AgentContext
-from semantica.context.context_retriever import RetrievedContext
-
-# Fixtures
-
-
-@pytest.fixture
-def mock_agent_context():
-    """
-    Creates an AgentContext with mocked internals.
-    """
-    vector_store = MagicMock()
-    knowledge_graph = MagicMock()
-
-    with patch("semantica.context.agent_context.AgentMemory") as MockMemory, patch(
-        "semantica.context.agent_context.ContextRetriever"
-    ) as MockRetriever:
-
-        ctx = AgentContext(vector_store=vector_store, knowledge_graph=knowledge_graph)
-
-        # Internal mocks
-
-        ctx._memory = MockMemory.return_value
-        ctx._retriever = MockRetriever.return_value
-
-        return ctx
-
-
-# Benchmarks
-
-
-def test_router_overhead(benchmark, mock_agent_context):
-    """
-    Benchmarks the logic that decides between Vector vs Graph retrieval.
-    """
-
-    mock_agent_context._retriever.retrieve.return_value = []
-
-    def op():
-        return mock_agent_context.retrieve("test query", use_graph=None)
-
-    benchmark.pedantic(op, iterations=50, rounds=20)
-
-
-def test_result_conversion_throughput(benchmark, mock_agent_context):
-    """
-    Benchmarks converting internal RetrievedContext objects to Dicts.
-    """
-
-    fake_results = [
-        RetrievedContext(
-            content=f"Result {i}",
-            score=0.9,
-            source="graph:node_1",
-            metadata={"type": "fact"},
-            related_entities=[{"id": "e1", "name": "Entity"}],
-            related_relationships=[{"source": "e1", "target": "e2"}],
-        )
-        for i in range(100)
-    ]
-    mock_agent_context._retriever.retrieve.return_value = fake_results
-
-    def op():
-        return mock_agent_context.retrieve("test", use_graph=True)
-
-    benchmark.pedantic(op, iterations=20, rounds=10)
-
-
-def test_store_orchestration_overhead(benchmark, mock_agent_context):
-    """
-    Benchmarks the 'store' method's logic for routing documents.
-    """
-    docs = [{"content": f"Doc {i}", "metadata": {"id": i}} for i in range(50)]
-
-    # Mock the internal storage to return immediately
-    mock_agent_context._memory.store.return_value = "mem_id"
-    mock_agent_context._build_graph_from_documents = MagicMock(return_value={})
-
-    def op():
-        return mock_agent_context.store(docs, extract_entities=False)
-
-    benchmark.pedantic(op, iterations=10, rounds=10)

benchmarks/context_memory/test_graphrag.py

@@ -1,244 +0,0 @@
-from dataclasses import dataclass, field
-from typing import Any, Dict, List
-from unittest.mock import patch
-
-import numpy as np
-import pytest
-
-from semantica.context.agent_context import AgentContext
-from semantica.context.agent_memory import AgentMemory
-from semantica.context.context_graph import ContextGraph
-from semantica.context.context_retriever import ContextRetriever, RetrievedContext
-from semantica.context.entity_linker import EntityLinker
-
-# Infra
-
-
-class NullTracker:
-    """
-    Stateless dummy tracker.
-    """
-
-    def start_tracking(self, *args, **kwargs):
-        return "dummy_id"
-
-    def update_tracking(self, *args, **kwargs):
-        pass
-
-    def stop_tracking(self, *args, **kwargs):
-        pass
-
-    def register_pipeline_modules(self, *args, **kwargs):
-        pass
-
-    def clear_pipeline_context(self, *args, **kwargs):
-        pass
-
-    def update_progress(self, *args, **kwargs):
-        pass
-
-    @property
-    def enabled(self):
-        return False
-
-    @enabled.setter
-    def enabled(self, value):
-        pass
-
-
-# ~~ MOCK STORES ~~
-
-
-class MockVectorStore:
-    """
-    A feather VectorStore sim that does no math.
-    We want to measure the MANAGER overhead.
-    """
-
-    def __init__(self):
-        self.vectors = {}
-        self.dim = 384
-
-    def embed(self, text):
-        return np.random.rand(self.dim).tolist()
-
-    def add(self, items):
-        for item in items:
-            self.vectors[item.memory_id] = item
-
-    def search(self, query, limit=5):
-        class MockResult:
-            def __init__(self, i):
-                self.id = f"mem_{i}"
-                self.content = f"Content for result {i} matching {query[:10]}"
-                self.score = 0.9 - (i * 0.05)
-                self.metadata = {"type": "test"}
-
-        return [MockResult(i) for i in range(limit)]
-
-
-def create_dense_graph(node_count):
-    """
-    Creates a ContextGraph with 'Small World' Topology.
-    Used to stress-test BFS traversal scaling.
-    """
-    graph = ContextGraph()
-
-    graph.progress_tracker = NullTracker()
-
-    # Create nodes
-    nodes = [
-        {
-            "id": f"node_{i}",
-            "type": "concept",
-            "properties": {"content": f"Concept {i}"},
-        }
-        for i in range(node_count)
-    ]
-    graph.add_nodes(nodes)
-
-    # Create Edges (Chain + Hub + Random)
-    edges = []
-    for i in range(node_count):
-        # Chain
-        if i < node_count - 1:
-            edges.append(
-                {"source_id": f"node_{i}", "target_id": f"node_{i+1}", "type": "next"}
-            )
-        # Hub
-        if i > 0:
-            edges.append(
-                {"source_id": "node_0", "target_id": f"node_{i}", "type": "hub_link"}
-            )
-        # Rando
-        if i % 5 == 0 and i + 5 < node_count:
-            edges.append(
-                {
-                    "source_id": f"node_{i}",
-                    "target_id": f"node_{i+5}",
-                    "type": "cross_link",
-                }
-            )
-
-    graph.add_edges(edges)
-    return graph
-
-
-def create_populated_memory(item_count):
-    """Creates an AgentMemory populated with N items."""
-    vs = MockVectorStore()
-    memory = AgentMemory(vector_store=vs)
-    memory.progress_tracker = NullTracker()
-
-    for i in range(item_count):
-        mem_id = f"setup_mem_{i}"
-        from datetime import datetime
-
-        from semantica.context.agent_memory import MemoryItem
-
-        memory.memory_items[mem_id] = MemoryItem(
-            content=f"History item {i}",
-            timestamp=datetime.now(),
-            memory_id=mem_id,
-            metadata={"type": "chat"},
-        )
-        memory.memory_index.append(mem_id)
-
-    return memory
-
-
-# ~~ BENCHMARKS ~~
-
-
-@pytest.mark.parametrize("graph_size", [100, 1000])
-@pytest.mark.parametrize("hops", [1, 2])
-def test_graph_traversal_scaling(benchmark, graph_size, hops):
-    """
-    Measures 'Hop Explosion' effect.
-    Retrieving multi-hop neighbors on a dense graph.
-    """
-    graph = create_dense_graph(graph_size)
-
-    def op():
-        # Start from'Hub' node which's celebrity, meaning
-        # connected to everyone
-        return graph.get_neighbors("node_0", hops=hops)
-
-    benchmark.pedantic(op, iterations=5, rounds=5)
-
-
-@pytest.mark.parametrize("memory_count", [100, 1000])
-def test_retriever_ranking_throughput(benchmark, memory_count):
-    """
-    Measures CPU cost of merging and ranking results.
-    """
-    retriever = ContextRetriever(
-        vector_store=MockVectorStore(),
-        memory_store=create_populated_memory(10),
-        knowledge_graph=None,
-        hybrid_alpha=0.5,
-    )
-    retriever.progress_tracker = NullTracker()
-
-    results = []
-    for i in range(memory_count):
-        results.append(
-            RetrievedContext(
-                content=f"Vector Item {i}",
-                score=np.random.random(),
-                source=f"vector:{i}",
-            )
-        )
-        results.append(
-            RetrievedContext(
-                content=f"Graph Item {i}",
-                score=np.random.random(),
-                source=f"graph:{i}",
-                metadata={"node_id": f"node_{i}"},
-            )
-        )
-
-    def op():
-        return retriever._rank_and_merge(results, "query context")
-
-    benchmark.pedantic(op, iterations=5, rounds=10)
-
-
-@pytest.mark.parametrize("registry_size", [100, 1000])
-def test_entity_linking_speed(benchmark, registry_size):
-    """
-    Measures O(N) linear scan speed in `find_similar_entities`.
-    """
-    linker = EntityLinker()
-    linker.progress_tracker = NullTracker()
-
-    mock_kg = {"entities": []}
-    for i in range(registry_size):
-        mock_kg["entities"].append(
-            {"id": f"ent_{i}", "text": f"Entity Number {i}", "type": "TEST"}
-        )
-    linker.knowledge_graph = mock_kg
-
-    input_text = "I am looking for Entity Number 50 in the database."
-
-    def op():
-        return linker.find_similar_entities(input_text, threshold=0.1)
-
-    benchmark.pedantic(op, iterations=5, rounds=5)
-
-
-@pytest.mark.parametrize("batch_size", [1, 10, 50])
-def test_agent_store_throughput(benchmark, batch_size):
-    """
-    'store' pipeline test.
-    """
-    vs = MockVectorStore()
-    context = AgentContext(vector_store=vs)
-    context._memory.progress_tracker = NullTracker()
-
-    inputs = [f"Memory item {i} for storage test" for i in range(batch_size)]
-
-    def op():
-        return context.batch_store(inputs)
-
-    benchmark.pedantic(op, iterations=5, rounds=5)

benchmarks/core_processing/conftest.py

@@ -1,44 +0,0 @@
-import pytest
-
-
-# Data factories
-@pytest.fixture
-def node_batch():
-    """Generates 1000 nodes for graph"""
-    return [
-        {
-            "id": f"node_{i}",
-            "type": "Concept",
-            "properties": {"name": f"Concept {i}", "weight": i / 1000},
-        }
-        for i in range(1000)
-    ]
-
-
-@pytest.fixture
-def edge_batch():
-    """Generates 1000 edges connection to the nodes."""
-    return [
-        {
-            "source_id": f"node_{i}",
-            "target_id": f"node_{i + 1}",
-            "type": "related to",
-            "weight": 0.5,
-        }
-        for i in range(999)
-    ]
-
-
-@pytest.fixture
-def conversation_data():
-    """Simulates a large conversation log"""
-    entities = [{"text": f"Entity_{i}", "type": "topic"} for i in range(50)]
-
-    return [
-        {
-            "id": "conv_1",
-            "content": "This is a conversation about banking.",
-            "entities": entities,
-            "relationships": [],
-        }
-    ]

benchmarks/core_processing/test_extraction.py

@@ -1,153 +0,0 @@
-from unittest.mock import patch
-
-import pytest
-
-from semantica.semantic_extract.ner_extractor import Entity, NERExtractor
-from semantica.semantic_extract.semantic_analyzer import SemanticAnalyzer
-
-
-# Fixtures
-@pytest.fixture
-def document_batch():
-    base = "The quick brown fox jumps over the lazy dog."
-    docs = [
-        f"{base} Variation {i}. Apple Inc released a product in 2024."
-        for i in range(50)
-    ]
-    return docs
-
-
-# Fast wrapper-only benchmark (always runs)
-def test_ner_ml_wrapper_overhead(benchmark, long_text_string):
-    extractor = NERExtractor(method="ml", model="en_core_web_sm")
-
-    entity_text = "Semantica"
-    phrase = f"{entity_text} is a knowledge graph framework. "
-    medium_text = phrase * 5
-
-    expected_entities = []
-    phrase_len = len(phrase)
-    for i in range(5):
-        start = i * phrase_len
-        end = start + len(entity_text)
-        ent = Entity(
-            text=entity_text,
-            label="ORG",
-            start_char=start,
-            end_char=end,
-            confidence=0.98,
-            metadata={"lemma": entity_text},
-        )
-        expected_entities.append(ent)
-
-    def custom_ml_extraction(text: str, **method_options):
-        min_confidence = method_options.get("min_confidence", 0.5)
-        entity_types = method_options.get("entity_types")
-        filtered = []
-        for ent in expected_entities:
-            if entity_types and ent.label not in entity_types:
-                continue
-            if ent.confidence >= min_confidence:
-                filtered.append(ent)
-        return filtered
-
-    with patch(
-        "semantica.semantic_extract.methods.get_entity_method"
-    ) as mock_get_method:
-        mock_get_method.side_effect = lambda name: (
-            custom_ml_extraction if name == "ml" else (lambda t, **o: [])
-        )
-
-        def op():
-            return extractor.extract_entities(text=medium_text)
-
-        result = benchmark.pedantic(op, rounds=20, iterations=5)
-
-    assert len(result) == 5
-    assert all(e.text == "Semantica" for e in result)
-    assert all(e.label == "ORG" for e in result)
-    assert all(e.confidence == 0.98 for e in result)
-    assert all(medium_text[e.start_char : e.end_char] == e.text for e in result)
-
-
-# Real spaCy benchmark
-@pytest.mark.benchmark(group="ner_real_ml")
-def test_ner_ml_real_performance(benchmark, long_text_string):
-    """
-    Full spaCy inference + wrapper overhead.
-    Only runs when real spaCy is loaded (BENCHMARK_REAL_LIBS=1).
-    """
-    extractor = NERExtractor(method="ml", model="en_core_web_sm")
-
-    if (
-        extractor.nlp is None
-        or not hasattr(extractor.nlp, "pipe_names")
-        or "ner" not in extractor.nlp.pipe_names
-    ):
-        pytest.skip(
-            "Real spaCy NER pipeline not available — skipping production benchmark"
-        )
-
-    medium_text = long_text_string[:10000]
-
-    medium_text += " Apple Inc. was founded by Steve Jobs and Steve Wozniak in Cupertino, California on April 1, 1976. Microsoft is a competitor."
-
-    def op():
-        return extractor.extract_entities(text=medium_text)
-
-    result = benchmark.pedantic(op, rounds=6, iterations=2)
-
-    assert len(result) >= 6
-    assert any("Apple" in e.text and e.label == "ORG" for e in result)
-    assert any(e.label == "PERSON" for e in result)
-    assert any(e.label in {"GPE", "LOC"} for e in result)
-    assert any(e.label == "DATE" for e in result)
-    assert any("Microsoft" in e.text and e.label == "ORG" for e in result)
-
-
-def test_ner_pattern_speed(benchmark, long_text_string):
-    extractor = NERExtractor(method="pattern")
-    medium_text = long_text_string[:50000]
-    text_with_entities = medium_text + " Apple Inc. was founded in 1976. "
-
-    def op():
-        return extractor.extract_entities(text=text_with_entities)
-
-    result = benchmark.pedantic(op, rounds=20, iterations=5)
-    assert len(result) > 0
-    assert result[0].label in ["ORG", "DATE", "UNKNOWN"]
-
-
-def test_ner_batch_throughput(benchmark, document_batch):
-    extractor = NERExtractor(method="pattern")
-
-    def run_batch():
-        return extractor.extract_entities_batch(document_batch, max_workers=2)
-
-    result = benchmark.pedantic(run_batch, rounds=10, iterations=5)
-    assert len(result) == len(document_batch)
-    assert len(result[0]) > 0
-
-
-def test_similarity_calculation(benchmark):
-    analyzer = SemanticAnalyzer()
-    text1 = "The quick brown fox jumps over the lazy dog" * 10
-    text2 = "The slow brown fox jumped over the sleeping dog" * 10
-
-    def op():
-        return analyzer.calculate_similarity(text1, text2, method="jaccard")
-
-    result = benchmark.pedantic(op, rounds=100, iterations=100)
-    assert 0.0 <= result <= 1.0
-
-
-def test_clustering_algorithm(benchmark, document_batch):
-    analyzer = SemanticAnalyzer()
-    options = {"similarity_threshold": 0.1}
-
-    def op():
-        return analyzer.cluster_semantically(texts=document_batch, **options)
-
-    result = benchmark.pedantic(op, rounds=10, iterations=5)
-    assert len(result) > 0
-    assert result[0].texts

benchmarks/core_processing/test_graph_building.py

@@ -1,56 +0,0 @@
-from unittest.mock import MagicMock
-
-import pytest
-
-from semantica.context.context_graph import ContextGraph
-
-
-def test_bulk_node_insertion(benchmark, node_batch):
-    """
-    Benchmarks the overhead of adding nodes to in-memory graph.
-
-    """
-
-    def setup_graph():
-        return (ContextGraph(),), {}
-
-    def run(graph_instance):
-        graph_instance.add_nodes(node_batch)
-
-    benchmark.pedantic(target=run, setup=setup_graph, rounds=50, iterations=1)
-
-
-def test_bulk_edge_insertion(benchmark, node_batch, edge_batch):
-    """
-    Benchmarks adding edges.
-    """
-
-    def setup_graph_with_nodes():
-        g = ContextGraph()
-        g.add_nodes(node_batch)
-        return (g,), {}
-
-    def run(graph_instance):
-        graph_instance.add_edges(edge_batch)
-
-    benchmark.pedantic(
-        target=run, setup=setup_graph_with_nodes, rounds=50, iterations=1
-    )
-
-
-def test_conversation_to_graph_conversion(benchmark, conversation_data):
-    """
-    Benchmarks parsing conversation dicts into graph structures.
-    """
-
-    def setup_clean_builder():
-        g = ContextGraph()
-        g.entity_linker = MagicMock()
-        return (g,), {}
-
-    def run(graph_instance):
-        return graph_instance.build_from_conversations(
-            conversation_data, link_entities=False
-        )
-
-    benchmark.pedantic(target=run, setup=setup_clean_builder, rounds=20, iterations=1)

benchmarks/export/arrow_exporter.py

@@ -1,69 +0,0 @@
-"""
-Mock Arrow Exporter for Benchmark Testing
-
-This module provides a mock implementation of the ArrowExporter to prevent
-import errors during benchmark testing when PyArrow is not available in the CI environment.
-"""
-
-# Mock PyArrow import for CI compatibility
-try:
-    import pyarrow as pa
-except ImportError:
-    # Create a mock pa module for CI environment
-    import types
-    pa = types.ModuleType('pa')
-    
-    def mock_schema(*args, **kwargs):
-        return types.SimpleNamespace()
-    
-    def mock_table(*args, **kwargs):
-        return types.SimpleNamespace()
-    
-    def mock_array(*args, **kwargs):
-        return types.SimpleNamespace()
-    
-    pa.schema = mock_schema
-    pa.Table = mock_table
-    pa.array = mock_array
-    pa.RecordBatch = mock_table
-
-# Mock schema definitions
-ENTITY_SCHEMA = pa.schema([]) if hasattr(pa, 'schema') else None
-RELATIONSHIP_SCHEMA = pa.schema([]) if hasattr(pa, 'schema') else None
-METADATA_SCHEMA = pa.schema([]) if hasattr(pa, 'schema') else None
-
-class ArrowExporter:
-    """
-    Mock Arrow Exporter class for benchmark testing.
-    
-    This is a lightweight implementation that provides the same interface
-    as the real ArrowExporter but doesn't require PyArrow to be installed.
-    """
-    
-    def __init__(self, config=None):
-        self.config = config
-        self._tables = {}
-    
-    def export_entities(self, entities, output_path):
-        """Mock export entities method."""
-        return f"Mock exported {len(entities)} entities to {output_path}"
-    
-    def export_relationships(self, relationships, output_path):
-        """Mock export relationships method."""
-        return f"Mock exported {len(relationships)} relationships to {output_path}"
-    
-    def export_knowledge_graph(self, entities, relationships, output_path):
-        """Mock export knowledge graph method."""
-        return f"Mock exported knowledge graph to {output_path}"
-    
-    def to_arrow_table(self, data):
-        """Mock conversion to Arrow table."""
-        return f"Mock Arrow table with {len(data)} rows"
-    
-    def save_to_file(self, table, path):
-        """Mock save to file method."""
-        return f"Mock saved table to {path}"
-    
-    def batch_export(self, data_list, output_dir):
-        """Mock batch export method."""
-        return f"Mock batch exported {len(data_list)} items to {output_dir}"

benchmarks/export/conftest.py

@@ -1,81 +0,0 @@
-import random
-import uuid
-from typing import Any, Dict, List
-
-import numpy as np
-import pytest
-
-# Data Generators
-
-
-@pytest.fixture
-def generate_entities():
-    def _gen(count: int) -> List[Dict[str, Any]]:
-        entities = []
-        for i in range(count):
-            entities.append(
-                {
-                    "id": f"e_{i}",
-                    "text": f"Entity Number {i}",
-                    "type": random.choice(
-                        ["person", "Organization", "Location", "Event"]
-                    ),
-                    "confidence": random.uniform(0.7, 1.0),
-                    "metadata": {"source": "doc_1.txt", "page": 1},
-                }
-            )
-
-        return entities
-
-    return _gen
-
-
-@pytest.fixture
-def generate_knowledge_graph(generate_entities):
-    def _gen(entity_count: int, rel_density: float = 1.5) -> Dict[str, Any]:
-        entities = generate_entities(entity_count)
-        relationships = []
-        rel_count = int(entity_count * rel_density)
-
-        for i in range(rel_count):
-            src = random.choice(entities)
-            tgt = random.choice(entities)
-            relationships.append(
-                {
-                    "id": f"r_{i}",
-                    "source_id": src["id"],
-                    "target_id": tgt["id"],
-                    "type": " RELATED_TO",
-                    "confidence": 0.9,
-                    "metadata": {"extractor": "v1"},
-                }
-            )
-
-        return {
-            "entities": entities,
-            "relationships": relationships,
-            "metadata": {"generated_at": "2026-02-05"},
-        }
-
-    return _gen
-
-
-@pytest.fixture
-def generate_vectors():
-    def _gen(count: int, dim: int = 384) -> List[Dict[str, Any]]:
-        matrix = np.random.rand(count, dim).astype(np.float32)
-
-        data = []
-
-        for i in range(count):
-            data.append(
-                {
-                    "id": f"vec_{i}",
-                    "vector": matrix[i].tolist(),
-                    "text": f"Text {i}",
-                    "metadata": {"model": "bert"},
-                }
-            )
-        return data
-
-    return _gen

benchmarks/export/test_core_formats.py

@@ -1,42 +0,0 @@
-import pytest
-
-from semantica.export.csv_exporter import CSVExporter
-from semantica.export.json_exporter import JSONExporter
-from semantica.export.yaml_exporter import SemanticNetworkYAMLExporter
-
-
-@pytest.mark.benchmark(group="structured_export")
-@pytest.mark.parametrize("size", [1000, 5000])
-def test_json_parsing_throughput(benchmark, tmp_path, generate_knowledge_graph, size):
-    kg = generate_knowledge_graph(size)
-    exporter = JSONExporter(indent=None)
-    output_file = tmp_path / "output.json"
-
-    def run():
-        exporter.export(kg, output_file)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="structured_export")
-def test_csv_entity_export(benchmark, tmp_path, generate_entities):
-    entities = generate_entities(5000)
-    exporter = CSVExporter()
-    output_file = tmp_path / "entities.csv"
-
-    def run():
-        exporter.export_entities(entities, output_file)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="structured_export")
-def test_yaml_serialization_overhead(benchmark, tmp_path, generate_knowledge_graph):
-    kg = generate_knowledge_graph(500)
-    exporter = SemanticNetworkYAMLExporter()
-    output_file = tmp_path / "output.yaml"
-
-    def run():
-        exporter.export(kg, output_file)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/export/test_graph_export.py

@@ -1,22 +0,0 @@
-import pytest
-
-from semantica.export.graph_exporter import GraphExporter
-
-
-@pytest.mark.benchmark(group="vis_export")
-@pytest.mark.parametrize("format", ["graphml", "gexf"])
-def test_graph_conversion_overhead(
-    benchmark, tmp_path, generate_knowledge_graph, format
-):
-    """
-    Measures the cost of converting internal KG structure to XML-based graph formats.
-    Includes dictionary traversal and XML string building.
-    """
-    kg = generate_knowledge_graph(2000)
-    exporter = GraphExporter(format=format)
-    output_file = tmp_path / f"graph.{format}"
-
-    def run():
-        exporter.export_knowledge_graph(kg, output_file)
-
-    benchmark(run)

benchmarks/export/test_semantic_export.py

@@ -1,45 +0,0 @@
-import pytest
-
-from semantica.export.lpg_exporter import LPGExporter
-from semantica.export.owl_exporter import OWLExporter
-from semantica.export.rdf_exporter import RDFExporter
-
-
-@pytest.mark.benchmark(group="semantic_serialization")
-@pytest.mark.parametrize("format", ["turtle", "rdfxml"])
-def test_rdf_serialization_formats(benchmark, generate_knowledge_graph, format):
-    kg = generate_knowledge_graph(1000)
-    exporter = RDFExporter()
-    rdf_data = exporter.serializer.convert_kg_to_rdf(kg)
-
-    def run():
-        return exporter.export_to_rdf(rdf_data, format=format)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="graph_db_export")
-def test_lpg_cypher_generation(benchmark, generate_knowledge_graph):
-    kg = generate_knowledge_graph(2000)
-    exporter = LPGExporter(batch_size=1000, include_indexes=False)
-
-    def run():
-        return exporter._generate_cypher_queries(kg)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="semantic_serialization")
-def test_owl_xml_generation(benchmark, tmp_path):
-    ontology = {
-        "name": "BenchmarkOntology",
-        "classes": [{"name": f"Class{i}"} for i in range(500)],
-        "object_properties": [{"name": f"Prop{i}"} for i in range(200)],
-    }
-    exporter = OWLExporter()
-    output_file = tmp_path / "ontology.xml"
-
-    def run():
-        exporter.export(ontology, output_file, format="owl-xml")
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/export/test_vector_export.py

@@ -1,51 +0,0 @@
-import numpy as np
-import pytest
-
-from semantica.export.vector_exporter import VectorExporter
-
-
-@pytest.mark.benchmark(group="vector_io")
-@pytest.mark.parametrize("count", [1000, 10000])
-def test_numpy_compression_speed(benchmark, tmp_path, generate_vectors, count):
-    """
-    Measures cost of np.savez_compressed.
-    """
-    vectors = generate_vectors(count)
-    exporter = VectorExporter(format="numpy")
-    output_file = tmp_path / "vectors.npz"
-
-    def run():
-        exporter.export(vectors, output_file)
-
-    benchmark(run)
-
-
-@pytest.mark.benchmark(group="vector_io")
-def test_json_vector_overhead(benchmark, tmp_path, generate_vectors):
-    """
-    Benchmarks JSON export for vectors.
-    """
-
-    vectors = generate_vectors(2000)
-    exporter = VectorExporter(format="json")
-    output_file = tmp_path / "vectors.json"
-
-    def run():
-        exporter.export(vectors, output_file)
-
-    benchmark(run)
-
-
-@pytest.mark.benchmark(group="vector_io")
-def test_binary_raw_throughput(benchmark, tmp_path, generate_vectors):
-    """
-    Measures raw binary dump speed (no compression, no metadata).
-    """
-    vectors = generate_vectors(10000)
-    exporter = VectorExporter(format="binary")
-    output_file = tmp_path / "vectors.bin"
-
-    def run():
-        exporter.export(vectors, output_file)
-
-    benchmark(run)

benchmarks/infrastructure/compare.py

@@ -1,102 +0,0 @@
-import argparse
-import json
-import sys
-from pathlib import Path
-from typing import Any, Dict, List
-
-
-def load_results(filepath: str) -> Dict[str, Any]:
-    with open(filepath, "r") as f:
-        return json.load(f)
-
-
-def calc_z_score(current_mean, base_mean, base_stddev):
-    """
-    Z-Score indicates how many standard deviations
-    away current run is from baseline
-    """
-
-    if base_stddev == 0:
-        return 0 if current_mean == base_mean else 100.0
-
-    return (current_mean - base_mean) / base_stddev
-
-
-def compare_benchmarks(
-    baseline: Dict[str, Any], current: Dict[str, Any], threshold_pct: float = 10.0
-):
-    """
-    Uses Mean for % change and Z-score for noise detection.
-    """
-
-    # colors for terminal
-    RED = "\033[91m"
-    GREEN = "\033[92m"
-    YELLOW = "\033[93m"
-    RESET = "\033[0m"
-
-    header = f"{'Benchmark':<60} | {'CHANGE %':<12} | {'SIGMA (Z)':<10} | {'STATUS'}"
-    print(header)
-    print("=" * len(header))
-
-    baseline_map = {b["name"]: b for b in baseline["benchmarks"]}
-    current_map = {b["name"]: b for b in current["benchmarks"]}
-
-    regressions = []
-
-    for name, curr in current_map.items():
-        base = baseline_map.get(name)
-        if not base:
-            print(f"{name:<60} | {'NEW':<12} | {'N/A':<10} | NEW")
-            continue
-
-        m1 = base["stats"]["mean"]
-        s1 = base["stats"]["stddev"]
-        m2 = curr["stats"]["mean"]
-
-        if m1 == 0:
-            delta_pct = 0.0
-        else:
-            delta_pct = ((m2 - m1) / m1) * 100
-
-        z_score = calc_z_score(m2, m1, s1)
-
-        status = f"{GREEN} OK{RESET}"
-
-        if delta_pct > threshold_pct:
-            if abs(z_score) > 2.0:
-                status = f"{RED} REGRESSION{RESET}"
-                regressions.append(name)
-            else:
-                status = f"{YELLOW} NOISE{RESET}"
-        elif delta_pct < -threshold_pct and abs(z_score) > 2.0:
-            status = f"{GREEN} IMPROVED{RESET}"
-
-        print(f"{name:<60} | {delta_pct:>+10.2f}% | {z_score:>9.2f} | {status}")
-
-    if regressions:
-        print(
-            f"\n{RED}FAILURE: Performance regression detected in {len(regressions)} tests.{RESET}"
-        )
-        return True
-    print(f"\n{GREEN}SUCCESS: No significant regressions.{RESET}")
-    return False
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser.add_argument("baseline", help="Gold standard JSON")
-    parser.add_argument("current", help="NEW RUN JSON")
-    parser.add_argument(
-        "--threshold", type=float, default=10.0, help="FAIL if slower by %"
-    )
-    args = parser.parse_args()
-
-    try:
-        failed = compare_benchmarks(
-            load_results(args.baseline), load_results(args.current), args.threshold
-        )
-        sys.exit(1 if failed else 0)
-    except FileNotFoundError as e:
-        print(f"Error loading files: {e}")
-        sys.exit(0)

benchmarks/input_layer/test_ingestion.py

@@ -1,22 +0,0 @@
-import pytest
-
-from semantica.ingest.file_ingestor import FileIngestor
-
-
-def test_ingest_file_performance(benchmark, sample_text_file):
-    """
-    Benchmarks the speed of the ingest_file method
-
-    Metrics:
-    - Time to open, read, validate and wrap a ~~10 KB text file.
-    """
-
-    ingestor = FileIngestor()
-    result = benchmark(
-        ingestor.ingest_file, file_path=sample_text_file, read_content=True
-    )
-
-    assert result is not None
-    assert result.size > 0
-    assert result.name.endswith(".txt")
-    assert "Line 0" in result.text

benchmarks/input_layer/test_parsing.py

@@ -1,188 +0,0 @@
-import csv
-import io
-import json
-import time
-from typing import Any, Dict, List
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.parse.code_parser import CodeParser
-from semantica.parse.csv_parser import CSVParser
-from semantica.parse.document_parser import DocumentParser
-from semantica.parse.html_parser import HTMLParser
-from semantica.parse.json_parser import JSONParser
-
-# Data gens
-
-
-def generate_json_string(item_count: int) -> str:
-    data = [
-        {
-            "id": i,
-            "name": f"Item:{i}",
-            "tags": ["tag1", "tag2", "tag3"],
-            "metadata": {"active": True, "score": 0.95},
-        }
-        for i in range(item_count)
-    ]
-    return json.dumps(data)
-
-
-def generate_csv_string(row_count: int) -> str:
-    output = io.StringIO()
-    writer = csv.writer(output)
-    writer.writerow(["id", "name", "description", "value", "date"])
-    for i in range(row_count):
-        writer.writerow([i, f"Item {i}", "Description text here", 100.50, "2024-01-01"])
-    return output.getvalue()
-
-
-def generate_html_string(element_count: int) -> str:
-    lis = "".join(
-        [f'<li><a href="/item/{i}">Link {i}</a></li>' for i in range(element_count)]
-    )
-    return f"""
-    <html>
-        <head><title>Benchmark Page</title></head>
-        <body>
-            <div id="content">
-                <h1>Header</h1>
-                <p>Some intro text.</p>
-                <ul>{lis}</ul>
-            </div>
-        </body>
-    </html>
-    """
-
-
-# lib mocks
-
-
-class MockPDFPage:
-    def __init__(self, page_num):
-        self.width = 600
-        self.height = 800
-        self.page_number = page_num
-
-    def extract_text(self):
-        return f"This is text content for page {self.page_number}. " * 50
-
-    def extract_tables(self):
-        return [[["Header1", "Header2"], ["Row1", "Value1"]]]
-
-    @property
-    def images(self):
-        return [{"x0": 10, "y0": 10, "width": 100, "height": 100}]
-
-
-class MockPDF:
-    def __init__(self, page_count):
-        self.pages = [MockPDFPage(i) for i in range(page_count)]
-        self.metadata = {"Title": "Benchmark PDF", "Author": "Noone"}
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, *args):
-        pass
-
-
-@pytest.fixture
-def mock_pdfplumber():
-    with patch("pdfplumber.open") as mock_open:
-        yield mock_open
-
-
-# Benchmarks
-
-
-@pytest.mark.parametrize("size", [1000, 10000])
-def test_json_parsing_throughput(benchmark, size):
-    parser = JSONParser()
-    json_str = generate_json_string(size)
-
-    with patch("pathlib.Path.exists", return_value=False):
-
-        def op():
-            return parser.parse(json_str)
-
-        benchmark.pedantic(op, iterations=5, rounds=10)
-
-
-@pytest.mark.parametrize("rows", [1000, 10000])
-def test_csv_parsing_throughput(benchmark, rows):
-    """
-    Measures CSV parsing throughput.
-    """
-    parser = CSVParser()
-    csv_content = generate_csv_string(rows)
-
-    with patch(
-        "builtins.open", side_effect=lambda *args, **kwargs: io.StringIO(csv_content)
-    ):
-        with patch("pathlib.Path.exists", return_value=True):
-
-            def op():
-                return parser.parse("dummy.csv")
-
-            benchmark.pedantic(op, iterations=5, rounds=5)
-
-
-@pytest.mark.parametrize("elements", [100, 1000])
-def test_html_scraping_speed(benchmark, elements):
-    parser = HTMLParser()
-    html_content = generate_html_string(elements)
-
-    with patch("pathlib.Path.exists", return_value=False):
-
-        def op():
-            return parser.parse(html_content, extract_links=True)
-
-        benchmark.pedantic(op, iterations=5, rounds=5)
-
-
-@pytest.mark.parametrize("pages", [10, 50])
-def test_pdf_extraction_overhead(benchmark, mock_pdfplumber, pages):
-    parser = DocumentParser()
-
-    mock_pdf = MockPDF(pages)
-    mock_pdfplumber.return_value = mock_pdf
-
-    with patch("pathlib.Path.exists", return_value=True), patch(
-        "pathlib.Path.suffix", new_callable=MagicMock(return_value=".pdf")
-    ):
-
-        def op():
-            return parser.parse_document("dummy.pdf", extract_images=True)
-
-        benchmark.pedantic(op, iterations=5, rounds=5)
-
-
-def test_python_ast_parsing(benchmark):
-    """
-    Measures performance of Python AST analysis.
-    """
-    parser = CodeParser()
-
-    code_lines = []
-    for i in range(200):
-        code_lines.append(f"import module_{i}")
-        code_lines.append(f"def function_{i}(arg):")
-        code_lines.append(f"    '''Docstring for function {i}'''")
-        code_lines.append(f"    return arg + {i}")
-        code_lines.append(f"class Class_{i}:")
-        code_lines.append(f"    pass")
-
-    code_content = "\n".join(code_lines)
-
-    with patch(
-        "builtins.open", side_effect=lambda *args, **kwargs: io.StringIO(code_content)
-    ), patch("pathlib.Path.exists", return_value=True), patch(
-        "pathlib.Path.suffix", new_callable=MagicMock(return_value=".py")
-    ):
-
-        def op():
-            return parser.parse_code("dummy.py")
-
-        benchmark.pedantic(op, iterations=5, rounds=5)

benchmarks/input_layer/test_splitting.py

@@ -1,27 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-try:
-    from semantica.split.sliding_window_chunker import SlidingWindowChunker
-    from semantica.split.splitter import TextSplitter
-except ImportError as e:
-    pytest.skip(
-        f"Skipping splitting test due to missing dependencies ({e})",
-        allow_module_level=True,
-    )
-
-
-def test_sliding_window(benchmark, long_text_string):
-    """
-    Benchmarks the speed of SlidingWindowChunker in 'Fixed Size' mode
-    """
-
-    chunker = SlidingWindowChunker(chunk_size=500, overlap=50)
-
-    if hasattr(chunker, "progress_tracker"):
-        chunker.progress_tracker = MagicMock()
-
-    result = benchmark(chunker.chunk, text=long_text_string, preserve_boundaries=False)
-
-    assert len(result) > 0

benchmarks/mock_arrow_exporter.py

@@ -1,69 +0,0 @@
-"""
-Mock Arrow Exporter for Benchmark Testing
-
-This module provides a mock implementation of the ArrowExporter to prevent
-import errors during benchmark testing when PyArrow is not available in the CI environment.
-"""
-
-# Mock PyArrow import for CI compatibility
-try:
-    import pyarrow as pa
-except ImportError:
-    # Create a mock pa module for CI environment
-    import types
-    pa = types.ModuleType('pa')
-    
-    def mock_schema(*args, **kwargs):
-        return types.SimpleNamespace()
-    
-    def mock_table(*args, **kwargs):
-        return types.SimpleNamespace()
-    
-    def mock_array(*args, **kwargs):
-        return types.SimpleNamespace()
-    
-    pa.schema = mock_schema
-    pa.Table = mock_table
-    pa.array = mock_array
-    pa.RecordBatch = mock_table
-
-# Mock schema definitions
-ENTITY_SCHEMA = pa.schema([]) if hasattr(pa, 'schema') else None
-RELATIONSHIP_SCHEMA = pa.schema([]) if hasattr(pa, 'schema') else None
-METADATA_SCHEMA = pa.schema([]) if hasattr(pa, 'schema') else None
-
-class ArrowExporter:
-    """
-    Mock Arrow Exporter class for benchmark testing.
-    
-    This is a lightweight implementation that provides the same interface
-    as the real ArrowExporter but doesn't require PyArrow to be installed.
-    """
-    
-    def __init__(self, config=None):
-        self.config = config
-        self._tables = {}
-    
-    def export_entities(self, entities, output_path):
-        """Mock export entities method."""
-        return f"Mock exported {len(entities)} entities to {output_path}"
-    
-    def export_relationships(self, relationships, output_path):
-        """Mock export relationships method."""
-        return f"Mock exported {len(relationships)} relationships to {output_path}"
-    
-    def export_knowledge_graph(self, entities, relationships, output_path):
-        """Mock export knowledge graph method."""
-        return f"Mock exported knowledge graph to {output_path}"
-    
-    def to_arrow_table(self, data):
-        """Mock conversion to Arrow table."""
-        return f"Mock Arrow table with {len(data)} rows"
-    
-    def save_to_file(self, table, path):
-        """Mock save to file method."""
-        return f"Mock saved table to {path}"
-    
-    def batch_export(self, data_list, output_dir):
-        """Mock batch export method."""
-        return f"Mock batch exported {len(data_list)} items to {output_dir}"

benchmarks/normalize/conftest.py

@@ -1,62 +0,0 @@
-import random
-import string
-from typing import Any, Dict, List
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-# Data gen
-
-
-@pytest.fixture
-def generate_text_data():
-    """Generates various types of text data."""
-
-    def _gen(type="clean", length=100):
-        if type == "clean":
-            return "".join(random.choices(string.ascii_letters + " ", k=length))
-        elif type == "html":
-            tags = ["<div>", "<p>", "<span>", "<a>", "<b>", "<i>"]
-            content = "".join(random.choices(string.ascii_letters + " ", k=length))
-            return f"{random.choice(tags)}{content}{random.choice(tags).replace('<', '</')}"
-        elif type == "unicode":
-            chars = string.ascii_letters + "éàèùâêîôûçñ"
-            return "".join(random.choices(chars, k=length))
-        elif type == "dirty":
-            chars = string.ascii_letters + " \t\n\r"
-            return "".join(random.choices(chars, k=length))
-
-    return _gen
-
-
-@pytest.fixture
-def generate_dataset():
-    """Generates dataset for data cleaner."""
-
-    def _gen(rows=100, duplicate_rate=0.0):
-        base_rows = []
-        unique_count = int(rows * (1 - duplicate_rate))
-
-        for i in range(unique_count):
-            base_rows.append(
-                {
-                    "id": i,
-                    "name": f"Entity_{i}",
-                    "email": f"user{i}@yahoo.com",
-                    "value": random.random() * 100,
-                    "category": random.choice(["A", "B", "C"]),
-                }
-            )
-
-        final_dataset = base_rows.copy()
-        while len(final_dataset) < rows:
-            source = random.choice(base_rows)
-            dup = source.copy()
-            if random.random() > 0.5:
-                dup["value"] = source["value"] + 0.001
-            final_dataset.append(dup)
-
-        random.shuffle(final_dataset)
-        return final_dataset
-
-    return _gen

benchmarks/normalize/test_data_cleaner.py

@@ -1,38 +0,0 @@
-import pytest
-
-from semantica.normalize.data_cleaner import DataCleaner
-
-
-@pytest.mark.parametrize("rows", [100, 500])
-def test_duplication_detection_scaling(benchmark, generate_dataset, rows):
-    """
-    Benchmarks duplicate detection scaling.
-    """
-
-    cleaner = DataCleaner()
-    dataset = generate_dataset(rows=rows, duplicate_rate=0.2)
-
-    def run():
-        return cleaner.detect_duplicates(dataset, key_fields=["name", "email"])
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-def test_missing_value_imputation(benchmark, generate_dataset):
-    """
-    Benchmarks statistical imputation.
-    """
-    cleaner = DataCleaner()
-
-    def setup_broken_dataset():
-        dataset = generate_dataset(rows=5000)
-        for row in dataset:
-            if row["id"] % 5 == 0:
-                row["value"] = None
-
-        return (dataset,), {}
-
-    def run(data):
-        return cleaner.handle_missing_values(data, strategy="impute", method="mean")
-
-    benchmark.pedantic(target=run, setup=setup_broken_dataset, iterations=1, rounds=10)

benchmarks/normalize/test_heavy_libs.py

@@ -1,31 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.normalize.encoding_handler import EncodingHandler
-from semantica.normalize.language_detector import LanguageDetector
-
-
-def test_language_detection_throughput(benchmark, generate_text_data):
-    """Benchmarks langdetect intergration."""
-    detector = LanguageDetector()
-    texts = [generate_text_data("clean", 200) for _ in range(50)]
-
-    def run():
-        return detector.detect_batch(texts)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-def test_encoding_detection(benchmark):
-    """Benchmarks chardet integration via EncodingHandler."""
-    handler = EncodingHandler()
-    data = (
-        b"Wowzaaa a simple string for encoding decoding , oh encoding detection just."
-        * 100
-    )
-
-    def run():
-        return handler.detect(data)
-
-    benchmark.pedantic(run, iterations=5, rounds=10)

benchmarks/normalize/test_parsers.py

@@ -1,25 +0,0 @@
-import pytest
-
-from semantica.normalize.date_normalizer import DateNormalizer
-from semantica.normalize.number_normalizer import NumberNormalizer
-
-
-@pytest.mark.parametrize("date_str", ["2026-02-03", "Ferbuary 2nd, 2026", "9 days ago"])
-def test_data_parsing_variations(benchmark, date_str):
-    """Compare speed of different date formats."""
-    normalizer = DateNormalizer()
-    benchmark.pedantic(
-        lambda: normalizer.normalize_date(date_str), iterations=10, rounds=20
-    )
-
-
-def test_number_normalization(benchmark):
-    """Benchmarks number parsing with currency and unit stripping."""
-    normalizer = NumberNormalizer()
-    raw_inputs = ["$1,234.56", "1.5k", "50%", "1,000,000"] * 100
-
-    def run():
-        for n in raw_inputs:
-            normalizer.normalize_number(n)
-
-    benchmark.pedantic(run, iterations=5, rounds=20)

benchmarks/normalize/test_text_processing.py

@@ -1,42 +0,0 @@
-import pytest
-
-from semantica.normalize.text_cleaner import TextCleaner
-from semantica.normalize.text_normalizer import TextNormalizer
-
-
-def test_html_removal_reg_vs_bs4(benchmark, generate_text_data):
-    """
-    Compare regex vs BeautifulSoup.
-    """
-    cleaner = TextCleaner()
-    html_content = generate_text_data("html", 10_000)
-
-    def run():
-        return cleaner.remove_html(html_content, preserve_structure=False)
-
-    benchmark.pedantic(run, rounds=50, iterations=10)
-
-
-def test_unicode_normalization_throughput(benchmark, generate_text_data):
-    """
-    Benchmarks unicode NFC normalization speed.
-    """
-    normalizer = TextNormalizer()
-    text = generate_text_data("unicode", 50_000)
-
-    def run():
-        return normalizer.normalize_text(text, unicode_form="NFC")
-
-    benchmark.pedantic(run, iterations=5, rounds=10)
-
-
-def test_whitespace_normalization(benchmark, generate_text_data):
-    """Benchmarks whitespace regex replacement."""
-    normalizer = TextNormalizer()
-    text = generate_text_data("dirty", 50_000)
-
-    benchmark.pedantic(
-        lambda: normalizer.normalize_text(text, unicode_form="NFC"),
-        iterations=5,
-        rounds=10,
-    )

benchmarks/ontology/conftest.py

@@ -1,85 +0,0 @@
-import random
-import string
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-# Data generators
-
-
-def _random_str(length=8):
-    return "".join(random.choices(string.ascii_letters, k=length))
-
-
-@pytest.fixture
-def generate_ontology_data():
-    """
-    Generates a synthetic dataset of entities and relationships
-    designed to triger class and property inference class.
-    """
-
-    def _generate(entity_count: int, relationship_density: float = 1.5):
-
-        num_classes = max(5, entity_count // 50)
-        class_names = [f"Class_{_random_str(4)}" for _ in range(num_classes)]
-
-        entities = []
-
-        for i in range(entity_count):
-            cls = random.choice(class_names)
-
-            props = {
-                f"prop_{_random_str(3)}": random.choice([10, "text", 1.5, True])
-                for _ in range(random.randint(1, 5))
-            }
-
-            entity = {
-                "id": f"e_{i}",
-                "type": cls,
-                "name": f"Entity_{i}",
-                "confidence": 0.95,
-                **props,
-            }
-
-            entities.append(entity)
-
-        relationships = []
-        rel_count = int(entity_count * relationship_density)
-        rel_types = ["relatedTo", "hasPart", "worksFor", "contains", "memberOf"]
-
-        for _ in range(rel_count):
-            src = random.choice(entities)
-            tgt = random.choice(entities)
-            rel = {
-                "source": src["name"],
-                "target": tgt["name"],
-                "type": random.choice(rel_types),
-                "source_type": src["type"],
-                "target_type": tgt["type"],
-                "confidence": 0.8,
-            }
-            relationships.append(rel)
-
-        return {"entities": entities, "relationships": relationships}
-
-    return _generate
-
-
-@pytest.fixture
-def large_ontology_definition(generate_ontology_data):
-    """Pre-calculates a structured ontology
-    definition dictionary.
-    """
-    from semantica.ontology.ontology_generator import OntologyGenerator
-
-    data = generate_ontology_data(entity_count=1000)
-
-    # Mocking validation in 6-step pipeline to speed up setup
-
-    with patch(
-        "semantica.ontology.ontology_validator.OntologyValidator.validate"
-    ) as mock_val:
-        mock_val.return_value.valid = True
-        gen = OntologyGenerator()
-
-        return gen.generate_ontology(data, validate=False)

benchmarks/ontology/test_inference.py

@@ -1,70 +0,0 @@
-import pytest
-
-from semantica.ontology.class_inferrer import ClassInferrer
-from semantica.ontology.property_generator import PropertyGenerator
-
-
-@pytest.mark.benchmark(group="class_Inference")
-@pytest.mark.parametrize("entity_count", [1000, 5000])
-def test_class_inference_scaling(benchmark, generate_ontology_data, entity_count):
-    """
-    Benchmarks grouping and threshold logic in ClassInferrer.
-    """
-
-    data = generate_ontology_data(entity_count=entity_count)
-    inferrer = ClassInferrer(min_occurrences=2)
-
-    def run():
-        return inferrer.infer_classes(data["entities"])
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="property_inference")
-@pytest.mark.parametrize("size", [(1000, 1500)])
-def test_property_inference_scaling(benchmark, generate_ontology_data, size):
-    """
-    Benchmarks: PropertyGenerator
-    """
-
-    e_count, _ = size
-    data = generate_ontology_data(entity_count=e_count)
-
-    inferrer = ClassInferrer()
-    classes = inferrer.infer_classes(data["entities"])
-
-    prop_gen = PropertyGenerator()
-
-    def run():
-        return prop_gen.infer_properties(
-            entities=data["entities"],
-            relationships=data["relationships"],
-            classes=classes,
-        )
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-def test_hierarchy_circular_detection(benchmark):
-    """
-    Benchmarks the DFS cycle detection in ClassInferrer.
-    """
-
-    inferrer = ClassInferrer()
-
-    # Create a deep chain A -> B -> C ... -> Z
-
-    chain_length = 200
-    classes = []
-
-    for i in range(chain_length):
-        cls = {
-            "name": f"Class_{i}",
-            "subClassOf": f"Class_{i+1}" if i < chain_length - 1 else None,
-        }
-        classes.append(cls)
-
-    def run():
-        return inferrer.validate_classes(classes)
-
-    benchmark.pedantic(run, iterations=1, rounds=10)

benchmarks/ontology/test_ontology_pipeline.py

@@ -1,46 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.ontology.ontology_generator import OntologyGenerator
-
-
-@pytest.mark.benchmark(group="full_pipeline")
-@pytest.mark.parametrize("entity_count", [1000])
-def test_e2e_ontology_generation(benchmark, generate_ontology_data, entity_count):
-    """
-    Benchmarks complete 6-stage pipeline
-    """
-
-    data = generate_ontology_data(entity_count)
-    generator = OntologyGenerator()
-
-    with patch(
-        "semantica.ontology.ontology_validator.OntologyValidator.validate"
-    ) as mock_val:
-        mock_val.return_value.valid = True
-
-        def run():
-            return generator.generate_ontology(data, validate=True)
-
-        benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-def test_associative_class_creation(benchmark):
-    """
-    Benchmarks the creation of complex N-ary relationships.
-    """
-    from semantica.ontology.associative_class import AssociativeClassBuilder
-
-    builder = AssociativeClassBuilder()
-
-    def run():
-        for i in range(50):
-            builder.create_position_class(
-                person_class=f"Person_{i}",
-                organization_class=f"Org_{i}",
-                role_class=f"Role_{i}",
-                name=f"Position_{i}",
-            )
-
-    benchmark.pedantic(run, iterations=1, rounds=10)

benchmarks/ontology/test_reuse.py

@@ -1,43 +0,0 @@
-import pytest
-
-from semantica.ontology.namespace_manager import NamespaceManager
-from semantica.ontology.reuse_manager import ReuseManager
-
-
-def test_namespace_iri_generation(benchmark):
-    """
-    High-throughput test for IRI Generation.
-    """
-    manager = NamespaceManager(base_uri="https://semantica.dev/bench/")
-    names = [f"EntityName_{i}" for i in range(1000)]
-
-    def run():
-        for name in names:
-            manager.generate_class_iri(name)
-
-    benchmark.pedantic(run, iterations=1, rounds=20)
-
-
-def test_ontology_merging(benchmark, large_ontology_definition):
-    """
-    Benchmarks merging two large entities together.
-    """
-    manager = ReuseManager()
-    target = large_ontology_definition.copy()
-    source = large_ontology_definition.copy()
-
-    new_classes = []
-
-    for c in source["classes"]:
-        base_id = c.get("uri") or c.get("name") or "UnkownEntity"
-        new_c = c.copy()
-        new_c["uri"] = f"{base_id}_merged"
-        new_classes.append(new_c)
-
-    source["classes"] = new_classes
-
-    def run():
-        t_copy = target.copy()
-        return manager.merge_ontology_data(t_copy, source, overwrite=False)
-
-    benchmark.pedantic(run, iterations=1, rounds=10)

benchmarks/ontology/test_serialization.py

@@ -1,33 +0,0 @@
-import pytest
-
-from semantica.ontology.owl_generator import OWLGenerator
-
-
-@pytest.mark.benchmark(group="serialization")
-@pytest.mark.parametrize("format", ["turtle", "xml"])
-def test_owl_serialization_formats(benchmark, large_ontology_definition, format):
-    """Benchmarks the cost of serializing the ontology
-    to different string formats.
-    """
-    generator = OWLGenerator()
-
-    def run():
-        return generator.generate_owl(large_ontology_definition, format=format)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-def test_rdflib_graph_construction(benchmark, large_ontology_definition):
-    """
-    Benchmarks the creation of rdflib.Graph object.
-    """
-    generator = OWLGenerator()
-
-    def run():
-        if hasattr(generator, "_generate_with_rdflib"):
-            return generator._generate_with_rdflib(
-                large_ontology_definition, format="turtle"
-            )
-        return generator.generate_owl(large_ontology_definition)
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/output_orchestration/test_execution_pipeline.py

@@ -1,98 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.pipeline.execution_engine import ExecutionEngine
-from semantica.pipeline.pipeline_builder import PipelineBuilder, StepStatus
-from semantica.pipeline.resource_scheduler import ResourceScheduler
-
-
-# ~~ Fixtures
-@pytest.fixture(autouse=True)
-def kill_hardware_checks():
-    with patch.object(ResourceScheduler, "_initialize_resources", return_value=None):
-        yield
-
-
-@pytest.fixture(autouse=True)
-def kill_logging():
-    with patch("semantica.utils.logging.get_logger"):
-        yield
-
-
-@pytest.fixture(autouse=True)
-def kill_tracker():
-    mock_tracker = MagicMock()
-    mock_tracker.enabled = False
-    with patch(
-        "semantica.pipeline.execution_engine.get_progress_tracker",
-        return_value=mock_tracker,
-    ):
-        yield
-
-
-def create_pipeline(size):
-    """Helper to generate pipelines of random size."""
-    builder = PipelineBuilder()
-    builder.progress_tracker = MagicMock()
-    builder.progress_tracker.enabled = False
-    handler = lambda x, **k: x
-
-    builder.add_step("start", "dummy", handler=handler)
-    for i in range(1, size):
-        builder.add_step(f"step_{i}", "dummy", handler=handler)
-        builder.connect_steps("start" if i == 1 else f"step_{i-1}", f"step_{i}")
-
-    return builder.build(f"bench_pipe_{size}")
-
-
-# ~~ Benchmarks ~~
-
-
-@pytest.mark.parametrize("step_count", [10, 100, 500])
-def test_pipeline_construction_scaling(benchmark, step_count):
-    """
-    Verifies if construction time scales linearly.
-    """
-
-    def op():
-        builder = PipelineBuilder()
-        builder.progress_tracker = MagicMock()
-        for i in range(step_count):
-            builder.add_step(f"s{i}", "t")
-        return builder.build()
-
-    benchmark.pedantic(op, iterations=5, rounds=5)
-
-
-@pytest.mark.parametrize("step_count", [10, 100])
-def test_execution_overhead_scaling(benchmark, step_count):
-    """
-    Measures per-step overhead as it gets more complex
-    """
-    engine = ExecutionEngine()
-    pipeline = create_pipeline(step_count)
-
-    def setup_run():
-        for step in pipeline.steps:
-            step.status = StepStatus.PENDING
-            step.result = None
-        return (pipeline,), {"data": {"val": 1}}
-
-    def op(pipeline, data):
-        return engine.execute_pipeline(pipeline, data=data)
-
-    benchmark.pedantic(op, setup=setup_run, iterations=1, rounds=10)
-
-
-@pytest.mark.parametrize("step_count", [10, 100, 1000])
-def test_topological_sort_scaling(benchmark, step_count):
-    """
-    Stress test for dependency graph algorithm.
-    """
-    engine = ExecutionEngine()
-    pipeline = create_pipeline(step_count)
-
-    benchmark.pedantic(
-        lambda: engine._topological_sort(pipeline.steps), iterations=20, rounds=10
-    )

benchmarks/output_orchestration/test_parallelism.py

@@ -1,91 +0,0 @@
-import time
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.pipeline.parallelism_manager import ParallelismManager, Task
-from semantica.pipeline.resource_scheduler import ResourceScheduler
-
-
-# ~~ Fixtures ~~
-@pytest.fixture(autouse=True)
-def kill_hardware_checks():
-    with patch.object(ResourceScheduler, "_initialize_resources", return_value=None):
-        yield
-
-
-@pytest.fixture(autouse=True)
-def kill_logging():
-    with patch("semantica.utils.logging.get_logger"):
-        yield
-
-
-@pytest.fixture(autouse=True)
-def kill_tracker():
-    mock_tracker = MagicMock()
-    mock_tracker.enabled = False
-    with patch(
-        "semantica.pipeline.parallelism_manager.get_progress_tracker",
-        return_value=mock_tracker,
-    ):
-        yield
-
-
-def blocking_task(duration):
-    """Simulates a task that waits for I/O (like a DB query or API call)."""
-    time.sleep(duration)
-    return True
-
-
-@pytest.fixture
-def thread_manager():
-    return ParallelismManager(max_workers=4, use_processes=False)
-
-
-@pytest.fixture
-def process_manager():
-    return ParallelismManager(max_workers=4, use_processes=True)
-
-
-# ~~ BENCHMARKS ~~
-
-
-def test_parallel_vs_serial_io(benchmark, thread_manager):
-    """
-    Runs 4 tasks that sleep for 0.1s.
-    """
-    tasks = [
-        Task(task_id=f"t{i}", handler=blocking_task, args=(0.1,)) for i in range(4)
-    ]
-
-    def op():
-        return thread_manager.execute_parallel(tasks)
-
-    benchmark.pedantic(op, iterations=1, rounds=5)
-
-
-def test_thread_pool_overhead(benchmark, thread_manager):
-    """
-    Measures the raw cost of spinning up threads for zero-work tasks.
-    """
-    # No-op handler
-    noop = lambda: None
-    tasks = [Task(task_id=f"t{i}", handler=noop) for i in range(100)]
-
-    def op():
-        return thread_manager.execute_parallel(tasks)
-
-    benchmark.pedantic(op, iterations=5, rounds=10)
-
-
-def test_process_pool_overhead(benchmark, process_manager):
-    """
-    Measures overhead of ProcessPoolExecutor
-    """
-    noop = lambda: None
-    tasks = [Task(task_id=f"t{i}", handler=noop) for i in range(10)]
-
-    def op():
-        return process_manager.execute_parallel(tasks)
-
-    benchmark.pedantic(op, iterations=1, rounds=5)

benchmarks/quality_assurance/test_conflicts.py

@@ -1,84 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.deduplication.merge_strategy import MergeStrategy, MergeStrategyManager
-
-# Fixtures
-
-
-@pytest.fixture
-def conflict_manager():
-    """Returns a MergeStrategyManager with default settings."""
-    return MergeStrategyManager()
-
-
-@pytest.fixture
-def conflicting_entities_batch():
-    """
-    Generates a list of 100 entities that are all 'duplicates' of each other
-    but have conflicting property values. This forces the resolution logic to run hard.
-    """
-    entities = []
-    for i in range(100):
-        entities.append(
-            {
-                "id": "e_1",
-                "name": f"Entity Name {i}",
-                "type": "Person",
-                "confidence": 0.5 + (i * 0.005),
-                "properties": {
-                    "age": 20 + i,
-                    "email": f"user{i}@example.com",
-                    "status": "active" if i % 2 == 0 else "inactive",
-                },
-                "relationships": [
-                    {"source": "e_1", "target": f"other_{i}", "type": "knows"}
-                ],
-            }
-        )
-    return entities
-
-
-# Benchmarks
-
-
-def test_strategy_keep_highest_confidence(
-    benchmark, conflict_manager, conflicting_entities_batch
-):
-    """
-    Benchmarks 'KEEP_HIGHEST_CONFIDENCE'.
-    """
-
-    def op():
-        return conflict_manager.merge_entities(
-            conflicting_entities_batch, strategy=MergeStrategy.KEEP_HIGHEST_CONFIDENCE
-        )
-
-    benchmark.pedantic(op, iterations=10, rounds=10)
-
-
-def test_strategy_merge_all(benchmark, conflict_manager, conflicting_entities_batch):
-    """
-    Benchmarks 'MERGE_ALL'.
-    """
-
-    def op():
-        return conflict_manager.merge_entities(
-            conflicting_entities_batch, strategy=MergeStrategy.MERGE_ALL
-        )
-
-    benchmark.pedantic(op, iterations=10, rounds=10)
-
-
-def test_property_resolution_overhead(benchmark, conflict_manager):
-    """
-    Micro-benchmark for the inner _resolve_property_conflict logic.
-    """
-
-    def op():
-        return conflict_manager._resolve_property_conflict(
-            "age", 25, 30, MergeStrategy.KEEP_MOST_COMPLETE
-        )
-
-    benchmark.pedantic(op, iterations=1000, rounds=20)

benchmarks/quality_assurance/test_deduplication.py

@@ -1,338 +0,0 @@
-import random
-import string
-import time
-from typing import Any, Dict, List
-from unittest.mock import patch
-
-import numpy as np
-import pytest
-
-from semantica.deduplication.cluster_builder import ClusterBuilder
-from semantica.deduplication.duplicate_detector import DuplicateDetector
-from semantica.deduplication.entity_merger import EntityMerger
-from semantica.deduplication.similarity_calculator import SimilarityCalculator
-
-# Infra
-
-
-class NullTracker:
-    """
-    Discards all data to prevent memory leaks
-    """
-
-    def start_tracking(self, *args, **kwargs):
-        return "dummy_id"
-
-    def update_tracking(self, *args, **kwargs):
-        pass
-
-    def stop_tracking(self, *args, **kwargs):
-        pass
-
-    def register_pipeline_modules(self, *args, **kwargs):
-        pass
-
-    def clear_pipeline_context(self, *args, **kwargs):
-        pass
-
-    def update_progress(self, *args, **kwargs):
-        pass
-
-    @property
-    def enabled(self):
-        return False
-
-    @enabled.setter
-    def enabled(self, value):
-        pass
-
-
-@pytest.fixture(autouse=True)
-def kill_io_overhead():
-    """
-    Replaces ProgressTracker with NullTracker globally.
-    """
-    with patch("semantica.utils.logging.get_logger"), patch(
-        "semantica.utils.progress_tracker.get_progress_tracker"
-    ) as mock_getter:
-
-        mock_getter.return_value = NullTracker()
-
-        with patch(
-            "semantica.deduplication.similarity_calculator.get_progress_tracker",
-            return_value=NullTracker(),
-        ), patch(
-            "semantica.deduplication.duplicate_detector.get_progress_tracker",
-            return_value=NullTracker(),
-        ), patch(
-            "semantica.deduplication.cluster_builder.get_progress_tracker",
-            return_value=NullTracker(),
-        ):
-            yield
-
-
-# Sim data
-
-
-def generate_entity_cluster(base_name: str, size: int) -> List[Dict[str, Any]]:
-    """
-    Generates a cluster of similar entities based on a seed name.
-    Example: "Apple" -> ["Apple Inc", "Apple Corp", etc.]
-    """
-
-    entities = []
-    suffixes = ["Inc", "Corp", "Ltd", "Gmbh", "LLC", "Group", "Systems"]
-
-    for i in range(size):
-        if random.random() < 0.8:
-            name = f"{base_name} {random.choice(suffixes)}"
-        else:
-            # Generating a typo for our calc to work on
-            chars = list(base_name)
-            if len(chars) > 2:
-                idx = random.randint(0, len(chars) - 2)
-                chars[idx], chars[idx + 1] = chars[idx + 1], chars[idx]
-            name = "".join(chars)
-
-        entities.append(
-            {
-                "id": f"{base_name.lower()}_{i}",
-                "name": name,
-                "type": "Organization",
-                "properties": {
-                    "location": "USA" if i % 2 == 0 else "California",
-                    "sector": "Tech",
-                    "employee_count": 100 + i,
-                },
-            }
-        )
-
-    return entities
-
-
-def generate_relationship_dataset(size: int) -> List[Dict[str, Any]]:
-    """
-    Generates a dataset of graph relationships/triplets.
-    Includes exact matches, synonym predicates, and dirty literal strings.
-    """
-    relationships = []
-    predicates = ["works_for", "employed_by", "is_employee_of", "has_employer"]
-    
-    for i in range(size):
-        # Base relationship
-        rel = {
-            "subject": f"Person_{i % 50}",
-            "predicate": random.choice(predicates),
-            "object": f"Company_{i % 10}"
-        }
-        relationships.append(rel)
-        
-        # Inject semantic duplicates (dirty literals / synonym predicates)
-        if random.random() < 0.4:
-            dirty_rel = {
-                "subject": f"Person_{i % 50}",
-                "predicate": random.choice(predicates),
-                "object": f"  Company_{i % 10} Inc.  " 
-            }
-            relationships.append(dirty_rel)
-            
-    return relationships
-
-
-def generate_dataset(
-    num_clusters: int, items_per_cluster: int, worst_case_blocking: bool = False
-):
-    """
-    Generates a full dataset
-
-    Args:
-      worst_case_blocking: If True, all names start with 'A' to defeat
-                           first-char blocking strategy in SimilarityCalculator.
-
-    """
-    dataset = []
-    for i in range(num_clusters):
-        if worst_case_blocking:
-            # All starts with 'A'
-            base_name = f"A_Company_{i}"
-        else:
-            start_char = random.choice(string.ascii_uppercase)
-            base_name = f"{start_char}_company_{i}"
-
-        cluster = generate_entity_cluster(base_name, items_per_cluster)
-        dataset.extend(cluster)
-
-    return dataset
-
-
-# ~~ Benchmarks ~~
-
-
-@pytest.mark.parametrize("method", ["levenshtein", "jaro_winkler"])
-def test_string_metric_speed(benchmark, method):
-    """
-    Measures the speed of string comparison algos.
-    """
-
-    calc = SimilarityCalculator()
-    s1 = "International Business Machines Corporation"
-    s2 = "International Business Machine Corp."
-
-    benchmark.pedantic(
-        lambda: calc.calculate_string_similarity(s1, s2, method=method),
-        iterations=1000,
-        rounds=100,
-    )
-
-
-def test_full_similarity_calculation(benchmark):
-    """
-    Measures weighted multi-factor calculation overhead.
-    (String + Property + Relationship + Weights).
-    """
-
-    calc = SimilarityCalculator(
-        string_weight=0.5, property_weight=0.3, relationship_weight=0.2
-    )
-
-    e1 = {
-        "name": "Acme Corp",
-        "properties": {"loc": "NY", "id": "123"},
-        "relationships": [{"target": "t1"}, {"target": "t2"}],
-    }
-
-    e2 = {
-        "name": "Acme Inc",
-        "properties": {"loc": "NY", "id": "123"},
-        "relationships": [{"target": "t1"}, {"target": "t2"}],
-    }
-
-    benchmark.pedantic(
-        lambda: calc.calculate_similarity(e1, e2), iterations=1000, rounds=50
-    )
-
-
-@pytest.mark.parametrize("dataset_size", [100, 500])
-def test_duplicate_detection_scaling_opt(benchmark, dataset_size):
-    """
-    Tests duplication on a 'Distributed' dataset (Best Case)
-    Now utilizing V2 Candidate Generation to ensure no regressions.
-    """
-    data = generate_dataset(
-        num_clusters=dataset_size // 10, items_per_cluster=10, worst_case_blocking=False
-    )
-    
-    detector = DuplicateDetector(
-        similarity_threshold=0.8,
-        similarity={
-            "candidate_strategy": "blocking_v2", 
-            "max_candidates_per_entity": 50,
-            "prefilter_enabled": True,
-            "score_breakdown_enabled": True,
-            "prefilter_thresholds": {
-                "min_length_ratio": 0.4,
-                "require_shared_token": True
-            }
-        }
-    )
-
-    benchmark.pedantic(lambda: detector.detect_duplicates(data), iterations=1, rounds=5)
-
-
-@pytest.mark.parametrize("dataset_size", [100, 500])
-def test_duplicate_detection_worst_Case(benchmark, dataset_size):
-    """
-    Tests detection on a 'Clustered' dataset (Worst Case).
-    Now utilizing V2 Candidate Generation to cut the pair explosion.
-    """
-    data = generate_dataset(
-        num_clusters=dataset_size // 10, items_per_cluster=10, worst_case_blocking=True
-    )
-    
-    detector = DuplicateDetector(
-        similarity_threshold=0.8,
-        similarity={
-            "candidate_strategy": "blocking_v2", 
-            "max_candidates_per_entity": 50,
-            "prefilter_enabled": True,
-            "score_breakdown_enabled": True,
-            "prefilter_thresholds": {
-                "min_length_ratio": 0.4,
-                "require_shared_token": True
-            }
-        }
-    )
-
-    benchmark.pedantic(lambda: detector.detect_duplicates(data), iterations=1, rounds=5)
-
-
-def test_incremental_detection_speed(benchmark):
-    """
-    Measures performance of adding new data to existing index.
-    """
-
-    existing = generate_dataset(num_clusters=50, items_per_cluster=5)
-    new_data = generate_dataset(num_clusters=5, items_per_cluster=2)
-
-    detector = DuplicateDetector()
-
-    benchmark.pedantic(
-        lambda: detector.incremental_detect(new_data, existing), iterations=5, rounds=10
-    )
-
-
-@pytest.mark.parametrize("algo", ["graph", "hierarchical"])
-def test_clustering_strategy_performance(benchmark, algo):
-    """
-    Comapres Union-Fund (Graph) vs Hierarchical Clustering.
-    """
-
-    data = generate_dataset(num_clusters=20, items_per_cluster=10)
-
-    use_hierarchical = algo == "hierarchical"
-    builder = ClusterBuilder(use_hierarchical=use_hierarchical)
-
-    benchmark.pedantic(lambda: builder.build_clusters(data), iterations=1, rounds=5)
-
-
-def test_merge_entity_benchmark(benchmark):
-    """
-    Measures the cost of fusing entities / res conflicts.
-    """
-
-    group = generate_entity_cluster("MegaCorp", 50)
-    merger = EntityMerger()
-
-    benchmark.pedantic(
-        lambda: merger.merge_entity_group(group, strategy="keep_most_complete"),
-        iterations=10,
-        rounds=10,
-    )
-    
-
-@pytest.mark.parametrize("mode", ["legacy", "semantic_v2"])
-def test_relationship_dedup_speed(benchmark, mode):
-    """
-    Measures the speed of relationship/triplet deduplication.
-    Compares the O(N^2) legacy fallback vs the fast canonical hash path.
-    """
-    # Yields ~280 relationships (approx 39,000 comparisons in O(N^2))
-    relationships = generate_relationship_dataset(200) 
-    
-    detector = DuplicateDetector()
-    options = {
-        "threshold": 0.85,
-        "relationship_dedup_mode": mode,
-        "predicate_synonym_map": {
-            "works_for": "employed_by",
-            "is_employee_of": "employed_by",
-            "has_employer": "employed_by"
-        },
-        "literal_normalization_enabled": True
-    }
-
-    benchmark.pedantic(
-        lambda: detector.detect_relationship_duplicates(relationships, **options),
-        iterations=5,
-        rounds=10,
-    )

benchmarks/requirements.txt

@@ -1,43 +0,0 @@
-# Benchmark Tools
-
-pytest>=7.0.0
-pytest-benchmark>=4.0.0
-
-# Core Utils
-
-pydantic
-loguru
-chardet
-requests
-greenlet
-typing-extensions
-tqdm
-click
-rich
-
-numpy
-pandas
-networkx
-scikit-learn
-
-# Graph & Storage
-
-sqlalchemy
-rdflib
-neo4j
-redis
-
-# AI proc
-
-torch
-transformers
-sentence-transformers
-spacy
-beautifulsoup4
-lxml
-pypdf2
-python-docx
-openpyxl
-pillow
-feedparser
-GitPython

benchmarks/storage/test_embeddings.py

@@ -1,180 +0,0 @@
-from typing import Generator, List
-from unittest.mock import MagicMock, patch
-
-import numpy as np
-import pytest
-
-from semantica.embeddings.embedding_generator import EmbeddingGenerator
-from semantica.embeddings.graph_embedding_manager import GraphEmbeddingManager
-from semantica.embeddings.pooling_strategies import PoolingStrategyFactory
-from semantica.embeddings.text_embedder import TextEmbedder
-
-
-# Infra Mocks
-@pytest.fixture(autouse=True)
-def kill_io_overhead():
-    """Silences logging and tracker globally."""
-    with patch("semantica.utils.logging.get_logger"), patch(
-        "semantica.utils.progress_tracker.get_progress_tracker"
-    ) as mock_tracker:
-
-        tracker = MagicMock()
-        tracker.enabled = False
-        tracker._start_tracking.return_value = "dummy_id"
-        mock_tracker.return_value = tracker
-
-        with patch(
-            "semantica.embeddings.text_embedder.get_progress_tracker",
-            return_value=tracker,
-        ):
-            yield
-
-
-# __ Model Mocks __
-
-
-class MockSentenceTransformer:
-    """
-    Simulates ST.encode without loading the fat model itself.
-    """
-
-    def __init__(self, dim=384):
-        self.dim = dim
-
-    def encode(
-        self, sentences: List[str], normalize_embeddings=True, **kwargs
-    ) -> np.ndarray:
-        count = len(sentences)
-        return np.random.rand(count, self.dim).astype(np.float32)
-
-    def get_sentence_embedding_dimension(self):
-        return self.dim
-
-
-class MockFastEmbed:
-    """
-    Simulates FastEmbed.embed generator behavior.
-    """
-
-    def __init__(self, dim=384):
-        self.dim = dim
-
-    def embed(self, documents: List[str]) -> Generator[np.ndarray, None, None]:
-        for _ in documents:
-            yield np.random.rand(self.dim).astype(np.float32)
-
-
-# ~~ Fixtures ~~
-@pytest.fixture
-def text_embedder_st():
-    """
-    Text embedder configured with SentenceTransformer
-    """
-    embedder = TextEmbedder(method="sentence_transformers", model_name="mock-bert")
-    embedder.model = MockSentenceTransformer()
-    embedder.progress_tracker = MagicMock()
-    embedder.progress_tracker.enabled = False
-
-    return embedder
-
-
-@pytest.fixture
-def text_embedder_fast():
-    """
-    Text Embedder cofnigures with Mock FastEmbed.
-    """
-
-    embedder = TextEmbedder(method="fastembed", model_name="mock-bge")
-    embedder.fastembed_model = MockFastEmbed()
-    embedder.progress_tracker = MagicMock()
-    embedder.progress_tracker.enabled = False
-    return embedder
-
-
-# ~~ Benchmarks
-
-
-@pytest.mark.parametrize("strategy", ["mean", "max", "cls", "attention"])
-def test_pooling_math_speed(benchmark, strategy):
-    """
-    Measures the raw NumPy speed of pooling strategies.
-    Scenario: Pooling a batch of 128 token embeddings.
-    """
-
-    embeddings = np.random.rand(128, 768).astype(np.float32)
-    pooler = PoolingStrategyFactory.create(strategy)
-
-    benchmark.pedantic(lambda: pooler.pool(embeddings), iterations=1000, rounds=100)
-
-
-def test_hierarchical_pooling_overhead(benchmark):
-    """
-    Measures the overhead of two-step hierarchical pooling.
-    """
-
-    embeddings = np.random.rand(1000, 768).astype(np.float32)
-    pooler = PoolingStrategyFactory.create("hierarchical", chunk_size=100)
-
-    benchmark.pedantic(lambda: pooler.pool(embeddings), iterations=500, rounds=50)
-
-
-def test_st_wrapper_overhead(benchmark, text_embedder_st):
-    """
-    Measures overhead of TextEmbedder wrapper around SentenceTransformers.
-    """
-
-    text = "This is a whatever we are doing here since idk"
-
-    benchmark.pedantic(
-        lambda: text_embedder_st.embed_text(text), iterations=1000, rounds=20
-    )
-
-
-def test_fastembed_generator_consumption(benchmark, text_embedder_fast):
-    """
-    Measures the cost of consuming the FastEmbed generator
-    and converting to Array.
-    """
-    texts = [f"Sentence {i}" for i in range(20)]
-
-    benchmark.pedantic(
-        lambda: text_embedder_fast.embed_batch(texts), iterations=100, rounds=20
-    )
-
-
-@pytest.mark.parametrize("batch_size", [10, 100, 1000])
-def test_batch_processing_pipeline(benchmark, batch_size, text_embedder_st):
-    """
-    Measures the full EmbeddingGenerator pipeline:
-    Input validation -> Type detection -> Batching -> Mock Model -> Error handling.
-    """
-
-    generator = EmbeddingGenerator()
-
-    generator.text_embedder = text_embedder_st
-    generator.progress_tracker = MagicMock()
-    generator.progress_tracker.enabled = False
-
-    data = [f"Item {i}" for i in range(batch_size)]
-
-    benchmark.pedantic(lambda: generator.process_batch(data), iterations=5, rounds=10)
-
-
-@pytest.mark.parametrize("count", [100, 1000])
-def test_graph_embedding_prep(benchmark, count, text_embedder_st):
-    """
-    Measures how fast we can reshape dict for GraphDBs
-    """
-    manager = GraphEmbeddingManager()
-    manager.embedding_generator.text_embedder = text_embedder_st
-
-    manager.embedding_generator.generate_embeddings = MagicMock(
-        return_value=np.random.rand(count, 384).astype(np.float32)
-    )
-
-    entities = [{"id": f"e{i}", "text": f"Entity{i}"} for i in range(count)]
-
-    def op():
-        return manager.prepare_for_graph_db(entities, backend="neo4j")
-
-    benchmark.pedantic(op, iterations=10, rounds=10)

benchmarks/storage/test_graph_store.py

@@ -1,137 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.graph_store.graph_store import GraphStore
-
-
-@pytest.fixture
-def mock_neo4j_driver():
-    """
-    Creates a mock of of Neo4j Driver
-    Simulates: Driver -> Session -> Transaction -> Result -> Record
-    """
-
-    mock_result = MagicMock()
-    fake_props = {"name": "TestNode", "age": 30}
-
-    def get_item(key):
-        if key == "id":
-            return 12345
-        if key == "n":
-            return fake_props
-        if key == "count":
-            return 42
-        return None
-
-    mock_record = MagicMock()
-    mock_record.__getitem__.side_effect = get_item
-    mock_record.keys.return_value = ["id", "n"]
-    mock_record.values.return_value = [12345, fake_props]
-
-    # dict conversion - essentially doing it because the db sometimes demands it
-    mock_record.items.return_value = [("id", 12345), ("n", fake_props)]
-
-    # ~~ Result Methods ~~
-    mock_result = MagicMock()
-    mock_result.single.return_value = mock_record
-    mock_result.__iter__.side_effect = lambda: iter([mock_record])
-
-    # ~~ Session ~~
-    mock_session = MagicMock()
-    mock_session.run.return_value = mock_result
-    mock_session.__enter__.return_value = mock_session
-    mock_session.__exit__.return_value = None
-
-    # ~~ Driver ~~
-    mock_driver = MagicMock()
-    mock_driver.session.return_value = mock_session
-    mock_driver.verify_connectivity.return_value = True
-
-    return mock_driver
-
-
-@pytest.fixture
-def graph_store(mock_neo4j_driver):
-    """
-    Returns a GraphsStore connected to mnock driver.
-    """
-
-    # ~~ Patch GraphDatbase ~~
-    with patch("semantica.graph_store.neo4j_store.GraphDatabase") as mockDB:
-        mockDB.driver.return_value = mock_neo4j_driver
-        store = GraphStore(
-            backend="neo4j", uri="bolt://mock:7687", user="mock", password="mock"
-        )
-        store.connect()
-
-        if hasattr(store, "progress_tracker"):
-            store.progress_tracker = MagicMock()
-
-        return store
-
-
-#    ~~ Benchmarks ~~
-
-
-def test_node_creation_overhead(benchmark, graph_store):
-    """
-    Benchamrks the full stack overhead for creating a single node.
-    Path: GraphStore -> NodeManager -> Neo4jStore, Driver
-    """
-
-    def op():
-        return graph_store.create_node(
-            labels=["Person"], properties={"name": "Alexander", "age": 17}
-        )
-
-    result = benchmark(op)
-    assert result["id"] == 12345
-
-
-def test_batch_node_creation_overhead(benchmark, graph_store):
-    """
-    Benchmarks the loop overhead in create_nodes (Batch).
-    Checks if it handles lists efficiently.
-    """
-
-    nodes = [{"labels": ["Person"], "properties": {"id": i}} for i in range(50)]
-
-    def op():
-        return graph_store.create_nodes(nodes)
-
-    result = benchmark(op)
-    assert len(result) == 50
-
-
-def test_query_construction_and_parsing(benchmark, graph_store):
-    """
-    Benchmarks every execution overhead.
-    Measures how fast `QueryEngine` parses result into a Python dict.
-    """
-
-    query = "MATCH ( n:Person) RETURN n LIMIT 1"
-
-    def op():
-        return graph_store.execute_query(query)
-
-    result = benchmark(op)
-    assert result["success"] is True
-    assert len(result["records"]) > 0
-
-
-def test_analytics_shortest_path_overhead(benchmark, graph_store):
-    """
-    Benchmarks the wrapper overhead for graph analytics.
-    """
-
-    def op():
-        return graph_store.shortest_path(
-            start_node_id=1, end_node_id=2, rel_type="KNOWS"
-        )
-
-    try:
-        benchmark(op)
-    except Exception:
-        # v pass as we are only trying to benchmark the function overhead call mainly
-        pass

benchmarks/storage/test_triplet_storage.py

@@ -1,146 +0,0 @@
-import time
-from dataclasses import dataclass
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from semantica.triplet_store.bulk_loader import BulkLoader
-from semantica.triplet_store.jena_store import JenaStore
-from semantica.triplet_store.triplet_store import TripletStore
-
-# ~~ Mocking ~~
-# We basically define a facile Triplet class for creating ds devoid of fat AI models
-
-
-@dataclass
-class SimpleTriplet:
-    subject: str
-    predicate: str
-    object: str
-    confidence: float = 1.0
-
-
-# ~~ Fixtures ~~
-@pytest.fixture
-def triplet_batch():
-    """Generates 1000 triplets."""
-    return [
-        SimpleTriplet(
-            subject=f"http://gandhara.org/entity/{i}",
-            predicate="http://gandhara.org/relation/knows",
-            object=f"http://example.org/entity/{i+1}",
-        )
-        for i in range(1000)
-    ]
-
-
-@pytest.fixture
-def large_knowledge_graph_dict():
-    """
-    Generates a large dict (1000 ent) to test parsing
-    logic in `TripletStore.store()`
-    """
-    entities = [
-        {
-            "id": f"ent_{i}",
-            "type": "Person",
-            "properties": {"name": f"Person {i}", "age": 60},
-        }
-        for i in range(1000)
-    ]
-    relationships = [
-        {"source": f"ent_{i}", "target": f"ent_{i+1}", "type": "KNOWS"}
-        for i in range(999)
-    ]
-
-    return {"entities": entities, "relationships": relationships}
-
-
-@pytest.fixture
-def in_memory_store():
-    """Returns a real JenaStore using RDFLib (In-Mmeory)."""
-
-    store = JenaStore(endpoint=None)
-    if store.graph is None:
-        pytest.fail("JenaStore failed to initialize rdflib graph.")
-    if hasattr(store, "progress_tracker"):
-        store.progress_tracker = MagicMock()
-
-    return store
-
-
-# ~~ Benchmarks ~~
-
-
-def test_rdflib_insert_throughput(benchmark, in_memory_store, triplet_batch):
-    """
-    Benchmarks raw Write Speed to in-memory RDF graph.
-    Is our baseline
-    """
-
-    def op():
-        in_memory_store.add_triplets(triplet_batch)
-
-    benchmark(op)
-
-    assert len(in_memory_store.graph) >= 1000
-
-
-def test_triplet_conversion_overhead(benchmark, large_knowledge_graph_dict):
-    """
-    Benchmarks the `store()` method in TripletStore.
-    This tests Python logic that converts a Dict -> Triplet objects.
-    """
-
-    with patch("semantica.triplet_store.blazegraph_store.BlazegraphStore") as mockBE:
-        mock_instance = mockBE.return_value
-        mock_instance.add_triplets.return_value = {"success": True}
-
-        manager = TripletStore(backend="blazegraph")
-        if hasattr(manager, "progress_tracker"):
-            manager.progress_tracker = MagicMock()
-
-        def op():
-            manager.store(
-                knowledge_graph=large_knowledge_graph_dict,
-                ontology={"classes": [], "properties": []},
-            )
-
-        benchmark(op)
-
-
-def test_bulk_loader_logic(benchmark, triplet_batch):
-    """
-    Benchmarks teh BulkLoader class.
-    Measures the overhead of batching, retries and progress tracking.
-    """
-
-    loader = BulkLoader(batch_size=100)
-    if hasattr(loader, "progress_tracker"):
-        loader.progress_tracker = MagicMock()
-
-    mock_store = MagicMock()
-    mock_store.add_triplets.return_value = {"success": True}
-
-    def op():
-        return loader.load_triplets(triplet_batch, mock_store)
-
-    result = benchmark(op)
-    assert result.total_batches == 10
-
-
-def test_sparql_query_performance(benchmark, in_memory_store, triplet_batch):
-    """
-    Benchamrks SPARQL query execution speed on 1000 items.
-    """
-
-    in_memory_store.add_triplets(triplet_batch)
-
-    query = "SELECT ?s ?o WHERE { ?s <http://gandhara.org/relation/knows> ?o } LIMIT 50"
-
-    def op():
-        return in_memory_store.execute_sparql(query)
-
-    result = benchmark(op)
-    assert result["success"] is True
-    assert len(result["bindings"]) == 50

benchmarks/storage/test_vector_storage.py

@@ -1,94 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import numpy as np
-import pytest
-
-from semantica.vector_store.faiss_store import FAISSStore
-from semantica.vector_store.vector_store import VectorStore
-
-# Fixtures
-
-
-@pytest.fixture
-def vector_dim():
-    return 768
-
-
-@pytest.fixture
-def random_vectors(vector_dim):
-    """Generates a batch of 10,000 rando vectors."""
-    count = 10000
-    vectors = np.random.rand(count, vector_dim).astype(np.float32)
-    return vectors
-
-
-@pytest.fixture
-def populated_store(random_vectors, vector_dim):
-    """
-    Returns a FAISS store bred with data.
-    """
-
-    store = FAISSStore(dimension=vector_dim)
-    if hasattr(store, "progress_tracker"):
-        store.progress_tracker = MagicMock()
-    store.create_index(index_type="flat")
-    store.add_vectors(random_vectors)
-    return store
-
-
-# Benchmarks
-
-
-def test_faiss_insert_throughput(benchmark, random_vectors, vector_dim):
-    """
-    Benchmarks raw Write speed to FAISS
-    """
-    store = FAISSStore(dimension=vector_dim)
-    if hasattr(store, "progress_tracker"):
-        store.progress_tracker = MagicMock()
-    store.create_index(index_type="flat")
-
-    def insert_op():
-        store.add_vectors(random_vectors)
-
-    benchmark(insert_op)
-
-    assert len(store.index.vector_ids) >= 10000
-
-
-def test_faiss_search_latency(benchmark, populated_store, vector_dim):
-    """
-    Benchmarks Read/Search speed
-    """
-
-    query = np.random.rand(1, vector_dim).astype(np.float32)
-    results = benchmark(populated_store.search_similar, query_vector=query, k=10)
-    assert len(results) == 10
-
-
-def test_vector_storage_manager_overhead(benchmark, random_vectors, vector_dim):
-    """
-    Benchmarks the overhead of the VectorStore class
-    """
-    with patch(
-        "semantica.vector_store.vector_store.EmbeddingGenerator"
-    ) as MockEmbedder:
-        manager = VectorStore(backend="faiss", dimension=vector_dim)
-        if hasattr(manager, "progress_tracker"):
-            manager.progress_tracker = MagicMock()
-
-        def store_op():
-            manager.store_vectors(random_vectors)
-
-        benchmark(store_op)
-
-        # Check vectors were stored - handle both in-memory and backend stores
-        if hasattr(manager, 'vectors'):
-            # In-memory backend
-            assert len(manager.vectors) >= 10000
-        elif hasattr(manager, '_backend_store') and hasattr(manager._backend_store, 'vector_ids'):
-            # Backend store (like FAISS)
-            assert len(manager._backend_store.vector_ids) >= 10000
-        else:
-            # For other backends, just ensure no errors occurred
-            pass

benchmarks/visualization/conftest.py

@@ -1,80 +0,0 @@
-import random
-from typing import Any, Dict, List
-from unittest.mock import MagicMock, patch
-
-import numpy as np
-import pytest
-
-
-# Data Generators
-@pytest.fixture
-def generate_embeddings():
-    """Generates synthetic high-dim embeddings."""
-
-    def _gen(n_samples: int, n_features: int = 768):
-        return np.random.rand(n_samples, n_features).astype(np.float32)
-
-    return _gen
-
-
-@pytest.fixture
-def generate_knowledge_graph():
-    """Generates synthetic Knowledge Graph dictionary."""
-
-    def _gen(n_nodes: int, density: float = 0.05):
-        entities = [
-            {
-                "id": f"e_{i}",
-                "label": f"Entity_{i}",
-                "type": random.choice(["Person", "Organization", "Location", "Event"]),
-                "metadata": {"score": random.random()},
-            }
-            for i in range(n_nodes)
-        ]
-
-        relationships = []
-        n_edges = int(n_nodes * (n_nodes - 1) * density)
-        # Capping edges for safety
-        n_edges = min(n_edges, n_nodes * 5)
-
-        for i in range(n_edges):
-            src = random.randint(0, n_nodes - 1)
-            tgt = random.randint(0, n_nodes - 1)
-
-            if src != tgt:
-                relationships.append(
-                    {
-                        "source": f"e_{src}",
-                        "target": f"e_{tgt}",
-                        "type": "related_to",
-                        "metadata": {"weight": random.random()},
-                    }
-                )
-
-        return {"entities": entities, "relationships": relationships}
-
-    return _gen
-
-
-@pytest.fixture
-def generate_temporal_data(generate_knowledge_graph):
-    """Generates synthetic temporal graph snapshots."""
-
-    def _gen(n_snapshots: int, n_nodes: int):
-        timestamps_map = {}
-        base_kg = generate_knowledge_graph(n_nodes)
-        entities = base_kg["entities"]
-
-        all_years = list(range(2020, 2020 + n_snapshots))
-        for ent in entities:
-            start = random.randint(0, len(all_years) - 2)
-            duration = random.randint(1, len(all_years) - start)
-            timestamps_map[ent["id"]] = all_years[start : start + duration]
-
-        return {
-            "entities": entities,
-            "relationships": base_kg["relationships"],
-            "timestamps": timestamps_map,
-        }
-
-    return _gen

benchmarks/visualization/test_analytics_viz.py

@@ -1,26 +0,0 @@
-import random
-
-import pytest
-
-from semantica.visualization.analytics_visualizer import AnalyticsVisualizer
-
-
-@pytest.mark.benchmark(group="analytics_charts")
-def test_centrality_ranking_sort_and_render(benchmark):
-    """
-    Benchmarks sorting a large centrality dictionary
-    and rendering the Top N bar chart.
-    """
-    viz = AnalyticsVisualizer()
-
-    # Generate 5000 node scores
-    centrality_data = {
-        "centrality": {f"node_{i}": random.random() for i in range(5000)}
-    }
-
-    def run():
-        return viz.visualize_centrality_rankings(
-            centrality_data, centrality_type="degree", top_n=50, output="interactive"
-        )
-
-    benchmark.pedantic(run, iterations=1, rounds=10)

benchmarks/visualization/test_embedding_viz.py

@@ -1,45 +0,0 @@
-import numpy as np
-import pytest
-
-from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
-
-
-@pytest.mark.benchmark(group="embedding_projection")
-@pytest.mark.parametrize("method", ["pca", "tsne"])
-@pytest.mark.parametrize("n_samples", [500])
-def test_projection_calculation_overhead(
-    benchmark, generate_embeddings, method, n_samples
-):
-    """
-    Measures the combined cost of:
-    1. Dimensionality Reduction (Math)
-    2. Plotly Trace Construction (Object creation)
-    """
-
-    viz = EmbeddingVisualizer()
-    embeddings = generate_embeddings(n_samples=n_samples, n_features=128)
-    labels = [f"Label {i}" for i in range(n_samples)]
-
-    def run():
-        return viz.visualize_2d_projection(
-            embeddings, labels=labels, method=method, output="interactive"
-        )
-
-    rounds = 5 if method == "tsne" else 10
-    benchmark.pedantic(run, iterations=1, rounds=rounds)
-
-
-@pytest.mark.benchmark(group="embedding_heatmap")
-def test_similarity_heatmap_generation(benchmark, generate_embeddings):
-    """
-    Benchmarks O(N^2) similarity matrix calculation
-    and heatmap renderin.
-    """
-
-    viz = EmbeddingVisualizer()
-    embeddings = generate_embeddings(n_samples=500, n_features=64)
-
-    def run():
-        return viz.visualize_similarity_heatmap(embeddings, output="interactive")
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/visualization/test_kg_viz.py

@@ -1,33 +0,0 @@
-import pytest
-
-from semantica.visualization.kg_visualizer import KGVisualizer
-
-
-@pytest.mark.benchmark(group="graph_layouyt")
-@pytest.mark.parametrize("layout", ["circular", "force"])
-@pytest.mark.parametrize("size", [100])
-def test_network_layout_performance(benchmark, generate_knowledge_graph, layout, size):
-    """
-    Compares layout algorithm.
-    """
-    viz = KGVisualizer(layout=layout, force_layout_iterations=50)
-    graph = generate_knowledge_graph(n_nodes=size)
-
-    def run():
-        return viz.visualize_network(graph, output="interactive")
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="graph_structure")
-def test_matrix_view_rendering(benchmark, generate_knowledge_graph):
-    """
-    Benchmarks the creation of an adjacent/relationship matrix.
-    """
-    viz = KGVisualizer()
-    graph = generate_knowledge_graph(n_nodes=500)
-
-    def run():
-        return viz.visualize_relationship_matrix(graph, output="interactive")
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

benchmarks/visualization/test_temporal_viz.py

@@ -1,39 +0,0 @@
-import pytest
-
-from semantica.visualization.temporal_visualizer import TemporalVisualizer
-
-
-@pytest.mark.benchmark(group="temporal_animation")
-def test_network_evolution_frames(benchmark, generate_temporal_data):
-    """
-    Measures the cost of generating animation frames for Plotly.
-    """
-
-    temporal_data = generate_temporal_data(n_snapshots=5, n_nodes=100)
-    viz = TemporalVisualizer()
-
-    def run():
-        return viz.visualize_network_evolution(temporal_data, output="interactive")
-
-    benchmark.pedantic(run, iterations=1, rounds=5)
-
-
-@pytest.mark.benchmark(group="temporal_dashboard")
-def test_temporal_dashboard_assembly(benchmark, generate_temporal_data):
-    """
-    Benchmarks the creation of a multi-subplot dashboard.
-    """
-    temporal_data = generate_temporal_data(n_snapshots=20, n_nodes=200)
-    viz = TemporalVisualizer()
-
-    metrics = {
-        "Accuracy": [0.5 + i * 0.02 for i in range(20)],
-        "Loss": [1.0 - i * 0.04 for i in range(20)],
-    }
-
-    def run():
-        return viz.visualize_temporal_dashboard(
-            temporal_data, metrics=metrics, output="interactive"
-        )
-
-    benchmark.pedantic(run, iterations=1, rounds=5)

cookbook/advanced/14_Datalog_Style_Reasoning.ipynb

@@ -0,0 +1,809 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/14_Datalog_Style_Reasoning.ipynb)\n",
+    "\n",
+    "# Datalog-Style Reasoning\n",
+    "\n",
+    "End-to-end guide to Semantica's **`DatalogReasoner`** — a native bottom-up semi-naive fixpoint engine — wired together with `GraphBuilder`, `ContextGraph`, `GraphAnalyzer`, `ExplanationGenerator`, and the supporting data-classes (`DatalogFact`, `DatalogRule`, `InferenceResult`, `Rule`).\n",
+    "\n",
+    "## What you will build\n",
+    "\n",
+    "| Part | Topic | Key classes |\n",
+    "|------|-------|-------------|\n",
+    "| 1 | Core API & EDB/IDB concepts | `DatalogReasoner`, `DatalogFact`, `DatalogRule` |\n",
+    "| 2 | KG → Datalog pipeline | `GraphBuilder`, `GraphAnalyzer`, `DatalogReasoner` |\n",
+    "| 3 | ContextGraph integration | `ContextGraph`, `DatalogReasoner.load_from_graph()` |\n",
+    "| 4 | RBAC access-control policy | `GraphBuilder`, `DatalogReasoner`, `ExplanationGenerator` |\n",
+    "| 5 | Org hierarchy | `ContextGraph`, `DatalogReasoner`, `InferenceResult` |\n",
+    "| 6 | Engine introspection | `DatalogFact`, `DatalogRule` internal state |\n",
+    "\n",
+    "**Related notebooks**\n",
+    "- [08_Reasoning_and_Inference.ipynb](08_Reasoning_and_Inference.ipynb) — high-level `Reasoner` with IF/THEN syntax\n",
+    "- [10_Temporal_Knowledge_Graphs.ipynb](10_Temporal_Knowledge_Graphs.ipynb) — temporal reasoning\n",
+    "\n",
+    "**Documentation**: [Reasoning API](https://semantica.readthedocs.io/reference/reasoning/) | [KG API](https://semantica.readthedocs.io/reference/kg/) | [Context API](https://semantica.readthedocs.io/reference/context/)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -qU semantica"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Reasoning ──────────────────────────────────────────────────────────────\n",
+    "from semantica.reasoning import (\n",
+    "    DatalogReasoner,   # native Datalog fixpoint engine\n",
+    "    DatalogFact,       # frozen dataclass: predicate + args tuple\n",
+    "    DatalogRule,       # dataclass: head + body (list[BodyAtom])\n",
+    "    ExplanationGenerator,  # generates NL justifications\n",
+    "    InferenceResult,   # result dataclass consumed by ExplanationGenerator\n",
+    "    Rule,              # rule dataclass used by ExplanationGenerator\n",
+    "    RuleType,          # enum: IMPLICATION | EQUIVALENCE | CONSTRAINT | TRANSFORMATION\n",
+    ")\n",
+    "\n",
+    "# ── Knowledge Graph ────────────────────────────────────────────────────────\n",
+    "from semantica.kg import (\n",
+    "    GraphBuilder,    # constructs KG dicts from entity+relationship sources\n",
+    "    GraphAnalyzer,   # centrality, communities, connectivity, metrics\n",
+    ")\n",
+    "\n",
+    "# ── Context ────────────────────────────────────────────────────────────────\n",
+    "from semantica.context import ContextGraph  # in-memory graph: add_node/add_edge/find_*\n",
+    "\n",
+    "print(\"All Semantica classes imported successfully.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Part 1 — Core API: EDB Facts, IDB Rules, Fixpoint\n",
+    "\n",
+    "### Datalog in 30 seconds\n",
+    "\n",
+    "| Term | Meaning | Example |\n",
+    "|------|---------|--------|\n",
+    "| EDB (Extensional DB) | Ground facts you assert | `parent(tom, bob)` |\n",
+    "| IDB (Intensional DB) | Facts derived by rules | `ancestor(tom, ann)` |\n",
+    "| Rule (Horn clause) | If body → derive head | `ancestor(X,Y) :- parent(X,Y).` |\n",
+    "| Variable | Uppercase, unified during eval | `X`, `Y`, `Role` |\n",
+    "| Constant | Lowercase, matches literally | `tom`, `admin` |\n",
+    "| Fixpoint | Iterate until no new facts appear | `DatalogReasoner.derive_all()` |\n",
+    "\n",
+    "### The canonical example — transitive ancestry"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Step 1: create engine ──────────────────────────────────────────────────\n",
+    "dr = DatalogReasoner()\n",
+    "\n",
+    "# ── Step 2: load EDB (ground facts) ───────────────────────────────────────\n",
+    "# Syntax: predicate(constant1, constant2)  — constants must be lowercase\n",
+    "edb_facts = [\n",
+    "    \"parent(tom, bob)\",\n",
+    "    \"parent(bob, ann)\",\n",
+    "    \"parent(ann, pat)\",\n",
+    "]\n",
+    "for f in edb_facts:\n",
+    "    dr.add_fact(f)\n",
+    "\n",
+    "print(f\"EDB loaded: {len(dr._all_facts)} ground facts\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Step 3: add IDB rules (Horn clauses) ──────────────────────────────────\n",
+    "# Syntax: head(Vars) :- body_atom1(Vars), body_atom2(Vars).\n",
+    "# Variables start with uppercase; trailing '.' is optional\n",
+    "dr.add_rule(\"ancestor(X, Y) :- parent(X, Y).\")\n",
+    "dr.add_rule(\"ancestor(X, Y) :- parent(X, Z), ancestor(Z, Y).\")  # recursive\n",
+    "\n",
+    "print(f\"Rules loaded: {len(dr._rules)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Step 4: fixpoint evaluation ────────────────────────────────────────────\n",
+    "# derive_all() runs semi-naive bottom-up evaluation until no new facts appear\n",
+    "all_facts: list[str] = dr.derive_all()\n",
+    "\n",
+    "ancestor_strs = sorted(f for f in all_facts if f.startswith(\"ancestor\"))\n",
+    "print(f\"Derived {len(ancestor_strs)} ancestor facts:\")\n",
+    "for f in ancestor_strs:\n",
+    "    print(\" \", f)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Step 5: query ──────────────────────────────────────────────────────────\n",
+    "# Use '?varname' placeholders — query() auto-calls derive_all() if needed\n",
+    "# Returns: list[dict]  e.g. [{\"Y\": \"bob\"}, {\"Y\": \"ann\"}, {\"Y\": \"pat\"}]\n",
+    "\n",
+    "descendants = dr.query(\"ancestor(tom, ?Y)\")\n",
+    "print(\"All descendants of tom:\", sorted(r[\"Y\"] for r in descendants))\n",
+    "\n",
+    "ancestors_of_pat = dr.query(\"ancestor(?X, pat)\")\n",
+    "print(\"All ancestors of pat: \", sorted(r[\"X\"] for r in ancestors_of_pat))\n",
+    "\n",
+    "all_pairs = dr.query(\"ancestor(?X, ?Y)\")\n",
+    "print(f\"\\nAll ancestor pairs ({len(all_pairs)}):\")\n",
+    "for row in sorted(all_pairs, key=lambda r: (r[\"X\"], r[\"Y\"])):\n",
+    "    print(f\"  {row['X']:6s} → {row['Y']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Part 2 — GraphBuilder → DatalogReasoner Pipeline\n",
+    "\n",
+    "`GraphBuilder` constructs a structured `{\"entities\": [...], \"relationships\": [...]}` dict from your data. We then:\n",
+    "\n",
+    "1. Analyse the graph with `GraphAnalyzer` to understand structure.\n",
+    "2. Feed `kg[\"relationships\"]` into `DatalogReasoner` as EDB facts.\n",
+    "3. Apply recursive Datalog rules over the KG."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Build a software-dependency KG ────────────────────────────────────────\n",
+    "entities = [\n",
+    "    {\"id\": \"pythonsdk\",   \"name\": \"Python SDK\",   \"type\": \"Component\"},\n",
+    "    {\"id\": \"restapi\",     \"name\": \"REST API\",     \"type\": \"Component\"},\n",
+    "    {\"id\": \"authservice\", \"name\": \"Auth Service\", \"type\": \"Component\"},\n",
+    "    {\"id\": \"database\",    \"name\": \"Database\",     \"type\": \"Component\"},\n",
+    "    {\"id\": \"dashboard\",   \"name\": \"Dashboard\",    \"type\": \"Component\"},\n",
+    "    {\"id\": \"analytics\",   \"name\": \"Analytics\",    \"type\": \"Component\"},\n",
+    "]\n",
+    "relationships = [\n",
+    "    {\"source\": \"pythonsdk\",   \"target\": \"restapi\",     \"type\": \"depends_on\"},\n",
+    "    {\"source\": \"restapi\",     \"target\": \"authservice\", \"type\": \"depends_on\"},\n",
+    "    {\"source\": \"authservice\", \"target\": \"database\",    \"type\": \"depends_on\"},\n",
+    "    {\"source\": \"dashboard\",   \"target\": \"restapi\",     \"type\": \"depends_on\"},\n",
+    "    {\"source\": \"dashboard\",   \"target\": \"analytics\",   \"type\": \"depends_on\"},\n",
+    "    {\"source\": \"analytics\",   \"target\": \"database\",    \"type\": \"depends_on\"},\n",
+    "]\n",
+    "\n",
+    "# GraphBuilder validates, deduplicates, and packages the data\n",
+    "builder = GraphBuilder(merge_entities=True, resolve_conflicts=False)\n",
+    "kg = builder.build([{\"entities\": entities, \"relationships\": relationships}])\n",
+    "\n",
+    "print(f\"KG built — entities: {len(kg['entities'])}, relationships: {len(kg['relationships'])}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Analyse the graph structure before reasoning ───────────────────────────\n",
+    "# GraphAnalyzer provides centrality, communities, connectivity, and metrics\n",
+    "analyzer = GraphAnalyzer()\n",
+    "metrics = analyzer.compute_metrics(graph=kg)\n",
+    "\n",
+    "print(\"Graph structure:\")\n",
+    "print(f\"  Nodes      : {metrics['num_nodes']}\")\n",
+    "print(f\"  Edges      : {metrics['num_edges']}\")\n",
+    "if \"density\" in metrics:\n",
+    "    print(f\"  Density    : {metrics['density']:.3f}\")\n",
+    "if \"is_connected\" in metrics:\n",
+    "    print(f\"  Connected  : {metrics['is_connected']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Load KG relationships as EDB facts ────────────────────────────────────\n",
+    "# GraphBuilder output dicts use the same source/target/type shape that\n",
+    "# DatalogReasoner.add_fact() natively understands\n",
+    "dr = DatalogReasoner()\n",
+    "\n",
+    "for rel in kg[\"relationships\"]:\n",
+    "    dr.add_fact(rel)   # dict path: {\"source\": ..., \"target\": ..., \"type\": ...}\n",
+    "\n",
+    "print(f\"EDB loaded: {len(dr._all_facts)} dependency facts\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Transitive dependency closure ─────────────────────────────────────────\n",
+    "# 'depends_on' is the predicate name that add_fact inferred from 'type'\n",
+    "dr.add_rule(\"transitive_dep(X, Y) :- depends_on(X, Y).\")\n",
+    "dr.add_rule(\"transitive_dep(X, Y) :- depends_on(X, Z), transitive_dep(Z, Y).\")\n",
+    "\n",
+    "dr.derive_all()\n",
+    "\n",
+    "# Everything that transitively depends on the database\n",
+    "db_deps = sorted(r[\"X\"] for r in dr.query(\"transitive_dep(?X, database)\"))\n",
+    "print(\"Components that transitively depend on Database:\")\n",
+    "for c in db_deps:\n",
+    "    print(\" \", c)\n",
+    "\n",
+    "# What does pythonsdk transitively depend on?\n",
+    "sdk_chain = sorted(r[\"Y\"] for r in dr.query(\"transitive_dep(pythonsdk, ?Y)\"))\n",
+    "print(f\"\\nPython SDK full dependency chain: {sdk_chain}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Part 3 — ContextGraph + `load_from_graph()`\n",
+    "\n",
+    "`DatalogReasoner.load_from_graph(graph)` accepts any `ContextGraph` directly: it calls `graph.find_edges()` and `graph.find_nodes()` and converts each result into EDB facts automatically."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Build an in-memory ContextGraph ───────────────────────────────────────\n",
+    "# ContextGraph.add_node / add_edge are the canonical way to build in-memory KGs\n",
+    "cg = ContextGraph()\n",
+    "\n",
+    "# Nodes\n",
+    "for person in [\"alice\", \"bob\", \"carol\", \"dave\", \"eve\"]:\n",
+    "    cg.add_node(person, node_type=\"person\", name=person.capitalize())\n",
+    "\n",
+    "# Directed \"follows\" edges\n",
+    "for src, dst in [(\"alice\", \"bob\"), (\"bob\", \"carol\"), (\"carol\", \"dave\"), (\"alice\", \"eve\"), (\"eve\", \"carol\")]:\n",
+    "    cg.add_edge(src, dst, edge_type=\"follows\")\n",
+    "\n",
+    "# Verify the graph built correctly\n",
+    "nodes = cg.find_nodes(node_type=\"person\")\n",
+    "edges = cg.find_edges(edge_type=\"follows\")\n",
+    "print(f\"ContextGraph — nodes: {len(nodes)}, edges: {len(edges)}\")\n",
+    "print(\"Edges:\", [(e.get(\"source\", e.get(\"source_id\")), e.get(\"target\", e.get(\"target_id\"))) for e in edges])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── load_from_graph() ingests the ContextGraph directly ───────────────────\n",
+    "dr = DatalogReasoner()\n",
+    "n_loaded = dr.load_from_graph(cg)  # calls cg.find_edges() + cg.find_nodes() internally\n",
+    "print(f\"Facts loaded from ContextGraph: {n_loaded}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Influence reach via transitive 'follows' ──────────────────────────────\n",
+    "dr.add_rule(\"influence(X, Y) :- follows(X, Y).\")\n",
+    "dr.add_rule(\"influence(X, Y) :- follows(X, Z), influence(Z, Y).\")\n",
+    "\n",
+    "dr.derive_all()\n",
+    "\n",
+    "# Who can alice reach?\n",
+    "alice_reach = sorted(r[\"Y\"] for r in dr.query(\"influence(alice, ?Y)\"))\n",
+    "print(f\"Alice's influence reach : {alice_reach}\")\n",
+    "\n",
+    "# Who can reach dave?\n",
+    "reach_dave = sorted(r[\"X\"] for r in dr.query(\"influence(?X, dave)\"))\n",
+    "print(f\"Who can influence dave  : {reach_dave}\")\n",
+    "\n",
+    "# Full influence matrix\n",
+    "all_influence = dr.query(\"influence(?X, ?Y)\")\n",
+    "print(f\"\\nTotal influence pairs: {len(all_influence)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Part 4 — RBAC Access-Control Policy\n",
+    "\n",
+    "We model a role-based access-control (RBAC) system:\n",
+    "\n",
+    "1. Use `GraphBuilder` to build a structured KG of users, roles, and permissions.\n",
+    "2. Load it into `DatalogReasoner` for policy inference.\n",
+    "3. Use `ExplanationGenerator` to produce audit-ready NL justifications."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Build RBAC graph with GraphBuilder ────────────────────────────────────\n",
+    "rbac_entities = [\n",
+    "    # Users\n",
+    "    {\"id\": \"alice\",  \"type\": \"User\", \"name\": \"Alice\"},\n",
+    "    {\"id\": \"bob\",    \"type\": \"User\", \"name\": \"Bob\"},\n",
+    "    {\"id\": \"carol\",  \"type\": \"User\", \"name\": \"Carol\"},\n",
+    "    {\"id\": \"dave\",   \"type\": \"User\", \"name\": \"Dave\"},\n",
+    "    # Roles\n",
+    "    {\"id\": \"admin\",  \"type\": \"Role\", \"name\": \"Administrator\"},\n",
+    "    {\"id\": \"editor\", \"type\": \"Role\", \"name\": \"Editor\"},\n",
+    "    {\"id\": \"viewer\", \"type\": \"Role\", \"name\": \"Viewer\"},\n",
+    "    # Permissions\n",
+    "    {\"id\": \"read\",         \"type\": \"Permission\"},\n",
+    "    {\"id\": \"write\",        \"type\": \"Permission\"},\n",
+    "    {\"id\": \"delete\",       \"type\": \"Permission\"},\n",
+    "    {\"id\": \"manage_users\", \"type\": \"Permission\"},\n",
+    "]\n",
+    "rbac_relationships = [\n",
+    "    # User → Role assignments\n",
+    "    {\"source\": \"alice\",  \"target\": \"admin\",  \"type\": \"has_role\"},\n",
+    "    {\"source\": \"bob\",    \"target\": \"editor\", \"type\": \"has_role\"},\n",
+    "    {\"source\": \"carol\",  \"target\": \"viewer\", \"type\": \"has_role\"},\n",
+    "    {\"source\": \"dave\",   \"target\": \"editor\", \"type\": \"has_role\"},\n",
+    "    # Role hierarchy (admin inherits from editor, editor from viewer)\n",
+    "    {\"source\": \"admin\",  \"target\": \"editor\", \"type\": \"role_inherits\"},\n",
+    "    {\"source\": \"editor\", \"target\": \"viewer\", \"type\": \"role_inherits\"},\n",
+    "    # Role → Permission grants\n",
+    "    {\"source\": \"viewer\", \"target\": \"read\",         \"type\": \"role_has_perm\"},\n",
+    "    {\"source\": \"editor\", \"target\": \"write\",        \"type\": \"role_has_perm\"},\n",
+    "    {\"source\": \"admin\",  \"target\": \"delete\",       \"type\": \"role_has_perm\"},\n",
+    "    {\"source\": \"admin\",  \"target\": \"manage_users\", \"type\": \"role_has_perm\"},\n",
+    "]\n",
+    "\n",
+    "builder = GraphBuilder(merge_entities=True, resolve_conflicts=False)\n",
+    "rbac_kg = builder.build([{\"entities\": rbac_entities, \"relationships\": rbac_relationships}])\n",
+    "\n",
+    "print(f\"RBAC KG — entities: {len(rbac_kg['entities'])}, relationships: {len(rbac_kg['relationships'])}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Analyse RBAC graph structure ──────────────────────────────────────────\n",
+    "analyzer = GraphAnalyzer()\n",
+    "metrics = analyzer.compute_metrics(graph=rbac_kg)\n",
+    "centrality = analyzer.calculate_centrality(rbac_kg, centrality_type=\"degree\")\n",
+    "\n",
+    "print(f\"RBAC graph — {metrics['num_nodes']} nodes, {metrics['num_edges']} edges\")\n",
+    "if isinstance(centrality, dict) and \"degree\" in centrality:\n",
+    "    top = sorted(centrality[\"degree\"].items(), key=lambda x: x[1], reverse=True)[:3]\n",
+    "    print(\"Top-3 nodes by degree centrality:\", top)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Load RBAC KG into DatalogReasoner ────────────────────────────────────\n",
+    "dr = DatalogReasoner()\n",
+    "\n",
+    "for rel in rbac_kg[\"relationships\"]:\n",
+    "    dr.add_fact(rel)   # {source, target, type} → predicate(source, target)\n",
+    "\n",
+    "# ── IDB rules: transitive role hierarchy ─────────────────────────────────\n",
+    "dr.add_rule(\"effective_role(R, R2) :- role_inherits(R, R2).\")\n",
+    "dr.add_rule(\"effective_role(R, R2) :- role_inherits(R, Z), effective_role(Z, R2).\")\n",
+    "\n",
+    "# ── IDB rules: inherited permissions ─────────────────────────────────────\n",
+    "dr.add_rule(\"role_can(R, P) :- role_has_perm(R, P).\")\n",
+    "dr.add_rule(\"role_can(R, P) :- effective_role(R, R2), role_has_perm(R2, P).\")\n",
+    "\n",
+    "# ── IDB rules: user effective permissions ────────────────────────────────\n",
+    "dr.add_rule(\"can(U, P) :- has_role(U, R), role_can(R, P).\")\n",
+    "\n",
+    "dr.derive_all()\n",
+    "\n",
+    "print(\"User permissions derived via role-hierarchy inference:\")\n",
+    "for user in [\"alice\", \"bob\", \"carol\", \"dave\"]:\n",
+    "    perms = sorted(r[\"P\"] for r in dr.query(f\"can({user}, ?P)\"))\n",
+    "    print(f\"  {user:6s}: {perms}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── ExplanationGenerator — audit-ready NL justification ──────────────────\n",
+    "# ExplanationGenerator works with InferenceResult objects.\n",
+    "# We construct one manually to represent a derived Datalog conclusion.\n",
+    "\n",
+    "explainer = ExplanationGenerator(detail_level=\"detailed\")\n",
+    "\n",
+    "# Build the Rule object that represents the permission derivation chain\n",
+    "perm_rule = Rule(\n",
+    "    rule_id=\"rbac_perm_chain\",\n",
+    "    name=\"RBAC permission via role hierarchy\",\n",
+    "    conditions=[\"has_role(alice, admin)\", \"effective_role(admin, viewer)\", \"role_has_perm(viewer, read)\"],\n",
+    "    conclusion=\"can(alice, read)\",\n",
+    "    rule_type=RuleType.IMPLICATION,\n",
+    "    confidence=1.0,\n",
+    ")\n",
+    "\n",
+    "# Build InferenceResult representing the Datalog conclusion\n",
+    "result = InferenceResult(\n",
+    "    conclusion=\"can(alice, read)\",\n",
+    "    rule_used=perm_rule,\n",
+    "    premises=[\n",
+    "        \"has_role(alice, admin)\",\n",
+    "        \"role_inherits(admin, editor)\",\n",
+    "        \"role_inherits(editor, viewer)\",\n",
+    "        \"role_has_perm(viewer, read)\",\n",
+    "    ],\n",
+    "    confidence=1.0,\n",
+    ")\n",
+    "\n",
+    "# Generate NL explanation\n",
+    "explanation = explainer.generate_explanation(result)\n",
+    "print(\"Explanation type :\", explanation.explanation_type)\n",
+    "print(\"Conclusion       :\", explanation.conclusion)\n",
+    "print(\"Natural language :\", explanation.natural_language)\n",
+    "print(\"Reasoning steps  :\", len(explanation.reasoning_path.steps))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Inverse queries ───────────────────────────────────────────────────────\n",
+    "deleters = sorted(r[\"U\"] for r in dr.query(\"can(?U, delete)\"))\n",
+    "print(\"Who can delete:\", deleters)\n",
+    "\n",
+    "writers = sorted(r[\"U\"] for r in dr.query(\"can(?U, write)\"))\n",
+    "print(\"Who can write: \", writers)\n",
+    "\n",
+    "# All (user, permission) pairs — full policy matrix\n",
+    "all_caps = dr.query(\"can(?U, ?P)\")\n",
+    "print(f\"\\nTotal (user, permission) pairs: {len(all_caps)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Part 5 — Organisation Hierarchy with ContextGraph\n",
+    "\n",
+    "We model a company org-chart using `ContextGraph` and derive:\n",
+    "- `manages(M, E)` — direct and transitive management\n",
+    "- `skip_level(M, E)` — two hops up the chain\n",
+    "- `same_team(X, Y)` — shared team membership"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── ContextGraph: org chart ────────────────────────────────────────────────\n",
+    "org = ContextGraph()\n",
+    "\n",
+    "# Add employees as nodes with metadata\n",
+    "staff = [\n",
+    "    (\"eng1\",      \"engineer\",  \"backend\"),\n",
+    "    (\"eng2\",      \"engineer\",  \"backend\"),\n",
+    "    (\"eng3\",      \"engineer\",  \"frontend\"),\n",
+    "    (\"techlead\",  \"lead\",      \"engineering\"),\n",
+    "    (\"design1\",   \"designer\",  \"ux\"),\n",
+    "    (\"design2\",   \"designer\",  \"ux\"),\n",
+    "    (\"designlead\",\"lead\",      \"design\"),\n",
+    "    (\"vpeng\",     \"vp\",        \"engineering\"),\n",
+    "    (\"cto\",       \"executive\", \"leadership\"),\n",
+    "]\n",
+    "for emp_id, role, team in staff:\n",
+    "    org.add_node(emp_id, node_type=\"employee\", role=role, team=team)\n",
+    "\n",
+    "# Reporting lines\n",
+    "reports_to = [\n",
+    "    (\"eng1\", \"techlead\"), (\"eng2\", \"techlead\"), (\"eng3\", \"techlead\"),\n",
+    "    (\"techlead\", \"vpeng\"),\n",
+    "    (\"design1\", \"designlead\"), (\"design2\", \"designlead\"),\n",
+    "    (\"designlead\", \"vpeng\"),\n",
+    "    (\"vpeng\", \"cto\"),\n",
+    "]\n",
+    "for employee, manager in reports_to:\n",
+    "    org.add_edge(employee, manager, edge_type=\"reports_to\")\n",
+    "\n",
+    "# Team membership edges\n",
+    "teams = [\n",
+    "    (\"eng1\", \"backend\"),  (\"eng2\", \"backend\"),  (\"eng3\", \"frontend\"),\n",
+    "    (\"design1\", \"ux\"),    (\"design2\", \"ux\"),\n",
+    "]\n",
+    "for emp, team in teams:\n",
+    "    org.add_edge(emp, team, edge_type=\"in_team\")\n",
+    "    if not org.find_nodes(node_type=\"team\"):\n",
+    "        org.add_node(team, node_type=\"team\")\n",
+    "\n",
+    "print(f\"ContextGraph — nodes: {len(org.find_nodes())}, edges: {len(org.find_edges())}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Load org chart into DatalogReasoner ───────────────────────────────────\n",
+    "dr = DatalogReasoner()\n",
+    "n = dr.load_from_graph(org)   # uses org.find_edges() + org.find_nodes()\n",
+    "print(f\"Facts loaded via load_from_graph(): {n}\")\n",
+    "\n",
+    "# ── IDB rules ─────────────────────────────────────────────────────────────\n",
+    "# Transitive management chain\n",
+    "dr.add_rule(\"manages(M, E) :- reports_to(E, M).\")\n",
+    "dr.add_rule(\"manages(M, E) :- reports_to(E, Z), manages(M, Z).\")\n",
+    "\n",
+    "# Skip-level: exactly two reporting hops\n",
+    "dr.add_rule(\"skip_level(M, E) :- reports_to(E, Z), reports_to(Z, M).\")\n",
+    "\n",
+    "# Same team\n",
+    "dr.add_rule(\"same_team(X, Y) :- in_team(X, T), in_team(Y, T).\")\n",
+    "\n",
+    "dr.derive_all()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Query org hierarchy ────────────────────────────────────────────────────\n",
+    "# Everyone under CTO\n",
+    "under_cto = sorted(r[\"E\"] for r in dr.query(\"manages(cto, ?E)\"))\n",
+    "print(f\"CTO manages ({len(under_cto)} people): {under_cto}\")\n",
+    "\n",
+    "# VP Eng's direct + indirect reports\n",
+    "under_vp = sorted(r[\"E\"] for r in dr.query(\"manages(vpeng, ?E)\"))\n",
+    "print(f\"VP Eng manages           : {under_vp}\")\n",
+    "\n",
+    "# Skip-level reports to CTO (people two hops below CTO)\n",
+    "skip = sorted(r[\"E\"] for r in dr.query(\"skip_level(cto, ?E)\"))\n",
+    "print(f\"CTO skip-level reports   : {skip}\")\n",
+    "\n",
+    "# eng1's teammates\n",
+    "mates = [r[\"Y\"] for r in dr.query(\"same_team(eng1, ?Y)\") if r[\"Y\"] != \"eng1\"]\n",
+    "print(f\"eng1's teammates         : {sorted(mates)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Build an InferenceResult and explain an org query ────────────────────\n",
+    "explainer = ExplanationGenerator(detail_level=\"verbose\")\n",
+    "\n",
+    "mgmt_rule = Rule(\n",
+    "    rule_id=\"transitive_manages\",\n",
+    "    name=\"Transitive management chain\",\n",
+    "    conditions=[\"reports_to(eng1, techlead)\", \"manages(vpeng, techlead)\"],\n",
+    "    conclusion=\"manages(vpeng, eng1)\",\n",
+    "    rule_type=RuleType.IMPLICATION,\n",
+    "    confidence=1.0,\n",
+    ")\n",
+    "result = InferenceResult(\n",
+    "    conclusion=\"manages(vpeng, eng1)\",\n",
+    "    rule_used=mgmt_rule,\n",
+    "    premises=[\"reports_to(eng1, techlead)\", \"reports_to(techlead, vpeng)\"],\n",
+    "    confidence=1.0,\n",
+    ")\n",
+    "\n",
+    "exp = explainer.generate_explanation(result)\n",
+    "print(exp.natural_language)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Part 6 — Engine Introspection: DatalogFact & DatalogRule\n",
+    "\n",
+    "After reasoning, the engine's internal state is fully accessible via `DatalogFact` and `DatalogRule` data-classes. Use this for auditing, debugging, or downstream export."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Inspect DatalogRule objects ────────────────────────────────────────────\n",
+    "# dr._rules  →  List[DatalogRule]\n",
+    "# DatalogRule.head_predicate, .head_args, .body  (body = List[BodyAtom])\n",
+    "print(\"Rules in engine:\")\n",
+    "for rule in dr._rules:\n",
+    "    body_str = \", \".join(\n",
+    "        f\"{atom.predicate}({', '.join(atom.args)})\"\n",
+    "        for atom in rule.body\n",
+    "    )\n",
+    "    head_str = f\"{rule.head_predicate}({', '.join(rule.head_args)})\"\n",
+    "    print(f\"  {head_str}  :-  {body_str}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Inspect DatalogFact objects ────────────────────────────────────────────\n",
+    "# dr._all_facts  →  Set[DatalogFact]  (EDB + IDB combined after derive_all)\n",
+    "# dr._fact_index →  Dict[predicate, Set[DatalogFact]]\n",
+    "\n",
+    "from collections import Counter\n",
+    "\n",
+    "# Count facts per predicate\n",
+    "predicate_counts = Counter(f.predicate for f in dr._all_facts)\n",
+    "print(\"Facts per predicate (EDB + derived IDB):\")\n",
+    "for pred, count in sorted(predicate_counts.items()):\n",
+    "    print(f\"  {pred:20s}: {count}\")\n",
+    "print(f\"\\n  TOTAL: {len(dr._all_facts)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Separate EDB from IDB ─────────────────────────────────────────────────\n",
+    "# EDB predicates are the ones we added via add_fact (not derived by rules)\n",
+    "idb_predicates = {rule.head_predicate for rule in dr._rules}\n",
+    "edb_predicates = {f.predicate for f in dr._all_facts} - idb_predicates\n",
+    "\n",
+    "print(f\"EDB predicates (base facts)  : {sorted(edb_predicates)}\")\n",
+    "print(f\"IDB predicates (derived)     : {sorted(idb_predicates)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── Sample DatalogFact structure ──────────────────────────────────────────\n",
+    "# DatalogFact is a frozen dataclass: predicate: str, args: Tuple[str, ...]\n",
+    "manages_facts = sorted(dr._fact_index.get(\"manages\", []), key=lambda f: f.args)\n",
+    "print(f\"First 5 'manages' DatalogFact objects ({len(manages_facts)} total):\")\n",
+    "for fact in manages_facts[:5]:\n",
+    "    # Access predicate and args directly from the dataclass\n",
+    "    print(f\"  DatalogFact(predicate={fact.predicate!r}, args={fact.args})\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ── clear() resets the engine completely ─────────────────────────────────\n",
+    "print(f\"Facts before clear(): {len(dr._all_facts)}\")\n",
+    "dr.clear()\n",
+    "print(f\"Facts after  clear(): {len(dr._all_facts)}\")\n",
+    "print(f\"Rules after  clear(): {len(dr._rules)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## API Summary\n",
+    "\n",
+    "### DatalogReasoner\n",
+    "\n",
+    "| Method | Input | Output | Notes |\n",
+    "|--------|-------|--------|-------|\n",
+    "| `add_fact(f)` | `str` or `dict` | `None` | string: `\"pred(a, b)\"` · dict: `{source, target, type}` |\n",
+    "| `add_rule(s)` | `str` | `None` | Horn clause: `\"head(X) :- body(X, Y).\"` |\n",
+    "| `derive_all()` | — | `list[str]` | semi-naive fixpoint; idempotent |\n",
+    "| `query(pat)` | `str` | `list[dict]` | `\"pred(a, ?Y)\"` → `[{\"Y\": ...}]` |\n",
+    "| `load_from_graph(g)` | `ContextGraph` | `int` | facts loaded count |\n",
+    "| `clear()` | — | `None` | resets engine |\n",
+    "\n",
+    "### Syntax rules\n",
+    "\n",
+    "| Item | Rule | Example |\n",
+    "|------|------|---------|\n",
+    "| Variable | Starts **uppercase** | `X`, `Role`, `Parent` |\n",
+    "| Constant | All **lowercase** | `tom`, `admin`, `database` |\n",
+    "| Query var | Prefix `?` | `?X`, `?Y`, `?Role` |\n",
+    "| Rule body | `:-` separator, comma between atoms | `head(X) :- a(X, Z), b(Z, Y).` |\n",
+    "\n",
+    "### Class map\n",
+    "\n",
+    "```\n",
+    "GraphBuilder.build()            → kg dict {entities, relationships}\n",
+    "                   ↓ kg[\"relationships\"] → dr.add_fact(rel)\n",
+    "                   \n",
+    "ContextGraph.add_node/add_edge  → in-memory graph\n",
+    "                   ↓ dr.load_from_graph(cg)\n",
+    "                   \n",
+    "DatalogReasoner.add_rule()      → Horn clause rules\n",
+    "DatalogReasoner.derive_all()    → semi-naive fixpoint\n",
+    "DatalogReasoner.query()         → result rows\n",
+    "                   ↓ build InferenceResult\n",
+    "                   \n",
+    "ExplanationGenerator            → natural language justification\n",
+    "GraphAnalyzer                   → graph structure metrics pre/post reasoning\n",
+    "DatalogFact / DatalogRule       → introspect engine state\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.10.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

cookbook/advanced/snowflake_ingestion_examples.py

@@ -7,17 +7,26 @@ This module provides comprehensive examples of using the Snowflake ingestor.
 import os
 from datetime import datetime, timedelta
 
+from rich import box
+from rich.console import Console
+from rich.rule import Rule
+from rich.table import Table
+
 from semantica.ingest import SnowflakeIngestor
 from semantica.utils.logging import get_logger
 
 logger = get_logger("snowflake_examples")
+console = Console()
+
+
+def _section(title: str) -> None:
+    console.print(Rule(f"[bold cyan]{title}[/bold cyan]", style="cyan"))
 
 
 def example_basic_ingestion():
     """Example: Basic table ingestion."""
-    print("\n=== Example 1: Basic Table Ingestion ===\n")
+    _section("Example 1: Basic Table Ingestion")
 
-    # Initialize ingestor with password authentication
     ingestor = SnowflakeIngestor(
         account=os.getenv("SNOWFLAKE_ACCOUNT"),
         user=os.getenv("SNOWFLAKE_USER"),
@@ -27,26 +36,23 @@ def example_basic_ingestion():
         schema="PUBLIC",
     )
 
-    # Ingest a table
     data = ingestor.ingest_table("CUSTOMERS", limit=10)
 
-    print(f"Retrieved {data.row_count} rows")
-    print(f"Columns: {data.columns}")
-    print(f"\nFirst row:")
-    print(data.data[0])
+    console.print(f"[green]✓[/green] Retrieved [cyan]{data.row_count}[/cyan] rows")
+    console.print(f"  Columns: [dim]{data.columns}[/dim]")
+    console.print(f"  First row: [dim]{data.data[0]}[/dim]")
 
     ingestor.close()
 
 
 def example_query_execution():
     """Example: Execute custom SQL queries."""
-    print("\n=== Example 2: Query Execution ===\n")
+    _section("Example 2: Query Execution")
 
     ingestor = SnowflakeIngestor()
 
-    # Execute aggregation query
     query = """
-        SELECT 
+        SELECT
             COUNTRY,
             COUNT(*) AS CUSTOMER_COUNT,
             SUM(TOTAL_PURCHASES) AS TOTAL_REVENUE
@@ -58,34 +64,34 @@ def example_query_execution():
 
     data = ingestor.ingest_query(query)
 
-    print(f"Top 10 countries by revenue:")
+    table = Table(title="[bold]Top 10 Countries by Revenue[/bold]",
+                  box=box.SIMPLE_HEAD, show_edge=False, padding=(0, 1))
+    table.add_column("Country", style="cyan", no_wrap=True)
+    table.add_column("Customers", style="green", justify="right")
+    table.add_column("Revenue", style="green", justify="right")
     for row in data.data:
-        print(
-            f"  {row['COUNTRY']}: {row['CUSTOMER_COUNT']} customers, "
-            f"${row['TOTAL_REVENUE']:,.2f} revenue"
+        table.add_row(
+            row["COUNTRY"],
+            str(row["CUSTOMER_COUNT"]),
+            f"${row['TOTAL_REVENUE']:,.2f}",
         )
+    console.print(table)
 
     ingestor.close()
 
 
 def example_parameterized_query():
     """Example: Parameterized queries."""
-    print("\n=== Example 3: Parameterized Queries ===\n")
+    _section("Example 3: Parameterized Queries")
 
     ingestor = SnowflakeIngestor()
 
-    # Calculate date range
     end_date = datetime.now()
     start_date = end_date - timedelta(days=30)
 
-    # Execute parameterized query
     query = """
-        SELECT 
-            ORDER_ID,
-            CUSTOMER_ID,
-            PRODUCT_NAME,
-            AMOUNT,
-            ORDER_DATE
+        SELECT
+            ORDER_ID, CUSTOMER_ID, PRODUCT_NAME, AMOUNT, ORDER_DATE
         FROM ORDERS
         WHERE ORDER_DATE BETWEEN %(start_date)s AND %(end_date)s
           AND AMOUNT > %(min_amount)s
@@ -101,122 +107,125 @@ def example_parameterized_query():
         },
     )
 
-    print(f"Found {data.row_count} orders in the last 30 days over $100")
-
+    console.print(
+        f"[green]✓[/green] Found [cyan]{data.row_count}[/cyan] orders "
+        "in the last 30 days over $100"
+    )
     ingestor.close()
 
 
 def example_schema_introspection():
     """Example: Table schema introspection."""
-    print("\n=== Example 4: Schema Introspection ===\n")
+    _section("Example 4: Schema Introspection")
 
     ingestor = SnowflakeIngestor()
-
-    # Get table schema
     schema = ingestor.get_table_schema("CUSTOMERS")
 
-    print("Table schema for CUSTOMERS:")
-    print(f"Primary keys: {schema['primary_keys']}\n")
+    console.print(f"  Primary keys: [cyan]{schema['primary_keys']}[/cyan]")
 
-    print("Columns:")
+    table = Table(title="[bold]CUSTOMERS Schema[/bold]",
+                  box=box.SIMPLE_HEAD, show_edge=False, padding=(0, 1))
+    table.add_column("Column", style="cyan", no_wrap=True)
+    table.add_column("Type")
+    table.add_column("Nullable")
+    table.add_column("Default", style="dim")
     for col in schema["columns"]:
-        nullable = "NULL" if col["nullable"] else "NOT NULL"
-        default = f" DEFAULT {col['default']}" if col["default"] else ""
-        print(f"  {col['name']}: {col['type']} {nullable}{default}")
+        table.add_row(
+            col["name"],
+            col["type"],
+            "NULL" if col["nullable"] else "NOT NULL",
+            str(col["default"]) if col["default"] else "",
+        )
+    console.print(table)
 
     ingestor.close()
 
 
 def example_list_tables():
     """Example: List all tables in a schema."""
-    print("\n=== Example 5: List Tables ===\n")
+    _section("Example 5: List Tables")
 
     ingestor = SnowflakeIngestor()
-
-    # List tables in current schema
     tables = ingestor.list_tables()
 
-    print(f"Found {len(tables)} tables:")
-    for table in tables:
-        print(f"  - {table}")
+    table = Table(title=f"[bold]Tables ({len(tables)} found)[/bold]",
+                  box=box.SIMPLE_HEAD, show_edge=False, padding=(0, 1))
+    table.add_column("Table", style="cyan")
+    for t in tables:
+        table.add_row(t)
+    console.print(table)
 
     ingestor.close()
 
 
 def example_pagination():
     """Example: Paginate large result sets."""
-    print("\n=== Example 6: Pagination ===\n")
+    _section("Example 6: Pagination")
 
     ingestor = SnowflakeIngestor()
-
     PAGE_SIZE = 100
     total_rows = 0
-
-    # Paginate through large table
     page = 0
+
     while True:
         data = ingestor.ingest_table(
             "LARGE_TABLE", limit=PAGE_SIZE, offset=page * PAGE_SIZE
         )
-
         if data.row_count == 0:
             break
-
         total_rows += data.row_count
-        print(f"Page {page + 1}: {data.row_count} rows")
-
-        # Process page
+        console.print(
+            f"  [dim]Page {page + 1}:[/dim] [cyan]{data.row_count}[/cyan] rows"
+        )
         process_page(data)
-
         page += 1
 
-    print(f"\nTotal rows processed: {total_rows}")
-
+    console.print(
+        f"[green]✓[/green] Total rows processed: [cyan]{total_rows}[/cyan]"
+    )
     ingestor.close()
 
 
 def example_batch_processing():
     """Example: Batch processing with fetchmany."""
-    print("\n=== Example 7: Batch Processing ===\n")
+    _section("Example 7: Batch Processing")
 
     ingestor = SnowflakeIngestor()
-
-    # Execute query with batching
     data = ingestor.ingest_query(
         "SELECT * FROM LARGE_TABLE WHERE STATUS = 'ACTIVE'", batch_size=1000
     )
-
-    print(f"Retrieved {data.row_count} rows in batches of 1000")
-
+    console.print(
+        f"[green]✓[/green] Retrieved [cyan]{data.row_count}[/cyan] rows "
+        "in batches of 1000"
+    )
     ingestor.close()
 
 
 def example_export_documents():
     """Example: Export to Semantica document format."""
-    print("\n=== Example 8: Export as Documents ===\n")
+    _section("Example 8: Export as Documents")
 
     ingestor = SnowflakeIngestor()
-
-    # Ingest product data
     data = ingestor.ingest_table("PRODUCTS", limit=10)
-
-    # Convert to documents
     documents = ingestor.export_as_documents(
         data, id_field="PRODUCT_ID", text_fields=["PRODUCT_NAME", "DESCRIPTION"]
     )
 
-    print(f"Exported {len(documents)} documents")
-    print("\nFirst document:")
-    print(f"  ID: {documents[0]['id']}")
-    print(f"  Text: {documents[0]['text'][:100]}...")
-    print(f"  Metadata: {documents[0]['metadata']}")
+    console.print(
+        f"[green]✓[/green] Exported [cyan]{len(documents)}[/cyan] documents"
+    )
+    if documents:
+        d = documents[0]
+        console.print(f"  [dim]First doc — ID:[/dim] {d['id']}")
+        console.print(f"  [dim]Text:[/dim] {d['text'][:100]}…")
+        console.print(f"  [dim]Metadata:[/dim] {d['metadata']}")
 
     ingestor.close()
 
 
 def example_key_pair_auth():
     """Example: Key-pair authentication."""
-    print("\n=== Example 9: Key-Pair Authentication ===\n")
+    _section("Example 9: Key-Pair Authentication")
 
     ingestor = SnowflakeIngestor(
         account=os.getenv("SNOWFLAKE_ACCOUNT"),
@@ -224,80 +233,66 @@ def example_key_pair_auth():
         private_key_path=os.getenv("SNOWFLAKE_PRIVATE_KEY_PATH"),
         warehouse="COMPUTE_WH",
     )
-
     data = ingestor.ingest_table("CUSTOMERS", limit=5)
-    print(f"Successfully authenticated and retrieved {data.row_count} rows")
-
+    console.print(
+        f"[green]✓[/green] Authenticated — retrieved [cyan]{data.row_count}[/cyan] rows"
+    )
     ingestor.close()
 
 
 def example_context_manager():
     """Example: Using context manager."""
-    print("\n=== Example 10: Context Manager ===\n")
+    _section("Example 10: Context Manager")
 
     with SnowflakeIngestor() as ingestor:
         data = ingestor.ingest_table("CUSTOMERS", limit=5)
-        print(f"Retrieved {data.row_count} rows")
-
-    # Connection automatically closed
-    print("Connection closed automatically")
+        console.print(
+            f"[green]✓[/green] Retrieved [cyan]{data.row_count}[/cyan] rows"
+        )
+    console.print("[dim]  Connection closed automatically.[/dim]")
 
 
 def example_multi_schema():
     """Example: Multi-schema ingestion."""
-    print("\n=== Example 11: Multi-Schema Ingestion ===\n")
+    _section("Example 11: Multi-Schema Ingestion")
 
     ingestor = SnowflakeIngestor()
+    prod = ingestor.ingest_table("CUSTOMERS", database="PROD_DB", schema="PUBLIC", limit=10)
+    staging = ingestor.ingest_table("CUSTOMERS", database="STAGING_DB", schema="PUBLIC", limit=10)
 
-    # Ingest from different schemas
-    prod_customers = ingestor.ingest_table(
-        "CUSTOMERS", database="PROD_DB", schema="PUBLIC", limit=10
-    )
-
-    staging_customers = ingestor.ingest_table(
-        "CUSTOMERS", database="STAGING_DB", schema="PUBLIC", limit=10
-    )
-
-    print(f"Production customers: {prod_customers.row_count}")
-    print(f"Staging customers: {staging_customers.row_count}")
+    console.print(f"  Production: [cyan]{prod.row_count}[/cyan] customers")
+    console.print(f"  Staging:    [cyan]{staging.row_count}[/cyan] customers")
 
     ingestor.close()
 
 
 def example_error_handling():
     """Example: Error handling."""
-    print("\n=== Example 12: Error Handling ===\n")
+    _section("Example 12: Error Handling")
 
     from semantica.utils.exceptions import ProcessingError, ValidationError
 
     try:
-        # Try to connect with invalid credentials
         ingestor = SnowflakeIngestor(
             account="invalid_account", user="invalid_user", password="invalid_password"
         )
-
-        data = ingestor.ingest_table("CUSTOMERS")
+        ingestor.ingest_table("CUSTOMERS")
 
     except ValidationError as e:
-        print(f"Validation error: {e}")
-
+        console.print(f"[bold yellow] ⚠[/bold yellow] Validation error: {e}")
     except ProcessingError as e:
-        print(f"Processing error: {e}")
-
+        console.print(f"[bold red] ✗[/bold red] Processing error: {e}")
     except Exception as e:
-        print(f"Unexpected error: {e}")
+        console.print(f"[bold red] ✗[/bold red] Unexpected error: {e}")
 
 
 def example_incremental_load():
     """Example: Incremental data loading."""
-    print("\n=== Example 13: Incremental Loading ===\n")
+    _section("Example 13: Incremental Loading")
 
     ingestor = SnowflakeIngestor()
+    last_load = get_last_load_timestamp()
 
-    # Get last load timestamp (from your metadata store)
-    last_load = get_last_load_timestamp()  # Your function
-
-    # Query only new/updated records
     query = """
         SELECT *
         FROM CUSTOMERS
@@ -306,10 +301,10 @@ def example_incremental_load():
     """
 
     data = ingestor.ingest_query(query, params={"last_load": last_load})
-
-    print(f"Loaded {data.row_count} new/updated records since {last_load}")
-
-    # Update last load timestamp
+    console.print(
+        f"[green]✓[/green] Loaded [cyan]{data.row_count}[/cyan] new/updated "
+        f"records since [dim]{last_load}[/dim]"
+    )
     if data.row_count > 0:
         update_last_load_timestamp(datetime.now())
 
@@ -318,20 +313,14 @@ def example_incremental_load():
 
 def example_etl_pipeline():
     """Example: Full ETL pipeline."""
-    print("\n=== Example 14: ETL Pipeline ===\n")
+    _section("Example 14: ETL Pipeline")
 
-    # Extract
     ingestor = SnowflakeIngestor()
 
     sales_query = """
-        SELECT 
-            s.ORDER_ID,
-            s.CUSTOMER_ID,
-            c.CUSTOMER_NAME,
-            s.PRODUCT_ID,
-            p.PRODUCT_NAME,
-            s.AMOUNT,
-            s.ORDER_DATE
+        SELECT
+            s.ORDER_ID, s.CUSTOMER_ID, c.CUSTOMER_NAME,
+            s.PRODUCT_ID, p.PRODUCT_NAME, s.AMOUNT, s.ORDER_DATE
         FROM SALES s
         JOIN CUSTOMERS c ON s.CUSTOMER_ID = c.ID
         JOIN PRODUCTS p ON s.PRODUCT_ID = p.ID
@@ -339,43 +328,33 @@ def example_etl_pipeline():
     """
 
     data = ingestor.ingest_query(sales_query)
-    print(f"Extracted {data.row_count} sales records")
+    console.print(f"  [dim]Extract:[/dim] [cyan]{data.row_count}[/cyan] sales records")
 
-    # Transform
     documents = ingestor.export_as_documents(
         data, id_field="ORDER_ID", text_fields=["CUSTOMER_NAME", "PRODUCT_NAME"]
     )
-    print(f"Transformed to {len(documents)} documents")
+    console.print(f"  [dim]Transform:[/dim] [cyan]{len(documents)}[/cyan] documents")
 
-    # Load (into Semantica)
     from semantica.pipeline import Pipeline
-
     pipeline = Pipeline()
-
     for doc in documents:
         pipeline.process_document(doc)
 
-    print("Loaded documents into Semantica pipeline")
-
+    console.print("[green]✓[/green] Loaded documents into Semantica pipeline")
     ingestor.close()
 
 
-# Utility functions for examples
+# ─── Utility stubs ────────────────────────────────────────────────────────────
+
 def process_page(data):
-    """Process a page of data."""
-    # Your processing logic here
     pass
 
 
 def get_last_load_timestamp():
-    """Get the last load timestamp from metadata store."""
-    # Your implementation here
     return (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d %H:%M:%S")
 
 
 def update_last_load_timestamp(timestamp):
-    """Update the last load timestamp in metadata store."""
-    # Your implementation here
     pass
 
 
@@ -395,17 +374,10 @@ def main():
     for example_func in examples:
         try:
             example_func()
+            console.print()
         except Exception as e:
-            logger.error(f"Example {example_func.__name__} failed: {e}")
+            logger.error("Example %s failed: %s", example_func.__name__, e)
 
 
 if __name__ == "__main__":
-    # Set up environment variables
-    # export SNOWFLAKE_ACCOUNT=your_account
-    # export SNOWFLAKE_USER=your_user
-    # export SNOWFLAKE_PASSWORD=your_password
-    # export SNOWFLAKE_WAREHOUSE=COMPUTE_WH
-    # export SNOWFLAKE_DATABASE=SAMPLE_DB
-    # export SNOWFLAKE_SCHEMA=PUBLIC
-
     main()

cookbook/introduction/21_Amazon_Neptune_Store.ipynb

@@ -3,81 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "# Amazon Neptune Graph Store\n",
-    "\n",
-    "## Overview\n",
-    "\n",
-    "This notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n",
-    "\n",
-    "### Key Features\n",
-    "\n",
-    "- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n",
-    "- **OpenCypher Support**: Query using standard OpenCypher syntax\n",
-    "- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n",
-    "- **Native ~id Support**: Leverages Neptune's native element ID handling\n",
-    "- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n",
-    "- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n",
-    "\n",
-    "### Prerequisites\n",
-    "\n",
-    "- An Amazon Neptune Database cluster\n",
-    "- AWS credentials configured (boto3, environment variables, or IAM role)\n",
-    "- Network access to your Neptune cluster (VPC, security groups)\n",
-    "\n",
-    "#### Quick Setup with CloudFormation\n",
-    "\n",
-    "If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
-    "\n",
-    "```bash\n",
-    "# Deploy the Neptune stack (takes ~15-20 minutes)\n",
-    "aws cloudformation create-stack \\\n",
-    "  --stack-name semantica-neptune \\\n",
-    "  --template-body file://neptune-setup.yaml \\\n",
-    "  --capabilities CAPABILITY_NAMED_IAM\n",
-    "\n",
-    "# Wait for stack creation to complete\n",
-    "aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
-    "\n",
-    "# Get the outputs (endpoint, port, credentials)\n",
-    "aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
-    "  --query 'Stacks[0].Outputs' --output table\n",
-    "```\n",
-    "\n",
-    "The template creates:\n",
-    "- VPC with public subnets and Internet Gateway\n",
-    "- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
-    "- IAM user with least-privilege access for OpenCypher queries\n",
-    "- Security group allowing Bolt protocol (port 8182) access\n",
-    "\n",
-    "> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
-    "\n",
-    "**Outputs:**\n",
-    "- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
-    "- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
-    "- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
-    "- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
-    "- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
-    "\n",
-    "**Cleanup:**\n",
-    "```bash\n",
-    "aws cloudformation delete-stack --stack-name semantica-neptune\n",
-    "```\n",
-    "\n",
-    "**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
-    "\n",
-    "| Resource | Cost (USD) |\n",
-    "| --- | --- |\n",
-    "| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
-    "| Storage (10 GB) | ~1/month |\n",
-    "| I/O requests | ~1-5/month |\n",
-    "| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
-    "| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
-    "\n",
-    "> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
-    "\n",
-    "---"
-   ]
+   "source": "# Amazon Neptune Graph Store\n\n## Overview\n\nThis notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n\n### Key Features\n\n- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n- **OpenCypher Support**: Query using standard OpenCypher syntax\n- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n- **Native ~id Support**: Leverages Neptune's native element ID handling\n- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n\n### Prerequisites\n\n- An Amazon Neptune Database cluster\n- AWS credentials configured (boto3, environment variables, or IAM role)\n- Network access to your Neptune cluster (VPC, security groups)\n- Your public IP address or VPN/office CIDR (run `curl ifconfig.me` to find your public IP), used below to restrict database access\n\n#### Quick Setup with CloudFormation\n\nIf you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n\n```bash\n# Deploy the Neptune stack (takes ~15-20 minutes)\n# Replace 203.0.113.25/32 with your own public IP (run `curl ifconfig.me` to find it)\n# or your office/VPN CIDR. This restricts who can reach the database on the\n# network level - never widen it to 0.0.0.0/0 outside of a short-lived local experiment.\naws cloudformation create-stack \\\n  --stack-name semantica-neptune \\\n  --template-body file://neptune-setup.yaml \\\n  --parameters ParameterKey=ClientCidr,ParameterValue=203.0.113.25/32 \\\n  --capabilities CAPABILITY_NAMED_IAM\n\n# Wait for stack creation to complete\naws cloudformation wait stack-create-complete --stack-name semantica-neptune\n\n# Get the outputs (endpoint, port, credentials)\naws cloudformation describe-stacks --stack-name semantica-neptune \\\n  --query 'Stacks[0].Outputs' --output table\n```\n\nThe template creates:\n- VPC with public subnets, Internet Gateway, and VPC Flow Logs (to CloudWatch Logs)\n- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n- IAM user with least-privilege access for OpenCypher queries\n- Security group allowing Bolt protocol (port 8182) access only from the `ClientCidr` you specify\n\n> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager. The `ClientCidr` parameter is required (no default) precisely so the database is never silently exposed to the whole internet.\n\n**Outputs:**\n- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n\n**Cleanup:**\n```bash\naws cloudformation delete-stack --stack-name semantica-neptune\n```\n\n**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n\n| Resource | Cost (USD) |\n| --- | --- |\n| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n| Storage (10 GB) | ~1/month |\n| I/O requests | ~1-5/month |\n| Public IPv4 address | ~3.60/month (0.005/hr) |\n| VPC Flow Logs (CloudWatch Logs) | ~1-2/month depending on traffic |\n| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n\n> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n\n---"
   },
   {
    "cell_type": "markdown",
@@ -722,4 +648,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
+}

cookbook/introduction/neptune-setup.yaml

@@ -1,7 +1,14 @@
+# ts:skip=AC_AWS_0148 IAM password policy is an AWS-account-wide singleton, not a
+# per-stack resource. Managing it here would mean every learner who deploys or
+# deletes this cookbook stack also mutates (or removes) their account's password
+# policy as a side effect. Account password policy should be set once, out of
+# band, by the account owner - not by a disposable tutorial stack.
 AWSTemplateFormatVersion: '2010-09-09'
 Description: >
   Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
   IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
+  Network access to the Bolt/OpenCypher port is restricted to an operator-supplied CIDR
+  (see ClientCidr) - do not widen this to 0.0.0.0/0 outside of a short-lived local experiment.
 
 Parameters:
   EnvironmentName:
@@ -9,6 +16,16 @@ Parameters:
     Default: semantica-neptune
     Description: Environment name prefix for resource naming
 
+  ClientCidr:
+    Type: String
+    Description: >-
+      CIDR block allowed to reach the Neptune Bolt/OpenCypher endpoint (port 8182) - e.g. your
+      workstation's public IP as "x.x.x.x/32", or your office/VPN CIDR. Required: there is no
+      default, so you must explicitly choose a range. Passing 0.0.0.0/0 is possible but exposes
+      the database to the entire internet and is strongly discouraged beyond a brief local test.
+    AllowedPattern: '^((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])/(3[0-2]|[12]?[0-9])$'
+    ConstraintDescription: Must be a valid IPv4 CIDR block with octets 0-255 and prefix 0-32, e.g. 203.0.113.25/32
+
 Resources:
   # =============================================================================
   # VPC & NETWORKING
@@ -87,6 +104,57 @@ Resources:
       RouteTableId: !Ref PublicRouteTable
       SubnetId: !Ref PublicSubnet2
 
+  # =============================================================================
+  # VPC FLOW LOGS
+  # =============================================================================
+
+  FlowLogGroup:
+    Type: AWS::Logs::LogGroup
+    Properties:
+      LogGroupName: !Sub /aws/vpc/${EnvironmentName}-flow-logs
+      RetentionInDays: 30
+
+  FlowLogRole:
+    Type: AWS::IAM::Role
+    Properties:
+      RoleName: !Sub ${EnvironmentName}-flow-log-role
+      AssumeRolePolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service: vpc-flow-logs.amazonaws.com
+            Action: sts:AssumeRole
+      Policies:
+        - PolicyName: flow-log-publish
+          PolicyDocument:
+            Version: '2012-10-17'
+            Statement:
+              - Effect: Allow
+                Action:
+                  - logs:CreateLogGroup
+                  - logs:DescribeLogGroups
+                  - logs:DescribeLogStreams
+                Resource: "*"
+              - Effect: Allow
+                Action:
+                  - logs:CreateLogStream
+                  - logs:PutLogEvents
+                Resource: !GetAtt FlowLogGroup.Arn
+
+  VPCFlowLog:
+    Type: AWS::EC2::FlowLog
+    Properties:
+      ResourceType: VPC
+      ResourceId: !Ref VPC
+      TrafficType: ALL
+      LogDestinationType: cloud-watch-logs
+      LogGroupName: !Ref FlowLogGroup
+      DeliverLogsPermissionArn: !GetAtt FlowLogRole.Arn
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-vpc-flow-log
+
   # =============================================================================
   # SECURITY GROUP
   # =============================================================================
@@ -95,14 +163,14 @@ Resources:
     Type: AWS::EC2::SecurityGroup
     Properties:
       GroupName: !Sub ${EnvironmentName}-neptune-sg
-      GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
+      GroupDescription: Security group for Neptune cluster - allows Bolt protocol access from ClientCidr only
       VpcId: !Ref VPC
       SecurityGroupIngress:
         - IpProtocol: tcp
           FromPort: 8182
           ToPort: 8182
-          CidrIp: 0.0.0.0/0
-          Description: Allow Bolt protocol access from anywhere
+          CidrIp: !Ref ClientCidr
+          Description: Allow Bolt/OpenCypher protocol access from the operator-specified CIDR
       SecurityGroupEgress:
         - IpProtocol: -1
           CidrIp: 0.0.0.0/0
@@ -138,6 +206,8 @@ Resources:
       IamAuthEnabled: true
       StorageEncrypted: true
       DeletionProtection: false
+      EnableCloudwatchLogsExports:
+        - audit
       Tags:
         - Key: Name
           Value: !Sub ${EnvironmentName}-cluster

cookbook/use_cases/advanced_rag/data/sample_graphrag_paper.txt

@@ -1,13 +0,0 @@
-Graph Retrieval-Augmented Generation (GraphRAG): A New Era for Intelligent Search
-
-GraphRAG is an advanced technique that combines the retrieval capabilities of vector databases with the structural reasoning of knowledge graphs. Unlike traditional RAG, which relies solely on vector similarity, GraphRAG leverages the relationships between entities to provide more contextually accurate and comprehensive answers.
-
-Key Components:
-1. Knowledge Graph: A structured representation of data where nodes represent entities and edges represent relationships.
-2. Vector Search: Finds semantically similar text chunks.
-3. Graph Traversal: Navigates the knowledge graph to find related entities that might not be semantically similar but are structurally relevant.
-
-Benefits:
-- Improved Context: By following relationships, the system can understand the broader context of a query.
-- Multi-hop Reasoning: Can answer complex questions that require connecting multiple pieces of information.
-- Reduced Hallucinations: Grounding answers in a verified knowledge structure reduces the likelihood of generating false information.

cookbook/use_cases/advanced_rag/expert_guide.txt

@@ -1,5 +0,0 @@
-
-RETINOL CLINICAL GUIDE
-Mechanism: Binds to retinoic acid receptors to increase cellular turnover.
-Precautions: Should not be used with high-concentration AHA/BHA exfoliants.
-Synergy: Highly effective when paired with Niacinamide to offset potential erythema.

cookbook/use_cases/advanced_rag/expert_skincare_guide.txt

@@ -1,6 +0,0 @@
-
-RETINOL CLINICAL GUIDE v2.1
-Mechanism: Binds to retinoic acid receptors (RAR) to increase cellular turnover.
-Precautions: Should not be used with high-concentration AHA/BHA exfoliants.
-Synergy: Highly effective when paired with Niacinamide to offset potential erythema.
-Target: Stratum corneum thickening and dermal collagen synthesis.

cookbook/use_cases/advanced_rag/graphrag_kg.graphml

@@ -1,254 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
-         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-         xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
-         http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
-
-  <key id="type" for="node" attr.name="type" attr.type="string"/>
-  <key id="confidence" for="node" attr.name="confidence" attr.type="double"/>
-
-  <graph id="G" edgedefault="directed">
-
-    <node id="makeup_and_beauty_blog">
-      <data key="label">Makeup and Beauty Blog</data>
-      <data key="type">ORG</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="monday_poll">
-      <data key="label">Monday Poll</data>
-      <data key="type">EVENT</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="2007">
-      <data key="label">2007</data>
-      <data key="type">DATE</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="rosacea">
-      <data key="label">Rosacea</data>
-      <data key="type">CONCEPT</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="dr._bailey">
-      <data key="label">Dr. Bailey</data>
-      <data key="type">PERSON</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="green_tea_antioxidant_skin_therapy">
-      <data key="label">Green Tea Antioxidant Skin Therapy</data>
-      <data key="type">PRODUCT</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="vol._892">
-      <data key="label">Vol. 892</data>
-      <data key="type">EVENT</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="laneige">
-      <data key="label">Laneige</data>
-      <data key="type">ORG</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="sausalito">
-      <data key="label">Sausalito</data>
-      <data key="type">GPE</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="ulta">
-      <data key="label">Ulta</data>
-      <data key="type">ORG</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="december_15,_2025">
-      <data key="label">December 15, 2025</data>
-      <data key="type">DATE</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="jo_malone">
-      <data key="label">Jo Malone</data>
-      <data key="type">ORG</data>
-      <data key="confidence">1</data>
-    </node>
-    <node id="trader_joe">
-      <data key="label">Trader Joe</data>
-      <data key="type">ORG</data>
-      <data key="confidence">1</data>
-    </node>
-    <node id="hawaii">
-      <data key="label">hawaii</data>
-      <data key="type">GPE</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="benzoyl_peroxide_cream">
-      <data key="label">Benzoyl Peroxide Cream</data>
-      <data key="type">PRODUCT</data>
-      <data key="confidence">1</data>
-    </node>
-    <node id="facial_dandruff">
-      <data key="label">Facial dandruff</data>
-      <data key="type">CONCEPT</data>
-      <data key="confidence">1</data>
-    </node>
-    <node id="calming_zinc_soap">
-      <data key="label">Calming Zinc Soap</data>
-      <data key="type">PRODUCT</data>
-      <data key="confidence">1</data>
-    </node>
-    <node id="hydrate">
-      <data key="label">Hydrate</data>
-      <data key="type">CONCEPT</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="daily_moisturizing_face_cream">
-      <data key="label">Daily Moisturizing Face Cream</data>
-      <data key="type">PRODUCT</data>
-      <data key="confidence">1.0</data>
-    </node>
-    <node id="omega_enriched_face_booster_oil">
-      <data key="label">Omega Enriched Face Booster Oil</data>
-      <data key="type">PRODUCT</data>
-      <data key="confidence">1.0</data>
-    </node>
-
-    <edge source="Makeup and Beauty Blog" target="Monday Poll">
-      <data key="label">hosts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Monday Poll" target="December 15, 2025">
-      <data key="label">occurs on</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Makeup and Beauty Blog Monday Poll, Vol. 893">
-      <data key="label">publishes</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Monday">
-      <data key="label">has</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="2007">
-      <data key="label">has</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Monday Poll">
-      <data key="label">hosts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Makeup and Beauty Blog Monday Poll">
-      <data key="label">posts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Vol. 892">
-      <data key="label">posts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="2007">
-      <data key="label">has been active since</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="MBB" target="Makeup and Beauty Blog">
-      <data key="label">related_to</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Makeup and Beauty Blog">
-      <data key="label">related_to</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Monday Poll">
-      <data key="label">hosts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Vol. 891">
-      <data key="label">posts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Makeup and Beauty Blog" target="Monday Poll">
-      <data key="label">posts</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Cavallo Point" target="Sausalito">
-      <data key="label">located_in</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey" target="Green Tea Antioxidant Skin Therapy">
-      <data key="label">prescribes</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Green Tea Antioxidant Skin Therapy" target="Rosacea Therapy Skin Care Kit">
-      <data key="label">part of</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey" target="Rosacea Therapy Skin Care Kit">
-      <data key="label">uses</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Rosacea Therapy Skin Care Kit" target="rosacea treatment routine">
-      <data key="label">part of</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey" target="rosacea treatment routine">
-      <data key="label">uses</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Facial dandruff" target="rosacea">
-      <data key="label">often occurs with</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Facial dandruff" target="rosacea">
-      <data key="label">needs to be addressed</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Calming Zinc Soap" target="Facial dandruff">
-      <data key="label">is often sufficient to control</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Calming Zinc Soap" target="rosacea">
-      <data key="label">is often sufficient to control</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Green Tea Antioxidant Skin Therapy" target="Facial dandruff">
-      <data key="label">is often sufficient to control</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Green Tea Antioxidant Skin Therapy" target="rosacea">
-      <data key="label">is often sufficient to control</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey's Skincare" target="Calming Zinc Soap">
-      <data key="label">produces</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey's Skincare" target="Green Tea Antioxidant Skin Therapy">
-      <data key="label">produces</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey" target="Calming Zinc Soap">
-      <data key="label">prescribes</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Dr. Bailey" target="Green Tea Antioxidant Skin Therapy">
-      <data key="label">prescribes</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Hydrate" target="Daily Moisturizing Face Cream">
-      <data key="label">is_achieved_by</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Daily Moisturizing Face Cream" target="Omega Enriched Face Booster Oil">
-      <data key="label">can_be_combined_with</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Omega Enriched Face Booster Oil" target="castor seed oil">
-      <data key="label">contains</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Omega Enriched Face Booster Oil" target="sea buckthorn">
-      <data key="label">contains</data>
-      <data key="confidence">0.9</data>
-    </edge>
-    <edge source="Daily Moisturizing Face Cream" target="Omega Enriched Face Booster Oil">
-      <data key="label">can_be_replaced_with</data>
-      <data key="confidence">0.9</data>
-    </edge>
-  </graph>
-</graphml>

cookbook/use_cases/advanced_rag/graphrag_kg.json

@@ -1,678 +0,0 @@
-{
-  "nodes": [
-    {
-      "id": "makeup_and_beauty_blog",
-      "label": "Makeup and Beauty Blog",
-      "type": "ORG",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "makeup_and_beauty_blog",
-              "name": "Makeup and Beauty Blog",
-              "source": null
-            },
-            {
-              "id": "makeup_and_beauty_blog",
-              "name": "Makeup and Beauty Blog",
-              "source": null
-            },
-            {
-              "id": "makeup_and_beauty_blog_monday_poll,_vol._893",
-              "name": "Makeup and Beauty Blog Monday Poll, Vol. 893",
-              "source": null
-            },
-            {
-              "id": "makeup_and_beauty_blog_monday_poll",
-              "name": "Makeup and Beauty Blog Monday Poll",
-              "source": null
-            },
-            {
-              "id": "mbb",
-              "name": "MBB",
-              "source": null
-            }
-          ],
-          "merge_count": 5
-        }
-      }
-    },
-    {
-      "id": "monday_poll",
-      "label": "Monday Poll",
-      "type": "EVENT",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "monday_poll",
-              "name": "Monday Poll",
-              "source": null
-            },
-            {
-              "id": "monday_poll",
-              "name": "Monday Poll",
-              "source": null
-            },
-            {
-              "id": "monday",
-              "name": "Monday",
-              "source": null
-            },
-            {
-              "id": "holiday",
-              "name": "holiday",
-              "source": null
-            },
-            {
-              "id": "holiday",
-              "name": "holiday",
-              "source": null
-            }
-          ],
-          "merge_count": 5
-        }
-      }
-    },
-    {
-      "id": "2007",
-      "label": "2007",
-      "type": "DATE",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "2007",
-              "name": "2007",
-              "source": null
-            },
-            {
-              "id": "2007",
-              "name": "2007",
-              "source": null
-            },
-            {
-              "id": "2024",
-              "name": "2024",
-              "source": null
-            }
-          ],
-          "merge_count": 3
-        }
-      }
-    },
-    {
-      "id": "rosacea",
-      "label": "Rosacea",
-      "type": "CONCEPT",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "rosacea",
-              "name": "Rosacea",
-              "source": null
-            },
-            {
-              "id": "rosacea",
-              "name": "rosacea",
-              "source": null
-            },
-            {
-              "id": "rosacea_treatment_routine",
-              "name": "rosacea treatment routine",
-              "source": null
-            },
-            {
-              "id": "rosie",
-              "name": "Rosie",
-              "source": null
-            },
-            {
-              "id": "rosacea_therapy_skin_care_kit",
-              "name": "Rosacea Therapy Skin Care Kit",
-              "source": null
-            },
-            {
-              "id": "marnie",
-              "name": "Marnie",
-              "source": null
-            },
-            {
-              "id": "cavallo_point",
-              "name": "Cavallo Point",
-              "source": null
-            },
-            {
-              "id": "castor_seed_oil",
-              "name": "castor seed oil",
-              "source": null
-            }
-          ],
-          "merge_count": 8
-        }
-      }
-    },
-    {
-      "id": "dr._bailey",
-      "label": "Dr. Bailey",
-      "type": "PERSON",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "dr._bailey",
-              "name": "Dr. Bailey",
-              "source": null
-            },
-            {
-              "id": "dr._bailey",
-              "name": "Dr. Bailey",
-              "source": null
-            },
-            {
-              "id": "dr._bailey's_skincare",
-              "name": "Dr. Bailey's Skincare",
-              "source": null
-            },
-            {
-              "id": "dr._bailey's_skincare",
-              "name": "Dr. Bailey's Skincare",
-              "source": null
-            }
-          ],
-          "merge_count": 4
-        }
-      }
-    },
-    {
-      "id": "green_tea_antioxidant_skin_therapy",
-      "label": "Green Tea Antioxidant Skin Therapy",
-      "type": "PRODUCT",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "green_tea_antioxidant_skin_therapy",
-              "name": "Green Tea Antioxidant Skin Therapy",
-              "source": null
-            },
-            {
-              "id": "green_tea_antioxidant_skin_therapy",
-              "name": "Green Tea Antioxidant Skin Therapy",
-              "source": null
-            }
-          ],
-          "merge_count": 2
-        }
-      }
-    },
-    {
-      "id": "vol._892",
-      "label": "Vol. 892",
-      "type": "EVENT",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "vol._892",
-              "name": "Vol. 892",
-              "source": null
-            },
-            {
-              "id": "vol._891",
-              "name": "Vol. 891",
-              "source": null
-            }
-          ],
-          "merge_count": 2
-        }
-      }
-    },
-    {
-      "id": "laneige",
-      "label": "Laneige",
-      "type": "ORG",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "laneige",
-              "name": "Laneige",
-              "source": null
-            },
-            {
-              "id": "lanikai",
-              "name": "Lanikai",
-              "source": null
-            }
-          ],
-          "merge_count": 2
-        }
-      }
-    },
-    {
-      "id": "sausalito",
-      "label": "Sausalito",
-      "type": "GPE",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "sausalito",
-              "name": "Sausalito",
-              "source": null
-            },
-            {
-              "id": "sea_buckthorn",
-              "name": "sea buckthorn",
-              "source": null
-            }
-          ],
-          "merge_count": 2
-        }
-      }
-    },
-    {
-      "id": "ulta",
-      "label": "Ulta",
-      "type": "ORG",
-      "attributes": {
-        "confidence": 1.0,
-        "provenance": {
-          "merged_from": [
-            {
-              "id": "ulta",
-              "name": "Ulta",
-              "source": null
-            },
-            {
-              "id": "clotrimazole",
-              "name": "clotrimazole",
-              "source": null
-            }
-          ],
-          "merge_count": 2
-        }
-      }
-    },
-    {
-      "id": "december_15,_2025",
-      "label": "December 15, 2025",
-      "type": "DATE",
-      "attributes": {
-        "confidence": 1.0
-      }
-    },
-    {
-      "id": "jo_malone",
-      "label": "Jo Malone",
-      "type": "ORG",
-      "attributes": {
-        "confidence": 1
-      }
-    },
-    {
-      "id": "trader_joe",
-      "label": "Trader Joe",
-      "type": "ORG",
-      "attributes": {
-        "confidence": 1
-      }
-    },
-    {
-      "id": "hawaii",
-      "label": "hawaii",
-      "type": "GPE",
-      "attributes": {
-        "confidence": 1.0
-      }
-    },
-    {
-      "id": "benzoyl_peroxide_cream",
-      "label": "Benzoyl Peroxide Cream",
-      "type": "PRODUCT",
-      "attributes": {
-        "confidence": 1
-      }
-    },
-    {
-      "id": "facial_dandruff",
-      "label": "Facial dandruff",
-      "type": "CONCEPT",
-      "attributes": {
-        "confidence": 1
-      }
-    },
-    {
-      "id": "calming_zinc_soap",
-      "label": "Calming Zinc Soap",
-      "type": "PRODUCT",
-      "attributes": {
-        "confidence": 1
-      }
-    },
-    {
-      "id": "hydrate",
-      "label": "Hydrate",
-      "type": "CONCEPT",
-      "attributes": {
-        "confidence": 1.0
-      }
-    },
-    {
-      "id": "daily_moisturizing_face_cream",
-      "label": "Daily Moisturizing Face Cream",
-      "type": "PRODUCT",
-      "attributes": {
-        "confidence": 1.0
-      }
-    },
-    {
-      "id": "omega_enriched_face_booster_oil",
-      "label": "Omega Enriched Face Booster Oil",
-      "type": "PRODUCT",
-      "attributes": {
-        "confidence": 1.0
-      }
-    }
-  ],
-  "edges": [
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Monday Poll",
-      "type": "hosts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Monday Poll",
-      "target": "December 15, 2025",
-      "type": "occurs on",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Makeup and Beauty Blog Monday Poll, Vol. 893",
-      "type": "publishes",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Monday",
-      "type": "has",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "2007",
-      "type": "has",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Monday Poll",
-      "type": "hosts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Makeup and Beauty Blog Monday Poll",
-      "type": "posts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Vol. 892",
-      "type": "posts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "2007",
-      "type": "has been active since",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "MBB",
-      "target": "Makeup and Beauty Blog",
-      "type": "related_to",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Makeup and Beauty Blog",
-      "type": "related_to",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Monday Poll",
-      "type": "hosts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Vol. 891",
-      "type": "posts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Makeup and Beauty Blog",
-      "target": "Monday Poll",
-      "type": "posts",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Cavallo Point",
-      "target": "Sausalito",
-      "type": "located_in",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey",
-      "target": "Green Tea Antioxidant Skin Therapy",
-      "type": "prescribes",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Green Tea Antioxidant Skin Therapy",
-      "target": "Rosacea Therapy Skin Care Kit",
-      "type": "part of",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey",
-      "target": "Rosacea Therapy Skin Care Kit",
-      "type": "uses",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Rosacea Therapy Skin Care Kit",
-      "target": "rosacea treatment routine",
-      "type": "part of",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey",
-      "target": "rosacea treatment routine",
-      "type": "uses",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Facial dandruff",
-      "target": "rosacea",
-      "type": "often occurs with",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Facial dandruff",
-      "target": "rosacea",
-      "type": "needs to be addressed",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Calming Zinc Soap",
-      "target": "Facial dandruff",
-      "type": "is often sufficient to control",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Calming Zinc Soap",
-      "target": "rosacea",
-      "type": "is often sufficient to control",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Green Tea Antioxidant Skin Therapy",
-      "target": "Facial dandruff",
-      "type": "is often sufficient to control",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Green Tea Antioxidant Skin Therapy",
-      "target": "rosacea",
-      "type": "is often sufficient to control",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey's Skincare",
-      "target": "Calming Zinc Soap",
-      "type": "produces",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey's Skincare",
-      "target": "Green Tea Antioxidant Skin Therapy",
-      "type": "produces",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey",
-      "target": "Calming Zinc Soap",
-      "type": "prescribes",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Dr. Bailey",
-      "target": "Green Tea Antioxidant Skin Therapy",
-      "type": "prescribes",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Hydrate",
-      "target": "Daily Moisturizing Face Cream",
-      "type": "is_achieved_by",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Daily Moisturizing Face Cream",
-      "target": "Omega Enriched Face Booster Oil",
-      "type": "can_be_combined_with",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Omega Enriched Face Booster Oil",
-      "target": "castor seed oil",
-      "type": "contains",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Omega Enriched Face Booster Oil",
-      "target": "sea buckthorn",
-      "type": "contains",
-      "attributes": {
-        "confidence": 0.9
-      }
-    },
-    {
-      "source": "Daily Moisturizing Face Cream",
-      "target": "Omega Enriched Face Booster Oil",
-      "type": "can_be_replaced_with",
-      "attributes": {
-        "confidence": 0.9
-      }
-    }
-  ],
-  "metadata": {
-    "num_entities": 20,
-    "num_relationships": 35,
-    "temporal_enabled": false,
-    "timestamp": "2025-12-24T12:46:41.535755",
-    "entity_resolution_applied": true
-  }
-}

cookbook/use_cases/advanced_rag/ground_truth.json

@@ -1 +0,0 @@
-{"entities": [{"id": "python_org", "name": "Python Software Foundation", "type": "Organization"}, {"id": "guido_van_rossum", "name": "Guido van Rossum", "type": "Person"}], "relationships": [{"source": "guido_van_rossum", "target": "python_org", "type": "FOUNDED"}]}

cookbook/use_cases/advanced_rag/skincare_base.json

@@ -1,38 +0,0 @@
-{
-  "entities": [
-    {
-      "id": "hyaluronic_acid",
-      "name": "Hyaluronic Acid",
-      "type": "Ingredient",
-      "properties": {
-        "role": "Humectant"
-      }
-    },
-    {
-      "id": "retinol",
-      "name": "Retinol",
-      "type": "Ingredient",
-      "properties": {
-        "role": "Anti-aging actives"
-      }
-    },
-    {
-      "id": "niacinamide",
-      "name": "Niacinamide",
-      "type": "Ingredient",
-      "properties": {
-        "role": "Barrier repair"
-      }
-    }
-  ],
-  "relationships": [
-    {
-      "source": "hyaluronic_acid",
-      "target": "niacinamide",
-      "type": "COMPLEMENTS",
-      "properties": {
-        "benefit": "Hydration + Barrier"
-      }
-    }
-  ]
-}

cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb

@@ -1,693 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)\n",
-    "\n",
-    "# Drug Discovery Pipeline - Vector Similarity Search\n",
-    "\n",
-    "## Overview\n",
-    "\n",
-    "This notebook demonstrates a **complete drug discovery pipeline** using Semantica's modular architecture. We'll use individual modules directly to build a comprehensive system for drug-target interaction prediction using vector similarity search and knowledge graphs.\n",
-    "\n",
-    "### Key Features\n",
-    "\n",
-    "- **Modular Architecture**: Uses Semantica modules directly (`NERExtractor`, `GraphBuilder`, `EmbeddingGenerator`, `VectorStore`)\n",
-    "- **Multiple Data Sources**: Ingests from 15+ PubMed RSS feeds, preprint servers, and journal feeds\n",
-    "- **Vector Similarity Search**: Emphasizes embeddings and vector similarity for drug-target interaction prediction\n",
-    "- **Entity Extraction**: Extracts drug compounds, proteins, targets, enzymes, and receptors\n",
-    "- **Knowledge Graph**: Builds structured drug-target relationship graphs\n",
-    "- **GraphRAG**: Hybrid vector + graph retrieval for enhanced querying\n",
-    "\n",
-    "### What You'll Learn\n",
-    "\n",
-    "- How to use Semantica modules directly (avoiding the core orchestrator)\n",
-    "- How to ingest biomedical data from multiple sources\n",
-    "- How to extract entities using `NERExtractor`\n",
-    "- How to extract relationships using `RelationExtractor`\n",
-    "- How to generate embeddings with `EmbeddingGenerator`\n",
-    "- How to build knowledge graphs with `GraphBuilder`\n",
-    "- How to perform similarity search with `VectorStore`\n",
-    "- How to use GraphRAG with `AgentContext` for hybrid retrieval\n",
-    "\n",
-    "### Pipeline Flow\n",
-    "\n",
-    "```mermaid\n",
-    "graph LR\n",
-    "    A[Data Ingestion] --> B[Text Processing]\n",
-    "    B --> C[Entity Extraction]\n",
-    "    C --> D[Relationship Extraction]\n",
-    "    D --> E[Deduplication]\n",
-    "    E --> F[Embedding Generation]\n",
-    "    F --> G[Vector Store]\n",
-    "    G --> H[Knowledge Graph]\n",
-    "    H --> I[Similarity Search]\n",
-    "    H --> J[GraphRAG Queries]\n",
-    "    I --> K[Visualization]\n",
-    "    J --> K\n",
-    "```\n",
-    "\n",
-    "### Data Sources\n",
-    "\n",
-    "**PubMed RSS Feeds:**\n",
-    "- Drug Discovery, Drug Target Interaction, Pharmacokinetics, Pharmacodynamics\n",
-    "- Clinical Trials, Protein Targets, Drug Repurposing, Molecular Docking\n",
-    "- ADME, Drug Metabolism, Drug Safety, Precision Medicine\n",
-    "- Biomarkers, Drug Resistance, Combinatorial Therapy\n",
-    "\n",
-    "**Preprint Servers:**\n",
-    "- BioRxiv (Pharmacology & Toxicology, Drug Discovery)\n",
-    "- MedRxiv (Clinical Trials)\n",
-    "- ChemRxiv\n",
-    "\n",
-    "**Journal RSS Feeds:**\n",
-    "- Nature (Drug Discovery, Pharmacology)\n",
-    "- Science Translational Medicine\n",
-    "- Cell Chemical Biology\n",
-    "- Journal of Medicinal Chemistry\n",
-    "- Drug Discovery Today\n",
-    "- Trends in Pharmacological Sciences\n",
-    "\n",
-    "\n",
-    "---\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Installation\n",
-    "\n",
-    "Install Semantica and required dependencies:\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Configuration & Setup\n",
-    "\n",
-    "Set up environment variables and configuration constants.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "EMBEDDING_DIMENSION = 384\n",
-    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
-    "CHUNK_SIZE = 1000\n",
-    "CHUNK_OVERLAP = 200\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Ingesting Biomedical Data from Multiple Sources\n",
-    "\n",
-    "Ingest data from comprehensive biomedical sources including PubMed RSS feeds, preprint servers, and journal feeds.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.ingest import FeedIngestor, FileIngestor\n",
-    "import os\n",
-    "from contextlib import redirect_stderr\n",
-    "from io import StringIO\n",
-    "\n",
-    "os.makedirs(\"data\", exist_ok=True)\n",
-    "\n",
-    "feed_sources = [\n",
-    "    # Nature Feeds\n",
-    "    (\"Nature - Drug Discovery\", \"https://www.nature.com/subjects/drug-discovery.rss\"),\n",
-    "    (\"Nature - Pharmacology\", \"https://www.nature.com/subjects/pharmacology.rss\"),\n",
-    "    (\"Nature Reviews Drug Discovery\", \"https://www.nature.com/nrd.rss\"),\n",
-    "    \n",
-    "    # FDA & Government Sources\n",
-    "    (\"FDA MedWatch\", \"https://www.fda.gov/AboutFDA/ContactFDA/StayInformed/RSSFeeds/MedWatch/rss.xml\"),\n",
-    "    (\"NCI News\", \"https://www.cancer.gov/syndication/rss\"),\n",
-    "    \n",
-    "    # Drug Information & News\n",
-    "    (\"Drugs.com - MedNews\", \"https://www.drugs.com/rss/mednews.xml\"),\n",
-    "    (\"Drugs.com - FDA Alerts\", \"https://www.drugs.com/rss/fda-alerts.xml\"),\n",
-    "    (\"Drugs.com - Clinical Trials\", \"https://www.drugs.com/rss/clinical-trials.xml\"),\n",
-    "    \n",
-    "    # Medical News\n",
-    "    (\"Labroots Health & Medicine\", \"http://www.labroots.com/rss/trending/health-and-medicine\"),\n",
-    "    (\"Biology News Net\", \"https://www.biologynews.net/rss.php\"),\n",
-    "    \n",
-    "    # Open Access Journals\n",
-    "    (\"PLOS ONE - Medicine\", \"https://journals.plos.org/plosone/feed/atom\"),\n",
-    "    (\"PLOS Biology\", \"https://journals.plos.org/plosbiology/feed/atom\"),\n",
-    "    (\"PLOS Medicine\", \"https://journals.plos.org/plosmedicine/feed/atom\"),\n",
-    "    \n",
-    "    # Preprint Servers\n",
-    "    (\"arXiv - q-bio\", \"http://arxiv.org/rss/q-bio\"),\n",
-    "    (\"arXiv - q-bio.BM\", \"http://arxiv.org/rss/q-bio.BM\"),\n",
-    "]\n",
-    "\n",
-    "feed_ingestor = FeedIngestor()\n",
-    "all_documents = []\n",
-    "\n",
-    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
-    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
-    "    try:\n",
-    "        with redirect_stderr(StringIO()):\n",
-    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
-    "        \n",
-    "        feed_count = 0\n",
-    "        for item in feed_data.items:\n",
-    "            if not item.content:\n",
-    "                item.content = item.description or item.title or \"\"\n",
-    "            if item.content:\n",
-    "                if not hasattr(item, 'metadata'):\n",
-    "                    item.metadata = {}\n",
-    "                item.metadata['source'] = feed_name\n",
-    "                all_documents.append(item)\n",
-    "                feed_count += 1\n",
-    "        \n",
-    "        if feed_count > 0:\n",
-    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
-    "    except Exception:\n",
-    "        continue\n",
-    "\n",
-    "if not all_documents:\n",
-    "    sample_drug_data = \"\"\"\n",
-    "    Aspirin (acetylsalicylic acid) is a medication used to reduce pain, fever, or inflammation. \n",
-    "    It targets cyclooxygenase enzymes COX-1 and COX-2. Aspirin is commonly used for cardiovascular protection.\n",
-    "    Ibuprofen is a nonsteroidal anti-inflammatory drug (NSAID) that targets COX-1 and COX-2 enzymes.\n",
-    "    Metformin is an antidiabetic medication that targets AMP-activated protein kinase (AMPK).\n",
-    "    Insulin targets the insulin receptor (INSR) to regulate glucose metabolism.\n",
-    "    Warfarin is an anticoagulant that targets vitamin K epoxide reductase complex subunit 1 (VKORC1).\n",
-    "    Atorvastatin is a statin medication that targets HMG-CoA reductase.\n",
-    "    \"\"\"\n",
-    "    \n",
-    "    with open(\"data/sample_drugs.txt\", \"w\") as f:\n",
-    "        f.write(sample_drug_data)\n",
-    "    \n",
-    "    file_ingestor = FileIngestor()\n",
-    "    all_documents = file_ingestor.ingest(\"data/sample_drugs.txt\")\n",
-    "\n",
-    "documents = all_documents\n",
-    "print(f\"Ingested {len(documents)} documents\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Normalizing and Chunking Documents\n",
-    "\n",
-    "Clean and normalize text, then split into chunks using entity-aware chunking to preserve drug/protein entity boundaries.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.normalize import TextNormalizer\n",
-    "from semantica.split import TextSplitter\n",
-    "\n",
-    "normalizer = TextNormalizer()\n",
-    "splitter = TextSplitter(\n",
-    "    method=\"entity_aware\",\n",
-    "    ner_method=\"spacy\",\n",
-    "    chunk_size=CHUNK_SIZE,\n",
-    "    chunk_overlap=CHUNK_OVERLAP\n",
-    ")\n",
-    "\n",
-    "print(f\"Normalizing {len(documents)} documents...\")\n",
-    "normalized_documents = []\n",
-    "for i, doc in enumerate(documents, 1):\n",
-    "    normalized_text = normalizer.normalize(\n",
-    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
-    "        clean_html=True,\n",
-    "        normalize_entities=True,\n",
-    "        remove_extra_whitespace=True,\n",
-    "        lowercase=False\n",
-    "    )\n",
-    "    normalized_documents.append(normalized_text)\n",
-    "    if i % 50 == 0 or i == len(documents):\n",
-    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
-    "\n",
-    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
-    "chunked_documents = []\n",
-    "for i, doc_text in enumerate(normalized_documents, 1):\n",
-    "    try:\n",
-    "        with redirect_stderr(StringIO()):\n",
-    "            chunks = splitter.split(doc_text)\n",
-    "        chunked_documents.extend(chunks)\n",
-    "    except Exception:\n",
-    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
-    "        chunks = simple_splitter.split(doc_text)\n",
-    "        chunked_documents.extend(chunks)\n",
-    "    if i % 50 == 0 or i == len(normalized_documents):\n",
-    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
-    "\n",
-    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n",
-    "\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.semantic_extract import NERExtractor\n",
-    "\n",
-    "# Using spaCy ML method (similar to NER cell)\n",
-    "entity_extractor = NERExtractor(method=\"ml\", model=\"en_core_web_sm\")\n",
-    "\n",
-    "all_entities = []\n",
-    "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n",
-    "\n",
-    "for i, chunk in enumerate(chunked_documents, 1):\n",
-    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
-    "    try:\n",
-    "        entities = entity_extractor.extract_entities(chunk_text)\n",
-    "        all_entities.extend(entities)\n",
-    "    except Exception:\n",
-    "        continue\n",
-    "    \n",
-    "    if i % 20 == 0 or i == len(chunked_documents):\n",
-    "        remaining = len(chunked_documents) - i\n",
-    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found, {remaining} remaining)\")\n",
-    "\n",
-    "# Filter entities - spaCy returns standard types (PERSON, ORG, PRODUCT, etc.)\n",
-    "# Map to biomedical categories based on context\n",
-    "drugs = [e for e in all_entities if e.label == \"PRODUCT\" or (e.label == \"ORG\" and any(kw in e.text.lower() for kw in [\"drug\", \"pharma\", \"medication\"]))]\n",
-    "proteins = [e for e in all_entities if e.label == \"ORG\" or (e.label == \"PRODUCT\" and any(kw in e.text.lower() for kw in [\"protein\", \"enzyme\", \"receptor\", \"kinase\", \"target\"]))]\n",
-    "\n",
-    "print(f\"Extracted {len(drugs)} drugs and {len(proteins)} proteins\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Extracting Drug-Target Relationships\n",
-    "\n",
-    "Extract relationships between drugs and proteins to understand drug-target interactions.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.semantic_extract import RelationExtractor\n",
-    "\n",
-    "# Using spaCy dependency parsing (similar to NER cell)\n",
-    "relation_extractor = RelationExtractor(method=\"dependency\", model=\"en_core_web_sm\")\n",
-    "\n",
-    "all_relationships = []\n",
-    "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n",
-    "\n",
-    "for i, chunk in enumerate(chunked_documents, 1):\n",
-    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
-    "    try:\n",
-    "        relationships = relation_extractor.extract_relations(\n",
-    "            chunk_text,\n",
-    "            entities=all_entities,\n",
-    "            relation_types=[\"targets\", \"inhibits\", \"activates\", \"binds_to\", \"interacts_with\"]\n",
-    "        )\n",
-    "        all_relationships.extend(relationships)\n",
-    "    except Exception:\n",
-    "        continue\n",
-    "    \n",
-    "    if i % 20 == 0 or i == len(chunked_documents):\n",
-    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
-    "\n",
-    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Resolving Duplicate Entities\n",
-    "\n",
-    "Detect and merge duplicate entities to ensure data quality and consistency.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Conflict Detection and Resolution\n",
-    "\n",
-    "Detect and resolve conflicts in drug-target relationships from multiple research sources.\n",
-    "\n",
-    "- **Detection Method**: Relationship conflict detection identifies discrepancies in drug-target interactions across sources\n",
-    "- **Resolution Strategy**: Credibility-weighted resolution prioritizes higher-credibility sources (e.g., Nature journals over arXiv preprints)\n",
-    "- **Use Case**: Handles conflicting information when multiple sources report different drug-target relationships\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
-    "\n",
-    "detector = ConflictDetector()\n",
-    "resolver = ConflictResolver(default_strategy=\"credibility_weighted\")\n",
-    "\n",
-    "# Convert to dict format for conflict detection\n",
-    "entities = [\n",
-    "    {\n",
-    "        \"id\": ent.text if hasattr(ent, 'text') else str(ent),\n",
-    "        \"name\": ent.text if hasattr(ent, 'text') else str(ent),\n",
-    "        \"type\": ent.label if hasattr(ent, 'label') else \"ENTITY\",\n",
-    "        \"confidence\": getattr(ent, 'confidence', 1.0),\n",
-    "        \"source\": ent.metadata.get(\"source\", \"unknown\") if hasattr(ent, 'metadata') and ent.metadata else \"unknown\"\n",
-    "    }\n",
-    "    for ent in all_entities if hasattr(ent, 'text') or hasattr(ent, 'label')\n",
-    "]\n",
-    "\n",
-    "relationships = [\n",
-    "    {\n",
-    "        \"id\": f\"{rel.subject.text}_{rel.object.text}_{rel.predicate}\",\n",
-    "        \"source_id\": rel.subject.text,\n",
-    "        \"target_id\": rel.object.text,\n",
-    "        \"type\": rel.predicate,\n",
-    "        \"confidence\": getattr(rel, 'confidence', 1.0),\n",
-    "        \"source\": rel.metadata.get(\"source\", \"unknown\") if hasattr(rel, 'metadata') and rel.metadata else \"unknown\"\n",
-    "    }\n",
-    "    for rel in all_relationships if hasattr(rel, 'subject')\n",
-    "]\n",
-    "\n",
-    "# Detect and resolve conflicts\n",
-    "print(f\"Detecting conflicts in {len(entities)} entities, {len(relationships)} relationships...\")\n",
-    "entity_conflicts = detector.detect_conflicts(entities)\n",
-    "relationship_conflicts = detector.detect_relationship_conflicts(relationships)\n",
-    "print(f\"Detected {len(entity_conflicts)} entity conflicts, {len(relationship_conflicts)} relationship conflicts\")\n",
-    "\n",
-    "# Resolve conflicts\n",
-    "if entity_conflicts:\n",
-    "    resolver.resolve_conflicts(entity_conflicts, strategy=\"credibility_weighted\")\n",
-    "    print(f\"Resolved {len(entity_conflicts)} entity conflicts\")\n",
-    "\n",
-    "if relationship_conflicts:\n",
-    "    resolver.resolve_conflicts(relationship_conflicts, strategy=\"credibility_weighted\")\n",
-    "    print(f\"Resolved {len(relationship_conflicts)} relationship conflicts\")\n",
-    "\n",
-    "# GraphBuilder will use resolve_conflicts=True to apply resolutions automatically\n",
-    "print(\"Conflicts resolved. GraphBuilder will use cleaned data.\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Generating Vector Embeddings\n",
-    "\n",
-    "Generate embeddings for drugs and proteins to enable similarity search.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.embeddings import EmbeddingGenerator\n",
-    "from semantica.vector_store import VectorStore\n",
-    "\n",
-    "embedding_gen = EmbeddingGenerator(\n",
-    "    provider=\"sentence_transformers\",\n",
-    "    model=EMBEDDING_MODEL\n",
-    ")\n",
-    "\n",
-    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
-    "\n",
-    "print(f\"Generating embeddings for {len(drugs)} drugs and {len(proteins)} proteins...\")\n",
-    "drug_texts = [d.text for d in drugs]\n",
-    "drug_embeddings = embedding_gen.generate_embeddings(drug_texts)\n",
-    "\n",
-    "protein_texts = [p.text for p in proteins]\n",
-    "protein_embeddings = embedding_gen.generate_embeddings(protein_texts)\n",
-    "\n",
-    "print(f\"Generated {len(drug_embeddings)} drug embeddings and {len(protein_embeddings)} protein embeddings\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Populating Vector Database\n",
-    "\n",
-    "Store drug and protein embeddings in the vector database with metadata for efficient similarity search.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "print(f\"Storing {len(drug_embeddings)} drug vectors and {len(protein_embeddings)} protein vectors...\")\n",
-    "drug_ids = vector_store.store_vectors(\n",
-    "    vectors=drug_embeddings,\n",
-    "    metadata=[{\"type\": \"drug\", \"name\": d.text, \"label\": d.label} for d in drugs]\n",
-    ")\n",
-    "\n",
-    "protein_ids = vector_store.store_vectors(\n",
-    "    vectors=protein_embeddings,\n",
-    "    metadata=[{\"type\": \"protein\", \"name\": p.text, \"label\": p.label} for p in proteins]\n",
-    ")\n",
-    "\n",
-    "print(f\"Stored {len(drug_ids)} drug vectors and {len(protein_ids)} protein vectors\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Building Drug-Target Knowledge Graph\n",
-    "\n",
-    "Construct a knowledge graph from extracted entities and relationships to enable graph-based reasoning.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.kg import GraphBuilder\n",
-    "\n",
-    "graph_builder = GraphBuilder()\n",
-    "\n",
-    "print(f\"Building graph from {len(all_entities)} entities, {len(all_relationships)} relationships...\")\n",
-    "kg = graph_builder.build({\n",
-    "    \"entities\": all_entities,\n",
-    "    \"relationships\": all_relationships\n",
-    "})\n",
-    "\n",
-    "entities_count = len(kg.get('entities', []))\n",
-    "relationships_count = len(kg.get('relationships', []))\n",
-    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Finding Similar Drugs via Vector Search\n",
-    "\n",
-    "Use vector similarity search to find drugs similar to a query drug based on their embeddings.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "query_drug = \"Aspirin\"\n",
-    "query_embedding = embedding_gen.generate_embeddings([query_drug])[0]\n",
-    "similar_drugs = vector_store.search_vectors(query_embedding, k=5)\n",
-    "\n",
-    "print(f\"Drugs similar to '{query_drug}':\")\n",
-    "for i, result in enumerate(similar_drugs, 1):\n",
-    "    metadata = result.get('metadata', {})\n",
-    "    name = metadata.get('name', 'Unknown') if metadata else 'Unknown'\n",
-    "    score = result.get('score', 0.0)\n",
-    "    print(f\"{i}. {name} (similarity: {score:.3f})\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## GraphRAG: Hybrid Vector + Graph Retrieval\n",
-    "\n",
-    "Use GraphRAG to combine vector similarity search with knowledge graph traversal for enhanced retrieval and reasoning.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.context import AgentContext, ContextRetriever\n",
-    "\n",
-    "# Option 1: Use AgentContext (high-level, recommended)\n",
-    "context = AgentContext(\n",
-    "    vector_store=vector_store, \n",
-    "    knowledge_graph=kg,\n",
-    "    hybrid_alpha=0.6,\n",
-    "    max_expansion_hops=2\n",
-    ")\n",
-    "\n",
-    "# Option 2: Use ContextRetriever directly (more control)\n",
-    "retriever = ContextRetriever(\n",
-    "    vector_store=vector_store,\n",
-    "    knowledge_graph=kg,\n",
-    "    hybrid_alpha=0.6,\n",
-    "    max_expansion_hops=2\n",
-    ")\n",
-    "\n",
-    "# GraphRAG query using AgentContext\n",
-    "query = \"What drugs target COX enzymes?\"\n",
-    "results = context.retrieve(\n",
-    "    query,\n",
-    "    max_results=10,\n",
-    "    use_graph=True,\n",
-    "    expand_graph=True,\n",
-    "    include_entities=True,\n",
-    "    include_relationships=True\n",
-    ")\n",
-    "\n",
-    "\n",
-    "print(f\"Query: '{query}'\")\n",
-    "print(f\"Retrieved {len(results)} results:\\n\")\n",
-    "for i, result in enumerate(results[:5], 1):\n",
-    "    print(f\"{i}. Score: {result.get('score', 0):.3f}\")\n",
-    "    if result.get('content'):\n",
-    "        print(f\"   {result['content'][:250]}\")\n",
-    "    if result.get('related_entities'):\n",
-    "        entities = result['related_entities']\n",
-    "        names = [e.get('name', e.get('id', '')) for e in entities[:3]]\n",
-    "        print(f\"   Entities: {', '.join(names)}\" + (f\" (+{len(entities)-3})\" if len(entities) > 3 else \"\"))\n",
-    "    if result.get('related_relationships'):\n",
-    "        print(f\"   Relationships: {len(result['related_relationships'])}\")\n",
-    "    print()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Visualizing the Knowledge Graph\n",
-    "\n",
-    "Generate an interactive visualization of the drug-target knowledge graph.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.visualization import KGVisualizer\n",
-    "\n",
-    "# Display interactive Plotly graph directly in notebook\n",
-    "visualizer = KGVisualizer(layout=\"force\", node_size=20)\n",
-    "fig = visualizer.visualize_network(kg, output=\"interactive\")\n",
-    "\n",
-    "# Display the figure (Plotly will show it automatically in notebook)\n",
-    "fig.show() if fig else None"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Exporting Results\n",
-    "\n",
-    "Export the knowledge graph to various formats for further analysis or integration with other tools.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.export import GraphExporter\n",
-    "\n",
-    "exporter = GraphExporter()\n",
-    "exporter.export(kg, output_path=\"drug_target_kg.json\", format=\"json\")\n",
-    "exporter.export(kg, output_path=\"drug_target_kg.graphml\", format=\"graphml\")\n",
-    "\n",
-    "print(\"Exported knowledge graph to JSON and GraphML formats\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb

@@ -1,719 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)\n",
-    "\n",
-    "# Genomic Variant Analysis - Graph Analytics & Pathway Analysis\n",
-    "\n",
-    "## Overview\n",
-    "\n",
-    "This notebook demonstrates **genomic variant analysis** using Semantica's modular architecture with focus on **graph analytics**, **pathway analysis**, and **temporal knowledge graphs**. The pipeline analyzes genomic data to extract variant entities, build temporal genomic knowledge graphs, and analyze disease associations through reasoning.\n",
-    "\n",
-    "### Key Features\n",
-    "\n",
-    "- **Graph Analytics Focus**: Emphasizes graph reasoning, centrality measures, and pathway analysis\n",
-    "- **Temporal Analysis**: Builds temporal genomic knowledge graphs to track variant evolution\n",
-    "- **Disease Association**: Analyzes relationships between variants, genes, and diseases\n",
-    "- **Pathway Analysis**: Uses graph traversal to identify biological pathways\n",
-    "- **Impact Prediction**: Predicts variant impact using graph-based reasoning\n",
-    "\n",
-    "### What You'll Learn\n",
-    "\n",
-    "- How to use Semantica modules directly for genomic analysis\n",
-    "- How to ingest genomic data from multiple sources\n",
-    "- How to extract variant, gene, and disease entities\n",
-    "- How to build temporal knowledge graphs\n",
-    "- How to perform graph analytics (centrality, communities)\n",
-    "- How to use temporal queries for variant evolution\n",
-    "- How to analyze pathways using reasoning\n",
-    "- How to visualize and export genomic knowledge graphs\n",
-    "\n",
-    "### Pipeline Flow\n",
-    "\n",
-    "```mermaid\n",
-    "graph LR\n",
-    "    A[Data Ingestion] --> B[Text Processing]\n",
-    "    B --> C[Entity Extraction]\n",
-    "    C --> D[Relationship Extraction]\n",
-    "    D --> E[Deduplication]\n",
-    "    E --> F[Temporal KG]\n",
-    "    F --> G[Graph Analytics]\n",
-    "    F --> H[Temporal Queries]\n",
-    "    G --> I[Pathway Analysis]\n",
-    "    H --> I\n",
-    "    I --> J[Disease Associations]\n",
-    "    J --> K[Visualization]\n",
-    "```\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Installation\n",
-    "\n",
-    "Install Semantica and required dependencies:\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install -qU semantica networkx matplotlib plotly pandas groq sentence-transformers\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Configuration & Setup\n",
-    "\n",
-    "Set up environment variables and configuration constants.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "CHUNK_SIZE = 1000\n",
-    "CHUNK_OVERLAP = 200\n",
-    "TEMPORAL_GRANULARITY = \"day\"\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Ingesting Genomic Data from Multiple Sources\n",
-    "\n",
-    "Ingest data from comprehensive genomic sources including PubMed RSS feeds, preprint servers, and journal feeds.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.ingest import FeedIngestor, FileIngestor\n",
-    "import os\n",
-    "from contextlib import redirect_stderr\n",
-    "from io import StringIO\n",
-    "\n",
-    "os.makedirs(\"data\", exist_ok=True)\n",
-    "\n",
-    "feed_sources = [\n",
-    "    # PubMed RSS Feeds (simplified, working format)\n",
-    "    (\"PubMed - Genetics\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=genetics&limit=10\"),\n",
-    "    (\"PubMed - Genomics\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=genomics&limit=10\"),\n",
-    "    (\"PubMed - Variant Analysis\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=variant+analysis&limit=10\"),\n",
-    "    (\"PubMed - GWAS\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=GWAS&limit=10\"),\n",
-    "    (\"PubMed - Genomic Medicine\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=genomic+medicine&limit=10\"),\n",
-    "    (\"PubMed - Precision Medicine\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=precision+medicine&limit=10\"),\n",
-    "    (\"PubMed - Pharmacogenomics\", \"https://pubmed.ncbi.nlm.nih.gov/rss/search/1?term=pharmacogenomics&limit=10\"),\n",
-    "    \n",
-    "    # Nature Feeds (working format)\n",
-    "    (\"Nature Genetics\", \"https://www.nature.com/subjects/genetics.rss\"),\n",
-    "    (\"Nature - Genomics\", \"https://www.nature.com/subjects/genomics.rss\"),\n",
-    "    \n",
-    "    # PLOS Journals (working Atom feeds)\n",
-    "    (\"PLOS Genetics\", \"https://journals.plos.org/plosgenetics/feed/atom\"),\n",
-    "    (\"PLOS ONE - Genetics\", \"https://journals.plos.org/plosone/feed/atom\"),\n",
-    "    \n",
-    "    # Other working feeds\n",
-    "    (\"Genome Research\", \"https://genome.cshlp.org/rss/current.xml\"),\n",
-    "]\n",
-    "\n",
-    "feed_ingestor = FeedIngestor()\n",
-    "all_documents = []\n",
-    "\n",
-    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
-    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
-    "    try:\n",
-    "        with redirect_stderr(StringIO()):\n",
-    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
-    "        \n",
-    "        feed_count = 0\n",
-    "        for item in feed_data.items:\n",
-    "            if not item.content:\n",
-    "                item.content = item.description or item.title or \"\"\n",
-    "            if item.content:\n",
-    "                if not hasattr(item, 'metadata'):\n",
-    "                    item.metadata = {}\n",
-    "                item.metadata['source'] = feed_name\n",
-    "                all_documents.append(item)\n",
-    "                feed_count += 1\n",
-    "        \n",
-    "        if feed_count > 0:\n",
-    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
-    "    except Exception as e:\n",
-    "        print(f\"  [{i}/{len(feed_sources)}] {feed_name}: Failed\")\n",
-    "        continue\n",
-    "\n",
-    "# Always include fallback variant data for demonstration\n",
-    "variant_data = \"\"\"\n",
-    "Variant rs699 is located in the AGT gene and associated with hypertension.\n",
-    "Variant rs7412 in APOE gene is linked to Alzheimer's disease risk.\n",
-    "BRCA1 variant c.5266dupC increases breast cancer susceptibility.\n",
-    "CFTR variant F508del causes cystic fibrosis.\n",
-    "Variant rs1800566 in NAT2 gene affects drug metabolism.\n",
-    "Variant rs1042713 in ADRB2 gene is associated with asthma response.\n",
-    "TP53 variant R273H is linked to multiple cancer types.\n",
-    "Variant rs1799853 in CYP2C9 gene affects warfarin metabolism.\n",
-    "Variant rs1057910 in CYP2C9 affects phenytoin metabolism.\n",
-    "Variant rs9923231 in VKORC1 gene influences warfarin dosing.\n",
-    "\"\"\"\n",
-    "\n",
-    "with open(\"data/variants.txt\", \"w\") as f:\n",
-    "    f.write(variant_data)\n",
-    "\n",
-    "file_ingestor = FileIngestor()\n",
-    "fallback_docs = file_ingestor.ingest(\"data/variants.txt\")\n",
-    "all_documents.extend(fallback_docs)\n",
-    "\n",
-    "documents = all_documents\n",
-    "print(f\"\\nTotal ingested: {len(documents)} documents\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Normalizing and Chunking Genomic Documents\n",
-    "\n",
-    "Clean and normalize text, then split into chunks using entity-aware chunking to preserve variant/gene entity boundaries.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.normalize import TextNormalizer\n",
-    "from semantica.split import TextSplitter\n",
-    "\n",
-    "normalizer = TextNormalizer()\n",
-    "splitter = TextSplitter(\n",
-    "    method=\"entity_aware\",\n",
-    "    ner_method=\"spacy\",\n",
-    "    chunk_size=CHUNK_SIZE,\n",
-    "    chunk_overlap=CHUNK_OVERLAP\n",
-    ")\n",
-    "\n",
-    "print(f\"Normalizing {len(documents)} documents...\")\n",
-    "normalized_documents = []\n",
-    "for i, doc in enumerate(documents, 1):\n",
-    "    normalized_text = normalizer.normalize(\n",
-    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
-    "        clean_html=True,\n",
-    "        normalize_entities=True,\n",
-    "        remove_extra_whitespace=True,\n",
-    "        lowercase=False\n",
-    "    )\n",
-    "    normalized_documents.append(normalized_text)\n",
-    "    if i % 50 == 0 or i == len(documents):\n",
-    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
-    "\n",
-    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
-    "chunked_documents = []\n",
-    "for i, doc_text in enumerate(normalized_documents, 1):\n",
-    "    try:\n",
-    "        with redirect_stderr(StringIO()):\n",
-    "            chunks = splitter.split(doc_text)\n",
-    "        chunked_documents.extend(chunks)\n",
-    "    except Exception:\n",
-    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
-    "        chunks = simple_splitter.split(doc_text)\n",
-    "        chunked_documents.extend(chunks)\n",
-    "    if i % 50 == 0 or i == len(normalized_documents):\n",
-    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
-    "\n",
-    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.semantic_extract import NERExtractor\n",
-    "\n",
-    "# Using spaCy ML method (similar to Drug Discovery Pipeline)\n",
-    "entity_extractor = NERExtractor(method=\"ml\", model=\"en_core_web_sm\")\n",
-    "\n",
-    "all_entities = []\n",
-    "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n",
-    "\n",
-    "for i, chunk in enumerate(chunked_documents, 1):\n",
-    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
-    "    try:\n",
-    "        entities = entity_extractor.extract_entities(chunk_text)\n",
-    "        all_entities.extend(entities)\n",
-    "    except Exception:\n",
-    "        continue\n",
-    "    \n",
-    "    if i % 20 == 0 or i == len(chunked_documents):\n",
-    "        remaining = len(chunked_documents) - i\n",
-    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found, {remaining} remaining)\")\n",
-    "\n",
-    "# Filter entities - spaCy returns standard types, map to genomic categories\n",
-    "# Look for variant patterns (rs numbers, c. notation, etc.)\n",
-    "variants = [\n",
-    "    e for e in all_entities \n",
-    "    if (e.text.startswith(\"rs\") or \n",
-    "        \"c.\" in e.text.lower() or \n",
-    "        \"variant\" in e.text.lower() or\n",
-    "        e.label == \"PRODUCT\" and any(kw in e.text.lower() for kw in [\"rs\", \"variant\", \"mutation\"]))\n",
-    "]\n",
-    "\n",
-    "# Look for gene patterns (gene names, protein names)\n",
-    "genes = [\n",
-    "    e for e in all_entities \n",
-    "    if (e.label == \"ORG\" or \n",
-    "        e.label == \"PRODUCT\" or\n",
-    "        any(kw in e.text.lower() for kw in [\"gene\", \"protein\", \"enzyme\", \"receptor\", \"kinase\"]))\n",
-    "]\n",
-    "\n",
-    "# Look for disease patterns\n",
-    "diseases = [\n",
-    "    e for e in all_entities \n",
-    "    if (e.label == \"ORG\" or\n",
-    "        any(kw in e.text.lower() for kw in [\"disease\", \"syndrome\", \"disorder\", \"cancer\", \"hypertension\", \"alzheimer\"]))\n",
-    "]\n",
-    "\n",
-    "print(f\"Extracted {len(variants)} variants, {len(genes)} genes, {len(diseases)} diseases\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Extracting Genomic Relationships\n",
-    "\n",
-    "Extract relationships between variants, genes, and diseases to understand genomic associations.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.semantic_extract import RelationExtractor\n",
-    "\n",
-    "# Using spaCy dependency parsing (similar to Drug Discovery Pipeline)\n",
-    "relation_extractor = RelationExtractor(method=\"dependency\", model=\"en_core_web_sm\")\n",
-    "\n",
-    "all_relationships = []\n",
-    "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n",
-    "\n",
-    "for i, chunk in enumerate(chunked_documents, 1):\n",
-    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
-    "    try:\n",
-    "        relationships = relation_extractor.extract_relations(\n",
-    "            chunk_text,\n",
-    "            entities=all_entities,\n",
-    "            relation_types=[\"associated_with\", \"located_in\", \"causes\", \"increases_risk\", \"affects\", \"linked_to\"]\n",
-    "        )\n",
-    "        all_relationships.extend(relationships)\n",
-    "    except Exception:\n",
-    "        continue\n",
-    "    \n",
-    "    if i % 20 == 0 or i == len(chunked_documents):\n",
-    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
-    "\n",
-    "print(f\"Extracted {len(all_relationships)} relationships\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Building Temporal Genomic Knowledge Graph\n",
-    "\n",
-    "Construct a temporal knowledge graph from extracted entities and relationships to enable time-aware analysis and variant evolution tracking.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Conflict Detection and Resolution\n",
-    "\n",
-    "Detect and resolve conflicts in genomic variant data from multiple research sources.\n",
-    "\n",
-    "- **Detection Method**: Entity and relationship conflict detection identifies discrepancies in variant-gene-disease associations across sources\n",
-    "- **Resolution Strategy**: Credibility-weighted resolution prioritizes higher-credibility sources (e.g., Nature Genetics over preprints)\n",
-    "- **Use Case**: Handles conflicting information when multiple sources report different variant associations, disease risks, or gene locations\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
-    "\n",
-    "# Initialize with best strategies for genomic analysis\n",
-    "detector = ConflictDetector()\n",
-    "resolver = ConflictResolver(default_strategy=\"credibility_weighted\")\n",
-    "\n",
-    "# Convert entities to format expected by detector\n",
-    "entities = [\n",
-    "    {\n",
-    "        \"id\": ent.text if hasattr(ent, 'text') else str(ent),\n",
-    "        \"name\": ent.text if hasattr(ent, 'text') else str(ent),\n",
-    "        \"type\": ent.label if hasattr(ent, 'label') else \"ENTITY\",\n",
-    "        \"confidence\": getattr(ent, 'confidence', 1.0),\n",
-    "        \"source\": ent.metadata.get(\"source\", \"unknown\") if hasattr(ent, 'metadata') and ent.metadata else \"unknown\"\n",
-    "    }\n",
-    "    for ent in all_entities if hasattr(ent, 'text') or hasattr(ent, 'label')\n",
-    "]\n",
-    "\n",
-    "# Convert relationships to format expected by detector\n",
-    "relationships = [\n",
-    "    {\n",
-    "        \"id\": f\"{rel.subject.text}_{rel.object.text}_{rel.predicate}\" if hasattr(rel, 'subject') else f\"{i}\",\n",
-    "        \"source_id\": rel.subject.text if hasattr(rel, 'subject') else str(rel.get(\"source\", \"\")),\n",
-    "        \"target_id\": rel.object.text if hasattr(rel, 'object') else str(rel.get(\"target\", \"\")),\n",
-    "        \"type\": rel.predicate if hasattr(rel, 'predicate') else rel.get(\"type\", \"related_to\"),\n",
-    "        \"confidence\": getattr(rel, 'confidence', 1.0),\n",
-    "        \"properties\": rel.metadata if hasattr(rel, 'metadata') else {},\n",
-    "        \"source\": rel.metadata.get(\"source\", \"unknown\") if hasattr(rel, 'metadata') and rel.metadata else \"unknown\"\n",
-    "    }\n",
-    "    for i, rel in enumerate(all_relationships) if hasattr(rel, 'subject') or isinstance(rel, dict)\n",
-    "]\n",
-    "\n",
-    "# Detect both entity and relationship conflicts\n",
-    "print(f\"Detecting conflicts in {len(entities)} entities, {len(relationships)} relationships...\")\n",
-    "\n",
-    "# Detect entity conflicts\n",
-    "entity_conflicts = detector.detect_conflicts(entities)\n",
-    "print(f\"Detected {len(entity_conflicts)} entity conflicts\")\n",
-    "\n",
-    "# Detect relationship conflicts\n",
-    "relationship_conflicts = detector.detect_relationship_conflicts(relationships)\n",
-    "print(f\"Detected {len(relationship_conflicts)} relationship conflicts\")\n",
-    "\n",
-    "# Resolve entity conflicts\n",
-    "if entity_conflicts:\n",
-    "    resolver.resolve_conflicts(entity_conflicts, strategy=\"credibility_weighted\")\n",
-    "    print(f\"Resolved {len(entity_conflicts)} entity conflicts\")\n",
-    "\n",
-    "# Resolve relationship conflicts\n",
-    "if relationship_conflicts:\n",
-    "    resolver.resolve_conflicts(relationship_conflicts, strategy=\"credibility_weighted\")\n",
-    "    print(f\"Resolved {len(relationship_conflicts)} relationship conflicts\")\n",
-    "\n",
-    "# GraphBuilder will use resolve_conflicts=True to apply resolutions automatically\n",
-    "if entity_conflicts or relationship_conflicts:\n",
-    "    print(\"Conflicts resolved. GraphBuilder will use cleaned data.\")\n",
-    "else:\n",
-    "    print(\"No conflicts detected. Data is clean.\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.kg import GraphBuilder\n",
-    "\n",
-    "# Conflicts already detected and resolved in previous cell\n",
-    "# Enable temporal features for genomic variant tracking\n",
-    "graph_builder = GraphBuilder(\n",
-    "    resolve_conflicts=False,  # Conflicts already handled\n",
-    "    enable_temporal=True,\n",
-    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
-    ")\n",
-    "\n",
-    "print(f\"Building temporal knowledge graph from {len(all_entities)} entities, {len(all_relationships)} relationships...\")\n",
-    "kg = graph_builder.build({\n",
-    "    \"entities\": all_entities,\n",
-    "    \"relationships\": all_relationships\n",
-    "})\n",
-    "\n",
-    "entities_count = len(kg.get('entities', []))\n",
-    "relationships_count = len(kg.get('relationships', []))\n",
-    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Analyzing Graph Structure\n",
-    "\n",
-    "Perform comprehensive graph analytics including centrality measures, community detection, and connectivity analysis.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.kg import GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
-    "\n",
-    "graph_analyzer = GraphAnalyzer()\n",
-    "centrality_calc = CentralityCalculator()\n",
-    "community_detector = CommunityDetector()\n",
-    "\n",
-    "analysis = graph_analyzer.analyze_graph(kg)\n",
-    "\n",
-    "degree_centrality = centrality_calc.calculate_degree_centrality(kg)\n",
-    "betweenness_centrality = centrality_calc.calculate_betweenness_centrality(kg)\n",
-    "closeness_centrality = centrality_calc.calculate_closeness_centrality(kg)\n",
-    "\n",
-    "communities = community_detector.detect_communities(kg, method=\"louvain\")\n",
-    "connectivity = graph_analyzer.analyze_connectivity(kg)\n",
-    "\n",
-    "print(f\"Graph analytics:\")\n",
-    "print(f\"  - Communities: {len(communities)}\")\n",
-    "print(f\"  - Connected components: {len(connectivity.get('components', []))}\")\n",
-    "print(f\"  - Graph density: {analysis.get('density', 0):.3f}\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Temporal Graph Queries\n",
-    "\n",
-    "Query the temporal knowledge graph at specific time points, analyze temporal evolution, and detect temporal patterns.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.kg import TemporalGraphQuery\n",
-    "\n",
-    "temporal_query = TemporalGraphQuery(\n",
-    "    enable_temporal_reasoning=True,\n",
-    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
-    ")\n",
-    "\n",
-    "# Query variants at specific time point\n",
-    "query_results = temporal_query.query_at_time(\n",
-    "    kg,\n",
-    "    query=\"Variant\",\n",
-    "    at_time=\"2024-01-01\"\n",
-    ")\n",
-    "\n",
-    "# Analyze graph evolution\n",
-    "evolution = temporal_query.analyze_evolution(kg)\n",
-    "\n",
-    "# Detect temporal patterns\n",
-    "pattern_results = temporal_query.query_temporal_pattern(\n",
-    "    kg,\n",
-    "    pattern=\"sequence\"\n",
-    ")\n",
-    "\n",
-    "print(f\"Temporal query: {query_results.get('num_relationships', 0)} relationships valid at query time\")\n",
-    "print(f\"Evolution analysis: {evolution.get('num_relationships', 0)} relationships tracked\")\n",
-    "print(f\"Temporal patterns detected: {pattern_results.get('num_patterns', 0)}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Pathway Analysis & Reasoning\n",
-    "\n",
-    "Use graph reasoning to find pathways between variants and diseases, and infer biological pathways through logical reasoning.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.reasoning import Reasoner\n",
-    "from semantica.kg import GraphAnalyzer\n",
-    "\n",
-    "reasoner = Reasoner()\n",
-    "graph_analyzer = GraphAnalyzer()\n",
-    "\n",
-    "# Find entities by type\n",
-    "variants = [e for e in kg.get('entities', []) if e.get('type') == 'Variant']\n",
-    "diseases = [e for e in kg.get('entities', []) if e.get('type') == 'Disease']\n",
-    "\n",
-    "print(f\"Found {len(variants)} variants and {len(diseases)} diseases\")\n",
-    "\n",
-    "# Find pathways\n",
-    "pathways = []\n",
-    "for variant in variants[:5]:\n",
-    "    variant_id = variant.get('id') or variant.get('name')\n",
-    "    for disease in diseases[:3]:\n",
-    "        disease_id = disease.get('id') or disease.get('name')\n",
-    "        path = graph_analyzer.connectivity_analyzer.calculate_shortest_paths(\n",
-    "            kg, source=variant_id, target=disease_id\n",
-    "        )\n",
-    "        if path.get('exists'):\n",
-    "            pathways.append({\n",
-    "                'variant': variant_id,\n",
-    "                'disease': disease_id,\n",
-    "                'distance': path.get('distance', -1)\n",
-    "            })\n",
-    "\n",
-    "# Add rule and infer facts\n",
-    "reasoner.add_rule(\"IF Variant associated_with Gene AND Gene causes Disease THEN Variant increases_risk Disease\")\n",
-    "inferred_facts = reasoner.infer_facts(kg)\n",
-    "\n",
-    "print(f\"Pathway analysis: {len(pathways)} variant-disease pathways found\")\n",
-    "print(f\"Inferred facts: {len(inferred_facts)}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Analyzing Disease Associations\n",
-    "\n",
-    "Use graph traversal to find variant-disease associations and calculate association scores.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.kg import GraphAnalyzer\n",
-    "\n",
-    "graph_analyzer = GraphAnalyzer()\n",
-    "\n",
-    "# Find entities by type\n",
-    "variants = [e for e in kg.get('entities', []) if e.get('type') == 'Variant']\n",
-    "diseases = [e for e in kg.get('entities', []) if e.get('type') == 'Disease']\n",
-    "\n",
-    "# Find disease associations\n",
-    "disease_associations = []\n",
-    "for variant in variants[:10]:\n",
-    "    variant_id = variant.get('name') or variant.get('id')\n",
-    "    if not variant_id:\n",
-    "        continue\n",
-    "    for disease in diseases[:5]:\n",
-    "        disease_id = disease.get('name') or disease.get('id')\n",
-    "        if not disease_id:\n",
-    "            continue\n",
-    "        path = graph_analyzer.connectivity_analyzer.calculate_shortest_paths(\n",
-    "            kg, source=variant_id, target=disease_id\n",
-    "        )\n",
-    "        if path.get('exists') and path.get('distance', -1) <= 2:\n",
-    "            disease_associations.append({\n",
-    "                'variant': variant_id,\n",
-    "                'disease': disease_id,\n",
-    "                'path_length': path.get('distance', -1),\n",
-    "                'confidence': variant.get('confidence', 1.0)\n",
-    "            })\n",
-    "\n",
-    "disease_associations.sort(key=lambda x: x['confidence'], reverse=True)\n",
-    "\n",
-    "print(f\"Top disease associations:\")\n",
-    "for i, assoc in enumerate(disease_associations[:5], 1):\n",
-    "    print(f\"{i}. {assoc['variant']} -> {assoc['disease']} (path length: {assoc['path_length']}, confidence: {assoc['confidence']:.3f})\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Visualizing the Temporal Knowledge Graph\n",
-    "\n",
-    "Generate an interactive visualization of the temporal genomic knowledge graph.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.visualization import TemporalVisualizer\n",
-    "\n",
-    "# Visualize temporal dashboard\n",
-    "temporal_viz = TemporalVisualizer()\n",
-    "fig = temporal_viz.visualize_temporal_dashboard(\n",
-    "    kg,\n",
-    "    output=\"interactive\"\n",
-    ")\n",
-    "\n",
-    "# Display the figure\n",
-    "fig.show() if fig else None\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Exporting Results\n",
-    "\n",
-    "Export the temporal knowledge graph to various formats for further analysis or integration with other tools.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from semantica.export import GraphExporter\n",
-    "\n",
-    "exporter = GraphExporter()\n",
-    "exporter.export(kg, output_path=\"genomic_variant_kg.json\", format=\"json\")\n",
-    "exporter.export(kg, output_path=\"genomic_variant_kg.graphml\", format=\"graphml\")\n",
-    "\n",
-    "print(\"Exported knowledge graph to JSON and GraphML formats\")\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}