diff --git a/cookbook/introduction/25_Seed_Data.ipynb b/cookbook/introduction/25_Seed_Data.ipynb new file mode 100644 index 00000000..bf962e5d --- /dev/null +++ b/cookbook/introduction/25_Seed_Data.ipynb @@ -0,0 +1,314 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "6eb4dfba", + "metadata": {}, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/25_Seed_Data.ipynb)\n", + "\n", + "# Seed Data — Practical Guide\n", + "\n", + "The `seed` module bootstraps a knowledge graph from **trusted, pre-known data** (CSV/JSON/database/API sources) before any extraction runs. This gives extraction a foundation to link against instead of starting from an empty graph.\n", + "\n", + "Key pieces:\n", + "\n", + "- **`SeedDataManager`** — registers data sources and builds foundation graphs\n", + "- **`create_foundation_graph()`** — turns registered sources into `entities` + `relationships` + `metadata`\n", + "- **`validate_quality()`** — checks a foundation graph before you commit it\n", + "\n", + "All examples below were executed against `semantica/seed/seed_manager.py`." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "32f80cc6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-26T18:51:18.716466Z", + "iopub.status.busy": "2026-08-26T18:51:18.716264Z", + "iopub.status.idle": "2026-08-26T18:51:20.533828Z", + "shell.execute_reply": "2026-08-26T18:51:20.531402Z" + } + }, + "outputs": [], + "source": [ + "!pip install -q semantica" + ] + }, + { + "cell_type": "markdown", + "id": "75136e5f", + "metadata": {}, + "source": [ + "## 1) Prepare a seed CSV and register the source\n", + "\n", + "`register_source(name, format, location, entity_type=...)` records where trusted data lives. `verified=True` (the default) marks the source as pre-validated." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "a8089e1f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-26T18:51:20.538772Z", + "iopub.status.busy": "2026-08-26T18:51:20.538323Z", + "iopub.status.idle": "2026-08-26T18:51:20.675403Z", + "shell.execute_reply": "2026-08-26T18:51:20.674060Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import csv\n", + "import tempfile\n", + "from pathlib import Path\n", + "from semantica.seed import SeedDataManager\n", + "\n", + "# Write the sample CSV into a session-scoped temp directory so we never\n", + "# clobber a companies.csv that might exist in the user's working directory.\n", + "seed_csv = Path(tempfile.mkdtemp(prefix=\"semantica-seed-\")) / \"companies.csv\"\n", + "with open(seed_csv, \"w\", newline=\"\") as f:\n", + " writer = csv.DictWriter(f, fieldnames=[\"id\", \"name\", \"type\", \"industry\"])\n", + " writer.writeheader()\n", + " writer.writerow({\"id\": \"c1\", \"name\": \"Acme\", \"type\": \"Company\", \"industry\": \"robotics\"})\n", + " writer.writerow({\"id\": \"c2\", \"name\": \"Globex\", \"type\": \"Company\", \"industry\": \"energy\"})\n", + "\n", + "manager = SeedDataManager()\n", + "manager.register_source(\"companies\", format=\"csv\", location=str(seed_csv), entity_type=\"Company\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "e87221ba", + "metadata": {}, + "source": [ + "## 2) Load records from a registered source\n", + "\n", + "`load_source(name)` reads the source and enriches each record with `entity_type` and `source` provenance keys." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "f932e550", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-26T18:51:20.679424Z", + "iopub.status.busy": "2026-08-26T18:51:20.679156Z", + "iopub.status.idle": "2026-08-26T18:51:20.690659Z", + "shell.execute_reply": "2026-08-26T18:51:20.688812Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "

🧠 Semantica - 📊 Current Progress

StatusActionModuleSubmoduleProgressETARateTimeExtracted
✅Semantica is seeding🌱 seedSeedDataManager100.0%--0.00s-
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "🔄 Semantica is seeding: Loading seed data from CSV: /var/folders/7s/bvvstgs10y963tz6_4bbnklr0000gn/T/semantica-seed-eu9__ep1/companies.csv 🌱 seed SeedDataManager |░░░░░░░░░░░░░░░| 0.0% ETA: - Rate: - Time: 0.00s Extracted: -" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "loaded 2 records\n" + ] + }, + { + "data": { + "text/plain": [ + "{'id': 'c1',\n", + " 'name': 'Acme',\n", + " 'type': 'Company',\n", + " 'industry': 'robotics',\n", + " 'entity_type': 'Company',\n", + " 'source': 'companies'}" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "records = manager.load_source(\"companies\")\n", + "print(f\"loaded {len(records)} records\")\n", + "records[0]" + ] + }, + { + "cell_type": "markdown", + "id": "f2ebce64", + "metadata": {}, + "source": [ + "## 3) Build the foundation graph\n", + "\n", + "`create_foundation_graph()` converts every registered source into graph-ready entities and relationships. Entities carry `confidence: 1.0` — seed data is trusted by definition." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "09388c31", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-26T18:51:20.695259Z", + "iopub.status.busy": "2026-08-26T18:51:20.694928Z", + "iopub.status.idle": "2026-08-26T18:51:20.708595Z", + "shell.execute_reply": "2026-08-26T18:51:20.707072Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['entities', 'metadata', 'relationships']" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "foundation = manager.create_foundation_graph()\n", + "sorted(foundation.keys())" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "4610a59f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-26T18:51:20.713136Z", + "iopub.status.busy": "2026-08-26T18:51:20.712795Z", + "iopub.status.idle": "2026-08-26T18:51:20.718637Z", + "shell.execute_reply": "2026-08-26T18:51:20.716835Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'id': 'c1',\n", + " 'text': 'Acme',\n", + " 'type': 'Company',\n", + " 'confidence': 1.0,\n", + " 'metadata': {'industry': 'robotics', 'source': 'companies'}}" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "foundation[\"entities\"][0]" + ] + }, + { + "cell_type": "markdown", + "id": "f3a52dc7", + "metadata": {}, + "source": [ + "## 4) Validate quality before committing\n", + "\n", + "`validate_quality(foundation_graph)` returns `valid`, `errors`, `warnings`, and `metrics` so you can gate bad seed data before it pollutes the graph." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "4eb7e664", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-26T18:51:20.722674Z", + "iopub.status.busy": "2026-08-26T18:51:20.722118Z", + "iopub.status.idle": "2026-08-26T18:51:20.732003Z", + "shell.execute_reply": "2026-08-26T18:51:20.730170Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "quality = manager.validate_quality(foundation)\n", + "quality[\"valid\"]" + ] + }, + { + "cell_type": "markdown", + "id": "b534be89", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "| Task | API |\n", + "|---|---|\n", + "| Register a trusted source | `register_source(name, format, location, entity_type=...)` |\n", + "| Load records | `load_source(name)` — adds `entity_type` / `source` keys |\n", + "| Direct file load | `load_from_csv(path)` / `load_from_json(path)` |\n", + "| Build the graph | `create_foundation_graph()` → `entities` / `relationships` / `metadata` |\n", + "| Gate bad data | `validate_quality(graph)` → `valid` / `errors` / `warnings` / `metrics` |\n", + "\n", + "See also `semantica/seed/seed_usage.md` for `load_from_database`, `load_from_api`, and `integrate_with_extracted`." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}