{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n",
    "\n",
    "Semantica is a **semantic intelligence and knowledge engineering framework**. It helps you:\n",
    "\n",
    "- Build **knowledge graphs** from unstructured and semi-structured data\n",
    "- Create a unified **semantic layer** on top of diverse data sources\n",
    "- Power **GraphRAG**, AI agents, and multi-agent systems with structured knowledge\n",
    "- Incorporate **temporal and quality-aware reasoning** into your applications\n",
    "\n",
    "### Core Capabilities\n",
    "\n",
    "- **Universal ingestion**: Files, web, feeds, databases, repositories, streams\n",
    "- **Rich parsing**: PDFs, Office documents, HTML, JSON, CSV, images, code\n",
    "- **Normalization**: Cleaning, language detection, entity normalization, date/number standardization\n",
    "- **Semantic extraction**: Named entities, relationships, events, semantic networks\n",
    "- **Knowledge graph construction**: Property graphs from entities and relations\n",
    "- **Embeddings and vector search**: Text and graph embeddings, hybrid retrieval\n",
    "- **Reasoning and ontology**: Rule-based inference, ontology generation and validation\n",
    "- **Visualization and analytics**: Graph visualizations and quality metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Who Is Semantica For?\n",
    "\n",
    "- **AI/ML engineers** building GraphRAG systems, agents, and tools that need long-term memory\n",
    "- **Data engineers** orchestrating semantic enrichment pipelines over large, heterogeneous datasets\n",
    "- **Knowledge engineers and ontologists** designing and maintaining formal knowledge structures\n",
    "- **Researchers and analysts** creating domain knowledge graphs from documents and data feeds\n",
    "- **Product and platform teams** embedding semantic intelligence into applications and services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Architecture Overview\n",
    "\n",
    "Semantica is organized as three conceptual layers and multiple concrete modules.\n",
    "\n",
    "### Layers\n",
    "\n",
    "- **Input Layer**\n",
    "  - Connects to files, web pages, APIs, databases, email, feeds, repositories, and streams\n",
    "  - Normalizes these different sources into a unified internal representation\n",
    "\n",
    "- **Semantic Layer**\n",
    "  - Performs parsing, cleaning, semantic extraction, graph construction, embeddings, and reasoning\n",
    "  - This is where **unstructured data becomes structured knowledge**\n",
    "\n",
    "- **Output Layer**\n",
    "  - Exposes knowledge graphs, embeddings, ontologies, and analytics\n",
    "  - Integrates with vector stores, graph databases, and downstream applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🧩 Semantica Modules Reference\n",
    "\n",
    "Semantica is modular by design. Here is a comprehensive guide to all available modules, grouped by functionality.\n",
    "\n",
    "### 📥 Ingestion & Parsing\n",
    "Modules that handle raw data input and structure.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`ingest`** | **Data Ingestion**<br>Connects to data sources. | • File, Web, Feed, Stream ingestion<br>• DB, Email, Repo, MCP support |\n",
    "| **`parse`** | **Document Parsing**<br>Parses raw content into structures. | • PDF, HTML, JSON, CSV, Excel<br>• Image & Code parsing |\n",
    "\n",
    "### ⚙️ Data Processing\n",
    "Modules that clean, normalize, and split data.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`normalize`** | **Data Normalization**<br>Cleans and standardizes text. | • Text cleaning & Language detection<br>• Entity, Date, Number normalization |\n",
    "| **`split`** | **Chunking**<br>Splits documents for RAG. | • Recursive character splitting<br>• Semantic & Token-based splitting |\n",
    "\n",
    "### 🧠 Extraction & Enrichment\n",
    "Modules that extract meaning, structure, and vectors from raw data.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`semantic_extract`** | **Information Extraction**<br>Extracts entities and relations. | • NER & Relation Extraction<br>• Event & Semantic Network detection |\n",
    "| **`context`** | **Agent Memory**<br>Manages state for AI agents. | • Long-term memory & history<br>• Context graph & RAG integration |\n",
    "\n",
    "### 🕸️ Knowledge Graph Core\n",
    "Modules for building, refining, and resolving knowledge graphs.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`kg`** | **Graph Construction**<br>Builds and analyzes graphs. | • Graph Building & Analysis<br>• Validation & Entity Resolution |\n",
    "| **`conflicts`** | **Conflict Resolution**<br>Resolves data contradictions. | • Source reliability scoring<br>• Truth discovery algorithms |\n",
    "| **`deduplication`** | **Entity Resolution**<br>Merges duplicate entities. | • Similarity-based blocking<br>• Clustering & Canonicalization |\n",
    "\n",
    "### 💾 Storage & Retrieval\n",
    "Modules for persisting and querying data.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`embeddings`** | **Vector Embeddings**<br>Generates semantic vectors. | • Text & Graph embeddings<br>• Multi-provider support (OpenAI, etc.) |\n",
    "| **`vector_store`** | **Vector Database**<br>Stores and searches vectors. | • Similarity search & Filtering<br>• Hybrid search (Vector + Keyword) |\n",
    "| **`graph_store`** | **Property Graph Store**<br>Persists graph data. | • Neo4j, FalkorDB adapters<br>• Cypher query support |\n",
    "| **`triplet_store`** | **RDF Store**<br>Persists semantic triplets. | • SPARQL endpoints<br>• BlazeGraph, Jena, Virtuoso adapters |\n",
    "\n",
    "### 🔎 Reasoning & Analysis\n",
    "Modules for deriving new knowledge and evaluating quality.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`reasoning`** | **Reasoner Facade**<br>Unified interface for inference. | • Datalog/Rule-based inference<br>• Forward/Backward chaining |\n",
    "| **`ontology`** | **Ontology Management**<br>Manages schema and definitions. | • Ontology generation from data<br>• Validation & Evolution |\n",
    "| **`visualization`** | **Visual Analytics**<br>Visualizes graphs and metrics. | • 2D/3D Graph visualization<br>• Interactive plots & dashboards |\n",
    "| **`evals`** | **Evaluation**<br>Benchmarks pipeline quality. | • RAG & Graph quality metrics<br>• Ground truth comparison |\n",
    "\n",
    "### 🛠️ Orchestration & Utils\n",
    "Modules for managing the framework and workflows.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`core`** | **Framework Core**<br>Main entry point and config. | • Lifecycle management<br>• Plugin system & Configuration |\n",
    "| **`pipeline`** | **Workflow Orchestration**<br>Manages complex flows. | • DAG execution & Retries<br>• Error handling & Observability |\n",
    "| **`seed`** | **Data Seeding**<br>Initializes knowledge bases. | • Taxonomy & Ontology seeding<br>• Reference data loading |\n",
    "| **`export`** | **Data Export**<br>Exports data to files. | • JSON, CSV, RDF, GEXF export<br>• Report generation |\n",
    "| **`utils`** | **Utilities**<br>Common helper functions. | • Logging, Async, Hashing<br>• Text processing helpers |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Core Concepts (High-Level)\n",
    "\n",
    "- **Knowledge graph**\n",
    "  - Nodes represent entities such as people, organizations, locations, events, or concepts\n",
    "  - Edges represent relationships such as `works_for`, `located_in`, `founded_by`\n",
    "  - Properties capture attributes and metadata such as timestamps, sources, and confidence\n",
    "\n",
    "- **Entities and relationships**\n",
    "  - Entities are extracted from text and data using NER\n",
    "  - Relationships connect entities and are extracted using pattern-based, model-based, or LLM-based methods\n",
    "\n",
    "- **Embeddings**\n",
    "  - Numerical vectors that encode semantic meaning of text or graph structures\n",
    "  - Used for semantic search, clustering, and similarity-based retrieval\n",
    "\n",
    "- **GraphRAG**\n",
    "  - Combines vector search with graph traversal\n",
    "  - Uses both embeddings and graph structure to retrieve rich, context-aware information\n",
    "\n",
    "- **Ontology**\n",
    "  - A formal model of classes, relationships, and constraints in a domain\n",
    "  - Used to standardize meaning, enable reasoning, and integrate heterogeneous data\n",
    "\n",
    "- **Quality and governance**\n",
    "  - Quality metrics (completeness, consistency, accuracy, coverage)\n",
    "  - Conflict detection and resolution at the knowledge graph level"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "You can install Semantica from PyPI. In this notebook, we use a pip cell so it can run in local Jupyter or Colab.\n",
    "\n",
    "Equivalent shell commands:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "pip install semantica[all]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Configuration\n",
    "\n",
    "Semantica uses configuration for API keys, embedding providers, and knowledge graph options. The example below mirrors a typical configuration while staying simple enough for a notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.core import Config\n",
    "config = Config()\n",
    "print(config.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to Semantica\n",
    "\n",
    "**Open Source Framework for Semantic Layer & Knowledge Engineering**\n",
    "\n",
    "Semantica is a Python framework for transforming raw, messy, multi-source data into **semantic layers** and **knowledge graphs** that are ready to power GraphRAG, AI agents, multi-agent systems, and analytical applications.\n",
    "\n",
    "This notebook is an executable introduction. It combines:\n",
    "\n",
    "- High-level explanation of what Semantica is and why it exists\n",
    "- A structured tour of the architecture and key modules\n",
    "- Small, runnable code snippets that show the end-to-end flow\n",
    "\n",
    "**You should use this notebook to understand the big picture, not to learn every API in depth.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U semantica\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Is Semantica?\n",
    "\n",
    "Semantica is a **semantic intelligence and knowledge engineering framework**. It helps you:\n",
    "\n",
    "- Build **knowledge graphs** from unstructured and semi-structured data\n",
    "- Create a unified **semantic layer** on top of diverse data sources\n",
    "- Power **GraphRAG**, AI agents, and multi-agent systems with structured knowledge\n",
    "- Incorporate **temporal and quality-aware reasoning** into your applications\n",
    "\n",
    "### Core Capabilities\n",
    "\n",
    "- **Universal ingestion**: Files, web, feeds, databases, repositories, streams\n",
    "- **Rich parsing**: PDFs, Office documents, HTML, JSON, CSV, images, code\n",
    "- **Normalization**: Cleaning, language detection, entity normalization, date/number standardization\n",
    "- **Semantic extraction**: Named entities, relationships, events, semantic networks\n",
    "- **Knowledge graph construction**: Property graphs from entities and relations\n",
    "- **Embeddings and vector search**: Text and graph embeddings, hybrid retrieval\n",
    "- **Reasoning and ontology**: Rule-based inference, ontology generation and validation\n",
    "- **Visualization and analytics**: Graph visualizations and quality metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Who Is Semantica For?\n",
    "\n",
    "- **AI/ML engineers** building GraphRAG systems, agents, and tools that need long-term memory\n",
    "- **Data engineers** orchestrating semantic enrichment pipelines over large, heterogeneous datasets\n",
    "- **Knowledge engineers and ontologists** designing and maintaining formal knowledge structures\n",
    "- **Researchers and analysts** creating domain knowledge graphs from documents and data feeds\n",
    "- **Product and platform teams** embedding semantic intelligence into applications and services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Architecture Overview\n",
    "\n",
    "Semantica is organized as three conceptual layers and multiple concrete modules.\n",
    "\n",
    "### Layers\n",
    "\n",
    "- **Input Layer**\n",
    "  - Connects to files, web pages, APIs, databases, email, feeds, repositories, and streams\n",
    "  - Normalizes these different sources into a unified internal representation\n",
    "\n",
    "- **Semantic Layer**\n",
    "  - Performs parsing, cleaning, semantic extraction, graph construction, embeddings, and reasoning\n",
    "  - This is where **unstructured data becomes structured knowledge**\n",
    "\n",
    "- **Output Layer**\n",
    "  - Exposes knowledge graphs, embeddings, ontologies, and analytics\n",
    "  - Integrates with vector stores, graph databases, and downstream applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🧩 Semantica Modules Reference\n",
    "\n",
    "Semantica is modular by design. Here is a comprehensive guide to all available modules, grouped by functionality.\n",
    "\n",
    "### 📥 Ingestion & Parsing\n",
    "Modules that handle raw data input and structure.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`ingest`** | **Data Ingestion**<br>Connects to data sources. | • File, Web, Feed, Stream ingestion<br>• DB, Email, Repo, MCP support |\n",
    "| **`parse`** | **Document Parsing**<br>Parses raw content into structures. | • PDF, HTML, JSON, CSV, Excel<br>• Image & Code parsing |\n",
    "\n",
    "### ⚙️ Data Processing\n",
    "Modules that clean, normalize, and split data.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`normalize`** | **Data Normalization**<br>Cleans and standardizes text. | • Text cleaning & Language detection<br>• Entity, Date, Number normalization |\n",
    "| **`split`** | **Chunking**<br>Splits documents for RAG. | • Recursive character splitting<br>• Semantic & Token-based splitting |\n",
    "\n",
    "### 🧠 Extraction & Enrichment\n",
    "Modules that extract meaning, structure, and vectors from raw data.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`semantic_extract`** | **Information Extraction**<br>Extracts entities and relations. | • NER & Relation Extraction<br>• Event & Semantic Network detection |\n",
    "| **`context`** | **Agent Memory**<br>Manages state for AI agents. | • Long-term memory & history<br>• Context graph & RAG integration |\n",
    "\n",
    "### 🕸️ Knowledge Graph Core\n",
    "Modules for building, refining, and resolving knowledge graphs.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`kg`** | **Graph Construction**<br>Builds and analyzes graphs. | • Graph Building & Analysis<br>• Validation & Entity Resolution |\n",
    "| **`conflicts`** | **Conflict Resolution**<br>Resolves data contradictions. | • Source reliability scoring<br>• Truth discovery algorithms |\n",
    "| **`deduplication`** | **Entity Resolution**<br>Merges duplicate entities. | • Similarity-based blocking<br>• Clustering & Canonicalization |\n",
    "\n",
    "### 💾 Storage & Retrieval\n",
    "Modules for persisting and querying data.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`embeddings`** | **Vector Embeddings**<br>Generates semantic vectors. | • Text & Graph embeddings<br>• Multi-provider support (OpenAI, etc.) |\n",
    "| **`vector_store`** | **Vector Database**<br>Stores and searches vectors. | • Similarity search & Filtering<br>• Hybrid search (Vector + Keyword) |\n",
    "| **`graph_store`** | **Property Graph Store**<br>Persists graph data. | • Neo4j, FalkorDB adapters<br>• Cypher query support |\n",
    "| **`triplet_store`** | **RDF Store**<br>Persists semantic triplets. | • SPARQL endpoints<br>• BlazeGraph, Jena, Virtuoso adapters |\n",
    "\n",
    "### 🔎 Reasoning & Analysis\n",
    "Modules for deriving new knowledge and evaluating quality.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`reasoning`** | **Reasoner Facade**<br>Unified interface for inference. | • Datalog/Rule-based inference<br>• Forward/Backward chaining<br>• Automated explanation generation |\n",
    "| **`ontology`** | **Ontology Management**<br>Manages schema and definitions. | • Ontology generation from data<br>• Validation & Evolution |\n",
    "| **`visualization`** | **Visual Analytics**<br>Visualizes graphs and metrics. | • 2D/3D Graph visualization<br>• Interactive plots & dashboards |\n",
    "| **`evals`** | **Evaluation**<br>Benchmarks pipeline quality. | • RAG & Graph quality metrics<br>• Ground truth comparison |\n",
    "\n",
    "### 🛠️ Orchestration & Utils\n",
    "Modules for managing the framework and workflows.\n",
    "\n",
    "| Module | Description | Key Capabilities |\n",
    "| :--- | :--- | :--- |\n",
    "| **`core`** | **Framework Core**<br>Main entry point and config. | • Lifecycle management<br>• Plugin system & Configuration |\n",
    "| **`pipeline`** | **Workflow Orchestration**<br>Manages complex flows. | • DAG execution & Retries<br>• Error handling & Observability |\n",
    "| **`seed`** | **Data Seeding**<br>Initializes knowledge bases. | • Taxonomy & Ontology seeding<br>• Reference data loading |\n",
    "| **`export`** | **Data Export**<br>Exports data to files. | • JSON, CSV, RDF, GEXF export<br>• Report generation |\n",
    "| **`utils`** | **Utilities**<br>Common helper functions. | • Logging, Async, Hashing<br>• Text processing helpers |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Core Concepts (High-Level)\n",
    "\n",
    "- **Knowledge graph**\n",
    "  - Nodes represent entities such as people, organizations, locations, events, or concepts\n",
    "  - Edges represent relationships such as `works_for`, `located_in`, `founded_by`\n",
    "  - Properties capture attributes and metadata such as timestamps, sources, and confidence\n",
    "\n",
    "- **Entities and relationships**\n",
    "  - Entities are extracted from text and data using NER\n",
    "  - Relationships connect entities and are extracted using pattern-based, model-based, or LLM-based methods\n",
    "\n",
    "- **Embeddings**\n",
    "  - Numerical vectors that encode semantic meaning of text or graph structures\n",
    "  - Used for semantic search, clustering, and similarity-based retrieval\n",
    "\n",
    "- **GraphRAG**\n",
    "  - Combines vector search with graph traversal\n",
    "  - Uses both embeddings and graph structure to retrieve rich, context-aware information\n",
    "\n",
    "- **Ontology**\n",
    "  - A formal model of classes, relationships, and constraints in a domain\n",
    "  - Used to standardize meaning, enable reasoning, and integrate heterogeneous data\n",
    "\n",
    "- **Quality and governance**\n",
    "  - Quality metrics (completeness, consistency, accuracy, coverage)\n",
    "  - Conflict detection and resolution at the knowledge graph level"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "You can install Semantica from PyPI. In this notebook, we use a pip cell so it can run in local Jupyter or Colab.\n",
    "\n",
    "Equivalent shell commands:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "pip install semantica[all]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Configuration\n",
    "\n",
    "Semantica uses configuration for API keys, embedding providers, and knowledge graph options. The example below mirrors a typical configuration while staying simple enough for a notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "os.environ[\"SEMANTICA_API_KEY\"] = \"your_openai_key\"\n",
    "os.environ[\"SEMANTICA_EMBEDDING_PROVIDER\"] = \"openai\"\n",
    "os.environ[\"SEMANTICA_MODEL_NAME\"] = \"gpt-4\"\n",
    "\n",
    "config_text = \"\"\"api_keys:\n",
    "  openai: your_key_here\n",
    "  anthropic: your_key_here\n",
    "embedding:\n",
    "  provider: openai\n",
    "  model: text-embedding-3-large\n",
    "  dimensions: 3072\n",
    "knowledge_graph:\n",
    "  backend: networkx\n",
    "  temporal: true\n",
    "\"\"\"\n",
    "Path(\"config.yaml\").write_text(config_text, encoding=\"utf-8\")\n",
    "Path(\"config.yaml\").read_text(encoding=\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup: Create Sample Data\n",
    "\n",
    "First, let's create a small sample document to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "docs_dir = Path(\"welcome_docs\")\n",
    "docs_dir.mkdir(exist_ok=True)\n",
    "text_path = docs_dir / \"apple.txt\"\n",
    "text_content = (\n",
    "    \"Apple Inc. was founded by Steve Jobs, Steve Wozniak and Ronald Wayne in\"\n",
    "    \" Cupertino, California.\"\n",
    ")\n",
    "text_path.write_text(text_content, encoding=\"utf-8\")\n",
    "print(f\"Created sample document at {text_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal End-to-End Pipeline\n",
    "\n",
    "The next example shows how to explicitly use several modules in sequence. This mirrors the architecture discussed earlier:\n",
    "\n",
    "1. Ingest a directory of documents\n",
    "2. Parse them into structured documents\n",
    "3. Normalize text\n",
    "4. Extract entities and relationships\n",
    "5. Build and analyze a knowledge graph\n",
    "6. Create embeddings and store them in a vector store\n",
    "7. Run a hybrid semantic search query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor\n",
    "from semantica.parse import DocumentParser\n",
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer\n",
    "from semantica.embeddings import EmbeddingGenerator\n",
    "from semantica.vector_store import VectorStore, HybridSearch\n",
    "\n",
    "ingestor = FileIngestor()\n",
    "documents = ingestor.ingest(str(docs_dir))\n",
    "\n",
    "parser = DocumentParser()\n",
    "parsed_docs = parser.parse(documents)\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "normalized_docs = normalizer.normalize(parsed_docs)\n",
    "\n",
    "ner = NERExtractor()\n",
    "entities = ner.extract(normalized_docs)\n",
    "rel_extractor = RelationExtractor()\n",
    "relationships = rel_extractor.extract(normalized_docs, entities)\n",
    "\n",
    "builder = GraphBuilder()\n",
    "kg = builder.build(entities, relationships)\n",
    "analyzer = GraphAnalyzer()\n",
    "metrics = analyzer.analyze(kg)\n",
    "\n",
    "emb_generator = EmbeddingGenerator()\n",
    "embeddings = emb_generator.generate_embeddings(documents, data_type=\"text\")\n",
    "\n",
    "vec_store = VectorStore()\n",
    "vec_store.store(embeddings, documents, metadata={})\n",
    "hybrid = HybridSearch(vec_store)\n",
    "search_results = hybrid.search(\"Apple founders\", top_k=3)\n",
    "len(search_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization\n",
    "\n",
    "Semantica includes a powerful visualization module. Here we create an interactive network graph from the knowledge graph built above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import KGVisualizer\n",
    "\n",
    "# Create a visualizer instance\n",
    "viz = KGVisualizer(layout=\"force\", color_scheme=\"vibrant\")\n",
    "\n",
    "# Generate an interactive network visualization\n",
    "# This returns a Plotly figure object that renders in the notebook\n",
    "fig = viz.visualize_network(kg, output=\"interactive\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ontology Generation\n",
    "\n",
    "You can also automatically generate an ontology (a formal model of your domain) from the extracted entities and relationships."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import OntologyGenerator\n",
    "\n",
    "generator = OntologyGenerator(base_uri=\"https://example.org/ontology/\")\n",
    "\n",
    "# Generate ontology from the extracted data\n",
    "ontology = generator.generate_ontology({\n",
    "    \"entities\": entities,\n",
    "    \"relationships\": relationships\n",
    "})\n",
    "\n",
    "# View inferred classes\n",
    "[cls[\"name\"] for cls in ontology.get(\"classes\", [])[:5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced: Data Splitting and Chunking\n",
    "\n",
    "For RAG applications, splitting documents into smaller chunks is essential. Semantica provides a `split` module for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.split import TextSplitter\n",
    "\n",
    "splitter = TextSplitter(chunk_size=100, chunk_overlap=20)\n",
    "chunks = splitter.split_documents(documents)\n",
    "\n",
    "print(f\"Original documents: {len(documents)}\")\n",
    "print(f\"Generated chunks: {len(chunks)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced: Reasoning and Inference\n",
    "\n",
    "The `reasoning` module allows you to derive new facts from existing knowledge using logic rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.reasoning import Reasoner\n",
    "\n",
    "# Simple rule: If X founded Y, then X works_for Y\n",
    "rule = \"\"\"\n",
    "IF (?x founded ?y) THEN (?x works_for ?y)\n",
    "\"\"\"\n",
    "\n",
    "reasoner = Reasoner()\n",
    "reasoner.add_rule(rule)\n",
    "inferred_facts = reasoner.infer_facts(kg)\n",
    "\n",
    "print(f\"Inferred {len(inferred_facts)} new facts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced: Export and Persistence\n",
    "\n",
    "You can save your knowledge graph to disk or export it to standard formats like CSV, JSON, or RDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter\n",
    "\n",
    "exporter = GraphExporter()\n",
    "exporter.export(kg, format=\"json\", output_path=\"knowledge_graph.json\")\n",
    "print(\"Graph exported to knowledge_graph.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Core `Semantica` Class\n",
    "\n",
    "For more complex systems, you can work directly with the `Semantica` core class and a configuration object. This gives you access to lifecycle management, plugin registration, and orchestration helpers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.core import Semantica, ConfigManager\n",
    "\n",
    "config_manager = ConfigManager()\n",
    "config = config_manager.load_from_file(\"config.yaml\")\n",
    "\n",
    "framework = Semantica(config=config)\n",
    "framework.initialize()\n",
    "\n",
    "kb_result = framework.build_knowledge_base(\n",
    "    sources=[str(docs_dir)],\n",
    "    embeddings=True,\n",
    "    graph=True,\n",
    ")\n",
    "\n",
    "framework.shutdown()\n",
    "sorted(kb_result.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where to Go Next\n",
    "\n",
    "- Run the notebooks under `cookbook/introduction` for focused module overviews\n",
    "- Explore `cookbook/use_cases` for domain-specific end-to-end workflows\n",
    "- Read the **Core Concepts** documentation for deeper theory and best practices"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
