{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
    "\n",
    "# Chunking and Splitting - Comprehensive Guide\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook provides a **comprehensive walkthrough** of Semantica's split module, demonstrating all chunking strategies and methods for optimal document processing. You'll learn to use 15+ splitting methods including standard, semantic, and knowledge graph-aware approaches.\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/split/)\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "By the end of this notebook, you will be able to:\n",
    "\n",
    "- Use `TextSplitter` with multiple methods\n",
    "- Apply standard splitting methods (recursive, token, sentence, paragraph)\n",
    "- Use semantic chunking for topic coherence\n",
    "- Apply KG-aware chunking (entity-aware, relation-aware, graph-based)\n",
    "- Use specialized chunkers (structural, sliding window, table, hierarchical)\n",
    "- Track provenance with `ProvenanceTracker`\n",
    "- Choose the right method for your use case\n",
    "\n",
    "### What You'll Learn\n",
    "\n",
    "| Component | Purpose | When to Use |\n",
    "|-----------|---------|-------------|\n",
    "| `TextSplitter` | Unified splitter | All chunking needs |\n",
    "| `SemanticChunker` | Semantic boundaries | Topic-based chunks |\n",
    "| `EntityAwareChunker` | Preserve entities | GraphRAG workflows |\n",
    "| `RelationAwareChunker` | Preserve triplets | KG construction |\n",
    "| `StructuralChunker` | Document structure | Formatted documents |\n",
    "| `HierarchicalChunker` | Multi-level chunks | Large documents |\n",
    "\n",
    "---\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n"
     ]
    }
   ],
   "source": [
    "!pip install -q semantica\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Basic Chunking with TextSplitter\n",
    "\n",
    "Let's start with the unified `TextSplitter` interface, which provides access to all chunking methods.\n",
    "\n",
    "### What is TextSplitter?\n",
    "\n",
    "`TextSplitter` is a unified interface that supports 15+ chunking methods:\n",
    "- **Standard**: recursive, token, sentence, paragraph, character, word\n",
    "- **Semantic**: semantic_transformer, llm, huggingface, nltk\n",
    "- **KG/Ontology**: entity_aware, relation_aware, graph_based, ontology_aware\n",
    "- **Advanced**: hierarchical, structural, sliding_window, table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split into 8 chunks using recursive method\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Length: 181 characters\n",
      "  Start: 0, End: 184\n",
      "  Text: Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
      "in Cupert...\n",
      "\n",
      "Chunk 2:\n",
      "  Length: 144 characters\n",
      "  Start: 134, End: 281\n",
      "  Text: The company's current CEO is Tim Cook, who took \n",
      "over from Steve Jobs in August 2011. Apple is headq...\n",
      "\n",
      "Chunk 3:\n",
      "  Length: 150 characters\n",
      "  Start: 231, End: 383\n",
      "  Text: eadquartered at One Apple Park Way in Cupertino.\n",
      "\n",
      "Apple develops and sells consumer electronics, com...\n",
      "\n",
      "Chunk 4:\n",
      "  Length: 151 characters\n",
      "  Start: 333, End: 486\n",
      "  Text: ter software, and online services. The company's \n",
      "hardware products include the iPhone smartphone, t...\n",
      "\n",
      "Chunk 5:\n",
      "  Length: 176 characters\n",
      "  Start: 436, End: 614\n",
      "  Text: iPad tablet computer, the Mac personal computer, \n",
      "the iPod portable media player, the Apple Watch sm...\n",
      "\n",
      "Chunk 6:\n",
      "  Length: 152 characters\n",
      "  Start: 564, End: 718\n",
      "  Text: al media player, and the \n",
      "HomePod smart speaker.\n",
      "\n",
      "Apple's software includes the macOS and iOS operat...\n",
      "\n",
      "Chunk 7:\n",
      "  Length: 150 characters\n",
      "  Start: 668, End: 820\n",
      "  Text: systems, the iTunes media player, the Safari web \n",
      "browser, and the iLife and iWork creativity and pr...\n",
      "\n",
      "Chunk 8:\n",
      "  Length: 126 characters\n",
      "  Start: 770, End: 896\n",
      "  Text: uctivity suites. Its online services include the \n",
      "iTunes Store, the iOS App Store and Mac App Store,...\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import TextSplitter\n",
    "\n",
    "# Sample long text\n",
    "text = \"\"\"\n",
    "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
    "in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
    "over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.\n",
    "\n",
    "Apple develops and sells consumer electronics, computer software, and online services. The company's \n",
    "hardware products include the iPhone smartphone, the iPad tablet computer, the Mac personal computer, \n",
    "the iPod portable media player, the Apple Watch smartwatch, the Apple TV digital media player, and the \n",
    "HomePod smart speaker.\n",
    "\n",
    "Apple's software includes the macOS and iOS operating systems, the iTunes media player, the Safari web \n",
    "browser, and the iLife and iWork creativity and productivity suites. Its online services include the \n",
    "iTunes Store, the iOS App Store and Mac App Store, Apple Music, and iCloud.\n",
    "\"\"\"\n",
    "\n",
    "# Basic recursive splitting\n",
    "splitter = TextSplitter(\n",
    "    method=\"recursive\",\n",
    "    chunk_size=200,\n",
    "    chunk_overlap=50\n",
    ")\n",
    "\n",
    "chunks = splitter.split(text)\n",
    "\n",
    "print(f\"Split into {len(chunks)} chunks using recursive method\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Length: {len(chunk.text)} characters\")\n",
    "    print(f\"  Start: {chunk.start_index}, End: {chunk.end_index}\")\n",
    "    print(f\"  Text: {chunk.text[:100]}...\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Standard Splitting Methods\n",
    "\n",
    "Let's compare different standard splitting methods.\n",
    "\n",
    "### Method Comparison\n",
    "\n",
    "| Method | Best For | Speed | Accuracy |\n",
    "|--------|----------|-------|----------|\n",
    "| **recursive** | General text | Fast | Good |\n",
    "| **sentence** | Coherent chunks | Medium | Very Good |\n",
    "| **token** | LLM context | Medium | Excellent |\n",
    "| **paragraph** | Natural breaks | Fast | Good |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Comparing Standard Splitting Methods:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Method: RECURSIVE\n",
      "----------------------------------------\n",
      "  Chunks created: 8\n",
      "  Avg chunk size: 154 chars\n",
      "  First chunk: Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ron...\n",
      "\n",
      "Method: SENTENCE\n",
      "----------------------------------------\n",
      "  Chunks created: 6\n",
      "  Avg chunk size: 148 chars\n",
      "  First chunk: Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ron...\n",
      "\n",
      "Method: PARAGRAPH\n",
      "----------------------------------------\n",
      "  Chunks created: 3\n",
      "  Avg chunk size: 297 chars\n",
      "  First chunk: Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ron...\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Compare different methods\n",
    "methods = [\"recursive\", \"sentence\", \"paragraph\"]\n",
    "\n",
    "print(\"Comparing Standard Splitting Methods:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for method in methods:\n",
    "    splitter = TextSplitter(\n",
    "        method=method,\n",
    "        chunk_size=200,\n",
    "        chunk_overlap=50\n",
    "    )\n",
    "    \n",
    "    chunks = splitter.split(text)\n",
    "    \n",
    "    print(f\"\\nMethod: {method.upper()}\")\n",
    "    print(\"-\" * 40)\n",
    "    print(f\"  Chunks created: {len(chunks)}\")\n",
    "    print(f\"  Avg chunk size: {sum(len(c.text) for c in chunks) / len(chunks):.0f} chars\")\n",
    "    print(f\"  First chunk: {chunks[0].text[:80]}...\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Token-Based Splitting\n",
    "\n",
    "Token-based splitting is crucial for LLM applications where you need to respect token limits.\n",
    "\n",
    "### Why Token-Based?\n",
    "\n",
    "- **LLM Context Windows**: GPT-4 has 8K/32K token limits\n",
    "- **Accurate Counting**: Character count ≠ token count\n",
    "- **Cost Optimization**: Tokens determine API costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token-Based Splitting Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Tokens: 100\n",
      "  Characters: 461\n",
      "  Ratio: 4.61 chars/token\n",
      "\n",
      "Chunk 2:\n",
      "  Tokens: 100\n",
      "  Characters: 501\n",
      "  Ratio: 5.01 chars/token\n",
      "\n",
      "Chunk 3:\n",
      "  Tokens: 31\n",
      "  Characters: 150\n",
      "  Ratio: 4.84 chars/token\n",
      "\n",
      "Chunk 4:\n",
      "  Tokens: 20\n",
      "  Characters: 78\n",
      "  Ratio: 3.90 chars/token\n",
      "\n",
      "Chunk 5:\n",
      "  Tokens: 19\n",
      "  Characters: 76\n",
      "  Ratio: 4.00 chars/token\n",
      "\n",
      "Chunk 6:\n",
      "  Tokens: 18\n",
      "  Characters: 74\n",
      "  Ratio: 4.11 chars/token\n",
      "\n",
      "Chunk 7:\n",
      "  Tokens: 17\n",
      "  Characters: 70\n",
      "  Ratio: 4.12 chars/token\n",
      "\n",
      "Chunk 8:\n",
      "  Tokens: 16\n",
      "  Characters: 64\n",
      "  Ratio: 4.00 chars/token\n",
      "\n",
      "Chunk 9:\n",
      "  Tokens: 15\n",
      "  Characters: 63\n",
      "  Ratio: 4.20 chars/token\n",
      "\n",
      "Chunk 10:\n",
      "  Tokens: 14\n",
      "  Characters: 59\n",
      "  Ratio: 4.21 chars/token\n",
      "\n",
      "Chunk 11:\n",
      "  Tokens: 13\n",
      "  Characters: 55\n",
      "  Ratio: 4.23 chars/token\n",
      "\n",
      "Chunk 12:\n",
      "  Tokens: 12\n",
      "  Characters: 51\n",
      "  Ratio: 4.25 chars/token\n",
      "\n",
      "Chunk 13:\n",
      "  Tokens: 11\n",
      "  Characters: 45\n",
      "  Ratio: 4.09 chars/token\n",
      "\n",
      "Chunk 14:\n",
      "  Tokens: 10\n",
      "  Characters: 41\n",
      "  Ratio: 4.10 chars/token\n",
      "\n",
      "Chunk 15:\n",
      "  Tokens: 9\n",
      "  Characters: 37\n",
      "  Ratio: 4.11 chars/token\n",
      "\n",
      "Chunk 16:\n",
      "  Tokens: 8\n",
      "  Characters: 33\n",
      "  Ratio: 4.12 chars/token\n",
      "\n",
      "Chunk 17:\n",
      "  Tokens: 7\n",
      "  Characters: 27\n",
      "  Ratio: 3.86 chars/token\n",
      "\n",
      "Chunk 18:\n",
      "  Tokens: 6\n",
      "  Characters: 26\n",
      "  Ratio: 4.33 chars/token\n",
      "\n",
      "Chunk 19:\n",
      "  Tokens: 5\n",
      "  Characters: 20\n",
      "  Ratio: 4.00 chars/token\n",
      "\n",
      "Chunk 20:\n",
      "  Tokens: 4\n",
      "  Characters: 14\n",
      "  Ratio: 3.50 chars/token\n",
      "\n",
      "Chunk 21:\n",
      "  Tokens: 3\n",
      "  Characters: 13\n",
      "  Ratio: 4.33 chars/token\n",
      "\n",
      "Chunk 22:\n",
      "  Tokens: 2\n",
      "  Characters: 9\n",
      "  Ratio: 4.50 chars/token\n",
      "\n",
      "Chunk 23:\n",
      "  Tokens: 1\n",
      "  Characters: 2\n",
      "  Ratio: 2.00 chars/token\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import split_by_tokens\n",
    "\n",
    "# Token-based splitting\n",
    "chunks = split_by_tokens(\n",
    "    text,\n",
    "    chunk_size=100,  # 100 tokens\n",
    "    chunk_overlap=20,\n",
    "    tokenizer=\"tiktoken\",\n",
    "    model=\"gpt-4\"\n",
    ")\n",
    "\n",
    "print(\"Token-Based Splitting Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    token_count = chunk.metadata.get('token_count', 'N/A')\n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Tokens: {token_count}\")\n",
    "    print(f\"  Characters: {len(chunk.text)}\")\n",
    "    print(f\"  Ratio: {len(chunk.text)/token_count if token_count != 'N/A' else 'N/A':.2f} chars/token\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Semantic Chunking\n",
    "\n",
    "Semantic chunking creates chunks based on semantic boundaries using embeddings.\n",
    "\n",
    "### How It Works\n",
    "\n",
    "1. Split text into sentences\n",
    "2. Generate embeddings for each sentence\n",
    "3. Calculate similarity between consecutive sentences\n",
    "4. Create boundaries where similarity drops below threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>File</th><th>Time</th></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>SemanticChunker</td><td>-</td><td>0.13s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>EntityAwareChunker</td><td>-</td><td>0.98s</td></tr><tr><td>✅</td><td>Semantica is extracting</td><td>🎯 semantic_extract</td><td>NERExtractor</td><td>-</td><td>0.47s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>RelationAwareChunker</td><td>-</td><td>1.39s</td></tr><tr><td>✅</td><td>Semantica is extracting</td><td>🎯 semantic_extract</td><td>RelationExtractor</td><td>-</td><td>0.45s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>StructuralChunker</td><td>-</td><td>0.01s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>HierarchicalChunker</td><td>-</td><td>0.00s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>SlidingWindowChunker</td><td>-</td><td>0.00s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>TableChunker</td><td>-</td><td>0.01s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>ChunkValidator</td><td>-</td><td>0.01s</td></tr></table></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Semantic Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Length: 133 chars\n",
      "  Coherence: N/A\n",
      "  Text: Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
      "in Cupert...\n",
      "\n",
      "Chunk 2:\n",
      "  Length: 194 chars\n",
      "  Coherence: N/A\n",
      "  Text: Wayne \n",
      "in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
      "o...\n",
      "\n",
      "Chunk 3:\n",
      "  Length: 136 chars\n",
      "  Coherence: N/A\n",
      "  Text: headquartered at One Apple Park Way in Cupertino. Apple develops and sells consumer electronics, com...\n",
      "\n",
      "Chunk 4:\n",
      "  Length: 295 chars\n",
      "  Coherence: N/A\n",
      "  Text: ectronics, computer software, and online services. The company's \n",
      "hardware products include the iPho...\n",
      "\n",
      "Chunk 5:\n",
      "  Length: 223 chars\n",
      "  Coherence: N/A\n",
      "  Text: ital media player, and the \n",
      "HomePod smart speaker. Apple's software includes the macOS and iOS opera...\n",
      "\n",
      "Chunk 6:\n",
      "  Length: 159 chars\n",
      "  Coherence: N/A\n",
      "  Text: Life and iWork creativity and productivity suites. Its online services include the \n",
      "iTunes Store, th...\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import SemanticChunker\n",
    "\n",
    "# Semantic chunking\n",
    "semantic_chunker = SemanticChunker(\n",
    "    chunk_size=200,\n",
    "    chunk_overlap=50,\n",
    "    embedding_model=\"all-MiniLM-L6-v2\",\n",
    "    similarity_threshold=0.7\n",
    ")\n",
    "\n",
    "chunks = semantic_chunker.chunk(text)\n",
    "\n",
    "print(\"Semantic Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    coherence = chunk.metadata.get('coherence_score', 'N/A')\n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Length: {len(chunk.text)} chars\")\n",
    "    print(f\"  Coherence: {coherence}\")\n",
    "    print(f\"  Text: {chunk.text[:100]}...\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Entity-Aware Chunking for GraphRAG\n",
    "\n",
    "Entity-aware chunking preserves entity boundaries, crucial for GraphRAG workflows.\n",
    "\n",
    "### Why Entity-Aware?\n",
    "\n",
    "- **Preserve Entities**: Don't split \"Steve Jobs\" across chunks\n",
    "- **Better Extraction**: Complete entities improve NER accuracy\n",
    "- **GraphRAG**: Essential for knowledge graph construction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entity-Aware Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Length: 892 chars\n",
      "  Entities: 28\n",
      "  Sample entities: [\"Entity(text='Apple Inc.', label='ORG', start_char=1, end_char=11, confidence=1.0, metadata={'extraction_method': 'ml', 'model': 'en_core_web_sm', 'lemma': 'Apple Inc.'})\", \"Entity(text='Steve Jobs', label='PERSON', start_char=47, end_char=57, confidence=1.0, metadata={'extraction_method': 'ml', 'model': 'en_core_web_sm', 'lemma': 'Steve Jobs'})\", \"Entity(text='Steve Wozniak', label='PERSON', start_char=59, end_char=72, confidence=1.0, metadata={'extraction_method': 'ml', 'model': 'en_core_web_sm', 'lemma': 'Steve Wozniak'})\"]\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import EntityAwareChunker\n",
    "\n",
    "# Entity-aware chunking\n",
    "entity_chunker = EntityAwareChunker(\n",
    "    chunk_size=200,\n",
    "    chunk_overlap=50,\n",
    "    ner_method=\"ml\",  # \"ml\" (spaCy), \"pattern\", or \"llm\"\n",
    "    preserve_entities=True\n",
    ")\n",
    "\n",
    "chunks = entity_chunker.chunk(text)\n",
    "\n",
    "print(\"Entity-Aware Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    entities = chunk.metadata.get('entities', [])\n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Length: {len(chunk.text)} chars\")\n",
    "    print(f\"  Entities: {len(entities)}\")\n",
    "    \n",
    "    if entities:\n",
    "        # Handle both Entity objects and dicts\n",
    "        entity_texts = [e.get('text', e.get('entity', '')) if isinstance(e, dict) else str(e) for e in entities[:3]]\n",
    "        print(f\"  Sample entities: {entity_texts}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Relation-Aware Chunking\n",
    "\n",
    "Relation-aware chunking preserves relationship triplets within chunks.\n",
    "\n",
    "### Why Relation-Aware?\n",
    "\n",
    "- **Preserve Triplets**: Keep (subject, predicate, object) together\n",
    "- **KG Construction**: Better for building knowledge graphs\n",
    "- **Context**: Relationships need complete context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relation-Aware Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Length: 133 chars\n",
      "  Triplets: 2\n",
      "  Relationships: 2\n",
      "\n",
      "Chunk 2:\n",
      "  Length: 144 chars\n",
      "  Triplets: 0\n",
      "  Relationships: 0\n",
      "\n",
      "Chunk 3:\n",
      "  Length: 86 chars\n",
      "  Triplets: 0\n",
      "  Relationships: 0\n",
      "\n",
      "Chunk 4:\n",
      "  Length: 244 chars\n",
      "  Triplets: 0\n",
      "  Relationships: 0\n",
      "\n",
      "Chunk 5:\n",
      "  Length: 172 chars\n",
      "  Triplets: 0\n",
      "  Relationships: 0\n",
      "\n",
      "Chunk 6:\n",
      "  Length: 108 chars\n",
      "  Triplets: 0\n",
      "  Relationships: 0\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import RelationAwareChunker\n",
    "\n",
    "# Relation-aware chunking\n",
    "relation_chunker = RelationAwareChunker(\n",
    "    chunk_size=200,\n",
    "    chunk_overlap=50,\n",
    "    preserve_triplets=True\n",
    ")\n",
    "\n",
    "chunks = relation_chunker.chunk(text)\n",
    "\n",
    "print(\"Relation-Aware Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    triplets = chunk.metadata.get('triplets', [])\n",
    "    relationships = chunk.metadata.get('relationships', [])\n",
    "    \n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Length: {len(chunk.text)} chars\")\n",
    "    print(f\"  Triplets: {len(triplets)}\")\n",
    "    print(f\"  Relationships: {len(relationships)}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Structural Chunking\n",
    "\n",
    "Structural chunking respects document structure like headings, paragraphs, and lists.\n",
    "\n",
    "### When to Use?\n",
    "\n",
    "- **Formatted Documents**: Markdown, HTML, structured text\n",
    "- **Preserve Hierarchy**: Keep sections together\n",
    "- **Better Context**: Headings provide context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Structural Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Section: N/A\n",
      "  Level: N/A\n",
      "  Text: # Apple Inc....\n",
      "\n",
      "Chunk 2:\n",
      "  Section: N/A\n",
      "  Level: N/A\n",
      "  Text: ## History\n",
      "\n",
      "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayn...\n",
      "\n",
      "Chunk 3:\n",
      "  Section: N/A\n",
      "  Level: N/A\n",
      "  Text: ## Products\n",
      "\n",
      "### Hardware\n",
      "\n",
      "- iPhone\n",
      "- iPad\n",
      "- Mac\n",
      "\n",
      "### Software\n",
      "\n",
      "- macOS\n",
      "- iOS\n",
      "- ...\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import StructuralChunker\n",
    "\n",
    "# Markdown text with structure\n",
    "markdown_text = \"\"\"\n",
    "# Apple Inc.\n",
    "\n",
    "## History\n",
    "\n",
    "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
    "\n",
    "## Products\n",
    "\n",
    "### Hardware\n",
    "- iPhone\n",
    "- iPad\n",
    "- Mac\n",
    "\n",
    "### Software\n",
    "- macOS\n",
    "- iOS\n",
    "- Safari\n",
    "\"\"\"\n",
    "\n",
    "# Structural chunking\n",
    "structural_chunker = StructuralChunker(\n",
    "    respect_headings=True,\n",
    "    respect_paragraphs=True,\n",
    "    respect_lists=True,\n",
    "    max_chunk_size=500\n",
    ")\n",
    "\n",
    "chunks = structural_chunker.chunk(markdown_text)\n",
    "\n",
    "print(\"Structural Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    section = chunk.metadata.get('section_title', 'N/A')\n",
    "    level = chunk.metadata.get('heading_level', 'N/A')\n",
    "    \n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Section: {section}\")\n",
    "    print(f\"  Level: {level}\")\n",
    "    print(f\"  Text: {chunk.text[:80]}...\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Hierarchical Chunking\n",
    "\n",
    "Hierarchical chunking creates multi-level chunks for large documents.\n",
    "\n",
    "### Benefits\n",
    "\n",
    "- **Multiple Granularities**: Document → Section → Paragraph\n",
    "- **Better Navigation**: Parent-child relationships\n",
    "- **Flexible Retrieval**: Query at different levels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hierarchical Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Level: N/A\n",
      "  Length: 278 chars\n",
      "  Parent: None (root)\n",
      "  Children: 0\n",
      "\n",
      "Chunk 2:\n",
      "  Level: N/A\n",
      "  Length: 331 chars\n",
      "  Parent: None (root)\n",
      "  Children: 0\n",
      "\n",
      "Chunk 3:\n",
      "  Level: N/A\n",
      "  Length: 281 chars\n",
      "  Parent: None (root)\n",
      "  Children: 0\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import HierarchicalChunker\n",
    "\n",
    "# Hierarchical chunking\n",
    "hierarchical_chunker = HierarchicalChunker(\n",
    "    chunk_sizes=[400, 200, 100],  # 3 levels\n",
    "    chunk_overlaps=[80, 40, 20],\n",
    "    create_parent_chunks=True\n",
    ")\n",
    "\n",
    "chunks = hierarchical_chunker.chunk(text)\n",
    "\n",
    "print(\"Hierarchical Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    level = chunk.metadata.get('level', 'N/A')\n",
    "    parent_id = chunk.metadata.get('parent_id', None)\n",
    "    child_ids = chunk.metadata.get('child_ids', [])\n",
    "    \n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Level: {level}\")\n",
    "    print(f\"  Length: {len(chunk.text)} chars\")\n",
    "    print(f\"  Parent: {parent_id if parent_id else 'None (root)'}\")\n",
    "    print(f\"  Children: {len(child_ids)}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Sliding Window Chunking\n",
    "\n",
    "Sliding window creates overlapping fixed-size chunks.\n",
    "\n",
    "### Use Cases\n",
    "\n",
    "- **Dense Retrieval**: Ensure no information is missed\n",
    "- **Fixed Context**: Consistent chunk sizes\n",
    "- **Overlap Control**: Precise overlap management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sliding Window Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Window 1:\n",
      "  Position: 0-134\n",
      "  Length: 133 chars\n",
      "  Overlap with previous: 0 chars\n",
      "\n",
      "Window 2:\n",
      "  Position: 100-220\n",
      "  Length: 120 chars\n",
      "  Overlap with previous: 34 chars\n",
      "\n",
      "Window 3:\n",
      "  Position: 200-279\n",
      "  Length: 79 chars\n",
      "  Overlap with previous: 20 chars\n",
      "\n",
      "Window 4:\n",
      "  Position: 300-383\n",
      "  Length: 81 chars\n",
      "  Overlap with previous: 0 chars\n",
      "\n",
      "Window 5:\n",
      "  Position: 400-486\n",
      "  Length: 83 chars\n",
      "  Overlap with previous: 0 chars\n",
      "\n",
      "Window 6:\n",
      "  Position: 500-612\n",
      "  Length: 112 chars\n",
      "  Overlap with previous: 0 chars\n",
      "\n",
      "Window 7:\n",
      "  Position: 600-718\n",
      "  Length: 116 chars\n",
      "  Overlap with previous: 12 chars\n",
      "\n",
      "Window 8:\n",
      "  Position: 700-786\n",
      "  Length: 86 chars\n",
      "  Overlap with previous: 18 chars\n",
      "\n",
      "Window 9:\n",
      "  Position: 800-896\n",
      "  Length: 95 chars\n",
      "  Overlap with previous: 0 chars\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import SlidingWindowChunker\n",
    "\n",
    "# Sliding window chunking\n",
    "sliding_chunker = SlidingWindowChunker(\n",
    "    chunk_size=150,\n",
    "    overlap=50\n",
    ")\n",
    "\n",
    "chunks = sliding_chunker.chunk(text)\n",
    "\n",
    "print(\"Sliding Window Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    # Calculate overlap manually\n",
    "    overlap = 0\n",
    "    if i > 1:\n",
    "        prev_chunk = chunks[i-2]\n",
    "        overlap = max(0, prev_chunk.end_index - chunk.start_index)\n",
    "    \n",
    "    print(f\"\\nWindow {i}:\")\n",
    "    print(f\"  Position: {chunk.start_index}-{chunk.end_index}\")\n",
    "    print(f\"  Length: {len(chunk.text)} chars\")\n",
    "    print(f\"  Overlap with previous: {overlap} chars\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 10: Table Chunking\n",
    "\n",
    "Table chunking preserves table structure while splitting large tables.\n",
    "\n",
    "### Features\n",
    "\n",
    "- **Preserve Headers**: Keep column headers in each chunk\n",
    "- **Row-Based Splitting**: Split by rows, not characters\n",
    "- **Context Inclusion**: Include surrounding text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table Chunking Results:\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Chunk 1:\n",
      "  Type: Text\n",
      "  Content: Product | Category | Release Year\n",
      "--- | --- | ---\n",
      "iPhone | Smartphone | 2007\n",
      "iPad | Tablet | 2010\n",
      "Ma...\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import TableChunker\n",
    "\n",
    "# Text with table\n",
    "text_with_table = \"\"\"\n",
    "Apple's product lineup includes:\n",
    "\n",
    "| Product | Category | Release Year |\n",
    "|---------|----------|-------------|\n",
    "| iPhone | Smartphone | 2007 |\n",
    "| iPad | Tablet | 2010 |\n",
    "| Mac | Computer | 1984 |\n",
    "| Apple Watch | Wearable | 2015 |\n",
    "| AirPods | Audio | 2016 |\n",
    "\n",
    "These products have revolutionized their respective categories.\n",
    "\"\"\"\n",
    "\n",
    "# Table chunking\n",
    "table_chunker = TableChunker(\n",
    "    preserve_headers=True,\n",
    "    max_rows_per_chunk=3,\n",
    "    include_context=True,\n",
    "    table_format=\"markdown\"\n",
    ")\n",
    "\n",
    "chunks = table_chunker.chunk(text_with_table)\n",
    "\n",
    "print(\"Table Chunking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for i, chunk in enumerate(chunks, 1):\n",
    "    is_table = chunk.metadata.get('is_table', False)\n",
    "    \n",
    "    print(f\"\\nChunk {i}:\")\n",
    "    print(f\"  Type: {'Table' if is_table else 'Text'}\")\n",
    "    \n",
    "    if is_table:\n",
    "        rows = chunk.metadata.get('row_count', 'N/A')\n",
    "        cols = chunk.metadata.get('column_count', 'N/A')\n",
    "        print(f\"  Rows: {rows}, Columns: {cols}\")\n",
    "    \n",
    "    print(f\"  Content: {chunk.text[:100]}...\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 11: Provenance Tracking\n",
    "\n",
    "Track chunk origins for data lineage and debugging.\n",
    "\n",
    "### Why Track Provenance?\n",
    "\n",
    "- **Data Lineage**: Know where chunks came from\n",
    "- **Debugging**: Trace issues back to source\n",
    "- **Compliance**: Required for some use cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chunk class fields: {'text': <class 'str'>, 'start_index': <class 'int'>, 'end_index': <class 'int'>, 'metadata': typing.Dict[str, typing.Any], 'id': typing.Optional[str]}\n",
      "Provenance Tracking Results:\n",
      "\n",
      "================================================================================\n",
      "Chunk ID: f0bddbe2-b717-47c7-8bdb-631a61660373\n",
      "\n",
      "Lineage for Chunk 1:\n",
      "  Source Document: apple_doc_001\n",
      "  File Path: data/apple.txt\n",
      "  Method: recursive\n",
      "  Timestamp: None\n",
      "\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import importlib\n",
    "\n",
    "# 1. Ensure local package is in path\n",
    "project_root = os.path.abspath(os.path.join(os.getcwd(), \"../..\"))\n",
    "if project_root not in sys.path:\n",
    "    sys.path.insert(0, project_root)\n",
    "\n",
    "# 2. Force unload modules to ensure clean reload\n",
    "modules_to_unload = [\n",
    "    'semantica.split.semantic_chunker', \n",
    "    'semantica.split.splitter', \n",
    "    'semantica.split.provenance_tracker',\n",
    "    'semantica.split'\n",
    "]\n",
    "for module in modules_to_unload:\n",
    "    if module in sys.modules:\n",
    "        del sys.modules[module]\n",
    "\n",
    "# 3. Import fresh modules\n",
    "import semantica.split.semantic_chunker\n",
    "import semantica.split.splitter\n",
    "import semantica.split.provenance_tracker\n",
    "from semantica.split import ProvenanceTracker, TextSplitter\n",
    "\n",
    "# 4. Verify Chunk class has id field\n",
    "from semantica.split.semantic_chunker import Chunk\n",
    "print(f\"Chunk class fields: {Chunk.__annotations__}\")\n",
    "if 'id' not in Chunk.__annotations__:\n",
    "    print(\"WARNING: Chunk class still missing 'id' field. Kernel restart required.\")\n",
    "\n",
    "# Create chunks\n",
    "splitter = TextSplitter(method=\"recursive\", chunk_size=200, chunk_overlap=50)\n",
    "chunks = splitter.split(text)\n",
    "\n",
    "# Track provenance\n",
    "tracker = ProvenanceTracker()\n",
    "\n",
    "for chunk in chunks:\n",
    "    tracker.track_chunk(\n",
    "        chunk=chunk,\n",
    "        source_document=\"apple_doc_001\",\n",
    "        source_path=\"data/apple.txt\",\n",
    "        timestamp=\"2024-01-01T00:00:00Z\",\n",
    "        method=\"recursive\"\n",
    "    )\n",
    "\n",
    "print(\"Provenance Tracking Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "# Get lineage for first chunk\n",
    "if chunks:\n",
    "    # Get provenance info using the chunk's ID\n",
    "    chunk_id = getattr(chunks[0], 'id', None)\n",
    "    print(f\"Chunk ID: {chunk_id}\")\n",
    "    \n",
    "    if chunk_id:\n",
    "        prov_info = tracker.get_provenance(chunk_id)\n",
    "        \n",
    "        if prov_info:\n",
    "            print(f\"\\nLineage for Chunk 1:\")\n",
    "            print(f\"  Source Document: {prov_info.source_document}\")\n",
    "            print(f\"  File Path: {prov_info.source_path}\")\n",
    "            print(f\"  Method: {prov_info.metadata.get('method')}\")\n",
    "            print(f\"  Timestamp: {prov_info.timestamp}\")\n",
    "    else:\n",
    "        print(\"Error: Chunk ID not found. The Chunk class definition might still be cached.\")\n",
    "        print(\"Please click 'Kernel' -> 'Restart Kernel' in the menu and run all cells again.\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 13: Method Comparison\n",
    "\n",
    "Let's compare all methods side-by-side to help you choose the right one.\n",
    "\n",
    "### Comparison Criteria\n",
    "\n",
    "- **Chunk Count**: Number of chunks created\n",
    "- **Average Size**: Average chunk size\n",
    "- **Processing Time**: Speed of chunking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Method Comparison:\n",
      "\n",
      "================================================================================\n",
      "Method          Chunks     Avg Size     Time (ms)   \n",
      "--------------------------------------------------------------------------------\n",
      "recursive       8          154          0.00        \n",
      "sentence        6          148          519.51      \n",
      "paragraph       3          297          0.00        \n",
      "token           51         129          0.00        \n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# Methods to compare\n",
    "methods_to_compare = [\n",
    "    (\"recursive\", {}),\n",
    "    (\"sentence\", {}),\n",
    "    (\"paragraph\", {}),\n",
    "    (\"token\", {\"tokenizer\": \"tiktoken\"}),\n",
    "]\n",
    "\n",
    "print(\"Method Comparison:\\n\")\n",
    "print(\"=\" * 80)\n",
    "print(f\"{'Method':<15} {'Chunks':<10} {'Avg Size':<12} {'Time (ms)':<12}\")\n",
    "print(\"-\" * 80)\n",
    "\n",
    "for method, kwargs in methods_to_compare:\n",
    "    try:\n",
    "        start_time = time.time()\n",
    "        \n",
    "        splitter = TextSplitter(\n",
    "            method=method,\n",
    "            chunk_size=200,\n",
    "            chunk_overlap=50,\n",
    "            **kwargs\n",
    "        )\n",
    "        \n",
    "        chunks = splitter.split(text)\n",
    "        \n",
    "        elapsed = (time.time() - start_time) * 1000\n",
    "        avg_size = sum(len(c.text) for c in chunks) / len(chunks) if chunks else 0\n",
    "        \n",
    "        print(f\"{method:<15} {len(chunks):<10} {avg_size:<12.0f} {elapsed:<12.2f}\")\n",
    "        \n",
    "    except Exception as e:\n",
    "        print(f\"{method:<15} Error: {str(e)[:40]}\")\n",
    "\n",
    "print(\"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 14: Best Practices\n",
    "\n",
    "### Choosing the Right Method\n",
    "\n",
    "1. **General Documents**: Use `recursive` for speed and simplicity\n",
    "2. **LLM Applications**: Use `token` to respect context windows\n",
    "3. **Semantic Search**: Use `semantic_transformer` for topic coherence\n",
    "4. **GraphRAG**: Use `entity_aware` or `relation_aware`\n",
    "5. **Structured Docs**: Use `structural` for formatted documents\n",
    "6. **Large Documents**: Use `hierarchical` for multi-level access\n",
    "\n",
    "### Chunk Size Guidelines\n",
    "\n",
    "| Use Case | Recommended Size | Overlap |\n",
    "|----------|------------------|----------|\n",
    "| Semantic Search | 512-1024 chars | 20% |\n",
    "| LLM Context | 2000-4000 chars | 10-20% |\n",
    "| Entity Extraction | 500-1500 chars | 15-25% |\n",
    "| Question Answering | 1000-2000 chars | 20% |\n",
    "\n",
    "### Overlap Recommendations\n",
    "\n",
    "- **10-15%**: Fast processing, less redundancy\n",
    "- **20-25%**: Balanced (recommended)\n",
    "- **30-40%**: Maximum context preservation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "### What You've Learned\n",
    "\n",
    "In this notebook, you've learned how to:\n",
    "\n",
    "- Use `TextSplitter` with multiple methods\n",
    "- Apply standard splitting (recursive, token, sentence, paragraph)\n",
    "- Use semantic chunking for topic coherence\n",
    "- Apply KG-aware chunking (entity-aware, relation-aware)\n",
    "- Use specialized chunkers (structural, hierarchical, sliding window, table)\n",
    "- Track provenance\n",
    "- Choose the right method for your use case\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "1. **Method Selection Matters**: Different methods for different needs\n",
    "2. **Chunk Size is Critical**: Balance between context and processing\n",
    "3. **Overlap Helps**: 20% overlap is a good default\n",
    "4. **Track Provenance**: Important for debugging and compliance\n",
    "5. **KG-Aware for GraphRAG**: Use entity/relation-aware for knowledge graphs\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "**Next Notebook**: [12_Embedding_Generation.ipynb](./12_Embedding_Generation.ipynb)  \n",
    "Learn how to generate embeddings for your chunks!\n",
    "\n",
    "**Further Reading**:\n",
    "- [Split Module API Reference](https://semantica.readthedocs.io/reference/split/)\n",
    "- [Advanced Chunking Strategies](../advanced/11_Text_Chunking_Strategies.ipynb)\n",
    "- [GraphRAG Pipeline](../use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n",
    "\n",
    "---\n",
    "\n",
    "**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
