{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
                "\n",
                "# Chunking and Splitting - Comprehensive Guide\n",
                "\n",
                "## Overview\n",
                "\n",
                "This notebook provides a **comprehensive walkthrough** of Semantica's split module, demonstrating all chunking strategies and methods for optimal document processing. You'll learn to use 15+ splitting methods including standard, semantic, and knowledge graph-aware approaches.\n",
                "\n",
                "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/split/)\n",
                "\n",
                "### Learning Objectives\n",
                "\n",
                "By the end of this notebook, you will be able to:\n",
                "\n",
                "- Use `TextSplitter` with multiple methods\n",
                "- Apply standard splitting methods (recursive, token, sentence, paragraph)\n",
                "- Use semantic chunking for topic coherence\n",
                "- Apply KG-aware chunking (entity-aware, relation-aware, graph-based)\n",
                "- Use specialized chunkers (structural, sliding window, table, hierarchical)\n",
                "- Validate chunk quality with `ChunkValidator`\n",
                "- Track provenance with `ProvenanceTracker`\n",
                "- Choose the right method for your use case\n",
                "\n",
                "### What You'll Learn\n",
                "\n",
                "| Component | Purpose | When to Use |\n",
                "|-----------|---------|-------------|\n",
                "| `TextSplitter` | Unified splitter | All chunking needs |\n",
                "| `SemanticChunker` | Semantic boundaries | Topic-based chunks |\n",
                "| `EntityAwareChunker` | Preserve entities | GraphRAG workflows |\n",
                "| `RelationAwareChunker` | Preserve triples | KG construction |\n",
                "| `StructuralChunker` | Document structure | Formatted documents |\n",
                "| `HierarchicalChunker` | Multi-level chunks | Large documents |\n",
                "\n",
                "---\n",
                "\n",
                "## Installation\n",
                "\n",
                "Install Semantica from PyPI:\n",
                "\n",
                "```bash\n",
                "pip install semantica\n",
                "# Or with all optional dependencies:\n",
                "pip install semantica[all]\n",
                "```\n",
                "\n",
                "---"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 1: Basic Chunking with TextSplitter\n",
                "\n",
                "Let's start with the unified `TextSplitter` interface, which provides access to all chunking methods.\n",
                "\n",
                "### What is TextSplitter?\n",
                "\n",
                "`TextSplitter` is a unified interface that supports 15+ chunking methods:\n",
                "- **Standard**: recursive, token, sentence, paragraph, character, word\n",
                "- **Semantic**: semantic_transformer, llm, huggingface, nltk\n",
                "- **KG/Ontology**: entity_aware, relation_aware, graph_based, ontology_aware\n",
                "- **Advanced**: hierarchical, structural, sliding_window, table"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import TextSplitter\n",
                "\n",
                "# Sample long text\n",
                "text = \"\"\"\n",
                "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
                "in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
                "over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.\n",
                "\n",
                "Apple develops and sells consumer electronics, computer software, and online services. The company's \n",
                "hardware products include the iPhone smartphone, the iPad tablet computer, the Mac personal computer, \n",
                "the iPod portable media player, the Apple Watch smartwatch, the Apple TV digital media player, and the \n",
                "HomePod smart speaker.\n",
                "\n",
                "Apple's software includes the macOS and iOS operating systems, the iTunes media player, the Safari web \n",
                "browser, and the iLife and iWork creativity and productivity suites. Its online services include the \n",
                "iTunes Store, the iOS App Store and Mac App Store, Apple Music, and iCloud.\n",
                "\"\"\"\n",
                "\n",
                "# Basic recursive splitting\n",
                "splitter = TextSplitter(\n",
                "    method=\"recursive\",\n",
                "    chunk_size=200,\n",
                "    chunk_overlap=50\n",
                ")\n",
                "\n",
                "chunks = splitter.split(text)\n",
                "\n",
                "print(f\"Split into {len(chunks)} chunks using recursive method\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Length: {len(chunk.text)} characters\")\n",
                "    print(f\"  Start: {chunk.start}, End: {chunk.end}\")\n",
                "    print(f\"  Text: {chunk.text[:100]}...\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 2: Standard Splitting Methods\n",
                "\n",
                "Let's compare different standard splitting methods.\n",
                "\n",
                "### Method Comparison\n",
                "\n",
                "| Method | Best For | Speed | Accuracy |\n",
                "|--------|----------|-------|----------|\n",
                "| **recursive** | General text | Fast | Good |\n",
                "| **sentence** | Coherent chunks | Medium | Very Good |\n",
                "| **token** | LLM context | Medium | Excellent |\n",
                "| **paragraph** | Natural breaks | Fast | Good |"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Compare different methods\n",
                "methods = [\"recursive\", \"sentence\", \"paragraph\"]\n",
                "\n",
                "print(\"Comparing Standard Splitting Methods:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for method in methods:\n",
                "    splitter = TextSplitter(\n",
                "        method=method,\n",
                "        chunk_size=200,\n",
                "        chunk_overlap=50\n",
                "    )\n",
                "    \n",
                "    chunks = splitter.split(text)\n",
                "    \n",
                "    print(f\"\\nMethod: {method.upper()}\")\n",
                "    print(\"-\" * 40)\n",
                "    print(f\"  Chunks created: {len(chunks)}\")\n",
                "    print(f\"  Avg chunk size: {sum(len(c.text) for c in chunks) / len(chunks):.0f} chars\")\n",
                "    print(f\"  First chunk: {chunks[0].text[:80]}...\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 3: Token-Based Splitting\n",
                "\n",
                "Token-based splitting is crucial for LLM applications where you need to respect token limits.\n",
                "\n",
                "### Why Token-Based?\n",
                "\n",
                "- **LLM Context Windows**: GPT-4 has 8K/32K token limits\n",
                "- **Accurate Counting**: Character count ≠ token count\n",
                "- **Cost Optimization**: Tokens determine API costs"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import split_by_tokens\n",
                "\n",
                "# Token-based splitting\n",
                "chunks = split_by_tokens(\n",
                "    text,\n",
                "    chunk_size=100,  # 100 tokens\n",
                "    chunk_overlap=20,\n",
                "    tokenizer=\"tiktoken\",\n",
                "    model=\"gpt-4\"\n",
                ")\n",
                "\n",
                "print(\"Token-Based Splitting Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    token_count = chunk.metadata.get('token_count', 'N/A')\n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Tokens: {token_count}\")\n",
                "    print(f\"  Characters: {len(chunk.text)}\")\n",
                "    print(f\"  Ratio: {len(chunk.text)/token_count if token_count != 'N/A' else 'N/A':.2f} chars/token\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 4: Semantic Chunking\n",
                "\n",
                "Semantic chunking creates chunks based on semantic boundaries using embeddings.\n",
                "\n",
                "### How It Works\n",
                "\n",
                "1. Split text into sentences\n",
                "2. Generate embeddings for each sentence\n",
                "3. Calculate similarity between consecutive sentences\n",
                "4. Create boundaries where similarity drops below threshold"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import SemanticChunker\n",
                "\n",
                "# Semantic chunking\n",
                "semantic_chunker = SemanticChunker(\n",
                "    chunk_size=200,\n",
                "    chunk_overlap=50,\n",
                "    embedding_model=\"all-MiniLM-L6-v2\",\n",
                "    similarity_threshold=0.7\n",
                ")\n",
                "\n",
                "chunks = semantic_chunker.chunk(text)\n",
                "\n",
                "print(\"Semantic Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    coherence = chunk.metadata.get('coherence_score', 'N/A')\n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Length: {len(chunk.text)} chars\")\n",
                "    print(f\"  Coherence: {coherence}\")\n",
                "    print(f\"  Text: {chunk.text[:100]}...\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 5: Entity-Aware Chunking for GraphRAG\n",
                "\n",
                "Entity-aware chunking preserves entity boundaries, crucial for GraphRAG workflows.\n",
                "\n",
                "### Why Entity-Aware?\n",
                "\n",
                "- **Preserve Entities**: Don't split \"Steve Jobs\" across chunks\n",
                "- **Better Extraction**: Complete entities improve NER accuracy\n",
                "- **GraphRAG**: Essential for knowledge graph construction"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import EntityAwareChunker\n",
                "\n",
                "# Entity-aware chunking\n",
                "entity_chunker = EntityAwareChunker(\n",
                "    chunk_size=200,\n",
                "    chunk_overlap=50,\n",
                "    ner_method=\"spacy\",  # or \"llm\" for better accuracy\n",
                "    preserve_entities=True\n",
                ")\n",
                "\n",
                "chunks = entity_chunker.chunk(text)\n",
                "\n",
                "print(\"Entity-Aware Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    entities = chunk.metadata.get('entities', [])\n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Length: {len(chunk.text)} chars\")\n",
                "    print(f\"  Entities: {len(entities)}\")\n",
                "    \n",
                "    if entities:\n",
                "        entity_texts = [e.get('text', e.get('entity', '')) if isinstance(e, dict) else str(e) for e in entities[:3]]\n",
                "        print(f\"  Sample entities: {entity_texts}\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 6: Relation-Aware Chunking\n",
                "\n",
                "Relation-aware chunking preserves relationship triples within chunks.\n",
                "\n",
                "### Why Relation-Aware?\n",
                "\n",
                "- **Preserve Triples**: Keep (subject, predicate, object) together\n",
                "- **KG Construction**: Better for building knowledge graphs\n",
                "- **Context**: Relationships need complete context"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import RelationAwareChunker\n",
                "\n",
                "# Relation-aware chunking\n",
                "relation_chunker = RelationAwareChunker(\n",
                "    chunk_size=200,\n",
                "    chunk_overlap=50,\n",
                "    preserve_triples=True\n",
                ")\n",
                "\n",
                "chunks = relation_chunker.chunk(text)\n",
                "\n",
                "print(\"Relation-Aware Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    triples = chunk.metadata.get('triples', [])\n",
                "    relationships = chunk.metadata.get('relationships', [])\n",
                "    \n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Length: {len(chunk.text)} chars\")\n",
                "    print(f\"  Triples: {len(triples)}\")\n",
                "    print(f\"  Relationships: {len(relationships)}\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 7: Structural Chunking\n",
                "\n",
                "Structural chunking respects document structure like headings, paragraphs, and lists.\n",
                "\n",
                "### When to Use?\n",
                "\n",
                "- **Formatted Documents**: Markdown, HTML, structured text\n",
                "- **Preserve Hierarchy**: Keep sections together\n",
                "- **Better Context**: Headings provide context"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import StructuralChunker\n",
                "\n",
                "# Markdown text with structure\n",
                "markdown_text = \"\"\"\n",
                "# Apple Inc.\n",
                "\n",
                "## History\n",
                "\n",
                "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
                "\n",
                "## Products\n",
                "\n",
                "### Hardware\n",
                "- iPhone\n",
                "- iPad\n",
                "- Mac\n",
                "\n",
                "### Software\n",
                "- macOS\n",
                "- iOS\n",
                "- Safari\n",
                "\"\"\"\n",
                "\n",
                "# Structural chunking\n",
                "structural_chunker = StructuralChunker(\n",
                "    respect_headings=True,\n",
                "    respect_paragraphs=True,\n",
                "    respect_lists=True,\n",
                "    max_chunk_size=500\n",
                ")\n",
                "\n",
                "chunks = structural_chunker.chunk(markdown_text)\n",
                "\n",
                "print(\"Structural Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    section = chunk.metadata.get('section_title', 'N/A')\n",
                "    level = chunk.metadata.get('heading_level', 'N/A')\n",
                "    \n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Section: {section}\")\n",
                "    print(f\"  Level: {level}\")\n",
                "    print(f\"  Text: {chunk.text[:80]}...\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 8: Hierarchical Chunking\n",
                "\n",
                "Hierarchical chunking creates multi-level chunks for large documents.\n",
                "\n",
                "### Benefits\n",
                "\n",
                "- **Multiple Granularities**: Document → Section → Paragraph\n",
                "- **Better Navigation**: Parent-child relationships\n",
                "- **Flexible Retrieval**: Query at different levels"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import HierarchicalChunker\n",
                "\n",
                "# Hierarchical chunking\n",
                "hierarchical_chunker = HierarchicalChunker(\n",
                "    chunk_sizes=[400, 200, 100],  # 3 levels\n",
                "    chunk_overlaps=[80, 40, 20],\n",
                "    create_parent_chunks=True\n",
                ")\n",
                "\n",
                "chunks = hierarchical_chunker.chunk(text)\n",
                "\n",
                "print(\"Hierarchical Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    level = chunk.metadata.get('level', 'N/A')\n",
                "    parent_id = chunk.metadata.get('parent_id', None)\n",
                "    child_ids = chunk.metadata.get('child_ids', [])\n",
                "    \n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Level: {level}\")\n",
                "    print(f\"  Length: {len(chunk.text)} chars\")\n",
                "    print(f\"  Parent: {parent_id if parent_id else 'None (root)'}\")\n",
                "    print(f\"  Children: {len(child_ids)}\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 9: Sliding Window Chunking\n",
                "\n",
                "Sliding window creates overlapping fixed-size chunks.\n",
                "\n",
                "### Use Cases\n",
                "\n",
                "- **Dense Retrieval**: Ensure no information is missed\n",
                "- **Fixed Context**: Consistent chunk sizes\n",
                "- **Overlap Control**: Precise overlap management"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import SlidingWindowChunker\n",
                "\n",
                "# Sliding window chunking\n",
                "sliding_chunker = SlidingWindowChunker(\n",
                "    window_size=150,\n",
                "    step_size=100,  # 50 char overlap\n",
                "    min_chunk_size=50\n",
                ")\n",
                "\n",
                "chunks = sliding_chunker.chunk(text)\n",
                "\n",
                "print(\"Sliding Window Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    overlap = chunk.metadata.get('overlap_chars', 0)\n",
                "    \n",
                "    print(f\"\\nWindow {i}:\")\n",
                "    print(f\"  Position: {chunk.start}-{chunk.end}\")\n",
                "    print(f\"  Length: {len(chunk.text)} chars\")\n",
                "    print(f\"  Overlap with previous: {overlap} chars\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 10: Table Chunking\n",
                "\n",
                "Table chunking preserves table structure while splitting large tables.\n",
                "\n",
                "### Features\n",
                "\n",
                "- **Preserve Headers**: Keep column headers in each chunk\n",
                "- **Row-Based Splitting**: Split by rows, not characters\n",
                "- **Context Inclusion**: Include surrounding text"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import TableChunker\n",
                "\n",
                "# Text with table\n",
                "text_with_table = \"\"\"\n",
                "Apple's product lineup includes:\n",
                "\n",
                "| Product | Category | Release Year |\n",
                "|---------|----------|-------------|\n",
                "| iPhone | Smartphone | 2007 |\n",
                "| iPad | Tablet | 2010 |\n",
                "| Mac | Computer | 1984 |\n",
                "| Apple Watch | Wearable | 2015 |\n",
                "| AirPods | Audio | 2016 |\n",
                "\n",
                "These products have revolutionized their respective categories.\n",
                "\"\"\"\n",
                "\n",
                "# Table chunking\n",
                "table_chunker = TableChunker(\n",
                "    preserve_headers=True,\n",
                "    max_rows_per_chunk=3,\n",
                "    include_context=True,\n",
                "    table_format=\"markdown\"\n",
                ")\n",
                "\n",
                "chunks = table_chunker.chunk(text_with_table)\n",
                "\n",
                "print(\"Table Chunking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "for i, chunk in enumerate(chunks, 1):\n",
                "    is_table = chunk.metadata.get('is_table', False)\n",
                "    \n",
                "    print(f\"\\nChunk {i}:\")\n",
                "    print(f\"  Type: {'Table' if is_table else 'Text'}\")\n",
                "    \n",
                "    if is_table:\n",
                "        rows = chunk.metadata.get('row_count', 'N/A')\n",
                "        cols = chunk.metadata.get('column_count', 'N/A')\n",
                "        print(f\"  Rows: {rows}, Columns: {cols}\")\n",
                "    \n",
                "    print(f\"  Content: {chunk.text[:100]}...\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 11: Chunk Validation\n",
                "\n",
                "Validate chunk quality to ensure optimal processing.\n",
                "\n",
                "### Validation Checks\n",
                "\n",
                "- **Size Constraints**: Min/max chunk size\n",
                "- **Overlap**: Appropriate overlap percentage\n",
                "- **Completeness**: Full text coverage\n",
                "- **Quality Score**: Overall quality metric"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import ChunkValidator\n",
                "\n",
                "# Create chunks\n",
                "splitter = TextSplitter(method=\"recursive\", chunk_size=200, chunk_overlap=50)\n",
                "chunks = splitter.split(text)\n",
                "\n",
                "# Validate chunks\n",
                "validator = ChunkValidator(\n",
                "    min_chunk_size=50,\n",
                "    max_chunk_size=300,\n",
                "    min_overlap=20,\n",
                "    max_overlap=100\n",
                ")\n",
                "\n",
                "validation_result = validator.validate(chunks)\n",
                "\n",
                "print(\"Chunk Validation Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "print(f\"\\nOverall Valid: {validation_result.get('valid', False)}\")\n",
                "print(f\"Quality Score: {validation_result.get('quality_score', 0):.2f}\")\n",
                "\n",
                "issues = validation_result.get('issues', [])\n",
                "if issues:\n",
                "    print(f\"\\nIssues Found: {len(issues)}\")\n",
                "    for issue in issues[:3]:\n",
                "        print(f\"  - {issue}\")\n",
                "else:\n",
                "    print(\"\\nNo issues found!\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 12: Provenance Tracking\n",
                "\n",
                "Track chunk origins for data lineage and debugging.\n",
                "\n",
                "### Why Track Provenance?\n",
                "\n",
                "- **Data Lineage**: Know where chunks came from\n",
                "- **Debugging**: Trace issues back to source\n",
                "- **Compliance**: Required for some use cases"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.split import ProvenanceTracker\n",
                "\n",
                "# Create chunks\n",
                "splitter = TextSplitter(method=\"recursive\", chunk_size=200, chunk_overlap=50)\n",
                "chunks = splitter.split(text)\n",
                "\n",
                "# Track provenance\n",
                "tracker = ProvenanceTracker()\n",
                "\n",
                "for chunk in chunks:\n",
                "    tracker.track(\n",
                "        chunk=chunk,\n",
                "        source={\n",
                "            \"document_id\": \"apple_doc_001\",\n",
                "            \"file_path\": \"data/apple.txt\",\n",
                "            \"timestamp\": \"2024-01-01T00:00:00Z\",\n",
                "            \"method\": \"recursive\"\n",
                "        }\n",
                "    )\n",
                "\n",
                "print(\"Provenance Tracking Results:\\n\")\n",
                "print(\"=\" * 80)\n",
                "\n",
                "# Get lineage for first chunk\n",
                "if chunks:\n",
                "    lineage = tracker.get_lineage(chunks[0].id)\n",
                "    \n",
                "    print(f\"\\nLineage for Chunk 1:\")\n",
                "    print(f\"  Source Document: {lineage.get('source', {}).get('document_id')}\")\n",
                "    print(f\"  File Path: {lineage.get('source', {}).get('file_path')}\")\n",
                "    print(f\"  Method: {lineage.get('source', {}).get('method')}\")\n",
                "    print(f\"  Timestamp: {lineage.get('source', {}).get('timestamp')}\")\n",
                "\n",
                "print(\"\\n\" + \"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 13: Method Comparison\n",
                "\n",
                "Let's compare all methods side-by-side to help you choose the right one.\n",
                "\n",
                "### Comparison Criteria\n",
                "\n",
                "- **Chunk Count**: Number of chunks created\n",
                "- **Average Size**: Average chunk size\n",
                "- **Processing Time**: Speed of chunking"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import time\n",
                "\n",
                "# Methods to compare\n",
                "methods_to_compare = [\n",
                "    (\"recursive\", {}),\n",
                "    (\"sentence\", {}),\n",
                "    (\"paragraph\", {}),\n",
                "    (\"token\", {\"tokenizer\": \"tiktoken\"}),\n",
                "]\n",
                "\n",
                "print(\"Method Comparison:\\n\")\n",
                "print(\"=\" * 80)\n",
                "print(f\"{'Method':<15} {'Chunks':<10} {'Avg Size':<12} {'Time (ms)':<12}\")\n",
                "print(\"-\" * 80)\n",
                "\n",
                "for method, kwargs in methods_to_compare:\n",
                "    try:\n",
                "        start_time = time.time()\n",
                "        \n",
                "        splitter = TextSplitter(\n",
                "            method=method,\n",
                "            chunk_size=200,\n",
                "            chunk_overlap=50,\n",
                "            **kwargs\n",
                "        )\n",
                "        \n",
                "        chunks = splitter.split(text)\n",
                "        \n",
                "        elapsed = (time.time() - start_time) * 1000\n",
                "        avg_size = sum(len(c.text) for c in chunks) / len(chunks) if chunks else 0\n",
                "        \n",
                "        print(f\"{method:<15} {len(chunks):<10} {avg_size:<12.0f} {elapsed:<12.2f}\")\n",
                "        \n",
                "    except Exception as e:\n",
                "        print(f\"{method:<15} Error: {str(e)[:40]}\")\n",
                "\n",
                "print(\"=\" * 80)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 14: Best Practices\n",
                "\n",
                "### Choosing the Right Method\n",
                "\n",
                "1. **General Documents**: Use `recursive` for speed and simplicity\n",
                "2. **LLM Applications**: Use `token` to respect context windows\n",
                "3. **Semantic Search**: Use `semantic_transformer` for topic coherence\n",
                "4. **GraphRAG**: Use `entity_aware` or `relation_aware`\n",
                "5. **Structured Docs**: Use `structural` for formatted documents\n",
                "6. **Large Documents**: Use `hierarchical` for multi-level access\n",
                "\n",
                "### Chunk Size Guidelines\n",
                "\n",
                "| Use Case | Recommended Size | Overlap |\n",
                "|----------|------------------|----------|\n",
                "| Semantic Search | 512-1024 chars | 20% |\n",
                "| LLM Context | 2000-4000 chars | 10-20% |\n",
                "| Entity Extraction | 500-1500 chars | 15-25% |\n",
                "| Question Answering | 1000-2000 chars | 20% |\n",
                "\n",
                "### Overlap Recommendations\n",
                "\n",
                "- **10-15%**: Fast processing, less redundancy\n",
                "- **20-25%**: Balanced (recommended)\n",
                "- **30-40%**: Maximum context preservation"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Summary\n",
                "\n",
                "### What You've Learned\n",
                "\n",
                "In this notebook, you've learned how to:\n",
                "\n",
                "- Use `TextSplitter` with multiple methods\n",
                "- Apply standard splitting (recursive, token, sentence, paragraph)\n",
                "- Use semantic chunking for topic coherence\n",
                "- Apply KG-aware chunking (entity-aware, relation-aware)\n",
                "- Use specialized chunkers (structural, hierarchical, sliding window, table)\n",
                "- Validate chunk quality\n",
                "- Track provenance\n",
                "- Choose the right method for your use case\n",
                "\n",
                "### Key Takeaways\n",
                "\n",
                "1. **Method Selection Matters**: Different methods for different needs\n",
                "2. **Chunk Size is Critical**: Balance between context and processing\n",
                "3. **Overlap Helps**: 20% overlap is a good default\n",
                "4. **Validate Quality**: Always validate chunks before use\n",
                "5. **Track Provenance**: Important for debugging and compliance\n",
                "6. **KG-Aware for GraphRAG**: Use entity/relation-aware for knowledge graphs\n",
                "\n",
                "### Next Steps\n",
                "\n",
                "**Next Notebook**: [12_Embedding_Generation.ipynb](./12_Embedding_Generation.ipynb)  \n",
                "Learn how to generate embeddings for your chunks!\n",
                "\n",
                "**Further Reading**:\n",
                "- [Split Module API Reference](https://semantica.readthedocs.io/reference/split/)\n",
                "- [Advanced Chunking Strategies](../advanced/11_Text_Chunking_Strategies.ipynb)\n",
                "- [GraphRAG Pipeline](../use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n",
                "\n",
                "---\n",
                "\n",
                "**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.10.0"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}