semantica/docs/cookbook/advanced/Text_Chunking_Strategies.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Text Chunking Strategies\n",
        "\n",
        "## Overview\n",
        "\n",
        "Explore different text chunking strategies: semantic, structural, sliding window, and table chunking for optimal document processing.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from semantica.parse import TextSplitter\n",
        "import re\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 1: Prepare Sample Document\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "document = \"\"\"\n",
        "# Introduction to Knowledge Graphs\n",
        "\n",
        "Knowledge graphs are powerful data structures that represent information as entities and their relationships. \n",
        "They enable semantic understanding and reasoning over complex data.\n",
        "\n",
        "## What are Knowledge Graphs?\n",
        "\n",
        "A knowledge graph is a graph-based data model used to represent knowledge. It consists of nodes (entities) \n",
        "and edges (relationships) that connect these entities. Knowledge graphs are widely used in search engines, \n",
        "recommendation systems, and AI applications.\n",
        "\n",
        "## Applications\n",
        "\n",
        "Knowledge graphs have numerous applications:\n",
        "- Search engines use them to understand user queries\n",
        "- Recommendation systems leverage them for personalized suggestions\n",
        "- AI systems use them for reasoning and inference\n",
        "\n",
        "## Conclusion\n",
        "\n",
        "In summary, knowledge graphs provide a flexible and powerful way to represent and reason about complex information.\n",
        "\"\"\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 2: Semantic Chunking\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class SemanticChunker:\n",
        "    def chunk(self, document, chunk_size=500):\n",
        "        paragraphs = [p.strip() for p in document.split('\\n\\n') if p.strip()]\n",
        "        \n",
        "        chunks = []\n",
        "        current_chunk = \"\"\n",
        "        \n",
        "        for para in paragraphs:\n",
        "            if len(current_chunk) + len(para) <= chunk_size:\n",
        "                current_chunk += para + \"\\n\\n\"\n",
        "            else:\n",
        "                if current_chunk:\n",
        "                    chunks.append(current_chunk.strip())\n",
        "                current_chunk = para + \"\\n\\n\"\n",
        "        \n",
        "        if current_chunk:\n",
        "            chunks.append(current_chunk.strip())\n",
        "        \n",
        "        return chunks\n",
        "\n",
        "semantic_chunker = SemanticChunker()\n",
        "semantic_chunks = semantic_chunker.chunk(document, chunk_size=500)\n",
        "\n",
        "for i, chunk in enumerate(semantic_chunks, 1):\n",
        "    print(f\"Chunk {i}: {len(chunk)} characters\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 3: Structural Chunking\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class StructuralChunker:\n",
        "    def chunk(self, document):\n",
        "        chunks = []\n",
        "        current_section = \"\"\n",
        "        current_header = \"\"\n",
        "        \n",
        "        lines = document.split('\\n')\n",
        "        \n",
        "        for line in lines:\n",
        "            if line.startswith('#'):\n",
        "                if current_section:\n",
        "                    chunks.append({\n",
        "                        'header': current_header,\n",
        "                        'content': current_section.strip()\n",
        "                    })\n",
        "                current_header = line.strip()\n",
        "                current_section = \"\"\n",
        "            else:\n",
        "                current_section += line + \"\\n\"\n",
        "        \n",
        "        if current_section:\n",
        "            chunks.append({\n",
        "                'header': current_header,\n",
        "                'content': current_section.strip()\n",
        "            })\n",
        "        \n",
        "        return chunks\n",
        "\n",
        "structural_chunker = StructuralChunker()\n",
        "structural_chunks = structural_chunker.chunk(document)\n",
        "\n",
        "for i, chunk in enumerate(structural_chunks, 1):\n",
        "    header = chunk['header'][:50] if chunk['header'] else \"No header\"\n",
        "    print(f\"Chunk {i}: {header}... ({len(chunk['content'])} chars)\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 4: Sliding Window Chunking\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class SlidingWindowChunker:\n",
        "    def chunk(self, document, window_size=200, overlap=50):\n",
        "        words = document.split()\n",
        "        chunks = []\n",
        "        \n",
        "        start = 0\n",
        "        while start < len(words):\n",
        "            end = min(start + window_size, len(words))\n",
        "            chunk_words = words[start:end]\n",
        "            chunks.append(' '.join(chunk_words))\n",
        "            \n",
        "            start += window_size - overlap\n",
        "        \n",
        "        return chunks\n",
        "\n",
        "sliding_chunker = SlidingWindowChunker()\n",
        "sliding_chunks = sliding_chunker.chunk(document, window_size=200, overlap=50)\n",
        "\n",
        "for i, chunk in enumerate(sliding_chunks[:3], 1):\n",
        "    print(f\"Chunk {i}: {len(chunk)} characters\")\n",
        "if len(sliding_chunks) > 3:\n",
        "    print(f\"... and {len(sliding_chunks) - 3} more chunks\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 5: Table Chunking\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "class TableChunker:\n",
        "    def chunk(self, table_data):\n",
        "        if isinstance(table_data, str):\n",
        "            rows = [row.strip() for row in table_data.split('\\n') if row.strip()]\n",
        "            chunks = []\n",
        "            for row in rows:\n",
        "                if '|' in row:\n",
        "                    chunks.append(row)\n",
        "            return chunks\n",
        "        elif isinstance(table_data, list):\n",
        "            return [str(row) for row in table_data]\n",
        "        else:\n",
        "            return [str(table_data)]\n",
        "\n",
        "table_data = \"\"\"\n",
        "| Name | Age | Role |\n",
        "|------|-----|------|\n",
        "| Alice | 30 | Engineer |\n",
        "| Bob | 35 | Manager |\n",
        "| Charlie | 28 | Developer |\n",
        "\"\"\"\n",
        "\n",
        "table_chunker = TableChunker()\n",
        "table_chunks = table_chunker.chunk(table_data)\n",
        "\n",
        "for i, chunk in enumerate(table_chunks, 1):\n",
        "    print(f\"Chunk {i}: {chunk[:50]}...\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "Chunking strategies:\n",
        "- Semantic Chunking (by meaning/paragraphs)\n",
        "- Structural Chunking (by document structure)\n",
        "- Sliding Window Chunking (with overlap)\n",
        "- Table Chunking (for structured data)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"Text Chunking Strategies Complete\")\n"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}