mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
KaifAhmad1
7a6f1d0417
Add a Cite Us section to the README with BibTeX citation info, and align it with docs/citation.md (author/organization: Semantica, 2026). Update LICENSE and docs/project-license.md copyright holder to Semantica, and replace the stale Hawksight-AI GitHub org slug with semantica-agi across READMEs, plugin manifests, cookbook notebooks, and GitHub templates.
846 lines
28 KiB
Plaintext
846 lines
28 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)\n",
|
|
"\n",
|
|
"# Chunking and Splitting - Comprehensive Guide\n",
|
|
"\n",
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook provides a **comprehensive walkthrough** of Semantica's split module, demonstrating all chunking strategies and methods for optimal document processing. You'll learn to use 15+ splitting methods including standard, semantic, and knowledge graph-aware approaches.\n",
|
|
"\n",
|
|
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/split/)\n",
|
|
"\n",
|
|
"### Learning Objectives\n",
|
|
"\n",
|
|
"By the end of this notebook, you will be able to:\n",
|
|
"\n",
|
|
"- Use `TextSplitter` with multiple methods\n",
|
|
"- Apply standard splitting methods (recursive, token, sentence, paragraph)\n",
|
|
"- Use semantic chunking for topic coherence\n",
|
|
"- Apply KG-aware chunking (entity-aware, relation-aware, graph-based)\n",
|
|
"- Use specialized chunkers (structural, sliding window, table, hierarchical)\n",
|
|
"- Track provenance with `ProvenanceTracker`\n",
|
|
"- Choose the right method for your use case\n",
|
|
"\n",
|
|
"### What You'll Learn\n",
|
|
"\n",
|
|
"| Component | Purpose | When to Use |\n",
|
|
"|-----------|---------|-------------|\n",
|
|
"| `TextSplitter` | Unified splitter | All chunking needs |\n",
|
|
"| `SemanticChunker` | Semantic boundaries | Topic-based chunks |\n",
|
|
"| `EntityAwareChunker` | Preserve entities | GraphRAG workflows |\n",
|
|
"| `RelationAwareChunker` | Preserve triplets | KG construction |\n",
|
|
"| `StructuralChunker` | Document structure | Formatted documents |\n",
|
|
"| `HierarchicalChunker` | Multi-level chunks | Large documents |\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"## Installation\n",
|
|
"\n",
|
|
"Install Semantica from PyPI:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"pip install semantica\n",
|
|
"# Or with all optional dependencies:\n",
|
|
"pip install semantica[all]\n",
|
|
"```\n",
|
|
"\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install -q semantica\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 1: Basic Chunking with TextSplitter\n",
|
|
"\n",
|
|
"Let's start with the unified `TextSplitter` interface, which provides access to all chunking methods.\n",
|
|
"\n",
|
|
"### What is TextSplitter?\n",
|
|
"\n",
|
|
"`TextSplitter` is a unified interface that supports 15+ chunking methods:\n",
|
|
"- **Standard**: recursive, token, sentence, paragraph, character, word\n",
|
|
"- **Semantic**: semantic_transformer, llm, huggingface, nltk\n",
|
|
"- **KG/Ontology**: entity_aware, relation_aware, graph_based, ontology_aware\n",
|
|
"- **Advanced**: hierarchical, structural, sliding_window, table"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import TextSplitter\n",
|
|
"\n",
|
|
"# Sample long text\n",
|
|
"text = \"\"\"\n",
|
|
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
|
|
"in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
|
|
"over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.\n",
|
|
"\n",
|
|
"Apple develops and sells consumer electronics, computer software, and online services. The company's \n",
|
|
"hardware products include the iPhone smartphone, the iPad tablet computer, the Mac personal computer, \n",
|
|
"the iPod portable media player, the Apple Watch smartwatch, the Apple TV digital media player, and the \n",
|
|
"HomePod smart speaker.\n",
|
|
"\n",
|
|
"Apple's software includes the macOS and iOS operating systems, the iTunes media player, the Safari web \n",
|
|
"browser, and the iLife and iWork creativity and productivity suites. Its online services include the \n",
|
|
"iTunes Store, the iOS App Store and Mac App Store, Apple Music, and iCloud.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# Basic recursive splitting\n",
|
|
"splitter = TextSplitter(\n",
|
|
" method=\"recursive\",\n",
|
|
" chunk_size=200,\n",
|
|
" chunk_overlap=50\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = splitter.split(text)\n",
|
|
"\n",
|
|
"print(f\"Split into {len(chunks)} chunks using recursive method\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Length: {len(chunk.text)} characters\")\n",
|
|
" print(f\" Start: {chunk.start_index}, End: {chunk.end_index}\")\n",
|
|
" print(f\" Text: {chunk.text[:100]}...\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 2: Standard Splitting Methods\n",
|
|
"\n",
|
|
"Let's compare different standard splitting methods.\n",
|
|
"\n",
|
|
"### Method Comparison\n",
|
|
"\n",
|
|
"| Method | Best For | Speed | Accuracy |\n",
|
|
"|--------|----------|-------|----------|\n",
|
|
"| **recursive** | General text | Fast | Good |\n",
|
|
"| **sentence** | Coherent chunks | Medium | Very Good |\n",
|
|
"| **token** | LLM context | Medium | Excellent |\n",
|
|
"| **paragraph** | Natural breaks | Fast | Good |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Compare different methods\n",
|
|
"methods = [\"recursive\", \"sentence\", \"paragraph\"]\n",
|
|
"\n",
|
|
"print(\"Comparing Standard Splitting Methods:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for method in methods:\n",
|
|
" splitter = TextSplitter(\n",
|
|
" method=method,\n",
|
|
" chunk_size=200,\n",
|
|
" chunk_overlap=50\n",
|
|
" )\n",
|
|
" \n",
|
|
" chunks = splitter.split(text)\n",
|
|
" \n",
|
|
" print(f\"\\nMethod: {method.upper()}\")\n",
|
|
" print(\"-\" * 40)\n",
|
|
" print(f\" Chunks created: {len(chunks)}\")\n",
|
|
" print(f\" Avg chunk size: {sum(len(c.text) for c in chunks) / len(chunks):.0f} chars\")\n",
|
|
" print(f\" First chunk: {chunks[0].text[:80]}...\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 3: Token-Based Splitting\n",
|
|
"\n",
|
|
"Token-based splitting is crucial for LLM applications where you need to respect token limits.\n",
|
|
"\n",
|
|
"### Why Token-Based?\n",
|
|
"\n",
|
|
"- **LLM Context Windows**: GPT-4 has 8K/32K token limits\n",
|
|
"- **Accurate Counting**: Character count ≠ token count\n",
|
|
"- **Cost Optimization**: Tokens determine API costs"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import split_by_tokens\n",
|
|
"\n",
|
|
"# Token-based splitting\n",
|
|
"chunks = split_by_tokens(\n",
|
|
" text,\n",
|
|
" chunk_size=100, # 100 tokens\n",
|
|
" chunk_overlap=20,\n",
|
|
" tokenizer=\"tiktoken\",\n",
|
|
" model=\"gpt-4\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\"Token-Based Splitting Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" token_count = chunk.metadata.get('token_count', 'N/A')\n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Tokens: {token_count}\")\n",
|
|
" print(f\" Characters: {len(chunk.text)}\")\n",
|
|
" print(f\" Ratio: {len(chunk.text)/token_count if token_count != 'N/A' else 'N/A':.2f} chars/token\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 4: Semantic Chunking\n",
|
|
"\n",
|
|
"Semantic chunking creates chunks based on semantic boundaries using embeddings.\n",
|
|
"\n",
|
|
"### How It Works\n",
|
|
"\n",
|
|
"1. Split text into sentences\n",
|
|
"2. Generate embeddings for each sentence\n",
|
|
"3. Calculate similarity between consecutive sentences\n",
|
|
"4. Create boundaries where similarity drops below threshold"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import SemanticChunker\n",
|
|
"\n",
|
|
"# Semantic chunking\n",
|
|
"semantic_chunker = SemanticChunker(\n",
|
|
" chunk_size=200,\n",
|
|
" chunk_overlap=50,\n",
|
|
" embedding_model=\"all-MiniLM-L6-v2\",\n",
|
|
" similarity_threshold=0.7\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = semantic_chunker.chunk(text)\n",
|
|
"\n",
|
|
"print(\"Semantic Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" coherence = chunk.metadata.get('coherence_score', 'N/A')\n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Length: {len(chunk.text)} chars\")\n",
|
|
" print(f\" Coherence: {coherence}\")\n",
|
|
" print(f\" Text: {chunk.text[:100]}...\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 5: Entity-Aware Chunking for GraphRAG\n",
|
|
"\n",
|
|
"Entity-aware chunking preserves entity boundaries, crucial for GraphRAG workflows.\n",
|
|
"\n",
|
|
"### Why Entity-Aware?\n",
|
|
"\n",
|
|
"- **Preserve Entities**: Don't split \"Steve Jobs\" across chunks\n",
|
|
"- **Better Extraction**: Complete entities improve NER accuracy\n",
|
|
"- **GraphRAG**: Essential for knowledge graph construction"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import EntityAwareChunker\n",
|
|
"\n",
|
|
"# Entity-aware chunking\n",
|
|
"entity_chunker = EntityAwareChunker(\n",
|
|
" chunk_size=200,\n",
|
|
" chunk_overlap=50,\n",
|
|
" ner_method=\"ml\", # \"ml\" (spaCy), \"pattern\", or \"llm\"\n",
|
|
" preserve_entities=True\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = entity_chunker.chunk(text)\n",
|
|
"\n",
|
|
"print(\"Entity-Aware Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" entities = chunk.metadata.get('entities', [])\n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Length: {len(chunk.text)} chars\")\n",
|
|
" print(f\" Entities: {len(entities)}\")\n",
|
|
" \n",
|
|
" if entities:\n",
|
|
" # Handle both Entity objects and dicts\n",
|
|
" entity_texts = [e.get('text', e.get('entity', '')) if isinstance(e, dict) else str(e) for e in entities[:3]]\n",
|
|
" print(f\" Sample entities: {entity_texts}\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 6: Relation-Aware Chunking\n",
|
|
"\n",
|
|
"Relation-aware chunking preserves relationship triplets within chunks.\n",
|
|
"\n",
|
|
"### Why Relation-Aware?\n",
|
|
"\n",
|
|
"- **Preserve Triplets**: Keep (subject, predicate, object) together\n",
|
|
"- **KG Construction**: Better for building knowledge graphs\n",
|
|
"- **Context**: Relationships need complete context"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import RelationAwareChunker\n",
|
|
"\n",
|
|
"# Relation-aware chunking\n",
|
|
"relation_chunker = RelationAwareChunker(\n",
|
|
" chunk_size=200,\n",
|
|
" chunk_overlap=50,\n",
|
|
" preserve_triplets=True\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = relation_chunker.chunk(text)\n",
|
|
"\n",
|
|
"print(\"Relation-Aware Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" triplets = chunk.metadata.get('triplets', [])\n",
|
|
" relationships = chunk.metadata.get('relationships', [])\n",
|
|
" \n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Length: {len(chunk.text)} chars\")\n",
|
|
" print(f\" Triplets: {len(triplets)}\")\n",
|
|
" print(f\" Relationships: {len(relationships)}\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 7: Structural Chunking\n",
|
|
"\n",
|
|
"Structural chunking respects document structure like headings, paragraphs, and lists.\n",
|
|
"\n",
|
|
"### When to Use?\n",
|
|
"\n",
|
|
"- **Formatted Documents**: Markdown, HTML, structured text\n",
|
|
"- **Preserve Hierarchy**: Keep sections together\n",
|
|
"- **Better Context**: Headings provide context"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import StructuralChunker\n",
|
|
"\n",
|
|
"# Markdown text with structure\n",
|
|
"markdown_text = \"\"\"\n",
|
|
"# Apple Inc.\n",
|
|
"\n",
|
|
"## History\n",
|
|
"\n",
|
|
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
|
|
"\n",
|
|
"## Products\n",
|
|
"\n",
|
|
"### Hardware\n",
|
|
"- iPhone\n",
|
|
"- iPad\n",
|
|
"- Mac\n",
|
|
"\n",
|
|
"### Software\n",
|
|
"- macOS\n",
|
|
"- iOS\n",
|
|
"- Safari\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# Structural chunking\n",
|
|
"structural_chunker = StructuralChunker(\n",
|
|
" respect_headings=True,\n",
|
|
" respect_paragraphs=True,\n",
|
|
" respect_lists=True,\n",
|
|
" max_chunk_size=500\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = structural_chunker.chunk(markdown_text)\n",
|
|
"\n",
|
|
"print(\"Structural Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" section = chunk.metadata.get('section_title', 'N/A')\n",
|
|
" level = chunk.metadata.get('heading_level', 'N/A')\n",
|
|
" \n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Section: {section}\")\n",
|
|
" print(f\" Level: {level}\")\n",
|
|
" print(f\" Text: {chunk.text[:80]}...\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 8: Hierarchical Chunking\n",
|
|
"\n",
|
|
"Hierarchical chunking creates multi-level chunks for large documents.\n",
|
|
"\n",
|
|
"### Benefits\n",
|
|
"\n",
|
|
"- **Multiple Granularities**: Document → Section → Paragraph\n",
|
|
"- **Better Navigation**: Parent-child relationships\n",
|
|
"- **Flexible Retrieval**: Query at different levels"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import HierarchicalChunker\n",
|
|
"\n",
|
|
"# Hierarchical chunking\n",
|
|
"hierarchical_chunker = HierarchicalChunker(\n",
|
|
" chunk_sizes=[400, 200, 100], # 3 levels\n",
|
|
" chunk_overlaps=[80, 40, 20],\n",
|
|
" create_parent_chunks=True\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = hierarchical_chunker.chunk(text)\n",
|
|
"\n",
|
|
"print(\"Hierarchical Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" level = chunk.metadata.get('level', 'N/A')\n",
|
|
" parent_id = chunk.metadata.get('parent_id', None)\n",
|
|
" child_ids = chunk.metadata.get('child_ids', [])\n",
|
|
" \n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Level: {level}\")\n",
|
|
" print(f\" Length: {len(chunk.text)} chars\")\n",
|
|
" print(f\" Parent: {parent_id if parent_id else 'None (root)'}\")\n",
|
|
" print(f\" Children: {len(child_ids)}\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 9: Sliding Window Chunking\n",
|
|
"\n",
|
|
"Sliding window creates overlapping fixed-size chunks.\n",
|
|
"\n",
|
|
"### Use Cases\n",
|
|
"\n",
|
|
"- **Dense Retrieval**: Ensure no information is missed\n",
|
|
"- **Fixed Context**: Consistent chunk sizes\n",
|
|
"- **Overlap Control**: Precise overlap management"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import SlidingWindowChunker\n",
|
|
"\n",
|
|
"# Sliding window chunking\n",
|
|
"sliding_chunker = SlidingWindowChunker(\n",
|
|
" chunk_size=150,\n",
|
|
" overlap=50\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = sliding_chunker.chunk(text)\n",
|
|
"\n",
|
|
"print(\"Sliding Window Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" # Calculate overlap manually\n",
|
|
" overlap = 0\n",
|
|
" if i > 1:\n",
|
|
" prev_chunk = chunks[i-2]\n",
|
|
" overlap = max(0, prev_chunk.end_index - chunk.start_index)\n",
|
|
" \n",
|
|
" print(f\"\\nWindow {i}:\")\n",
|
|
" print(f\" Position: {chunk.start_index}-{chunk.end_index}\")\n",
|
|
" print(f\" Length: {len(chunk.text)} chars\")\n",
|
|
" print(f\" Overlap with previous: {overlap} chars\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 10: Table Chunking\n",
|
|
"\n",
|
|
"Table chunking preserves table structure while splitting large tables.\n",
|
|
"\n",
|
|
"### Features\n",
|
|
"\n",
|
|
"- **Preserve Headers**: Keep column headers in each chunk\n",
|
|
"- **Row-Based Splitting**: Split by rows, not characters\n",
|
|
"- **Context Inclusion**: Include surrounding text"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.split import TableChunker\n",
|
|
"\n",
|
|
"# Text with table\n",
|
|
"text_with_table = \"\"\"\n",
|
|
"Apple's product lineup includes:\n",
|
|
"\n",
|
|
"| Product | Category | Release Year |\n",
|
|
"|---------|----------|-------------|\n",
|
|
"| iPhone | Smartphone | 2007 |\n",
|
|
"| iPad | Tablet | 2010 |\n",
|
|
"| Mac | Computer | 1984 |\n",
|
|
"| Apple Watch | Wearable | 2015 |\n",
|
|
"| AirPods | Audio | 2016 |\n",
|
|
"\n",
|
|
"These products have revolutionized their respective categories.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# Table chunking\n",
|
|
"table_chunker = TableChunker(\n",
|
|
" preserve_headers=True,\n",
|
|
" max_rows_per_chunk=3,\n",
|
|
" include_context=True,\n",
|
|
" table_format=\"markdown\"\n",
|
|
")\n",
|
|
"\n",
|
|
"chunks = table_chunker.chunk(text_with_table)\n",
|
|
"\n",
|
|
"print(\"Table Chunking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"for i, chunk in enumerate(chunks, 1):\n",
|
|
" is_table = chunk.metadata.get('is_table', False)\n",
|
|
" \n",
|
|
" print(f\"\\nChunk {i}:\")\n",
|
|
" print(f\" Type: {'Table' if is_table else 'Text'}\")\n",
|
|
" \n",
|
|
" if is_table:\n",
|
|
" rows = chunk.metadata.get('row_count', 'N/A')\n",
|
|
" cols = chunk.metadata.get('column_count', 'N/A')\n",
|
|
" print(f\" Rows: {rows}, Columns: {cols}\")\n",
|
|
" \n",
|
|
" print(f\" Content: {chunk.text[:100]}...\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 11: Provenance Tracking\n",
|
|
"\n",
|
|
"Track chunk origins for data lineage and debugging.\n",
|
|
"\n",
|
|
"### Why Track Provenance?\n",
|
|
"\n",
|
|
"- **Data Lineage**: Know where chunks came from\n",
|
|
"- **Debugging**: Trace issues back to source\n",
|
|
"- **Compliance**: Required for some use cases"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"import os\n",
|
|
"import importlib\n",
|
|
"\n",
|
|
"# 1. Ensure local package is in path\n",
|
|
"project_root = os.path.abspath(os.path.join(os.getcwd(), \"../..\"))\n",
|
|
"if project_root not in sys.path:\n",
|
|
" sys.path.insert(0, project_root)\n",
|
|
"\n",
|
|
"# 2. Force unload modules to ensure clean reload\n",
|
|
"modules_to_unload = [\n",
|
|
" 'semantica.split.semantic_chunker', \n",
|
|
" 'semantica.split.splitter', \n",
|
|
" 'semantica.split.provenance_tracker',\n",
|
|
" 'semantica.split'\n",
|
|
"]\n",
|
|
"for module in modules_to_unload:\n",
|
|
" if module in sys.modules:\n",
|
|
" del sys.modules[module]\n",
|
|
"\n",
|
|
"# 3. Import fresh modules\n",
|
|
"import semantica.split.semantic_chunker\n",
|
|
"import semantica.split.splitter\n",
|
|
"import semantica.split.provenance_tracker\n",
|
|
"from semantica.split import ProvenanceTracker, TextSplitter\n",
|
|
"\n",
|
|
"# 4. Verify Chunk class has id field\n",
|
|
"from semantica.split.semantic_chunker import Chunk\n",
|
|
"print(f\"Chunk class fields: {Chunk.__annotations__}\")\n",
|
|
"if 'id' not in Chunk.__annotations__:\n",
|
|
" print(\"WARNING: Chunk class still missing 'id' field. Kernel restart required.\")\n",
|
|
"\n",
|
|
"# Create chunks\n",
|
|
"splitter = TextSplitter(method=\"recursive\", chunk_size=200, chunk_overlap=50)\n",
|
|
"chunks = splitter.split(text)\n",
|
|
"\n",
|
|
"# Track provenance\n",
|
|
"tracker = ProvenanceTracker()\n",
|
|
"\n",
|
|
"for chunk in chunks:\n",
|
|
" tracker.track_chunk(\n",
|
|
" chunk=chunk,\n",
|
|
" source_document=\"apple_doc_001\",\n",
|
|
" source_path=\"data/apple.txt\",\n",
|
|
" timestamp=\"2024-01-01T00:00:00Z\",\n",
|
|
" method=\"recursive\"\n",
|
|
" )\n",
|
|
"\n",
|
|
"print(\"Provenance Tracking Results:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"\n",
|
|
"# Get lineage for first chunk\n",
|
|
"if chunks:\n",
|
|
" # Get provenance info using the chunk's ID\n",
|
|
" chunk_id = getattr(chunks[0], 'id', None)\n",
|
|
" print(f\"Chunk ID: {chunk_id}\")\n",
|
|
" \n",
|
|
" if chunk_id:\n",
|
|
" prov_info = tracker.get_provenance(chunk_id)\n",
|
|
" \n",
|
|
" if prov_info:\n",
|
|
" print(f\"\\nLineage for Chunk 1:\")\n",
|
|
" print(f\" Source Document: {prov_info.source_document}\")\n",
|
|
" print(f\" File Path: {prov_info.source_path}\")\n",
|
|
" print(f\" Method: {prov_info.metadata.get('method')}\")\n",
|
|
" print(f\" Timestamp: {prov_info.timestamp}\")\n",
|
|
" else:\n",
|
|
" print(\"Error: Chunk ID not found. The Chunk class definition might still be cached.\")\n",
|
|
" print(\"Please click 'Kernel' -> 'Restart Kernel' in the menu and run all cells again.\")\n",
|
|
"\n",
|
|
"print(\"\\n\" + \"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 13: Method Comparison\n",
|
|
"\n",
|
|
"Let's compare all methods side-by-side to help you choose the right one.\n",
|
|
"\n",
|
|
"### Comparison Criteria\n",
|
|
"\n",
|
|
"- **Chunk Count**: Number of chunks created\n",
|
|
"- **Average Size**: Average chunk size\n",
|
|
"- **Processing Time**: Speed of chunking"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import time\n",
|
|
"\n",
|
|
"# Methods to compare\n",
|
|
"methods_to_compare = [\n",
|
|
" (\"recursive\", {}),\n",
|
|
" (\"sentence\", {}),\n",
|
|
" (\"paragraph\", {}),\n",
|
|
" (\"token\", {\"tokenizer\": \"tiktoken\"}),\n",
|
|
"]\n",
|
|
"\n",
|
|
"print(\"Method Comparison:\\n\")\n",
|
|
"print(\"=\" * 80)\n",
|
|
"print(f\"{'Method':<15} {'Chunks':<10} {'Avg Size':<12} {'Time (ms)':<12}\")\n",
|
|
"print(\"-\" * 80)\n",
|
|
"\n",
|
|
"for method, kwargs in methods_to_compare:\n",
|
|
" try:\n",
|
|
" start_time = time.time()\n",
|
|
" \n",
|
|
" splitter = TextSplitter(\n",
|
|
" method=method,\n",
|
|
" chunk_size=200,\n",
|
|
" chunk_overlap=50,\n",
|
|
" **kwargs\n",
|
|
" )\n",
|
|
" \n",
|
|
" chunks = splitter.split(text)\n",
|
|
" \n",
|
|
" elapsed = (time.time() - start_time) * 1000\n",
|
|
" avg_size = sum(len(c.text) for c in chunks) / len(chunks) if chunks else 0\n",
|
|
" \n",
|
|
" print(f\"{method:<15} {len(chunks):<10} {avg_size:<12.0f} {elapsed:<12.2f}\")\n",
|
|
" \n",
|
|
" except Exception as e:\n",
|
|
" print(f\"{method:<15} Error: {str(e)[:40]}\")\n",
|
|
"\n",
|
|
"print(\"=\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 14: Best Practices\n",
|
|
"\n",
|
|
"### Choosing the Right Method\n",
|
|
"\n",
|
|
"1. **General Documents**: Use `recursive` for speed and simplicity\n",
|
|
"2. **LLM Applications**: Use `token` to respect context windows\n",
|
|
"3. **Semantic Search**: Use `semantic_transformer` for topic coherence\n",
|
|
"4. **GraphRAG**: Use `entity_aware` or `relation_aware`\n",
|
|
"5. **Structured Docs**: Use `structural` for formatted documents\n",
|
|
"6. **Large Documents**: Use `hierarchical` for multi-level access\n",
|
|
"\n",
|
|
"### Chunk Size Guidelines\n",
|
|
"\n",
|
|
"| Use Case | Recommended Size | Overlap |\n",
|
|
"|----------|------------------|----------|\n",
|
|
"| Semantic Search | 512-1024 chars | 20% |\n",
|
|
"| LLM Context | 2000-4000 chars | 10-20% |\n",
|
|
"| Entity Extraction | 500-1500 chars | 15-25% |\n",
|
|
"| Question Answering | 1000-2000 chars | 20% |\n",
|
|
"\n",
|
|
"### Overlap Recommendations\n",
|
|
"\n",
|
|
"- **10-15%**: Fast processing, less redundancy\n",
|
|
"- **20-25%**: Balanced (recommended)\n",
|
|
"- **30-40%**: Maximum context preservation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"### What You've Learned\n",
|
|
"\n",
|
|
"In this notebook, you've learned how to:\n",
|
|
"\n",
|
|
"- Use `TextSplitter` with multiple methods\n",
|
|
"- Apply standard splitting (recursive, token, sentence, paragraph)\n",
|
|
"- Use semantic chunking for topic coherence\n",
|
|
"- Apply KG-aware chunking (entity-aware, relation-aware)\n",
|
|
"- Use specialized chunkers (structural, hierarchical, sliding window, table)\n",
|
|
"- Track provenance\n",
|
|
"- Choose the right method for your use case\n",
|
|
"\n",
|
|
"### Key Takeaways\n",
|
|
"\n",
|
|
"1. **Method Selection Matters**: Different methods for different needs\n",
|
|
"2. **Chunk Size is Critical**: Balance between context and processing\n",
|
|
"3. **Overlap Helps**: 20% overlap is a good default\n",
|
|
"4. **Track Provenance**: Important for debugging and compliance\n",
|
|
"5. **KG-Aware for GraphRAG**: Use entity/relation-aware for knowledge graphs\n",
|
|
"\n",
|
|
"### Next Steps\n",
|
|
"\n",
|
|
"**Next Notebook**: [12_Embedding_Generation.ipynb](./12_Embedding_Generation.ipynb) \n",
|
|
"Learn how to generate embeddings for your chunks!\n",
|
|
"\n",
|
|
"**Further Reading**:\n",
|
|
"- [Split Module API Reference](https://semantica.readthedocs.io/reference/split/)\n",
|
|
"- [Advanced Chunking Strategies](../advanced/11_Text_Chunking_Strategies.ipynb)\n",
|
|
"- [GraphRAG Pipeline](../use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|