semantica/cookbook/introduction/11_Chunking_and_Splitting.ipynb

Chunking and Splitting - Comprehensive Guide

Overview

This notebook provides a comprehensive walkthrough of Semantica's split module, demonstrating all chunking strategies and methods for optimal document processing. You'll learn to use 15+ splitting methods including standard, semantic, and knowledge graph-aware approaches.

Documentation: API Reference

Learning Objectives

By the end of this notebook, you will be able to:

• Use TextSplitter with multiple methods
• Apply standard splitting methods (recursive, token, sentence, paragraph)
• Use semantic chunking for topic coherence
• Apply KG-aware chunking (entity-aware, relation-aware, graph-based)
• Use specialized chunkers (structural, sliding window, table, hierarchical)
• Track provenance with ProvenanceTracker
• Choose the right method for your use case

What You'll Learn

Component	Purpose	When to Use
TextSplitter	Unified splitter	All chunking needs
SemanticChunker	Semantic boundaries	Topic-based chunks
EntityAwareChunker	Preserve entities	GraphRAG workflows
RelationAwareChunker	Preserve triplets	KG construction
StructuralChunker	Document structure	Formatted documents
HierarchicalChunker	Multi-level chunks	Large documents

---

Installation

Install Semantica from PyPI:

pip install semantica
# Or with all optional dependencies:
pip install semantica[all]

---

In [ ]:

!pip install -q semantica

Step 1: Basic Chunking with TextSplitter

Let's start with the unified TextSplitter interface, which provides access to all chunking methods.

What is TextSplitter?

TextSplitter is a unified interface that supports 15+ chunking methods:

• Standard: recursive, token, sentence, paragraph, character, word
• Semantic: semantic_transformer, llm, huggingface, nltk
• KG/Ontology: entity_aware, relation_aware, graph_based, ontology_aware
• Advanced: hierarchical, structural, sliding_window, table

In [ ]:

from semantica.split import TextSplitter

# Sample long text
text = """
Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne 
in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took 
over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.

Apple develops and sells consumer electronics, computer software, and online services. The company's 
hardware products include the iPhone smartphone, the iPad tablet computer, the Mac personal computer, 
the iPod portable media player, the Apple Watch smartwatch, the Apple TV digital media player, and the 
HomePod smart speaker.

Apple's software includes the macOS and iOS operating systems, the iTunes media player, the Safari web 
browser, and the iLife and iWork creativity and productivity suites. Its online services include the 
iTunes Store, the iOS App Store and Mac App Store, Apple Music, and iCloud.
"""

# Basic recursive splitting
splitter = TextSplitter(
    method="recursive",
    chunk_size=200,
    chunk_overlap=50
)

chunks = splitter.split(text)

print(f"Split into {len(chunks)} chunks using recursive method\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    print(f"\nChunk {i}:")
    print(f"  Length: {len(chunk.text)} characters")
    print(f"  Start: {chunk.start_index}, End: {chunk.end_index}")
    print(f"  Text: {chunk.text[:100]}...")

print("\n" + "=" * 80)

Step 2: Standard Splitting Methods

Let's compare different standard splitting methods.

Method Comparison

Method	Best For	Speed	Accuracy
recursive	General text	Fast	Good
sentence	Coherent chunks	Medium	Very Good
token	LLM context	Medium	Excellent
paragraph	Natural breaks	Fast	Good

In [ ]:

# Compare different methods
methods = ["recursive", "sentence", "paragraph"]

print("Comparing Standard Splitting Methods:\n")
print("=" * 80)

for method in methods:
    splitter = TextSplitter(
        method=method,
        chunk_size=200,
        chunk_overlap=50
    )
    
    chunks = splitter.split(text)
    
    print(f"\nMethod: {method.upper()}")
    print("-" * 40)
    print(f"  Chunks created: {len(chunks)}")
    print(f"  Avg chunk size: {sum(len(c.text) for c in chunks) / len(chunks):.0f} chars")
    print(f"  First chunk: {chunks[0].text[:80]}...")

print("\n" + "=" * 80)

Step 3: Token-Based Splitting

Token-based splitting is crucial for LLM applications where you need to respect token limits.

Why Token-Based?

• LLM Context Windows: GPT-4 has 8K/32K token limits
• Accurate Counting: Character count ≠ token count
• Cost Optimization: Tokens determine API costs

In [ ]:

from semantica.split import split_by_tokens

# Token-based splitting
chunks = split_by_tokens(
    text,
    chunk_size=100,  # 100 tokens
    chunk_overlap=20,
    tokenizer="tiktoken",
    model="gpt-4"
)

print("Token-Based Splitting Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    token_count = chunk.metadata.get('token_count', 'N/A')
    print(f"\nChunk {i}:")
    print(f"  Tokens: {token_count}")
    print(f"  Characters: {len(chunk.text)}")
    print(f"  Ratio: {len(chunk.text)/token_count if token_count != 'N/A' else 'N/A':.2f} chars/token")

print("\n" + "=" * 80)

Step 4: Semantic Chunking

Semantic chunking creates chunks based on semantic boundaries using embeddings.

How It Works

1. Split text into sentences
2. Generate embeddings for each sentence
3. Calculate similarity between consecutive sentences
4. Create boundaries where similarity drops below threshold

In [ ]:

from semantica.split import SemanticChunker

# Semantic chunking
semantic_chunker = SemanticChunker(
    chunk_size=200,
    chunk_overlap=50,
    embedding_model="all-MiniLM-L6-v2",
    similarity_threshold=0.7
)

chunks = semantic_chunker.chunk(text)

print("Semantic Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    coherence = chunk.metadata.get('coherence_score', 'N/A')
    print(f"\nChunk {i}:")
    print(f"  Length: {len(chunk.text)} chars")
    print(f"  Coherence: {coherence}")
    print(f"  Text: {chunk.text[:100]}...")

print("\n" + "=" * 80)

Step 5: Entity-Aware Chunking for GraphRAG

Entity-aware chunking preserves entity boundaries, crucial for GraphRAG workflows.

Why Entity-Aware?

• Preserve Entities: Don't split "Steve Jobs" across chunks
• Better Extraction: Complete entities improve NER accuracy
• GraphRAG: Essential for knowledge graph construction

In [ ]:

from semantica.split import EntityAwareChunker

# Entity-aware chunking
entity_chunker = EntityAwareChunker(
    chunk_size=200,
    chunk_overlap=50,
    ner_method="ml",  # "ml" (spaCy), "pattern", or "llm"
    preserve_entities=True
)

chunks = entity_chunker.chunk(text)

print("Entity-Aware Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    entities = chunk.metadata.get('entities', [])
    print(f"\nChunk {i}:")
    print(f"  Length: {len(chunk.text)} chars")
    print(f"  Entities: {len(entities)}")
    
    if entities:
        # Handle both Entity objects and dicts
        entity_texts = [e.get('text', e.get('entity', '')) if isinstance(e, dict) else str(e) for e in entities[:3]]
        print(f"  Sample entities: {entity_texts}")

print("\n" + "=" * 80)

Step 6: Relation-Aware Chunking

Relation-aware chunking preserves relationship triplets within chunks.

Why Relation-Aware?

• Preserve Triplets: Keep (subject, predicate, object) together
• KG Construction: Better for building knowledge graphs
• Context: Relationships need complete context

In [ ]:

from semantica.split import RelationAwareChunker

# Relation-aware chunking
relation_chunker = RelationAwareChunker(
    chunk_size=200,
    chunk_overlap=50,
    preserve_triplets=True
)

chunks = relation_chunker.chunk(text)

print("Relation-Aware Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    triplets = chunk.metadata.get('triplets', [])
    relationships = chunk.metadata.get('relationships', [])
    
    print(f"\nChunk {i}:")
    print(f"  Length: {len(chunk.text)} chars")
    print(f"  Triplets: {len(triplets)}")
    print(f"  Relationships: {len(relationships)}")

print("\n" + "=" * 80)

Step 7: Structural Chunking

Structural chunking respects document structure like headings, paragraphs, and lists.

When to Use?

• Formatted Documents: Markdown, HTML, structured text
• Preserve Hierarchy: Keep sections together
• Better Context: Headings provide context

In [ ]:

from semantica.split import StructuralChunker

# Markdown text with structure
markdown_text = """
# Apple Inc.

## History

Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.

## Products

### Hardware
- iPhone
- iPad
- Mac

### Software
- macOS
- iOS
- Safari
"""

# Structural chunking
structural_chunker = StructuralChunker(
    respect_headings=True,
    respect_paragraphs=True,
    respect_lists=True,
    max_chunk_size=500
)

chunks = structural_chunker.chunk(markdown_text)

print("Structural Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    section = chunk.metadata.get('section_title', 'N/A')
    level = chunk.metadata.get('heading_level', 'N/A')
    
    print(f"\nChunk {i}:")
    print(f"  Section: {section}")
    print(f"  Level: {level}")
    print(f"  Text: {chunk.text[:80]}...")

print("\n" + "=" * 80)

Step 8: Hierarchical Chunking

Hierarchical chunking creates multi-level chunks for large documents.

Benefits

• Multiple Granularities: Document → Section → Paragraph
• Better Navigation: Parent-child relationships
• Flexible Retrieval: Query at different levels

In [ ]:

from semantica.split import HierarchicalChunker

# Hierarchical chunking
hierarchical_chunker = HierarchicalChunker(
    chunk_sizes=[400, 200, 100],  # 3 levels
    chunk_overlaps=[80, 40, 20],
    create_parent_chunks=True
)

chunks = hierarchical_chunker.chunk(text)

print("Hierarchical Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    level = chunk.metadata.get('level', 'N/A')
    parent_id = chunk.metadata.get('parent_id', None)
    child_ids = chunk.metadata.get('child_ids', [])
    
    print(f"\nChunk {i}:")
    print(f"  Level: {level}")
    print(f"  Length: {len(chunk.text)} chars")
    print(f"  Parent: {parent_id if parent_id else 'None (root)'}")
    print(f"  Children: {len(child_ids)}")

print("\n" + "=" * 80)

Step 9: Sliding Window Chunking

Sliding window creates overlapping fixed-size chunks.

Use Cases

• Dense Retrieval: Ensure no information is missed
• Fixed Context: Consistent chunk sizes
• Overlap Control: Precise overlap management

In [ ]:

from semantica.split import SlidingWindowChunker

# Sliding window chunking
sliding_chunker = SlidingWindowChunker(
    chunk_size=150,
    overlap=50
)

chunks = sliding_chunker.chunk(text)

print("Sliding Window Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    # Calculate overlap manually
    overlap = 0
    if i > 1:
        prev_chunk = chunks[i-2]
        overlap = max(0, prev_chunk.end_index - chunk.start_index)
    
    print(f"\nWindow {i}:")
    print(f"  Position: {chunk.start_index}-{chunk.end_index}")
    print(f"  Length: {len(chunk.text)} chars")
    print(f"  Overlap with previous: {overlap} chars")

print("\n" + "=" * 80)

Step 10: Table Chunking

Table chunking preserves table structure while splitting large tables.

Features

• Preserve Headers: Keep column headers in each chunk
• Row-Based Splitting: Split by rows, not characters
• Context Inclusion: Include surrounding text

In [ ]:

from semantica.split import TableChunker

# Text with table
text_with_table = """
Apple's product lineup includes:

| Product | Category | Release Year |
|---------|----------|-------------|
| iPhone | Smartphone | 2007 |
| iPad | Tablet | 2010 |
| Mac | Computer | 1984 |
| Apple Watch | Wearable | 2015 |
| AirPods | Audio | 2016 |

These products have revolutionized their respective categories.
"""

# Table chunking
table_chunker = TableChunker(
    preserve_headers=True,
    max_rows_per_chunk=3,
    include_context=True,
    table_format="markdown"
)

chunks = table_chunker.chunk(text_with_table)

print("Table Chunking Results:\n")
print("=" * 80)

for i, chunk in enumerate(chunks, 1):
    is_table = chunk.metadata.get('is_table', False)
    
    print(f"\nChunk {i}:")
    print(f"  Type: {'Table' if is_table else 'Text'}")
    
    if is_table:
        rows = chunk.metadata.get('row_count', 'N/A')
        cols = chunk.metadata.get('column_count', 'N/A')
        print(f"  Rows: {rows}, Columns: {cols}")
    
    print(f"  Content: {chunk.text[:100]}...")

print("\n" + "=" * 80)

Step 11: Provenance Tracking

Track chunk origins for data lineage and debugging.

Why Track Provenance?

• Data Lineage: Know where chunks came from
• Debugging: Trace issues back to source
• Compliance: Required for some use cases

In [ ]:

import sys
import os
import importlib

# 1. Ensure local package is in path
project_root = os.path.abspath(os.path.join(os.getcwd(), "../.."))
if project_root not in sys.path:
    sys.path.insert(0, project_root)

# 2. Force unload modules to ensure clean reload
modules_to_unload = [
    'semantica.split.semantic_chunker', 
    'semantica.split.splitter', 
    'semantica.split.provenance_tracker',
    'semantica.split'
]
for module in modules_to_unload:
    if module in sys.modules:
        del sys.modules[module]

# 3. Import fresh modules
import semantica.split.semantic_chunker
import semantica.split.splitter
import semantica.split.provenance_tracker
from semantica.split import ProvenanceTracker, TextSplitter

# 4. Verify Chunk class has id field
from semantica.split.semantic_chunker import Chunk
print(f"Chunk class fields: {Chunk.__annotations__}")
if 'id' not in Chunk.__annotations__:
    print("WARNING: Chunk class still missing 'id' field. Kernel restart required.")

# Create chunks
splitter = TextSplitter(method="recursive", chunk_size=200, chunk_overlap=50)
chunks = splitter.split(text)

# Track provenance
tracker = ProvenanceTracker()

for chunk in chunks:
    tracker.track_chunk(
        chunk=chunk,
        source_document="apple_doc_001",
        source_path="data/apple.txt",
        timestamp="2024-01-01T00:00:00Z",
        method="recursive"
    )

print("Provenance Tracking Results:\n")
print("=" * 80)

# Get lineage for first chunk
if chunks:
    # Get provenance info using the chunk's ID
    chunk_id = getattr(chunks[0], 'id', None)
    print(f"Chunk ID: {chunk_id}")
    
    if chunk_id:
        prov_info = tracker.get_provenance(chunk_id)
        
        if prov_info:
            print(f"\nLineage for Chunk 1:")
            print(f"  Source Document: {prov_info.source_document}")
            print(f"  File Path: {prov_info.source_path}")
            print(f"  Method: {prov_info.metadata.get('method')}")
            print(f"  Timestamp: {prov_info.timestamp}")
    else:
        print("Error: Chunk ID not found. The Chunk class definition might still be cached.")
        print("Please click 'Kernel' -> 'Restart Kernel' in the menu and run all cells again.")

print("\n" + "=" * 80)

Step 13: Method Comparison

Let's compare all methods side-by-side to help you choose the right one.

Comparison Criteria

• Chunk Count: Number of chunks created
• Average Size: Average chunk size
• Processing Time: Speed of chunking

In [ ]:

import time

# Methods to compare
methods_to_compare = [
    ("recursive", {}),
    ("sentence", {}),
    ("paragraph", {}),
    ("token", {"tokenizer": "tiktoken"}),
]

print("Method Comparison:\n")
print("=" * 80)
print(f"{'Method':<15} {'Chunks':<10} {'Avg Size':<12} {'Time (ms)':<12}")
print("-" * 80)

for method, kwargs in methods_to_compare:
    try:
        start_time = time.time()
        
        splitter = TextSplitter(
            method=method,
            chunk_size=200,
            chunk_overlap=50,
            **kwargs
        )
        
        chunks = splitter.split(text)
        
        elapsed = (time.time() - start_time) * 1000
        avg_size = sum(len(c.text) for c in chunks) / len(chunks) if chunks else 0
        
        print(f"{method:<15} {len(chunks):<10} {avg_size:<12.0f} {elapsed:<12.2f}")
        
    except Exception as e:
        print(f"{method:<15} Error: {str(e)[:40]}")

print("=" * 80)

Step 14: Best Practices

Choosing the Right Method

1. General Documents: Use recursive for speed and simplicity
2. LLM Applications: Use token to respect context windows
3. Semantic Search: Use semantic_transformer for topic coherence
4. GraphRAG: Use entity_aware or relation_aware
5. Structured Docs: Use structural for formatted documents
6. Large Documents: Use hierarchical for multi-level access

Chunk Size Guidelines

Use Case	Recommended Size	Overlap
Semantic Search	512-1024 chars	20%
LLM Context	2000-4000 chars	10-20%
Entity Extraction	500-1500 chars	15-25%
Question Answering	1000-2000 chars	20%

Overlap Recommendations

• 10-15%: Fast processing, less redundancy
• 20-25%: Balanced (recommended)
• 30-40%: Maximum context preservation

Summary

What You've Learned

In this notebook, you've learned how to:

• Use TextSplitter with multiple methods
• Apply standard splitting (recursive, token, sentence, paragraph)
• Use semantic chunking for topic coherence
• Apply KG-aware chunking (entity-aware, relation-aware)
• Use specialized chunkers (structural, hierarchical, sliding window, table)
• Track provenance
• Choose the right method for your use case

Key Takeaways

1. Method Selection Matters: Different methods for different needs
2. Chunk Size is Critical: Balance between context and processing
3. Overlap Helps: 20% overlap is a good default
4. Track Provenance: Important for debugging and compliance
5. KG-Aware for GraphRAG: Use entity/relation-aware for knowledge graphs

Next Steps

Next Notebook: 12_Embedding_Generation.ipynb
Learn how to generate embeddings for your chunks!

Further Reading:

• Split Module API Reference
• Advanced Chunking Strategies
• GraphRAG Pipeline

---

Questions or Issues? Check out our GitHub repository or documentation.