semantica/docs/reference/parse.md

Parse Module

The parse module provides comprehensive document parsing capabilities for extracting content, metadata, and structure from various file formats.

Overview

• Document Parsing: PDF, DOCX, XLSX, PPTX, and more
• OCR Processing: Extract text from images and scanned documents
• Table Extraction: Extract tables with structure preservation
• Metadata Extraction: Author, title, dates, and custom metadata
• Structure Analysis: Headings, sections, lists, and document hierarchy
• Multi-Format Support: 50+ file formats with specialized parsers

---

Algorithms Used

PDF Parsing

• Text Extraction: PDFMiner algorithm for text layout analysis
• OCR: Tesseract with preprocessing (deskewing, noise reduction)
• Table Detection: Hough transform for line detection + clustering
• Layout Analysis: XY-cut algorithm for reading order determination

OCR Algorithms

• Preprocessing: Otsu's thresholding, morphological operations
• Text Detection: EAST/CRAFT deep learning models
• Recognition: Tesseract LSTM + language models
• Post-processing: Spell correction, confidence filtering

Table Extraction

• Rule-based: Line detection + cell clustering
• ML-based: Deep learning table structure recognition
• Hybrid: Combine rules + ML for robustness

---

Quick Start

from semantica.parse import DocumentParser

# Initialize parser
parser = DocumentParser()

# Parse with options
docs = parser.parse(
    sources=["documents/"],
    formats=["pdf", "docx", "xlsx"],
    metadata={"source": "company_docs"}
)

# Access parsed data
for doc in docs:
    print(f"Title: {doc.metadata.get('title')}")
    print(f"Author: {doc.metadata.get('author')}")
    print(f"Pages: {doc.metadata.get('pages')}")
    print(f"Text length: {len(doc.text)} characters")

---

PDFParser

Example Usage:

from semantica.parse import PDFParser

# Basic PDF parsing
parser = PDFParser()
doc = parser.parse("document.pdf")

# Advanced PDF parsing with OCR
parser = PDFParser(
    ocr_enabled=True,
    ocr_language="eng",
    extract_tables=True,
    extract_images=True,
    dpi=300
)

doc = parser.parse("scanned_document.pdf")

# Access PDF-specific data
print(f"Pages: {doc.metadata['pages']}")
print(f"PDF version: {doc.metadata['pdf_version']}")
print(f"Tables extracted: {len(doc.tables)}")

# Access tables
for i, table in enumerate(doc.tables):
    print(f"Table {i+1}:")
    print(table.to_dataframe())

---

DOCXParser

Example Usage:

from semantica.parse import DOCXParser

# Parse Word document
parser = DOCXParser()
doc = parser.parse("document.docx")

# With structure preservation
parser = DOCXParser(
    preserve_structure=True,
    extract_tables=True,
    extract_images=True
)

doc = parser.parse("report.docx")

# Access structure
print(f"Headings: {doc.structure['headings']}")
print(f"Sections: {len(doc.structure['sections'])}")
print(f"Tables: {len(doc.tables)}")

---

ExcelParser

Example Usage:

from semantica.parse import ExcelParser

# Parse Excel file
parser = ExcelParser()
doc = parser.parse("data.xlsx")

# Parse specific sheets
parser = ExcelParser(
    sheets=["Sheet1", "Data"],
    header_row=0,
    skip_empty_rows=True
)

doc = parser.parse("workbook.xlsx")

# Access data
for sheet_name, df in doc.dataframes.items():
    print(f"Sheet: {sheet_name}")
    print(f"Rows: {len(df)}, Columns: {len(df.columns)}")
    print(df.head())

---

HTMLParser

Example Usage:

from semantica.parse import HTMLParser

# Parse HTML
parser = HTMLParser()
doc = parser.parse("page.html")

# Advanced parsing
parser = HTMLParser(
    extract_metadata=True,
    extract_links=True,
    remove_scripts=True,
    remove_styles=True
)

doc = parser.parse("article.html")

# Access extracted data
print(f"Title: {doc.metadata['title']}")
print(f"Links: {len(doc.links)}")
print(f"Main content: {doc.main_content}")

---

JSONParser

Example Usage:

from semantica.parse import JSONParser

# Parse JSON
parser = JSONParser()
doc = parser.parse("data.json")

# Flatten nested JSON
parser = JSONParser(
    flatten=True,
    separator=".",
    extract_schema=True
)

doc = parser.parse("nested_data.json")

# Access schema
print(f"Schema: {doc.schema}")
print(f"Flattened keys: {list(doc.flattened_data.keys())}")

---

ImageParser

Example Usage:

from semantica.parse import ImageParser

# Parse image with OCR
parser = ImageParser(
    ocr_enabled=True,
    ocr_language="eng",
    detect_orientation=True
)

doc = parser.parse("scanned_page.jpg")

# Access OCR results
print(f"Extracted text: {doc.text}")
print(f"Confidence: {doc.metadata['ocr_confidence']}")
print(f"Orientation: {doc.metadata['orientation']}")

---

CodeParser

Example Usage:

from semantica.parse import CodeParser

# Parse source code
parser = CodeParser(
    language="python",
    extract_functions=True,
    extract_classes=True,
    extract_docstrings=True
)

doc = parser.parse("module.py")

# Access code structure
print(f"Functions: {len(doc.functions)}")
print(f"Classes: {len(doc.classes)}")

for func in doc.functions:
    print(f"Function: {func.name}")
    print(f"  Parameters: {func.parameters}")
    print(f"  Docstring: {func.docstring}")

---

Common Patterns

Pattern 1: Batch Parsing

from semantica.parse import DocumentParser
from pathlib import Path

parser = DocumentParser()

# Get all files
files = list(Path("documents/").rglob("*.*"))

# Parse in batches
batch_size = 10
for i in range(0, len(files), batch_size):
    batch = files[i:i+batch_size]
    docs = parser.parse(batch)
    # Process docs

Pattern 2: Format-Specific Parsing

from semantica.parse import PDFParser, DOCXParser, ExcelParser

# Route to appropriate parser
def parse_document(file_path):
    if file_path.endswith('.pdf'):
        parser = PDFParser(ocr_enabled=True)
    elif file_path.endswith('.docx'):
        parser = DOCXParser(preserve_structure=True)
    elif file_path.endswith('.xlsx'):
        parser = ExcelParser()
    else:
        raise ValueError(f"Unsupported format: {file_path}")
    
    return parser.parse(file_path)

Pattern 3: Error Handling

from semantica.parse import DocumentParser, ParseError

parser = DocumentParser()

successful = []
failed = []

for file_path in file_paths:
    try:
        doc = parser.parse(file_path)
        successful.append(doc)
    except ParseError as e:
        print(f"Failed to parse {file_path}: {e}")
        failed.append((file_path, str(e)))

print(f"Parsed: {len(successful)}, Failed: {len(failed)}")

---

Configuration

# config.yaml - Parse Configuration

parse:
  pdf:
    ocr_enabled: true
    ocr_language: eng
    extract_tables: true
    extract_images: true
    dpi: 300
    
  docx:
    preserve_structure: true
    extract_tables: true
    extract_images: true
    
  excel:
    header_row: 0
    skip_empty_rows: true
    infer_types: true
    
  html:
    extract_metadata: true
    extract_links: true
    remove_scripts: true
    remove_styles: true
    
  image:
    ocr_enabled: true
    ocr_language: eng
    detect_orientation: true
    
  code:
    extract_functions: true
    extract_classes: true
    extract_docstrings: true

---

See Also

• Ingest Module - Data ingestion
• Normalize Module - Data cleaning and normalization
• Semantic Extract Module - Entity and relationship extraction