semantica/docs/reference/normalize.md

Normalize

Clean, standardize, and prepare text and data for semantic processing with comprehensive normalization capabilities.

---

🎯 Overview

• :material-text-box-remove:{ .lg .middle } Text Cleaning

  ---

  Remove noise, fix encoding issues, and standardize whitespace for clean text

• :material-format-text:{ .lg .middle } Entity Normalization

  ---

  Standardize entity names, abbreviations, and formats across documents

• :material-calendar-clock:{ .lg .middle } Date & Time

  ---

  Parse and standardize date/time formats to ISO 8601

• :material-numeric:{ .lg .middle } Number Normalization

  ---

  Standardize numeric values, units, and measurements

• :material-translate:{ .lg .middle } Language Detection

  ---

  Automatically detect document language with confidence scoring

• :material-file-code:{ .lg .middle } Encoding Handling

  ---

  Fix character encoding issues and ensure UTF-8 compliance

!!! tip "Why Normalize?" Normalization is crucial for:

- **Consistency**: Ensure uniform data representation
- **Accuracy**: Improve entity extraction and matching
- **Quality**: Reduce noise and errors in downstream processing
- **Performance**: Enable better deduplication and search

---

⚙️ Algorithms Used

Text Normalization

• Unicode Normalization: NFC, NFD, NFKC, NFKD forms using Unicode standard
• Whitespace Normalization: Regex-based cleanup (\s+ → single space)
• Case Folding: Locale-aware case normalization (Unicode case folding)
• Diacritic Removal: Unicode decomposition and combining character removal
• Punctuation Handling: Smart punctuation normalization preserving sentence structure

Entity Normalization

• Fuzzy Matching: Levenshtein distance with configurable threshold (default: 0.85)
• Phonetic Matching: Soundex and Metaphone algorithms for name variants
• Abbreviation Expansion: Dictionary-based expansion with context awareness
• Canonical Form Selection: Frequency-based or confidence-based selection
• Entity Linking: Hash-based entity ID generation for cross-document linking

Date/Time Normalization

• Parsing: dateutil parser with 100+ format support
• Timezone Handling: pytz for timezone conversion and DST handling
• Standardization: ISO 8601 format output (YYYY-MM-DDTHH:MM:SSZ)
• Relative Date Resolution: Convert "yesterday", "last week" to absolute dates
• Fuzzy Date Parsing: Handle incomplete dates (e.g., "March 2024")

Number Normalization

• Numeric Parsing: Handle various formats (1,000.00, 1.000,00, 1 000.00)
• Unit Conversion: Standardize units (km → meters, lbs → kg)
• Scientific Notation: Parse and normalize scientific notation
• Percentage Handling: Normalize percentage representations
• Currency Normalization: Standardize currency symbols and amounts

Language Detection

• N-gram Analysis: Character and word n-gram frequency analysis
• Statistical Models: Language-specific statistical models
• Confidence Scoring: Probability-based confidence scores
• Multi-language Support: 100+ languages supported

---

Main Classes

TextNormalizer

Main text normalization orchestrator with comprehensive cleaning capabilities.

Methods:

Method	Description	Algorithm
normalize(text)	Normalize single text	Full normalization pipeline
normalize_documents(docs)	Batch normalize documents	Parallel processing
clean(text)	Clean text only	Noise removal
fix_encoding(text)	Fix encoding issues	Character encoding detection
normalize_whitespace(text)	Standardize whitespace	Regex-based cleanup
remove_diacritics(text)	Remove accents	Unicode decomposition

Configuration Options:

TextNormalizer(
    lowercase=False,              # Convert to lowercase
    remove_punctuation=False,     # Remove all punctuation
    remove_numbers=False,         # Remove numeric values
    remove_whitespace=True,       # Normalize whitespace
    fix_encoding=True,            # Fix encoding issues
    remove_diacritics=False,      # Remove accents/diacritics
    normalize_unicode=True,       # Unicode normalization (NFC)
    remove_urls=False,            # Remove URLs
    remove_emails=False,          # Remove email addresses
    remove_phone_numbers=False,   # Remove phone numbers
    expand_contractions=False,    # Expand contractions (don't → do not)
    remove_html=True,             # Remove HTML tags
    remove_extra_spaces=True,     # Remove extra spaces
    strip=True                    # Strip leading/trailing whitespace
)

Example:

from semantica.normalize import TextNormalizer

# Basic normalization
normalizer = TextNormalizer(
    lowercase=False,
    remove_punctuation=False,
    fix_encoding=True,
    normalize_whitespace=True
)

text = "  Apple Inc.  was founded in 1976.  "
normalized = normalizer.normalize(text)
print(normalized)
# Output: "Apple Inc. was founded in 1976."

# Aggressive cleaning
aggressive_normalizer = TextNormalizer(
    lowercase=True,
    remove_punctuation=True,
    remove_numbers=True,
    remove_urls=True,
    expand_contractions=True
)

text = "Check out https://example.com! It's amazing (founded in 2020)."
cleaned = aggressive_normalizer.normalize(text)
print(cleaned)
# Output: "check out it is amazing founded in"

---

EntityNormalizer

Standardize entity names and resolve variations to canonical forms.

Methods:

Method	Description	Algorithm
normalize(entities)	Normalize entity list	Fuzzy + phonetic matching
normalize_single(entity)	Normalize one entity	Canonical form lookup
add_canonical(entity, canonical)	Add canonical mapping	Dictionary update
get_canonical(entity)	Get canonical form	Fuzzy search

Configuration Options:

EntityNormalizer(
    fuzzy_matching=True,          # Enable fuzzy matching
    similarity_threshold=0.85,    # Similarity threshold (0-1)
    phonetic_matching=False,      # Enable phonetic matching
    case_sensitive=False,         # Case-sensitive matching
    preserve_case=True,           # Preserve original case in output
    expand_abbreviations=True,    # Expand common abbreviations
    canonical_dict=None           # Custom canonical mappings
)

Example:

from semantica.normalize import EntityNormalizer

normalizer = EntityNormalizer(
    fuzzy_matching=True,
    similarity_threshold=0.85,
    expand_abbreviations=True
)

# Normalize entity variations
entities = [
    "Apple Inc.",
    "Apple",
    "AAPL",
    "Apple Computer",
    "Apple, Inc."
]

normalized = normalizer.normalize(entities)
print(normalized)
# All mapped to canonical form: "Apple Inc."

# Add custom canonical mappings
normalizer.add_canonical("AAPL", "Apple Inc.")
normalizer.add_canonical("Microsoft Corp", "Microsoft Corporation")

# Get canonical form
canonical = normalizer.get_canonical("MSFT")
print(canonical)  # "Microsoft Corporation"

---

DateNormalizer

Parse and standardize date/time formats to ISO 8601.

Methods:

Method	Description	Algorithm
normalize(date_string)	Parse and normalize date	dateutil parser
normalize_batch(dates)	Batch normalize dates	Parallel processing
parse_relative(relative_date)	Parse relative dates	Date arithmetic
detect_format(date_string)	Detect date format	Pattern matching

Configuration Options:

DateNormalizer(
    output_format="ISO8601",      # ISO8601, UNIX, custom format
    timezone="UTC",               # Target timezone
    handle_relative=True,         # Parse "yesterday", "last week"
    fuzzy=True,                   # Fuzzy parsing
    default_day=1,                # Default day for incomplete dates
    default_month=1               # Default month for incomplete dates
)

Example:

from semantica.normalize import DateNormalizer

normalizer = DateNormalizer(
    output_format="ISO8601",
    timezone="UTC",
    handle_relative=True
)

# Normalize various date formats
dates = [
    "Jan 1, 2024",
    "01/01/2024",
    "2024-01-01",
    "1st January 2024",
    "yesterday",
    "March 2024"
]

normalized = [normalizer.normalize(d) for d in dates]
for original, norm in zip(dates, normalized):
    print(f"{original:20} → {norm}")

# Output:
# Jan 1, 2024          → 2024-01-01T00:00:00Z
# 01/01/2024           → 2024-01-01T00:00:00Z
# 2024-01-01           → 2024-01-01T00:00:00Z
# 1st January 2024     → 2024-01-01T00:00:00Z
# yesterday            → 2024-11-29T00:00:00Z
# March 2024           → 2024-03-01T00:00:00Z

# Custom format
custom_normalizer = DateNormalizer(output_format="%Y/%m/%d")
result = custom_normalizer.normalize("Jan 1, 2024")
print(result)  # "2024/01/01"

---

NumberNormalizer

Standardize numeric values, units, and measurements.

Methods:

Method	Description	Algorithm
normalize(number_string)	Parse and normalize number	Regex parsing
convert_units(value, from_unit, to_unit)	Convert units	Unit conversion
parse_currency(currency_string)	Parse currency	Currency parsing
normalize_percentage(percent_string)	Normalize percentage	Percentage parsing

Example:

from semantica.normalize import NumberNormalizer

normalizer = NumberNormalizer(
    decimal_separator=".",
    thousands_separator=",",
    normalize_units=True
)

# Normalize various number formats
numbers = [
    "1,000.50",
    "1.000,50",
    "1 000.50",
    "$1,234.56",
    "50%",
    "1.5e3"
]

normalized = [normalizer.normalize(n) for n in numbers]
for original, norm in zip(numbers, normalized):
    print(f"{original:15} → {norm}")

# Unit conversion
distance_km = normalizer.convert_units(5, "km", "m")
print(f"5 km = {distance_km} m")  # 5000 m

---

LanguageDetector

Detect document language with confidence scoring.

Methods:

Method	Description	Algorithm
detect(text)	Detect language	N-gram analysis
detect_batch(texts)	Batch detect languages	Parallel processing
get_confidence(text, language)	Get confidence score	Statistical model

Example:

from semantica.normalize import LanguageDetector

detector = LanguageDetector()

# Detect language
texts = [
    "Hello, how are you?",
    "Bonjour, comment allez-vous?",
    "Hola, ¿cómo estás?",
    "Hallo, wie geht es dir?",
    "こんにちは、お元気ですか？"
]

for text in texts:
    result = detector.detect(text)
    print(f"{text[:30]:30} → {result['language']} ({result['confidence']:.2f})")

# Output:
# Hello, how are you?           → en (0.99)
# Bonjour, comment allez-vous?  → fr (0.98)
# Hola, ¿cómo estás?            → es (0.97)
# Hallo, wie geht es dir?       → de (0.96)
# こんにちは、お元気ですか？         → ja (0.99)

---

Convenience Functions

Quick access to normalization operations:

from semantica.normalize import (
    normalize_text,
    normalize_entities,
    normalize_dates,
    normalize_numbers,
    detect_language,
    fix_encoding
)

# Normalize text
clean_text = normalize_text("  Messy   text  ", remove_extra_spaces=True)

# Normalize entities
canonical_entities = normalize_entities(["Apple Inc.", "Apple", "AAPL"])

# Normalize dates
iso_dates = normalize_dates(["Jan 1, 2024", "01/01/2024"])

# Normalize numbers
standard_numbers = normalize_numbers(["1,000.50", "$1,234.56"])

# Detect language
language = detect_language("Hello, world!")

# Fix encoding
fixed_text = fix_encoding("Caf\u00e9")  # Café

---

Configuration

Environment Variables

# Normalization settings
export NORMALIZE_DEFAULT_LOWERCASE=false
export NORMALIZE_DEFAULT_ENCODING=utf-8
export NORMALIZE_DEFAULT_TIMEZONE=UTC

# Entity normalization
export NORMALIZE_ENTITY_SIMILARITY_THRESHOLD=0.85
export NORMALIZE_ENTITY_FUZZY_MATCHING=true

# Language detection
export NORMALIZE_LANGUAGE_DETECTOR=langdetect
export NORMALIZE_LANGUAGE_CONFIDENCE_THRESHOLD=0.8

YAML Configuration

# config.yaml - Normalize Module Configuration

normalize:
  text:
    lowercase: false
    remove_punctuation: false
    fix_encoding: true
    normalize_whitespace: true
    remove_urls: false
    expand_contractions: false
    
  entity:
    fuzzy_matching: true
    similarity_threshold: 0.85
    phonetic_matching: false
    expand_abbreviations: true
    
  date:
    output_format: "ISO8601"
    timezone: "UTC"
    handle_relative: true
    fuzzy: true
    
  number:
    decimal_separator: "."
    thousands_separator: ","
    normalize_units: true
    
  language:
    detector: "langdetect"  # langdetect, fasttext
    confidence_threshold: 0.8
    fallback_language: "en"

---

Integration Examples

Complete Document Normalization Pipeline

from semantica.normalize import TextNormalizer, EntityNormalizer, DateNormalizer, LanguageDetector
from semantica.parse import DocumentParser

# Parse documents
parser = DocumentParser()
documents = parser.parse(["document1.pdf", "document2.docx"])

# Detect language
detector = LanguageDetector()
for doc in documents:
    lang_result = detector.detect(doc.content)
    doc.metadata["language"] = lang_result["language"]
    doc.metadata["language_confidence"] = lang_result["confidence"]

# Normalize text
text_normalizer = TextNormalizer(
    fix_encoding=True,
    normalize_whitespace=True,
    remove_html=True
)

for doc in documents:
    doc.content = text_normalizer.normalize(doc.content)

# Normalize dates in metadata
date_normalizer = DateNormalizer(output_format="ISO8601")
for doc in documents:
    if "date" in doc.metadata:
        doc.metadata["date"] = date_normalizer.normalize(doc.metadata["date"])

# Normalize entities
entity_normalizer = EntityNormalizer(similarity_threshold=0.85)
# ... entity normalization logic

Multi-Language Document Processing

from semantica.normalize import LanguageDetector, TextNormalizer

detector = LanguageDetector()
normalizers = {
    "en": TextNormalizer(expand_contractions=True),
    "fr": TextNormalizer(remove_diacritics=False),
    "de": TextNormalizer(lowercase=False)
}

def process_multilingual_document(text):
    # Detect language
    lang_result = detector.detect(text)
    language = lang_result["language"]
    
    # Use language-specific normalizer
    normalizer = normalizers.get(language, TextNormalizer())
    normalized = normalizer.normalize(text)
    
    return {
        "text": normalized,
        "language": language,
        "confidence": lang_result["confidence"]
    }

# Process documents
documents = ["Hello world", "Bonjour le monde", "Hallo Welt"]
results = [process_multilingual_document(doc) for doc in documents]

---

Best Practices

1. Choose Appropriate Normalization Level

# Minimal normalization for entity extraction
minimal = TextNormalizer(
    fix_encoding=True,
    normalize_whitespace=True
)

# Moderate normalization for search
moderate = TextNormalizer(
    fix_encoding=True,
    normalize_whitespace=True,
    lowercase=True,
    remove_urls=True
)

# Aggressive normalization for topic modeling
aggressive = TextNormalizer(
    lowercase=True,
    remove_punctuation=True,
    remove_numbers=True,
    remove_urls=True,
    expand_contractions=True
)

2. Preserve Original Data

# Always keep original text
doc.original_content = doc.content
doc.content = normalizer.normalize(doc.content)

# Store normalization metadata
doc.metadata["normalized"] = True
doc.metadata["normalization_config"] = normalizer.config

3. Batch Processing for Performance

# Batch normalize for better performance
texts = [doc.content for doc in documents]
normalized_texts = normalizer.normalize_documents(texts)

for doc, normalized in zip(documents, normalized_texts):
    doc.content = normalized

---

Troubleshooting

Common Issues

Issue: Encoding errors with special characters

# Solution: Enable encoding fix
normalizer = TextNormalizer(fix_encoding=True)

# Or manually fix encoding
from semantica.normalize import fix_encoding
fixed_text = fix_encoding(problematic_text)

Issue: Over-normalization losing important information

# Solution: Use conservative settings
normalizer = TextNormalizer(
    lowercase=False,           # Keep case
    remove_punctuation=False,  # Keep punctuation
    remove_numbers=False       # Keep numbers
)

Issue: Slow processing for large documents

# Solution: Use batch processing
normalizer = TextNormalizer()
normalized = normalizer.normalize_documents(
    documents,
    batch_size=100,
    n_jobs=4  # Parallel processing
)

---

Performance Tips

Memory Optimization

# Process documents in chunks
def normalize_large_corpus(documents, chunk_size=1000):
    normalizer = TextNormalizer()
    
    for i in range(0, len(documents), chunk_size):
        chunk = documents[i:i + chunk_size]
        normalized_chunk = normalizer.normalize_documents(chunk)
        yield from normalized_chunk

Speed Optimization

# Disable unnecessary features
fast_normalizer = TextNormalizer(
    fix_encoding=False,        # Skip if encoding is known good
    normalize_unicode=False,   # Skip if not needed
    remove_diacritics=False    # Skip if not needed
)

# Use parallel processing
normalizer.normalize_documents(
    documents,
    n_jobs=-1  # Use all CPU cores
)

---

See Also

• Parse Module - Document parsing and extraction
• Semantic Extract Module - Entity and relation extraction
• Split Module - Text chunking and splitting
• Ingest Module - Data ingestion