semantica/docs/reference/normalize.md

title, description, icon

title	description	icon
Normalize Module	Text cleaning, entity canonicalization, date normalization, number/unit conversion, language detection, and encoding repair.	broom

Clean, standardize, and prepare text and data for semantic processing.

---

Overview

The Normalize Module standardizes raw data before extraction and graph construction — fixing encodings, canonicalizing entity names, normalizing dates, and detecting languages. All normalizers expose both convenience functions and stateful class instances.

Unicode, whitespace, HTML stripping, smart-quote/dash replacement. Alias resolution, disambiguation, name variant handling. ISO 8601 output, timezone conversion, relative date parsing. Currency, unit conversion, scientific notation. Duplicate detection, schema validation, missing value handling. 50+ language detection with confidence scoring.

---

Convenience Functions

The quickest way to normalize — dispatch via function with a method parameter:

from semantica.normalize import (
    normalize_text, normalize_entity, normalize_date,
    normalize_number, clean_data, detect_language, handle_encoding
)

clean  = normalize_text("  Hello,   World!!  \n\n")
# → "Hello, World!!"

entity = normalize_entity("Apple Computer Inc.", entity_type="Organization")
# → "Apple Inc."

date   = normalize_date("Jan 1st, 2020")
# → "2020-01-01"

num    = normalize_number("$1,234.56")
# → 1234.56

lang   = detect_language("Bonjour le monde")
# → {"language": "fr", "confidence": 0.98}

---

TextNormalizer

from semantica.normalize import TextNormalizer

normalizer = TextNormalizer()

normalized = normalizer.normalize_text(
    raw_text,
    lowercase=False,
    remove_punctuation=False,
    remove_extra_whitespace=True,
    strip_html=True,            # remove HTML tags
    normalize_unicode=True,     # NFC normalization
)

from semantica.normalize import UnicodeNormalizer

normalizer = UnicodeNormalizer(form="NFC")
text = normalizer.normalize("café")   # NFC → canonical composition

from semantica.normalize import WhitespaceNormalizer

normalizer = WhitespaceNormalizer()
text = normalizer.normalize("Hello   World\t\n")  # → "Hello World"

from semantica.normalize import SpecialCharacterProcessor

processor = SpecialCharacterProcessor()
text = processor.process("‘Hello’")  # '' → ''

**v0.5.0 fix:** Encoding repair now handles cp1252/latin-1 characters that previously caused crashes on Windows when processing documents with non-ASCII content.

---

EntityNormalizer

from semantica.normalize import EntityNormalizer

normalizer = EntityNormalizer()

# Normalize company names — handles suffixes, punctuation, case
companies = ["Apple Computer, Inc.", "Apple Inc", "APPLE INC."]
normalized = [normalizer.normalize_entity(c) for c in companies]
# All → "Apple Inc."

# Person names
name = normalizer.normalize_entity("JOBS, STEVE", entity_type="Person")
# → "Steve Jobs"

from semantica.normalize import AliasResolver

resolver = AliasResolver(aliases={
    "ML": "Machine Learning",
    "AI": "Artificial Intelligence",
    "DL": "Deep Learning",
})

resolved = resolver.resolve("ML and DL are subsets of AI")
# → "Machine Learning and Deep Learning are subsets of Artificial Intelligence"

from semantica.normalize import EntityDisambiguator

disambiguator = EntityDisambiguator()
result = disambiguator.disambiguate(
    "Apple", context="Steve Jobs founded Apple in Cupertino"
)
# → {"entity": "Apple Inc.", "type": "Organization", "confidence": 0.96}

from semantica.normalize import NameVariantHandler

handler = NameVariantHandler()
canonical = handler.normalize("Dr. JOHN P. SMITH Jr.")
# → "John P. Smith"

---

DateNormalizer

from semantica.normalize import DateNormalizer

normalizer = DateNormalizer()

dates = [
    "January 1st, 2020",
    "01/01/2020",
    "2020-01-01T00:00:00Z",
    "yesterday",        # relative dates supported
    "3 weeks ago",
]
normalized = [normalizer.normalize_date(d) for d in dates]
# All → ISO 8601 strings

# With timezone conversion to UTC
normalizer_utc = DateNormalizer(target_timezone="UTC")
utc_date = normalizer_utc.normalize_date("2024-01-01 09:00 EST")

from semantica.normalize import TimeZoneNormalizer

tz_normalizer = TimeZoneNormalizer(target_tz="UTC")
utc_dt = tz_normalizer.normalize("2024-01-01 09:00", source_tz="America/New_York")

from semantica.normalize import RelativeDateProcessor
from datetime import datetime

processor = RelativeDateProcessor(reference_date=datetime(2025, 1, 15))
result = processor.process("3 days ago")
# → datetime(2025, 1, 12)

from semantica.normalize import TemporalExpressionParser

parser = TemporalExpressionParser()
result = parser.parse("from January 2020 to March 2021")
# → {"start": "2020-01-01", "end": "2021-03-31", "type": "range"}

---

NumberNormalizer

from semantica.normalize import NumberNormalizer

normalizer = NumberNormalizer()

# Currency
normalizer.normalize_number("$1,234.56")    # → 1234.56
normalizer.normalize_number("€42K")          # → 42000.0
normalizer.normalize_number("$1.2B")         # → 1200000000.0

# Scientific notation
normalizer.normalize_number("3.14e-2")       # → 0.0314

# Percentages
normalizer.normalize_number("42%")           # → 0.42

from semantica.normalize import UnitConverter

converter = UnitConverter()
result = converter.convert(100, from_unit="km/h", to_unit="m/s")
# → 27.78

# All supported categories: length, weight, volume, temperature, speed, area
categories = converter.list_categories()

from semantica.normalize import CurrencyNormalizer

normalizer = CurrencyNormalizer()
result = normalizer.normalize("$42.50")
# → {"amount": 42.50, "currency": "USD", "raw": "$42.50"}

---

DataCleaner

from semantica.normalize import DataCleaner, DataValidator, DuplicateDetector

cleaner = DataCleaner()

# Remove duplicates from a dataset
deduped = cleaner.remove_duplicates(records, similarity_threshold=0.9)

# Fill missing values
filled = cleaner.fill_missing(records, strategy="mean")  # or "median", "mode", "remove"

# Validate schema
validator = DataValidator()
result = validator.validate(records, schema={"name": str, "age": int})
print(result.valid_count, result.errors)

---

LanguageDetector

from semantica.normalize import LanguageDetector

detector = LanguageDetector()

# Single text
lang = detector.detect("Bonjour le monde")
# → {"language": "fr", "confidence": 0.98}

# Top N languages
langs = detector.detect_top_n("This might be mixed", n=3)
# → [{"language": "en", "probability": 0.85}, ...]

# Batch
results = detector.detect_batch(["Hello", "Hola", "Bonjour"])

---

EncodingHandler

from semantica.normalize import EncodingHandler

handler = EncodingHandler()

# Detect encoding
encoding = handler.detect_encoding(raw_bytes)
# → {"encoding": "windows-1252", "confidence": 0.73}

# Convert to UTF-8
utf8_text = handler.to_utf8(raw_bytes)

# Remove BOM
clean = handler.remove_bom(text_with_bom)

---

Batch Processing

All normalizers support batch input:

from semantica.normalize import normalize_text

texts = ["Text 1...", "Text 2...", "Text 3..."]
normalized = [normalize_text(t) for t in texts]

For large datasets, use the pipeline:

from semantica.pipeline import Pipeline
from semantica.normalize import TextNormalizer

pipeline = Pipeline()
pipeline.add_step("normalize", TextNormalizer())
result = pipeline.run(documents)

---

See Also

Parse documents before normalization. Chunk normalized text. Resolve duplicate entities post-normalization. Include normalization in a pipeline.