mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
KaifAhmad1
409f3fdd27
- Added grid‑card layouts, icons, and admonitions to all reference pages - Removed all mkdocstrings ::: directives that caused alias‑resolution errors - Fixed export module alias issue (RDFExporter) by cleaning the docs - Updated README‑style sections for better readability - Added a PowerShell helper script (remove_mkdocstrings.ps1) for future clean‑ups
2.6 KiB
2.6 KiB
Normalize Module
The normalize module provides data cleaning, normalization, and standardization capabilities to prepare text and data for semantic processing.
Overview
- Text Cleaning: Remove noise, fix encoding, standardize whitespace
- Entity Normalization: Standardize entity names and formats
- Date Normalization: Parse and standardize date formats
- Number Normalization: Standardize numeric values and units
- Language Detection: Identify document language
- Encoding Handling: Fix character encoding issues
Algorithms Used
Text Normalization
- Unicode Normalization: NFC, NFD, NFKC, NFKD forms
- Whitespace Normalization: Regex-based cleanup
- Case Folding: Locale-aware case normalization
- Diacritic Removal: Unicode decomposition
Entity Normalization
- Fuzzy Matching: Levenshtein distance < threshold
- Phonetic Matching: Soundex, Metaphone algorithms
- Abbreviation Expansion: Dictionary-based expansion
Date/Time Normalization
- Parsing: dateutil parser with multiple format support
- Timezone Handling: pytz for timezone conversion
- Standardization: ISO 8601 format output
Quick Start
from semantica.normalize import TextNormalizer
# Initialize normalizer
normalizer = TextNormalizer()
# Normalize text
normalized_text = normalizer.normalize(raw_text)
# Normalize documents
normalized_docs = normalizer.normalize_documents(documents)
Main Classes
TextNormalizer
Example:
from semantica.normalize import TextNormalizer
normalizer = TextNormalizer(
lowercase=False,
remove_punctuation=False,
remove_numbers=False,
remove_whitespace=True,
fix_encoding=True
)
text = " Apple Inc. was founded in 1976. "
normalized = normalizer.normalize(text)
# Output: "Apple Inc. was founded in 1976."
EntityNormalizer
Example:
from semantica.normalize import EntityNormalizer
normalizer = EntityNormalizer(
fuzzy_matching=True,
similarity_threshold=0.85
)
# Normalize entity names
entities = ["Apple Inc.", "Apple", "AAPL", "Apple Computer"]
normalized = normalizer.normalize(entities)
# All mapped to canonical form: "Apple Inc."
DateNormalizer
Example:
from semantica.normalize import DateNormalizer
normalizer = DateNormalizer(output_format="ISO8601")
dates = ["Jan 1, 2024", "01/01/2024", "2024-01-01"]
normalized = [normalizer.normalize(d) for d in dates]
# All output: "2024-01-01T00:00:00Z"