mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
KaifAhmad1
409f3fdd27
- Added grid‑card layouts, icons, and admonitions to all reference pages - Removed all mkdocstrings ::: directives that caused alias‑resolution errors - Fixed export module alias issue (RDFExporter) by cleaning the docs - Updated README‑style sections for better readability - Added a PowerShell helper script (remove_mkdocstrings.ps1) for future clean‑ups
116 lines
2.6 KiB
Markdown
116 lines
2.6 KiB
Markdown
# Normalize Module
|
|
|
|
The `normalize` module provides data cleaning, normalization, and standardization capabilities to prepare text and data for semantic processing.
|
|
|
|
## Overview
|
|
|
|
- **Text Cleaning**: Remove noise, fix encoding, standardize whitespace
|
|
- **Entity Normalization**: Standardize entity names and formats
|
|
- **Date Normalization**: Parse and standardize date formats
|
|
- **Number Normalization**: Standardize numeric values and units
|
|
- **Language Detection**: Identify document language
|
|
- **Encoding Handling**: Fix character encoding issues
|
|
|
|
---
|
|
|
|
## Algorithms Used
|
|
|
|
### Text Normalization
|
|
- **Unicode Normalization**: NFC, NFD, NFKC, NFKD forms
|
|
- **Whitespace Normalization**: Regex-based cleanup
|
|
- **Case Folding**: Locale-aware case normalization
|
|
- **Diacritic Removal**: Unicode decomposition
|
|
|
|
### Entity Normalization
|
|
- **Fuzzy Matching**: Levenshtein distance < threshold
|
|
- **Phonetic Matching**: Soundex, Metaphone algorithms
|
|
- **Abbreviation Expansion**: Dictionary-based expansion
|
|
|
|
### Date/Time Normalization
|
|
- **Parsing**: dateutil parser with multiple format support
|
|
- **Timezone Handling**: pytz for timezone conversion
|
|
- **Standardization**: ISO 8601 format output
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from semantica.normalize import TextNormalizer
|
|
|
|
# Initialize normalizer
|
|
normalizer = TextNormalizer()
|
|
|
|
# Normalize text
|
|
normalized_text = normalizer.normalize(raw_text)
|
|
|
|
# Normalize documents
|
|
normalized_docs = normalizer.normalize_documents(documents)
|
|
```
|
|
|
|
---
|
|
|
|
## Main Classes
|
|
|
|
### TextNormalizer
|
|
|
|
|
|
**Example:**
|
|
|
|
```python
|
|
from semantica.normalize import TextNormalizer
|
|
|
|
normalizer = TextNormalizer(
|
|
lowercase=False,
|
|
remove_punctuation=False,
|
|
remove_numbers=False,
|
|
remove_whitespace=True,
|
|
fix_encoding=True
|
|
)
|
|
|
|
text = " Apple Inc. was founded in 1976. "
|
|
normalized = normalizer.normalize(text)
|
|
# Output: "Apple Inc. was founded in 1976."
|
|
```
|
|
|
|
### EntityNormalizer
|
|
|
|
|
|
**Example:**
|
|
|
|
```python
|
|
from semantica.normalize import EntityNormalizer
|
|
|
|
normalizer = EntityNormalizer(
|
|
fuzzy_matching=True,
|
|
similarity_threshold=0.85
|
|
)
|
|
|
|
# Normalize entity names
|
|
entities = ["Apple Inc.", "Apple", "AAPL", "Apple Computer"]
|
|
normalized = normalizer.normalize(entities)
|
|
# All mapped to canonical form: "Apple Inc."
|
|
```
|
|
|
|
### DateNormalizer
|
|
|
|
|
|
**Example:**
|
|
|
|
```python
|
|
from semantica.normalize import DateNormalizer
|
|
|
|
normalizer = DateNormalizer(output_format="ISO8601")
|
|
|
|
dates = ["Jan 1, 2024", "01/01/2024", "2024-01-01"]
|
|
normalized = [normalizer.normalize(d) for d in dates]
|
|
# All output: "2024-01-01T00:00:00Z"
|
|
```
|
|
|
|
---
|
|
|
|
## See Also
|
|
|
|
- [Parse Module](parse.md)
|
|
- [Semantic Extract Module](semantic_extract.md)
|