semantica/docs/reference/normalize.md at 946a1089c81b6c8a0e146e15c21d05225fb5dec2

mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00

Files

T

KaifAhmad1andClaude Sonnet 4.6 946a1089c8 docs: premium redesign — Mintlify v4, dark/cream theme, full module coverage

- Migrate from mint.json to docs.json (Mintlify v4)
- Theme: maple, emerald green + near-black dark / cream light palette
  (#059669 primary, #0A0A0A dark bg, #FAF7F0 light bg)
- Typography: Lexend headings, Inter body
- 5-tab navigation: Documentation, Quick Start, API Reference, Cookbook, FAQ
- Homepage: removed badge stickers, redundant h2, added blockquote tagline,
  full 27-module reference table with semantica.mcp_server added
- quickstart.md: CodeGroup per pipeline step, pattern vs LLM options,
  AccordionGroup for patterns and troubleshooting
- faq.md: full AccordionGroup structure across 5 sections
- reference/explorer.md: NEW — FastAPI explorer, Ontology Hub, Distance
  Intelligence, CLI reference, REST API endpoints
- reference/mcp_server.md: NEW — MCP stdio server, 12 tools with I/O
  examples, 3 resources, Claude Desktop/VS Code/Windsurf/Cline config
- docs.json: explorer added to Output group, mcp_server to Utilities group
- Chat, feedback (thumbs/suggest/raise), OG/Twitter metadata, search topbar
- All reference pages reformatted with Mintlify JSX components

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-22 21:52:50 +05:30

2.8 KiB

Raw Blame History

title, description, icon

title	description	icon
Normalize Module	Text cleaning, entity standardization, date normalization, and encoding repair.	broom

Clean, standardize, and prepare text and data for semantic processing.

DataNormalizer

from semantica.normalize import DataNormalizer

normalizer = DataNormalizer()

# Text cleaning
clean = normalizer.normalize_text("  Hello,   World!!  \n\n")
# → "Hello, World!!"

# Date standardization
date = normalizer.normalize_date("Jan 1st, 2020")
# → "2020-01-01"

# Entity normalization
entity = normalizer.normalize_entity("Apple Computer Inc.")
# → "Apple Inc."

# Number normalization
num = normalizer.normalize_number("$1,234.56")
# → 1234.56

Text Normalization

text = normalizer.normalize_text(
    raw_text,
    lowercase=False,           # convert to lowercase
    remove_punctuation=False,  # strip punctuation
    remove_extra_whitespace=True,
    fix_encoding=True,         # repair cp1252/latin-1 mojibake
    strip_html=True,           # remove HTML tags
    normalize_unicode=True     # NFC normalization
)

**v0.5.0 fix:** The encoding repair now handles cp1252/latin-1 characters that previously caused crashes on Windows when processing documents with non-ASCII content.

Entity Normalization

# Company name normalization
companies = [
    "Apple Computer, Inc.",
    "Apple Inc",
    "APPLE INC.",
]
normalized = [normalizer.normalize_entity(c) for c in companies]
# All → "Apple Inc."

# Person name normalization
name = normalizer.normalize_person_name("JOBS, STEVE")
# → "Steve Jobs"

Date & Time Normalization

dates = [
    "January 1st, 2020",
    "01/01/2020",
    "2020-01-01T00:00:00Z",
    "yesterday",   # relative dates supported
]
normalized = [normalizer.normalize_date(d) for d in dates]
# All → "2020-01-01" (ISO 8601)

Quantity Normalization

# Currency
normalizer.normalize_number("$1,234.56M")   # → 1234560000.0
normalizer.normalize_number("€42K")          # → 42000.0

# Units
normalizer.normalize_unit("100 km/h")        # → {"value": 100, "unit": "km/h", "si": 27.78}

Batch Processing

texts = ["Text 1...", "Text 2...", "Text 3..."]
normalized = normalizer.normalize_batch(texts, batch_size=100)

2.8 KiB Raw Blame History