semantica/cookbook/introduction/04_Data_Normalization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/05_Data_Normalization.ipynb)\n",
    "\n",
    "# Data Normalization\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates how to normalize and clean data using Semantica's normalization modules. You'll learn to normalize text, entities, dates, numbers, and handle encoding issues.\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/normalize/)\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Use `TextNormalizer` for text cleaning and normalization\n",
    "- Use `EntityNormalizer` for entity name standardization\n",
    "- Use `DateNormalizer` for date format normalization\n",
    "- Use `NumberNormalizer` for number and quantity normalization\n",
    "- Use `DataCleaner` for general data cleaning\n",
    "- Use `LanguageDetector` and `EncodingHandler` for data quality\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Text Normalization\n",
    "\n",
    "Normalize text content for consistency.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install semantica\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "\n",
    "text_normalizer = TextNormalizer()\n",
    "\n",
    "sample_text = \"Hello   World!!!  This is a test.\"\n",
    "\n",
    "normalized = text_normalizer.normalize_text(sample_text, case=\"lower\")\n",
    "cleaned = text_normalizer.clean_text(sample_text, remove_special_chars=False)\n",
    "\n",
    "sample_text, normalized, cleaned\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Entity Normalization\n",
    "\n",
    "Normalize entity names to canonical forms.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import EntityNormalizer\n",
    "\n",
    "entity_normalizer = EntityNormalizer()\n",
    "\n",
    "entity_variants = [\"Apple Inc.\", \"Apple Inc\", \"Apple\", \"Apple Incorporated\"]\n",
    "\n",
    "normalized_entities = []\n",
    "for entity in entity_variants:\n",
    "    normalized = entity_normalizer.normalize_entity(entity, entity_type=\"Organization\")\n",
    "    normalized_entities.append(normalized)\n",
    "    print(f\"{entity} -> {normalized}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Date Normalization\n",
    "\n",
    "Normalize dates to standard formats.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import DateNormalizer\n",
    "\n",
    "date_normalizer = DateNormalizer()\n",
    "\n",
    "date_formats = [\"2023-12-25\", \"12/25/2023\", \"December 25, 2023\", \"25 Dec 2023\"]\n",
    "\n",
    "for date_str in date_formats:\n",
    "    normalized = date_normalizer.normalize_date(date_str)\n",
    "    print(f\"{date_str} -> {normalized}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Number Normalization\n",
    "\n",
    "Normalize numbers and quantities.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib\n",
    "import semantica.normalize.number_normalizer\n",
    "importlib.reload(semantica.normalize.number_normalizer)\n",
    "\n",
    "from semantica.normalize import NumberNormalizer\n",
    "\n",
    "number_normalizer = NumberNormalizer()\n",
    "\n",
    "numbers = [\"1,000\", \"1.5M\", \"$100\", \"50%\", \"3.14e2\"]\n",
    "\n",
    "for num_str in numbers:\n",
    "    normalized = number_normalizer.normalize_number(num_str)\n",
    "    print(f\"{num_str} -> {normalized}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Data Cleaning\n",
    "\n",
    "Clean data using DataCleaner.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import DataCleaner\n",
    "\n",
    "data_cleaner = DataCleaner()\n",
    "\n",
    "data = [\n",
    "    {\"name\": \"Apple Inc.\", \"value\": 100},\n",
    "    {\"name\": \"Apple Inc\", \"value\": 100},\n",
    "    {\"name\": \"Microsoft\", \"value\": 200}\n",
    "]\n",
    "\n",
    "cleaned_data = data_cleaner.clean_data(data, remove_duplicates=True)\n",
    "\n",
    "print(f\"Original records: {len(data)}\")\n",
    "print(f\"Cleaned records: {len(cleaned_data)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Language Detection and Encoding\n",
    "\n",
    "Detect language and handle encoding.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import LanguageDetector, EncodingHandler\n",
    "\n",
    "language_detector = LanguageDetector()\n",
    "encoding_handler = EncodingHandler()\n",
    "\n",
    "text_samples = [\n",
    "    \"Hello, this is English text.\",\n",
    "    \"Bonjour, ceci est du texte français.\",\n",
    "    \"Hola, este es texto en español.\"\n",
    "]\n",
    "\n",
    "for text in text_samples:\n",
    "    detected_lang = language_detector.detect(text)\n",
    "    print(f\"Text: {text[:30]}... -> Language: {detected_lang}\")\n",
    "\n",
    "sample_bytes = \"Hello World\".encode('utf-8')\n",
    "detected_encoding = encoding_handler.detect(sample_bytes)\n",
    "print(f\"\\nDetected encoding: {detected_encoding}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "You've learned how to normalize and clean data:\n",
    "\n",
    "- **TextNormalizer**: Text cleaning and normalization\n",
    "- **EntityNormalizer**: Entity name standardization\n",
    "- **DateNormalizer**: Date format normalization\n",
    "- **NumberNormalizer**: Number and quantity normalization\n",
    "- **DataCleaner**: General data cleaning\n",
    "- **LanguageDetector**: Language detection\n",
    "- **EncodingHandler**: Encoding detection and conversion\n",
    "\n",
    "Next: Learn how to extract entities in the Entity_Extraction notebook.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}