{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)\n",
                "\n",
                "# Data Ingestion - Comprehensive Guide\n",
                "\n",
                "## Overview\n",
                "\n",
                "This notebook provides a comprehensive guide to Semantica's data ingestion capabilities. It covers all submodules, classes, and helper functions available in the `semantica.ingest` module.\n",
                "\n",
                "**Documentation**: [Ingest API Reference](https://semantica.readthedocs.io/reference/ingest/)\n",
                "\n",
                "### Table of Contents\n",
                "\n",
                "1.  **Unified Ingestion**: `ingest` function\n",
                "2.  **File Ingestion**: `FileIngestor`, `FileTypeDetector`, `CloudStorageIngestor`\n",
                "3.  **Web Ingestion**: `WebIngestor`, `ContentExtractor`, `SitemapCrawler`, `RobotsChecker`\n",
                "4.  **Feed Ingestion**: `FeedIngestor`, `FeedMonitor`\n",
                "5.  **Stream Ingestion**: `StreamIngestor`, `StreamMonitor`\n",
                "6.  **Repository Ingestion**: `RepoIngestor`, `CodeExtractor`, `GitAnalyzer`\n",
                "7.  **Email Ingestion**: `EmailIngestor`, `AttachmentProcessor`\n",
                "8.  **Database Ingestion**: `DBIngestor`, `DatabaseConnector`\n",
                "9.  **MCP Ingestion**: `MCPIngestor`\n",
                "10. **Configuration**: `IngestConfig`\n",
                "\n",
                "## Installation\n",
                "\n",
                "Install Semantica with all dependencies:\n",
                "\n",
                "```bash\n",
                "pip install semantica[all]\n",
                "```\n",
                "\n",
                "---\n",
                "\n",
                "## 1. Unified Ingestion\n",
                "\n",
                "The `ingest` function is the main entry point for quick data loading. It automatically detects the source type.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import ingest\n",
                "import tempfile\n",
                "import os\n",
                "import json\n",
                "\n",
                "# Setup temporary directory for examples\n",
                "temp_dir = tempfile.mkdtemp()\n",
                "sample_file = os.path.join(temp_dir, \"sample.txt\")\n",
                "with open(sample_file, 'w') as f:\n",
                "    f.write(\"Semantica Unified Ingestion Example\")\n",
                "\n",
                "# Auto-detect file source\n",
                "result = ingest(sample_file)\n",
                "print(f\"Ingested: {result.name} (Type: {result.file_type})\")\n",
                "\n",
                "# Explicit source type\n",
                "result_explicit = ingest(sample_file, source_type=\"file\")\n",
                "print(f\"Explicit Ingest: {result_explicit.name}\")\n",
                "\n",
                "# Ingest web URL (auto-detected)\n",
                "# Note: This will fail if no internet connection\n",
                "try:\n",
                "    result_web = ingest(\"https://example.com\")\n",
                "    print(f\"Ingested Web: {result_web.title}\")\n",
                "except Exception as e:\n",
                "    print(f\"Web ingestion skipped: {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2. File Ingestion\n",
                "\n",
                "Detailed control over file processing using `FileIngestor` and helper classes.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import FileIngestor, FileTypeDetector, CloudStorageIngestor\n",
                "\n",
                "# --- FileTypeDetector ---\n",
                "detector = FileTypeDetector()\n",
                "detected_type = detector.detect_type(sample_file)\n",
                "print(f\"Detected Type: {detected_type}\")\n",
                "\n",
                "# --- FileIngestor ---\n",
                "file_ingestor = FileIngestor()\n",
                "\n",
                "# Ingest Directory\n",
                "subdir = os.path.join(temp_dir, \"docs\")\n",
                "os.makedirs(subdir, exist_ok=True)\n",
                "with open(os.path.join(subdir, \"note.md\"), 'w') as f:\n",
                "    f.write(\"# Note\\nThis is a markdown file.\")\n",
                "\n",
                "files = file_ingestor.ingest_directory(temp_dir, recursive=True)\n",
                "print(f\"Ingested {len(files)} files from directory\")\n",
                "\n",
                "# --- CloudStorageIngestor (Mock Config) ---\n",
                "s3_config = {\n",
                "    \"aws_access_key_id\": \"mock_key\",\n",
                "    \"aws_secret_access_key\": \"mock_secret\",\n",
                "    \"region_name\": \"us-east-1\"\n",
                "}\n",
                "cloud_ingestor = CloudStorageIngestor(provider=\"s3\", **s3_config)\n",
                "\n",
                "# Example call (will raise error without real credentials)\n",
                "try:\n",
                "    result = cloud_ingestor.ingest(\"s3://my-bucket/data.csv\")\n",
                "    print(f\"Cloud Ingest: {result.name}\")\n",
                "except Exception as e:\n",
                "    print(f\"Cloud ingestion skipped (Mock Config): {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 3. Web Ingestion\n",
                "\n",
                "Scraping and crawling with `WebIngestor`, `ContentExtractor`, and `SitemapCrawler`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import WebIngestor, ContentExtractor, SitemapCrawler, RobotsChecker\n",
                "\n",
                "# --- ContentExtractor ---\n",
                "extractor = ContentExtractor()\n",
                "html_content = \"<html><body><h1>Hello World</h1><p>This is a test.</p><a href='/link'>Link</a></body></html>\"\n",
                "text = extractor.extract_text(html_content)\n",
                "links = extractor.extract_links(html_content, base_url=\"https://example.com\")\n",
                "print(f\"Extracted Text: {text}\")\n",
                "print(f\"Extracted Links: {links}\")\n",
                "\n",
                "# --- RobotsChecker ---\n",
                "checker = RobotsChecker()\n",
                "can_fetch = checker.can_fetch(\"https://www.google.com/search\", \"MyBot\")\n",
                "print(f\"Can fetch google search? {can_fetch}\")\n",
                "\n",
                "# --- WebIngestor ---\n",
                "web_ingestor = WebIngestor(delay=1.0)\n",
                "try:\n",
                "    web_content = web_ingestor.ingest_url(\"https://example.com\")\n",
                "    print(f\"Web Content Title: {web_content.title}\")\n",
                "except Exception as e:\n",
                "    print(f\"Web ingest failed: {e}\")\n",
                "\n",
                "# --- SitemapCrawler ---\n",
                "crawler = SitemapCrawler()\n",
                "try:\n",
                "    urls = crawler.parse_sitemap(\"https://www.google.com/sitemap.xml\")\n",
                "    print(f\"Found {len(urls)} URLs in sitemap\")\n",
                "except Exception as e:\n",
                "    print(f\"Sitemap crawl failed: {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 4. Feed Ingestion\n",
                "\n",
                "Consuming RSS/Atom feeds with `FeedIngestor` and monitoring with `FeedMonitor`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import FeedIngestor, FeedMonitor\n",
                "import time\n",
                "\n",
                "# --- FeedIngestor ---\n",
                "feed_ingestor = FeedIngestor()\n",
                "try:\n",
                "    feed_data = feed_ingestor.ingest_feed(\"https://feeds.feedburner.com/oreilly/radar\")\n",
                "    print(f\"Feed Title: {feed_data.title}\")\n",
                "except Exception as e:\n",
                "    print(f\"Feed ingest failed: {e}\")\n",
                "\n",
                "# --- FeedMonitor ---\n",
                "def feed_callback(feed_data):\n",
                "    print(f\"Feed Updated: {feed_data.title} with {len(feed_data.items)} items\")\n",
                "\n",
                "monitor = FeedMonitor(check_interval=5)\n",
                "try:\n",
                "    monitor.monitor(\"https://feeds.feedburner.com/oreilly/radar\", callback=feed_callback)\n",
                "    time.sleep(2) # Let it run briefly\n",
                "    monitor.stop()\n",
                "except Exception as e:\n",
                "    print(f\"Feed monitor failed: {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 5. Stream Ingestion\n",
                "\n",
                "Real-time processing with `StreamIngestor` and `StreamMonitor`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import StreamIngestor, StreamMonitor\n",
                "\n",
                "stream_ingestor = StreamIngestor()\n",
                "\n",
                "# --- Kafka Processor ---\n",
                "kafka_config = {\"bootstrap_servers\": [\"localhost:9092\"]}\n",
                "kafka_processor = stream_ingestor.ingest_kafka(\"my-topic\", **kafka_config)\n",
                "\n",
                "# --- RabbitMQ Processor ---\n",
                "rabbitmq_processor = stream_ingestor.ingest_rabbitmq(\"my-queue\", \"amqp://guest:guest@localhost:5672/\")\n",
                "\n",
                "# --- Stream Monitor ---\n",
                "monitor = stream_ingestor.monitor\n",
                "health = monitor.check_health()\n",
                "print(f\"Stream Health: {health['overall']}\")\n",
                "print(f\"Processors: {list(health['processors'].keys())}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 6. Repository Ingestion\n",
                "\n",
                "Analyzing codebases with `RepoIngestor`, `CodeExtractor`, and `GitAnalyzer`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import RepoIngestor, CodeExtractor, GitAnalyzer\n",
                "\n",
                "# --- CodeExtractor ---\n",
                "code_extractor = CodeExtractor()\n",
                "py_code = \"class MyClass:\\n    def my_method(self):\\n        pass\"\n",
                "structure = code_extractor.extract_structure(py_code, language=\"python\")\n",
                "print(f\"Classes: {structure.get('classes')}\")\n",
                "print(f\"Functions: {structure.get('functions')}\")\n",
                "\n",
                "# --- RepoIngestor ---\n",
                "repo_ingestor = RepoIngestor()\n",
                "try:\n",
                "    repo_data = repo_ingestor.ingest_repository(\"https://github.com/Hawksight-AI/semantica.git\")\n",
                "    print(f\"Repo Name: {repo_data['name']}\")\n",
                "except Exception as e:\n",
                "    print(f\"Repo ingest failed: {e}\")\n",
                "\n",
                "# --- GitAnalyzer ---\n",
                "try:\n",
                "    analyzer = GitAnalyzer(\".\")\n",
                "    stats = analyzer.get_statistics()\n",
                "    print(f\"Commits in current repo: {stats.get('total_commits', 'N/A')}\")\n",
                "except Exception as e:\n",
                "    print(f\"Git analysis failed: {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 7. Email Ingestion\n",
                "\n",
                "Processing emails with `EmailIngestor` and `AttachmentProcessor`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import EmailIngestor, AttachmentProcessor\n",
                "\n",
                "# --- AttachmentProcessor ---\n",
                "att_processor = AttachmentProcessor()\n",
                "dummy_content = b\"PDF Content\"\n",
                "saved_path = att_processor.save_attachment(dummy_content, \"doc.pdf\", temp_dir)\n",
                "print(f\"Saved attachment to: {saved_path}\")\n",
                "\n",
                "# --- EmailIngestor ---\n",
                "email_ingestor = EmailIngestor()\n",
                "try:\n",
                "    email_ingestor.connect_imap(\"imap.gmail.com\", \"user\", \"pass\")\n",
                "    emails = email_ingestor.ingest_mailbox(\"INBOX\", max_emails=5)\n",
                "    print(f\"Fetched {len(emails)} emails\")\n",
                "except Exception as e:\n",
                "    print(f\"Email ingest failed (Auth required): {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 8. Database Ingestion\n",
                "\n",
                "Connecting to SQL databases with `DBIngestor` and `DatabaseConnector`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import DBIngestor, DatabaseConnector\n",
                "import sqlite3\n",
                "\n",
                "# Setup SQLite DB\n",
                "db_path = os.path.join(temp_dir, \"test.db\")\n",
                "conn = sqlite3.connect(db_path)\n",
                "conn.execute(\"CREATE TABLE items (id INT, name TEXT)\")\n",
                "conn.execute(\"INSERT INTO items VALUES (1, 'Item 1'), (2, 'Item 2')\")\n",
                "conn.commit()\n",
                "conn.close()\n",
                "\n",
                "# --- DatabaseConnector ---\n",
                "connector = DatabaseConnector()\n",
                "engine = connector.create_engine(f\"sqlite:///{db_path}\")\n",
                "print(f\"Connected to DB: {engine.name}\")\n",
                "\n",
                "# --- DBIngestor ---\n",
                "db_ingestor = DBIngestor()\n",
                "table_data = db_ingestor.ingest_database(f\"sqlite:///{db_path}\", table=\"items\")\n",
                "print(f\"Table: {table_data.table_name}\")\n",
                "print(f\"Rows: {table_data.row_count}\")\n",
                "for row in table_data.rows:\n",
                "    print(f\" - {row}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 9. MCP Ingestion\n",
                "\n",
                "Integrating with Model Context Protocol servers using `MCPIngestor`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import MCPIngestor\n",
                "\n",
                "mcp_ingestor = MCPIngestor()\n",
                "\n",
                "try:\n",
                "    # Connect\n",
                "    mcp_ingestor.connect(\"weather_server\", url=\"http://localhost:8000/mcp\")\n",
                "\n",
                "    # Ingest Resources\n",
                "    resources = mcp_ingestor.ingest_resources(\"weather_server\")\n",
                "    print(f\"Resources: {len(resources)}\")\n",
                "\n",
                "    # Call Tool\n",
                "    result = mcp_ingestor.ingest_tool_output(\"weather_server\", \"get_forecast\", {\"city\": \"NYC\"})\n",
                "    print(f\"Tool Result: {result.content}\")\n",
                "except Exception as e:\n",
                "    print(f\"MCP ingest failed (Server required): {e}\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 10. Configuration\n",
                "\n",
                "Managing ingestion settings with `IngestConfig`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.ingest import IngestConfig, ingest_config\n",
                "\n",
                "# Global config\n",
                "print(f\"Default Source Type: {ingest_config.get('default_source_type')}\")\n",
                "\n",
                "# Custom config instance\n",
                "config = IngestConfig()\n",
                "config.set(\"max_file_size\", 1024 * 1024) # 1MB\n",
                "print(f\"Max File Size: {config.get('max_file_size')} bytes\")\n"
            ]
        }
    ],
    "metadata": {
        "language_info": {
            "name": "python"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}