semantica/docs/reference/ingest.md

title, description, icon

title	description	icon
Ingest Module	Universal data ingestion from files, Parquet, XML, web, feeds, streams, repositories, email, and databases.	database

Universal data ingestion — the entry point for loading data into Semantica.

---

Overview

The Ingest Module handles loading data from every common source into Semantica. All ingestors return a list of DataSource objects with content and metadata.

---

FileIngestor

from semantica.ingest import FileIngestor

ingestor = FileIngestor()

# Single file
sources = ingestor.ingest("data/report.pdf")

# Directory (recursive)
sources = ingestor.ingest_directory("data/", recursive=True)

# Glob pattern
sources = ingestor.ingest("data/**/*.docx")

Supported formats: PDF, DOCX, TXT, HTML, JSON, CSV, Excel (XLSX/XLS), PPTX, ZIP/TAR archives. File type is auto-detected.

---

ParquetIngestor (v0.5.0)

PyArrow-based ingestion for Apache Parquet files.

from semantica.ingest import ParquetIngestor

ingestor = ParquetIngestor()

# Single file
sources = ingestor.ingest("data/events.parquet")

# Partitioned directory (Hive-style: year=2024/month=01/...)
sources = ingestor.ingest("data/partitioned/")

# Selective columns
sources = ingestor.ingest("data/events.parquet", columns=["id", "text", "timestamp"])

---

XMLIngestor (v0.5.0)

XXE-safe lxml-based ingestion with optional XSD/DTD validation.

from semantica.ingest import XMLIngestor

# Basic ingestion
ingestor = XMLIngestor()
sources = ingestor.ingest("data/records.xml")

# With XSD validation
ingestor = XMLIngestor(validate_xsd="schema.xsd")
sources = ingestor.ingest("data/records/")

# With DTD validation
ingestor = XMLIngestor(validate_dtd=True)
sources = ingestor.ingest("data/feed.xml")

The `XMLIngestor` uses lxml with XXE disabled (`resolve_entities=False`) to prevent XML External Entity injection attacks.

---

WebIngestor

from semantica.ingest import WebIngestor

ingestor = WebIngestor(
    rate_limit=1.0,        # seconds between requests
    respect_robots=True,   # honor robots.txt
    max_depth=2            # crawl depth
)

# Single URL
sources = ingestor.ingest("https://example.com/about")

# Multiple URLs
sources = ingestor.ingest_urls([
    "https://example.com/page1",
    "https://example.com/page2",
])

---

FeedIngestor (RSS/Atom)

from semantica.ingest import FeedIngestor

ingestor = FeedIngestor()
sources = ingestor.ingest("https://feeds.example.com/rss")

# Monitor for updates
ingestor.monitor("https://feeds.example.com/rss", interval=300, callback=process_new_items)

---

DBIngestor (SQL / NoSQL)

from semantica.ingest import DBIngestor

# PostgreSQL
ingestor = DBIngestor(
    connection_string="postgresql://user:pass@localhost/db",
    query="SELECT id, content, created_at FROM documents WHERE status='active'"
)
sources = ingestor.ingest()

---

SnowflakeIngestor

from semantica.ingest import SnowflakeIngestor

ingestor = SnowflakeIngestor(
    account=os.getenv("SNOWFLAKE_ACCOUNT"),
    user=os.getenv("SNOWFLAKE_USER"),
    password=os.getenv("SNOWFLAKE_PASSWORD"),
    warehouse="COMPUTE_WH",
    database="ANALYTICS",
    schema="PUBLIC"
)
sources = ingestor.ingest(query="SELECT * FROM documents")

---

Other Ingestors

Class	Source
StreamIngestor	Kafka, RabbitMQ, Kinesis, Pulsar
RepoIngestor	Git repositories (GitHub, GitLab)
EmailIngestor	IMAP/POP3 servers
MCPIngestor	Model Context Protocol servers
S3Ingestor	AWS S3 buckets
GCSIngestor	Google Cloud Storage

---

DataSource Object

All ingestors return a list of DataSource objects:

@dataclass
class DataSource:
    content: str            # raw text content
    source_id: str          # unique identifier
    source_type: str        # "file", "web", "database", etc.
    metadata: Dict          # title, author, url, date, etc.
    raw_bytes: Optional[bytes]

---

See Also

Parse raw sources into structured text. Orchestrate ingest as part of a pipeline. Snowflake-specific setup guide. Track lineage from ingest to inference.