mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: highlight Databricks/Snowflake enterprise data ingestion, fix ingest doc bugs

Browse Source 
Makes enterprise lakehouse/warehouse ingestion (Databricks Unity Catalog +
Delta Lake, Snowflake) a first-class, prominently documented capability
across the README and guides, and adds matching runnable examples to
docs/guides/ingest.md. Also fixes several pre-existing inaccuracies caught
while auditing the ingest module docs against the actual source:
WebIngestor has no ingest_urls() (only singular ingest_url()), XMLIngestor's
XSD option is schema_path (not validate_xsd) and belongs on ingest() not the
constructor, and the "Available ingestors" list was missing DatabricksIngestor
while listing several classes not actually exported from semantica.ingest.
This commit is contained in:
KaifAhmad1
2026-07-28 11:58:09 +05:30
parent 80bce453c3
commit 6ed5aea993
7 changed files with 132 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+32 -2
View File
@@ -49,6 +49,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
**Who it's for:**
- **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index
- **Data platform teams on Databricks or Snowflake** who need to turn tables already sitting in Unity Catalog or a Snowflake warehouse into a governed, lineage-tracked knowledge graph, without exporting that data to a third-party SaaS first
- **Compliance, risk, and audit teams** who need a straight answer to "why did the AI do that?" in a format a regulator will actually accept
- **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box, and can't send their data to someone else's SaaS to get one
- **Platform and infra engineers** who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend
@@ -66,6 +67,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
- **Full Auditability:** W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF
- **Deterministic Reasoning:** Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes
- **Knowledge Pipeline:** Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout
- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection) and Snowflake (warehouse/database/schema, key-pair and OAuth auth), so tables already living in your lakehouse or warehouse become graph nodes with provenance, not another export/import hop
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
@@ -309,7 +311,7 @@ Every module below is independently importable, with working code samples verifi
| Module | What it does |
| --- | --- |
| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Snowflake, MCP |
| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, MCP |
| [`semantica.semantic_extract`](#semanticasemantic_extract-ner-relations-events-triplets) | NER, relation extraction, event detection, triplet generation |
| [`semantica.kg`](#semanticakg-knowledge-graph-construction--analysis) | Graph construction, centrality, communities, link prediction |
| [`semantica.reasoning`](#semanticareasoning-forward-chaining-rete-datalog-sparql) | Forward chaining, Rete, Datalog, SPARQL, fully explainable |
@@ -360,7 +362,34 @@ rows = DBIngestor().ingest_database(
)
```
**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`)
```python
# Enterprise data platforms - pull tables straight out of your lakehouse
# or warehouse, with lineage, instead of exporting to CSV first
from semantica.ingest import DatabricksIngestor, SnowflakeIngestor
# pip install "semantica[db-databricks]"
databricks = DatabricksIngestor(
host="https://adb-xxx.azuredatabricks.net",
token="dapi-xxxxxxxx", # or client_id/client_secret for OAuth M2M
http_path="/sql/1.0/warehouses/xxxxxxxx",
catalog="main",
)
customers = databricks.ingest_table("customers", limit=10_000)
sales = databricks.ingest_query("SELECT * FROM sales WHERE region = 'EMEA'")
table_lineage = databricks.get_table_lineage("main", "sales", "customers") # Unity Catalog lineage
# pip install semantica[db-snowflake]
snowflake = SnowflakeIngestor(
account="myaccount",
user="myuser",
password="mypassword", # or private_key / token for key-pair / OAuth auth
warehouse="COMPUTE_WH",
database="MYDB",
)
orders = snowflake.ingest_table("ORDERS", limit=10_000)
```
**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`)
Elasticsearch and Google Drive ingestion also ship (`ElasticIngestor`, `GDriveIngestor`) but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly: `from semantica.ingest.elastic_ingestor import ElasticIngestor`.
@@ -1111,6 +1140,7 @@ if report.valid:
| **Vector Store** | FAISS · Pinecone · Weaviate · Qdrant · Milvus · PgVector · hybrid + filtered search |
| **Graph Databases (LPG)** | Neo4j · FalkorDB · Apache AGE · AWS Neptune |
| **Triple Stores (RDF)** | Blazegraph · Apache Jena · Eclipse RDF4J · unified `TripletStore` interface · SPARQL query & bulk load |
| **Enterprise Data Platforms** | Databricks (`DatabricksIngestor`: Unity Catalog + Delta Lake, PAT/OAuth M2M, table/query ingestion, catalog/schema/table/lineage introspection) · Snowflake (`SnowflakeIngestor`: warehouse/database/schema, password/key-pair/OAuth auth) |
| **LLM Providers** | **All already supported today:** OpenAI (GPT-4o, o1, o3) · Anthropic (Claude) · Google Gemini · Mistral · Meta Llama · Groq · Cohere · Azure OpenAI · AWS Bedrock · Ollama · DeepSeek · Perplexity · Together AI · Fireworks AI · Replicate · HuggingFace · via `semantica.llms` and LiteLLM |
---
docs/architecture.md
+1 -1
View File
@@ -23,7 +23,7 @@ Loads data from any source into the pipeline as a unified `SourceDocument`.
| Parquet | `ingest.ParquetIngestor` | PyArrow, Hive-style partitions (v0.5.0) |
| XML | `ingest.XMLIngestor` | XXE-safe lxml, XSD/DTD validation (v0.5.0) |
| Web pages | `ingest.WebIngestor` | Configurable depth, link filtering |
| SQL / Snowflake | `ingest.DBIngestor` / `ingest.SnowflakeIngestor` | Custom SQL, schema introspection |
| SQL / Snowflake / Databricks | `ingest.DBIngestor` / `ingest.SnowflakeIngestor` / `ingest.DatabricksIngestor` | Custom SQL, schema introspection, Unity Catalog lineage |
| Kafka / streams | `ingest.StreamIngestor` | Real-time feed ingestion |
| Email | `ingest.EmailIngestor` | IMAP/SMTP with attachment extraction |
| Repositories | `ingest.RepoIngestor` | Git repos, code structure |
docs/choose-your-module.md
+1 -1
View File
@@ -18,7 +18,7 @@ Find your goal below. The **Module** column is your import path; **Key class** i
| Crawl a website | `ingest` | `WebIngestor` |
| Load Parquet files or partitioned datasets | `ingest` | `ParquetIngestor` |
| Ingest XML with schema validation | `ingest` | `XMLIngestor` |
| Ingest from SQL, Snowflake, Kafka, or email | `ingest` | `DBIngestor`, `SnowflakeIngestor`, `StreamIngestor` |
| Ingest from SQL, Snowflake, Databricks, Kafka, or email | `ingest` | `DBIngestor`, `SnowflakeIngestor`, `DatabricksIngestor`, `StreamIngestor` |
| Extract clean text and tables from a document | `parse` | `DocumentParser` |
| Parse complex PDFs with OCR or multi-column layout | `parse` | `DoclingParser` |
| Chunk text for embedding or RAG | `split` | `TextSplitter` |
docs/faq.md
+1 -1
View File
@@ -129,7 +129,7 @@ If you're on an older version, install extras individually: `pip install "semant
| :-------- | :------- |
| **Files** | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, Parquet (v0.5.0), XML (v0.5.0), archives |
| **Web** | `WebIngestor` crawl, RSS feeds, sitemaps |
| **Databases** | PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor` |
| **Databases** | PostgreSQL, MySQL, Snowflake, Databricks via `DBIngestor` / `SnowflakeIngestor` / `DatabricksIngestor` |
| **NoSQL** | MongoDB via `MongoIngestor`, DuckDB via `DuckDBIngestor` |
| **Streams** | Kafka, real-time ingestion via `StreamIngestor` |
| **Protocols** | MCP (Model Context Protocol) via `MCPIngestor` |
docs/guides/ingest.md
+84
View File
@@ -50,6 +50,7 @@ Use the ingest module when your data lives outside Semantica and you need to bri
- **Web content** — public documentation sites, regulatory publication pages, news feeds, or any URL you can crawl.
- **REST APIs** — internal platforms (SIEM, EDR, ITSM, CRM), threat intelligence feeds, or any paginated HTTP endpoint.
- **Databases** — existing SQL databases where relevant records can be fetched with a targeted query.
- **Enterprise data platforms** — tables already living in a Databricks lakehouse (Unity Catalog + Delta Lake) or a Snowflake warehouse, without exporting to CSV first.
- **Live streams** — Kafka or other message brokers where you need to process events as they arrive.
- **Git repositories** — source code, documentation, or configuration files tracked in version control.
@@ -298,6 +299,87 @@ for bundle in stix_xml_files:
print(f"{bundle.source_path}: {len(bundle.elements)} elements parsed")
```
## Source 6 — Enterprise Data Platforms (Databricks & Snowflake)
`DatabricksIngestor` and `SnowflakeIngestor` return the same shape as `DBIngestor` — a typed object (`DatabricksData` / `SnowflakeData`) whose `.data` field is `List[Dict]`, one dict per row. The same "transform to text, then store" pattern from Source 3 applies: pull only the tables and columns you need with a targeted query, then build a sentence per record before handing it to `AgentContext.store()`.
```python
from semantica.ingest import DatabricksIngestor
# Unity Catalog + Delta Lake — PAT or OAuth M2M auth
databricks = DatabricksIngestor(
host="https://adb-xxx.azuredatabricks.net",
token="dapi-xxxxxxxx",
http_path="/sql/1.0/warehouses/xxxxxxxx",
catalog="main",
)
# .data is List[Dict] — one dict per row, same shape as DBIngestor.execute_query()
customers = databricks.ingest_query(
"SELECT customer_id, name, industry, arr FROM main.default.customers "
"WHERE churn_risk_score > 0.7"
)
customer_texts = [
f"Customer {r['customer_id']} ({r['name']}, {r['industry']}): "
f"ARR ${r['arr']:,}, flagged high churn risk"
for r in customers.data
]
# Unity Catalog lineage — build Table --DEPENDS_ON--> Table edges directly from
# Unity Catalog's own lineage tracking, instead of re-deriving them from query logs
lineage = databricks.get_table_lineage("customers", catalog="main", schema="default")
lineage_texts = [
f"Table main.default.customers depends on {upstream}"
for upstream in lineage["upstream"]
]
```
```python
from semantica.ingest import SnowflakeIngestor
snowflake = SnowflakeIngestor(
account="myaccount",
user="myuser",
password="mypassword", # or private_key / token for key-pair / OAuth auth
warehouse="COMPUTE_WH",
database="ANALYTICS",
schema="PUBLIC",
)
# Snowflake uppercases unquoted identifiers, so unquoted columns come back
# as ORDER_ID, PRODUCT, etc. unless the source table quotes them lowercase
orders = snowflake.ingest_query(
"SELECT order_id, product, region, amount FROM orders "
"WHERE order_date >= DATEADD(day, -30, CURRENT_DATE())"
)
order_texts = [
f"Order {r['ORDER_ID']}: {r['PRODUCT']} in {r['REGION']}, ${r['AMOUNT']}"
for r in orders.data
]
```
Feed the resulting text lists into `AgentContext.store()` exactly like any other structured source:
```python
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
graph = ContextGraph(advanced_analytics=True)
context = AgentContext(
vector_store = VectorStore(backend="faiss"),
knowledge_graph = graph,
)
context.store(
customer_texts + lineage_texts + order_texts,
extract_entities=True,
extract_relationships=True,
)
print(f"Enterprise data graph: {graph.stats()['node_count']} nodes")
```
For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-pair vs. OAuth for Snowflake), schema/catalog introspection, and troubleshooting, see the dedicated [Databricks Integration](../integrations/databricks) and [Snowflake Integration](../integrations/snowflake) guides.
## Combining All Five Sources
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
@@ -831,3 +913,5 @@ print(f"Compliance graph: {graph.stats()['node_count']} nodes, "
- [Context Graphs](context-graphs) — storing and querying the entities you ingest as a typed property graph
- [Semantic Extraction](semantic-extraction) — NER, relation extraction, and triplet extraction from ingested text
- [Provenance](provenance) — tracking the origin document, confidence score, and ingestion timestamp for every extracted entity
- [Databricks Integration](../integrations/databricks) — Unity Catalog setup, PAT/OAuth M2M authentication, and lineage introspection
- [Snowflake Integration](../integrations/snowflake) — warehouse setup and password/key-pair/OAuth authentication
docs/modules.md
+12 -4
View File
@@ -39,18 +39,26 @@ documents = ingestor.ingest_directory("data/")
# Web crawl
web_ingestor = WebIngestor()
pages = web_ingestor.ingest_urls(["https://example.com"])
page = web_ingestor.ingest_url("https://example.com")
# Parquet: single file, partitioned directory, Hive-style (v0.5.0)
parquet = ParquetIngestor()
sources = parquet.ingest("data/events.parquet")
# XML with XSD/DTD validation, namespace handling (v0.5.0)
xml = XMLIngestor(validate_xsd="schema.xsd")
sources = xml.ingest("data/records/")
xml = XMLIngestor()
sources = xml.ingest("data/records/", schema_path="schema.xsd")
# Enterprise lakehouse/warehouse — Unity Catalog + Delta Lake, or a Snowflake warehouse
databricks = DatabricksIngestor(host="...", token="...", http_path="...")
customers = databricks.ingest_table("customers")
```
**Available ingestors:** `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor`, `RESTIngestor`, `PublicAPIIngestor`, `DBIngestor`, `DuckDBIngestor`, `ElasticIngestor`, `EmailIngestor`, `FeedIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MCPIngestor`, `MongoIngestor`, `OntologyIngestor`, `PandasIngestor`, `RepoIngestor`, `SnowflakeIngestor`, `StreamIngestor`
**Available ingestors:** `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor`, `RESTIngestor`, `PublicAPIIngestor`, `DBIngestor`, `DatabricksIngestor`, `SnowflakeIngestor`, `EmailIngestor`, `FeedIngestor`, `MCPIngestor`, `OntologyIngestor`, `RepoIngestor`, `StreamIngestor`, `ArrowIngestor`, `CloudStorageIngestor`
<Note>
`DuckDBIngestor`, `ElasticIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MongoIngestor`, and `PandasIngestor` also ship but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly, e.g. `from semantica.ingest.duckdb_ingestor import DuckDBIngestor`.
</Note>
### Parse
docs/reference/ingest.md
+1 -1
View File
@@ -6,7 +6,7 @@ icon: "database"
**`semantica.ingest`** is the **universal entry point** for loading data into Semantica:
- 15+ ingestion adapters: files, web, SQL, Snowflake, Kafka, MCP, Git repos, email
- 15+ ingestion adapters: files, web, SQL, Databricks, Snowflake, Kafka, MCP, Git repos, email
- PyArrow Parquet with column selection and partitioned dataset support
- XXE-safe lxml XML with optional XSD schema validation
- `ingest()` unified dispatcher: auto-detects source type from path or URL