Add Databricks connector (Unity Catalog + Delta Lake ingestion)

Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.

Closes #747
This commit is contained in:
KaifAhmad1
2026-07-15 22:07:25 +05:30
parent fabff5d9ec
commit d74b650643
9 changed files with 1907 additions and 2 deletions
+20
View File
@@ -27,6 +27,7 @@ icon: "database"
| `RepoIngestor` | Git repositories: source files, commit history, and metadata |
| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
| `EmailIngestor` | IMAP/POP3 email ingestion with attachment extraction |
@@ -440,6 +441,24 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
result = ingestor.ingest_query("SELECT * FROM documents")
result = ingestor.ingest_table("documents")
```
### DatabricksIngestor
```python
from semantica.ingest import DatabricksIngestor
import os
ingestor = DatabricksIngestor(
host=os.getenv("DATABRICKS_HOST"),
token=os.getenv("DATABRICKS_TOKEN"),
http_path=os.getenv("DATABRICKS_HTTP_PATH"),
catalog="main",
schema="default",
)
result = ingestor.ingest_query("SELECT * FROM documents")
result = ingestor.ingest_table("documents")
lineage = ingestor.get_table_lineage("documents")
```
</Tab>
<Tab title="Stream">
### StreamIngestor
@@ -628,4 +647,5 @@ result = ingest_file("source_path", method="my_format")
- [Parse](parse) — Parse raw sources into structured text and tables.
- [Pipeline](pipeline) — Orchestrate ingest as the first pipeline step.
- [Snowflake Integration](../integrations/snowflake) — Snowflake-specific setup and authentication guide.
- [Databricks Integration](../integrations/databricks) — Databricks Unity Catalog setup, authentication, and lineage guide.
- [Provenance](provenance) — Track lineage from ingest through to inference.