diff --git a/README.md b/README.md index d02a7020..23eadf4d 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter **Who it's for:** - **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index +- **Data platform teams on Databricks or Snowflake** who need to turn tables already sitting in Unity Catalog or a Snowflake warehouse into a governed, lineage-tracked knowledge graph, without exporting that data to a third-party SaaS first - **Compliance, risk, and audit teams** who need a straight answer to "why did the AI do that?" in a format a regulator will actually accept - **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box, and can't send their data to someone else's SaaS to get one - **Platform and infra engineers** who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend @@ -66,6 +67,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter - **Full Auditability:** W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF - **Deterministic Reasoning:** Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes - **Knowledge Pipeline:** Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout +- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection) and Snowflake (warehouse/database/schema, key-pair and OAuth auth), so tables already living in your lakehouse or warehouse become graph nodes with provenance, not another export/import hop - **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built - **Polyglot Graph Storage:** Native RDF (Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code - **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench @@ -309,7 +311,7 @@ Every module below is independently importable, with working code samples verifi | Module | What it does | | --- | --- | -| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Snowflake, MCP | +| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, MCP | | [`semantica.semantic_extract`](#semanticasemantic_extract-ner-relations-events-triplets) | NER, relation extraction, event detection, triplet generation | | [`semantica.kg`](#semanticakg-knowledge-graph-construction--analysis) | Graph construction, centrality, communities, link prediction | | [`semantica.reasoning`](#semanticareasoning-forward-chaining-rete-datalog-sparql) | Forward chaining, Rete, Datalog, SPARQL, fully explainable | @@ -360,7 +362,34 @@ rows = DBIngestor().ingest_database( ) ``` -**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`) +```python +# Enterprise data platforms - pull tables straight out of your lakehouse +# or warehouse, with lineage, instead of exporting to CSV first +from semantica.ingest import DatabricksIngestor, SnowflakeIngestor + +# pip install "semantica[db-databricks]" +databricks = DatabricksIngestor( + host="https://adb-xxx.azuredatabricks.net", + token="dapi-xxxxxxxx", # or client_id/client_secret for OAuth M2M + http_path="/sql/1.0/warehouses/xxxxxxxx", + catalog="main", +) +customers = databricks.ingest_table("customers", limit=10_000) +sales = databricks.ingest_query("SELECT * FROM sales WHERE region = 'EMEA'") +table_lineage = databricks.get_table_lineage("main", "sales", "customers") # Unity Catalog lineage + +# pip install semantica[db-snowflake] +snowflake = SnowflakeIngestor( + account="myaccount", + user="myuser", + password="mypassword", # or private_key / token for key-pair / OAuth auth + warehouse="COMPUTE_WH", + database="MYDB", +) +orders = snowflake.ingest_table("ORDERS", limit=10_000) +``` + +**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`) Elasticsearch and Google Drive ingestion also ship (`ElasticIngestor`, `GDriveIngestor`) but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly: `from semantica.ingest.elastic_ingestor import ElasticIngestor`. @@ -1111,6 +1140,7 @@ if report.valid: | **Vector Store** | FAISS · Pinecone · Weaviate · Qdrant · Milvus · PgVector · hybrid + filtered search | | **Graph Databases (LPG)** | Neo4j · FalkorDB · Apache AGE · AWS Neptune | | **Triple Stores (RDF)** | Blazegraph · Apache Jena · Eclipse RDF4J · unified `TripletStore` interface · SPARQL query & bulk load | +| **Enterprise Data Platforms** | Databricks (`DatabricksIngestor`: Unity Catalog + Delta Lake, PAT/OAuth M2M, table/query ingestion, catalog/schema/table/lineage introspection) · Snowflake (`SnowflakeIngestor`: warehouse/database/schema, password/key-pair/OAuth auth) | | **LLM Providers** | **All already supported today:** OpenAI (GPT-4o, o1, o3) · Anthropic (Claude) · Google Gemini · Mistral · Meta Llama · Groq · Cohere · Azure OpenAI · AWS Bedrock · Ollama · DeepSeek · Perplexity · Together AI · Fireworks AI · Replicate · HuggingFace · via `semantica.llms` and LiteLLM | --- diff --git a/docs/architecture.md b/docs/architecture.md index 4d7e9faa..25612793 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -23,7 +23,7 @@ Loads data from any source into the pipeline as a unified `SourceDocument`. | Parquet | `ingest.ParquetIngestor` | PyArrow, Hive-style partitions (v0.5.0) | | XML | `ingest.XMLIngestor` | XXE-safe lxml, XSD/DTD validation (v0.5.0) | | Web pages | `ingest.WebIngestor` | Configurable depth, link filtering | -| SQL / Snowflake | `ingest.DBIngestor` / `ingest.SnowflakeIngestor` | Custom SQL, schema introspection | +| SQL / Snowflake / Databricks | `ingest.DBIngestor` / `ingest.SnowflakeIngestor` / `ingest.DatabricksIngestor` | Custom SQL, schema introspection, Unity Catalog lineage | | Kafka / streams | `ingest.StreamIngestor` | Real-time feed ingestion | | Email | `ingest.EmailIngestor` | IMAP/SMTP with attachment extraction | | Repositories | `ingest.RepoIngestor` | Git repos, code structure | diff --git a/docs/choose-your-module.md b/docs/choose-your-module.md index 397f24be..0a4a9d2c 100644 --- a/docs/choose-your-module.md +++ b/docs/choose-your-module.md @@ -18,7 +18,7 @@ Find your goal below. The **Module** column is your import path; **Key class** i | Crawl a website | `ingest` | `WebIngestor` | | Load Parquet files or partitioned datasets | `ingest` | `ParquetIngestor` | | Ingest XML with schema validation | `ingest` | `XMLIngestor` | -| Ingest from SQL, Snowflake, Kafka, or email | `ingest` | `DBIngestor`, `SnowflakeIngestor`, `StreamIngestor` | +| Ingest from SQL, Snowflake, Databricks, Kafka, or email | `ingest` | `DBIngestor`, `SnowflakeIngestor`, `DatabricksIngestor`, `StreamIngestor` | | Extract clean text and tables from a document | `parse` | `DocumentParser` | | Parse complex PDFs with OCR or multi-column layout | `parse` | `DoclingParser` | | Chunk text for embedding or RAG | `split` | `TextSplitter` | diff --git a/docs/faq.md b/docs/faq.md index 9195d100..8cead87c 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -129,7 +129,7 @@ If you're on an older version, install extras individually: `pip install "semant | :-------- | :------- | | **Files** | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, Parquet (v0.5.0), XML (v0.5.0), archives | | **Web** | `WebIngestor` crawl, RSS feeds, sitemaps | -| **Databases** | PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor` | +| **Databases** | PostgreSQL, MySQL, Snowflake, Databricks via `DBIngestor` / `SnowflakeIngestor` / `DatabricksIngestor` | | **NoSQL** | MongoDB via `MongoIngestor`, DuckDB via `DuckDBIngestor` | | **Streams** | Kafka, real-time ingestion via `StreamIngestor` | | **Protocols** | MCP (Model Context Protocol) via `MCPIngestor` | diff --git a/docs/guides/ingest.md b/docs/guides/ingest.md index 62549dd6..87f066fb 100644 --- a/docs/guides/ingest.md +++ b/docs/guides/ingest.md @@ -50,6 +50,7 @@ Use the ingest module when your data lives outside Semantica and you need to bri - **Web content** — public documentation sites, regulatory publication pages, news feeds, or any URL you can crawl. - **REST APIs** — internal platforms (SIEM, EDR, ITSM, CRM), threat intelligence feeds, or any paginated HTTP endpoint. - **Databases** — existing SQL databases where relevant records can be fetched with a targeted query. +- **Enterprise data platforms** — tables already living in a Databricks lakehouse (Unity Catalog + Delta Lake) or a Snowflake warehouse, without exporting to CSV first. - **Live streams** — Kafka or other message brokers where you need to process events as they arrive. - **Git repositories** — source code, documentation, or configuration files tracked in version control. @@ -298,6 +299,87 @@ for bundle in stix_xml_files: print(f"{bundle.source_path}: {len(bundle.elements)} elements parsed") ``` +## Source 6 — Enterprise Data Platforms (Databricks & Snowflake) + +`DatabricksIngestor` and `SnowflakeIngestor` return the same shape as `DBIngestor` — a typed object (`DatabricksData` / `SnowflakeData`) whose `.data` field is `List[Dict]`, one dict per row. The same "transform to text, then store" pattern from Source 3 applies: pull only the tables and columns you need with a targeted query, then build a sentence per record before handing it to `AgentContext.store()`. + +```python +from semantica.ingest import DatabricksIngestor + +# Unity Catalog + Delta Lake — PAT or OAuth M2M auth +databricks = DatabricksIngestor( + host="https://adb-xxx.azuredatabricks.net", + token="dapi-xxxxxxxx", + http_path="/sql/1.0/warehouses/xxxxxxxx", + catalog="main", +) + +# .data is List[Dict] — one dict per row, same shape as DBIngestor.execute_query() +customers = databricks.ingest_query( + "SELECT customer_id, name, industry, arr FROM main.default.customers " + "WHERE churn_risk_score > 0.7" +) +customer_texts = [ + f"Customer {r['customer_id']} ({r['name']}, {r['industry']}): " + f"ARR ${r['arr']:,}, flagged high churn risk" + for r in customers.data +] + +# Unity Catalog lineage — build Table --DEPENDS_ON--> Table edges directly from +# Unity Catalog's own lineage tracking, instead of re-deriving them from query logs +lineage = databricks.get_table_lineage("customers", catalog="main", schema="default") +lineage_texts = [ + f"Table main.default.customers depends on {upstream}" + for upstream in lineage["upstream"] +] +``` + +```python +from semantica.ingest import SnowflakeIngestor + +snowflake = SnowflakeIngestor( + account="myaccount", + user="myuser", + password="mypassword", # or private_key / token for key-pair / OAuth auth + warehouse="COMPUTE_WH", + database="ANALYTICS", + schema="PUBLIC", +) + +# Snowflake uppercases unquoted identifiers, so unquoted columns come back +# as ORDER_ID, PRODUCT, etc. unless the source table quotes them lowercase +orders = snowflake.ingest_query( + "SELECT order_id, product, region, amount FROM orders " + "WHERE order_date >= DATEADD(day, -30, CURRENT_DATE())" +) +order_texts = [ + f"Order {r['ORDER_ID']}: {r['PRODUCT']} in {r['REGION']}, ${r['AMOUNT']}" + for r in orders.data +] +``` + +Feed the resulting text lists into `AgentContext.store()` exactly like any other structured source: + +```python +from semantica.context import AgentContext, ContextGraph +from semantica.vector_store import VectorStore + +graph = ContextGraph(advanced_analytics=True) +context = AgentContext( + vector_store = VectorStore(backend="faiss"), + knowledge_graph = graph, +) + +context.store( + customer_texts + lineage_texts + order_texts, + extract_entities=True, + extract_relationships=True, +) +print(f"Enterprise data graph: {graph.stats()['node_count']} nodes") +``` + +For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-pair vs. OAuth for Snowflake), schema/catalog introspection, and troubleshooting, see the dedicated [Databricks Integration](../integrations/databricks) and [Snowflake Integration](../integrations/snowflake) guides. + ## Combining All Five Sources Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly. @@ -831,3 +913,5 @@ print(f"Compliance graph: {graph.stats()['node_count']} nodes, " - [Context Graphs](context-graphs) — storing and querying the entities you ingest as a typed property graph - [Semantic Extraction](semantic-extraction) — NER, relation extraction, and triplet extraction from ingested text - [Provenance](provenance) — tracking the origin document, confidence score, and ingestion timestamp for every extracted entity +- [Databricks Integration](../integrations/databricks) — Unity Catalog setup, PAT/OAuth M2M authentication, and lineage introspection +- [Snowflake Integration](../integrations/snowflake) — warehouse setup and password/key-pair/OAuth authentication diff --git a/docs/modules.md b/docs/modules.md index 5be77677..92d510d2 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -39,18 +39,26 @@ documents = ingestor.ingest_directory("data/") # Web crawl web_ingestor = WebIngestor() -pages = web_ingestor.ingest_urls(["https://example.com"]) +page = web_ingestor.ingest_url("https://example.com") # Parquet: single file, partitioned directory, Hive-style (v0.5.0) parquet = ParquetIngestor() sources = parquet.ingest("data/events.parquet") # XML with XSD/DTD validation, namespace handling (v0.5.0) -xml = XMLIngestor(validate_xsd="schema.xsd") -sources = xml.ingest("data/records/") +xml = XMLIngestor() +sources = xml.ingest("data/records/", schema_path="schema.xsd") + +# Enterprise lakehouse/warehouse — Unity Catalog + Delta Lake, or a Snowflake warehouse +databricks = DatabricksIngestor(host="...", token="...", http_path="...") +customers = databricks.ingest_table("customers") ``` -**Available ingestors:** `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor`, `RESTIngestor`, `PublicAPIIngestor`, `DBIngestor`, `DuckDBIngestor`, `ElasticIngestor`, `EmailIngestor`, `FeedIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MCPIngestor`, `MongoIngestor`, `OntologyIngestor`, `PandasIngestor`, `RepoIngestor`, `SnowflakeIngestor`, `StreamIngestor` +**Available ingestors:** `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor`, `RESTIngestor`, `PublicAPIIngestor`, `DBIngestor`, `DatabricksIngestor`, `SnowflakeIngestor`, `EmailIngestor`, `FeedIngestor`, `MCPIngestor`, `OntologyIngestor`, `RepoIngestor`, `StreamIngestor`, `ArrowIngestor`, `CloudStorageIngestor` + + + `DuckDBIngestor`, `ElasticIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MongoIngestor`, and `PandasIngestor` also ship but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly, e.g. `from semantica.ingest.duckdb_ingestor import DuckDBIngestor`. + ### Parse diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index fdd62df3..99eef7b2 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -6,7 +6,7 @@ icon: "database" **`semantica.ingest`** is the **universal entry point** for loading data into Semantica: -- 15+ ingestion adapters: files, web, SQL, Snowflake, Kafka, MCP, Git repos, email +- 15+ ingestion adapters: files, web, SQL, Databricks, Snowflake, Kafka, MCP, Git repos, email - PyArrow Parquet with column selection and partitioned dataset support - XXE-safe lxml XML with optional XSD schema validation - `ingest()` unified dispatcher: auto-detects source type from path or URL