feat(ingest): add Salesforce ingestor
Adds first-class Salesforce ingestion support, following the existing
Connector + Data + Ingestor architecture already used by the
Snowflake and Databricks integrations: SalesforceConnector /
SalesforceData / SalesforceIngestor, exposed lazily from
semantica.ingest so the base install stays unaffected.
SalesforceConnector supports both auth landscapes Salesforce actually
uses in practice: username + password + security token (SOAP login,
on-prem/sandbox), and session_id + instance_url for reusing an
existing authenticated session. Production and sandbox are selected
through domain, credentials can come from environment variables, and
the connector never intentionally puts credential material into logs,
exceptions, or its own repr.
SalesforceIngestor covers ingest_sobject(), ingest_query(),
list_sobjects(), get_sobject_schema(), and export_as_documents(),
against standard sObjects, custom objects (__c), custom metadata
objects (__mdt), platform events (__e), namespaced objects, and
relationship-field traversal (Owner.Name). Pagination follows
nextRecordsUrl/query_more() automatically and stops once a caller's
limit is satisfied rather than continuing to fetch full pages past it.
Dynamically constructed SOQL is validated before it's sent: sObject
names, field names, relationship paths, ORDER BY expressions, and
numeric limits are checked, and WHERE fragments are screened against
common injection primitives after masking quoted string literals so a
value like status = 'union' doesn't false-positive. Raw SOQL passed
directly to ingest_query() stays intentionally caller-controlled,
since that method is documented as the advanced/unvalidated escape
hatch.
Salesforce-specific attributes metadata is stripped from returned
records before they're handed to the rest of the pipeline, while
relationship data, normal field values, and datetime normalization
are preserved. export_as_documents() uses the Salesforce Id as the
stable document identifier and keeps the source record in document
metadata for provenance.
Wired into the unified ingestion API via ingest_salesforce() and
ingest(source_type="salesforce", ...), registered with
MethodRegistry under sobject/query/list_sobjects/schema/documents.
Isolated behind the semantica[db-salesforce] extra
(simple-salesforce>=1.12.0), included in db-all.
JWT Bearer authentication and Bulk API 2.0 are intentionally out of
scope for this first connector; both are documented as deliberate
follow-ups rather than gaps.
fix(ingest): address Salesforce review findings
- limit now validates as a non-negative integer before use; negative,
string, and float values raise ValidationError instead of silently
returning an empty result, raising a bare TypeError, or building an
invalid LIMIT 0 query
- fields is validated as a non-empty list of strings; a bare string
(e.g. "Id") no longer gets iterated character-by-character into
nonsense field names, and an empty list no longer builds a
syntactically invalid SELECT
- the generic connection-failure path now raises with `from None`
instead of chaining the original exception, so credential or
request detail from the underlying library can't surface through a
traceback
- the unified ingest() dispatch no longer coerces a non-dict source
into None and silently falling back to environment credentials; an
invalid source now raises
- _validate_order_by rewritten to validate each dot-separated
component through _validate_field_name, rejecting malformed
fragments like "Name." or "Owner..Name" that the previous regex let
through
- CI conflicts from parallel merges resolved; upstream markdown
dependency changes preserved
test(ingest): add Salesforce JWT coverage
Adds construction and connect() coverage for the JWT Bearer auth path
(consumer_key + privatekey/privatekey_file), the one auth mode that
had no dedicated tests despite handling private key material.
Also removes _SAFE_ORDER_RE, left behind as dead code once
_validate_order_by was rewritten to use _validate_field_name per
component, and fixes a test-isolation leak where an earlier test left
SALESFORCE_AVAILABLE=True behind for a later test that expected it
False when simple-salesforce isn't installed.
* feat(crewai): add first-class CrewAI integration (#962)
Add native CrewAI support so Crew agents can share a ContextGraph and
AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching
the existing agno integration pattern.
- SemanticaKGTool: 5 KG actions (extract_entities, extract_relations,
add_to_graph, query_graph, find_related) with sync run()/async arun()
- SemanticaDecisionTool: 5 decision-intelligence actions
(record_decision, find_precedents, trace_causal_chain,
analyze_impact, check_policy) over AgentContext
- SemanticaKnowledgeSource: serializes a ContextGraph into crew
knowledge storage; bridges legacy load_content() and current
validate_content()/aadd() contracts for crewai>=0.80.0
- All classes degrade gracefully when crewai is absent
- New pip extra crewai=... included in the all bundle
- 70 new tests (stub-based present-case + subprocess degradation path)
- Docs: integrations/crewai.md, docs.json nav, README matrix updates
* fix(crewai): harden tools against real Semantica dataclass shapes (#962)
Bugs found during live testing with crewai 1.15.16:
- SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses
('str' object has no attribute 'end_char'): string names were passed to
extract_relations(entities=...), which requires Entity objects, and the
tool read .name/.source/.target instead of Entity's .text/.label and
Relation's .subject/.object. Add shape-agnostic field helpers.
- SemanticaDecisionTool() created an AgentContext without a knowledge_graph,
so _decision_backend was never set and record_decision raised 'Decision
tracking is not enabled'. Wire in a ContextGraph.
- record_decision hard-failed when the agent omitted optional fields; fall
back to category='general', reasoning='agent decision',
outcome='recorded'.
Add tests covering real Entity/Relation dataclass shapes and the live
auto-created AgentContext path (now 77 crewai tests, 212 total).
* fix(crewai): make find_related traverse edges undirected (#962)
ContextGraph.get_neighbors only follows outgoing edges, so a node whose
only edge is incoming (A -> B) reported no related concepts. Rebuild a
bidirectional adjacency from find_edges() in SemanticaKGTool._find_related
so 'related' honors both directions.
* fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962)
- Exclude live graph/context/extractor state from JSON serialization
(model_dump(mode="json")) so CrewAI checkpointing no longer raises
PydanticSerializationError; model_post_init self-heals defaults on restore
- query_graph now searches node content via graph.query() plus id/type
- trace_causal_chain returns an explicit error when causal tracing is
unavailable instead of substituting similarity precedents; call
trace_decision_causality(..., max_depth=...) with the correct kwarg name
- find_precedents propagates max_precedents/limit to the backend instead of
being silently capped at 10
- Serialize add_to_graph batches under a module lock to prevent concurrent
double-counting; skip nameless entities instead of creating repr()-junk nodes
- aadd() runs CPU-bound serialization in a thread executor
- Mirror crewai args_schema serialize/restore in the conftest stub and add
serialization regression tests (crewai: 92 tests)
* fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962)
- _eval_rule now coerces rule values type-aware: bool("false") was truthy, so
'enabled == false' reported a violation for enabled=false, and string datums
like "0.90" were compared lexicographically instead of numerically
- _trace_causal_chain no longer raises AttributeError (which escaped _run) when
the decision context lacks knowledge_graph; returns honest error JSON
- SemanticaKnowledgeSource storage failures log an actionable ERROR; without a
configured crew embedder agents previously retrieved nothing silently
- add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a
process-global one: independent graphs no longer serialize each other and
re-entrant extractor callbacks cannot deadlock
- entity/relation confidence=None normalizes to 1.0 instead of failing the
whole extraction with float(None)
- add subprocess integration test against real crewai covering Crew-level
serialization round-trip and checkpoint restore (stub tests cannot see it)
- docs: embedder requirement for SemanticaKnowledgeSource; resume contract note
* fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962)
Re-verification against real crewai showed the embedder-missing failure raises
ValueError even though storage IS wired, so the old except-ValueError branch
mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure.
Distinguish by storage presence instead of exception type: storage is None ->
DEBUG keep-in-memory (legitimate standalone use); storage wired but save()
raises -> actionable ERROR. Add regression test mirroring real crewai's
ValueError-on-missing-embedder behavior.
* fix(crewai): expose run()/arun() entry points in degraded mode (#962)
The public crewai contract is run()/arun(); without crewai installed they were
missing (only the private _run existed), so the documented 'usable without
crewai' path raised AttributeError at the entry point. Define them in degraded
mode only, leaving crewai's BaseTool implementations untouched when present.
Extend the degradation subprocess test to exercise run() and arun().
* fix(crewai): standardize query shape, field-name rules, and restore-state flag
- _query_graph: id/type matches now return the same schema as content
matches (id/type/label/content/score) instead of a bare list
- _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys
(e.g. "risk-score >= 0.9") are addressable in policy rules
- add had_live_state/reconstructed_state so checkpoint-restored tools
and knowledge sources signal that their live graph/context was lost
and an empty one reconstructed; knowledge source no longer hides the
loss by eagerly rebuilding its graph inside __init__ (pydantic calls
__init__ during model_validate)
* fix(crewai): address Qodo review — confidence errors, string trim, holistic availability
- record_decision: stop calling float() in _run, so malformed confidence
values surface as JSON errors (via _record_decision's handling) instead
of crashing the tool
- _coerce_value: return the stripped string for non-numeric literals so
whitespace-padded decision_data fields match policy rules
- centralize crewai availability in _availability.py so the exported
CREWAI_AVAILABLE flag is holistic across tools and knowledge source
(previously each module probed crewai independently and the package
flag came from decision_tool only)
* ci: regenerate requirements-ci.txt for the crewai extra
The crewai extra in pyproject.toml brings in crewai, crewai-tools and
transitive deps (chromadb, lancedb, ...). Recompile with
uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes.
* ci: keep crewai out of the locked CI dependency set
crewai (all versions) hard-requires chromadb~=1.1.0, which carries a
pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c)
with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in
the 'all' extra failed pip-audit and the safety check on requirements-ci.txt.
- drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is
unchanged and still installs crewai)
- stop listing crewai-tools in the extra: the integration only uses crewai core
(BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps
- regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0
vulnerabilities, staleness check matches
* docs(crewai): document crewai extra scope and chromadb CVE-2026-45829
- CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not
part of the 'all' bundle, with the chromadb CVE-2026-45829 reason
- integrations/crewai/README.md: add a security warning that installing
the extra pulls chromadb~=1.1.0, which is affected by the unpatched
pre-auth code-injection CVE-2026-45829
---------
- get_table_lineage() gains include_column_lineage=True, resolving
per-column upstream/downstream references via Unity Catalog's
column-lineage API (one request per column, opt-in)
- DatabricksConnector.connect() now reuses an already-open connection
instead of opening a second one; ingest_table()/ingest_query() only
close the connection they opened themselves, so using the ingestor
as a context manager no longer leaks the connection opened by
__enter__
- get_table_schema()/get_table_lineage()/list_tables() now validate
both catalog and schema are resolved before calling Unity Catalog,
matching list_tables()'s existing catalog check
- 8 new regression tests (35 total)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
- Fix What's new → link in Info banner (now a proper <a> tag, always clickable)
- Replace 4-stat CardGroup on index with inline premium stats row
- Convert every <CardGroup>/<Card> block site-wide to markdown bullet lists:
content sections → bold-title bullets with sub-bullets, nav cards → [Title](href) — description
- Add cursor-animated list item hover effects to custom.css:
green inset left border, subtle background tint, marker color change on hover
- Affects index, getting-started, quickstart, concepts, modules, faq, architecture,
installation, cookbook, glossary, learning-more, explorer-setup, cli-setup,
community, contributing-guide, governance, citation, project-license,
all integrations pages, and all 20+ reference module pages
* fix: replace invalid Mintlify theme 'venus' with 'mint'
* docs: replace em dashes with colons across all docs files
* fix: strip UTF-8 BOM from all docs files (broke frontmatter detection)
- Rewrote index.md to match README (tagline, badges, Problem/Solution text)
- Improved getting-started, concepts, quickstart, installation, faq, use-cases, contributing, glossary, learning-more, examples, modules, architecture, cookbook, deep-dive pages: tighter prose, fixed headings/bullets, removed inconsistencies and duplicate sections
- Removed overuse of emojis from headings in integration pages (docling, snowflake)
- Fixed change_management reference page: closed unclosed JSON code block that broke the right TOC, demoted noisy sub-headings to bold text
- CSS layout: widened content area (max-width 1440px grid, left sidebar 11rem, right TOC narrowed to 11rem for broader content), tightened TOC spacing and font size, fixed word-wrap/overflow on TOC links
- Added mkdocs_local.yml for local serving without mkdocs-jupyter plugin
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sameer Kadam