Commit Graph
165 Commits
Author SHA1 Message Date
KaifAhmad1 d74b650643 Add Databricks connector (Unity Catalog + Delta Lake ingestion)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.

Closes #747
2026-07-15 22:07:25 +05:30
Sameer KadamandKaifAhmad1 7a4810893a docs: improve agent memory guide onboarding and usage guidance (#691)
* docs: improve agent memory guide onboarding and usage guidance

* docs: align Agent Memory guide with persistence implementation

* docs: fix misleading index_path persistence claim across guides

VectorStore's index_path kwarg is silently absorbed into FAISSStore's
**config and never read anywhere in faiss_store.py, so it does not make
the FAISS index persist across restarts as several docs implied. Real
persistence requires an explicit VectorStore.save()/.load() call, or
AgentContext.save()/.load() which cascades to it.

- docs/reference/context.md: rewrite the "Persist your vector store"
  tip to explain the actual save()/load() mechanism instead of the
  dead index_path kwarg.
- docs/guides/graphrag.md, decision-intelligence.md, ingest.md,
  semantic-extraction.md: drop the dead index_path=... kwarg from
  VectorStore(backend="faiss", ...) constructor calls.

Follow-up to #691, which fixed the same false claim in
docs/guides/agent-memory.md but missed these other files.

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-07-02 17:40:42 +05:30
KaifAhmad1 b2c949f7de fix(deploy): harden security in deployment templates and explorer app
- GCP: remove --allow-unauthenticated, restrict ingress to
  internal-and-cloud-load-balancing, replace wildcard ALLOWED_ORIGINS=*
  with a substitution variable (_ALLOWED_ORIGINS) so operators supply a
  real URL at deploy time; same fix in cloudrun-service.yaml
- Fly.io: replace hardcoded FALKORDB_HOST=localhost with the correct
  .internal private-network hostname pattern; update README accordingly
- docker-compose.dev.yml: add missing top-level networks: block so the
  frontend service can join the semantica network without --file layering
- K8s/Helm: add readOnlyRootFilesystem: true + runAsUser: 1000 to
  container securityContext; mount an emptyDir /tmp so uvicorn can write
  temp files
- app.py: fix _read_explorer_settings() or-chain, use in os.environ
  checks so an explicit ALLOWED_ORIGINS="" produces an empty allow-list
  instead of silently falling through to localhost defaults; remove dead
  app.state.falkordb_host/port attributes
- docs: update four locations that still documented {"status":"healthy"}
  to reflect the new {"status":"ok"} health response
- tests: update test assertion to read falkordb settings from
  app.state.explorer_settings instead of removed top-level attributes
2026-06-24 12:51:09 +05:30
Mohd Kaif 12d61b92df docs: add Temporal & Distance Intelligence reference pages with accurate API (#650)
- Add docs/reference/temporal.md: full Temporal Intelligence reference covering
  bi-temporal model (TemporalBound.OPEN sentinel, BiTemporalFact.from_relationship()
  factory), TemporalGraphQuery (query_at_time, reconstruct_at_time, query_time_range,
  find_temporal_paths, analyze_evolution, validate_temporal_consistency),
  TemporalPatternDetector, TemporalReasoningEngine with all 13 Allen interval
  relations over TemporalInterval objects, TemporalNormalizer (returns
  Optional[Tuple[datetime, datetime]]), TemporalQueryRewriter.rewrite() returning
  TemporalQueryResult, and TemporalVersionManager with SQLite storage and correct
  method names (list_versions, compare_versions, get_version, apply_revision,
  validate_snapshot, verify_checksum)

- Add docs/reference/distance.md: Distance Intelligence reference with corrected
  SimilarityCalculator API (pairwise_similarity, batch_similarity, find_most_similar)
  and semantic neighborhood / proximity-blended retrieval patterns

- Update docs/reference/kg.md: expand Exported Classes table to include all
  TemporalPatternDetector, TemporalInterval, IntervalRelation, TemporalQueryResult,
  AlgorithmTrackerWithProvenance, AlgorithmRegistry, ProvenanceTracker, SeedManager,
  KGConfig; fix all temporal code examples to use correct constructors and method names

- Update docs/reference/context.md: add Distance Intelligence section

- Update docs/index.md: add v0.3.0 release accordion with feature highlights

- Update docs/docs.json: wire temporal and distance pages into Modules navigation
2026-06-18 13:37:24 +05:30
Mohd Kaif 25289023fe docs: replace all CardGroup/Card blocks with animated bullet points across all 50 docs pages (#648)
- Fix What's new → link in Info banner (now a proper <a> tag, always clickable)
- Replace 4-stat CardGroup on index with inline premium stats row
- Convert every <CardGroup>/<Card> block site-wide to markdown bullet lists:
  content sections → bold-title bullets with sub-bullets, nav cards → [Title](href) — description
- Add cursor-animated list item hover effects to custom.css:
  green inset left border, subtle background tint, marker color change on hover
- Affects index, getting-started, quickstart, concepts, modules, faq, architecture,
  installation, cookbook, glossary, learning-more, explorer-setup, cli-setup,
  community, contributing-guide, governance, citation, project-license,
  all integrations pages, and all 20+ reference module pages
2026-06-17 23:18:40 +05:30
Mohd Kaif e04dc12e6e docs: premium UI improvements — navbar links, hover effects, inline tips, accordion troubleshooting (#646)
- Move Discord, GitHub, PyPI, and Follow on X links from sidebar anchors to top-right navbar
- Lock dark mode as default via appearance.strict and hide theme toggle
- Add custom.css with hover highlighting for tables, code blocks, cards, callouts, and inline code
- Move all Tips and Common Pitfalls sections inline next to their relevant content across all 25 reference docs
- Polish context.md: remove duplicates, condense callouts, upgrade Cookbooks to CardGroup
- Convert Troubleshooting and Performance Optimization sections in installation.md, cli-setup.md, explorer-setup.md, learning-more.md, and faq.md from plain headers to AccordionGroup
- Change navigation-hint Tip callouts to Info in concepts.md, faq.md, glossary.md, and modules.md
2026-06-17 18:59:25 +05:30
Mohd Kaif a326c7d3bd Fix/mintlify theme (#645)
* fix: replace invalid Mintlify theme 'venus' with 'mint'

* docs: replace em dashes with colons across all docs files

* fix: strip UTF-8 BOM from all docs files (broke frontmatter detection)
2026-06-17 13:26:19 +05:30
Mohd Kaif 1f3cea5f0a docs: upgrade all docs pages with Mintlify premium components (#642)
Replace plain markdown lists, tables, and numbered steps with interactive
Mintlify v3 MDX components across all 50+ documentation files:

- Tabs: provider/parser/method selection guides, citation formats, component details
- Steps: setup flows, pipeline stages, connection initialization
- CardGroup/Card: feature overviews, "what you get" sections, navigation footers
- AccordionGroup: FAQ entries
- Check/Warning/Tip/Note/Info: callouts replacing plain bold text and inline notes

Files improved span the full docs surface: reference modules (context, llms,
kg, reasoning, embeddings, deduplication, provenance, parse, ontology, core,
semantic_extract), integrations (agno, docling, snowflake), graph/vector
store backends (apache_age, pgvector), and top-level guides (contributing,
governance, glossary, citation, community-projects, learning-more).
2026-06-17 12:58:44 +05:30
KaifAhmad1 850f47625f docs: address review feedback across 7 modules
- llms.md: use showcase models (llama-3.3-70b-versatile, gpt-4o) in
  provider examples and use-case tables; clarify defaults vs recommended
  in Defaults and Reproducibility section
- split.md: document that chunk_size is in characters with migration note
- ingest.md: add Note that glob patterns are not supported by ingest()
- explorer-setup.md: remove hardcoded "1.5 seconds" timing claim
- cli-setup.md: expand semantica-worker description with concrete usage
- mcp_server.md: clarify turtle/ttl are aliases for the same RDF format
- semantic_extract.md: remove emoji from code comments
2026-06-16 21:49:20 +05:30
Sameer6305 fc22805e44 docs(change_management): align versioning APIs with implementation 2026-06-16 15:17:38 +05:30
Sameer6305 f2bf1e2159 docs(conflicts): align conflict APIs with implementation 2026-06-16 15:03:28 +05:30
Sameer6305 7e5db2b38d docs(utils): align utility APIs with implementation 2026-06-16 14:49:26 +05:30
Sameer6305 74b8170015 docs(mcp_server): align tools and resources with implementation 2026-06-16 14:38:14 +05:30
Sameer6305 2b661f1c2c docs(visualization): align visualizer APIs with implementation 2026-06-16 14:13:22 +05:30
Sameer6305 50d4c1e849 docs(export): align exporters and format support with implementation 2026-06-16 13:41:57 +05:30
Sameer6305 c62ff0238a docs(explorer): align explorer routes, CLI flags, and APIs with implementation 2026-06-16 11:34:14 +05:30
Sameer6305 d9f3e11eb3 docs(context): align context and policy APIs with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 4275d6ee34 docs(deduplication): align entity resolution APIs with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 55ea3fb4d5 docs(normalize): align normalization APIs with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 4cb2154803 docs(ingest): align ingestion APIs and return types with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 babeaf0e61 docs(evals): align placeholder documentation with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 777deaa4e3 docs(provenance): align lineage and provenance APIs with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 21b15f4b8e docs(reasoning): fix Python 3.8 type annotation compatibility 2026-06-16 11:18:19 +05:30
Sameer6305 ca0133227b docs(reasoning): align inference and reasoning APIs with implementation 2026-06-16 11:18:19 +05:30
Sameer6305 f7e8e87734 docs(triplet_store): align SPARQL and storage APIs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 10c53c9dd7 docs(graph_store): align graph APIs and backend documentation 2026-06-16 11:18:18 +05:30
Sameer6305 05ff590212 docs(vector_store): align vector store docs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 a28cb68098 docs(embeddings): align embedding docs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 13805d40e3 docs(ontology): align ontology docs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 2fc3261b89 docs(seed): align seed data docs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 441515a66c docs(kg): align graph and temporal APIs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 4982ce8d4b docs(pipeline): align examples and templates with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 273f37dd05 docs(core): align configuration examples with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 7e0e29282e docs(split): remove unsupported APIs and fix examples 2026-06-16 11:18:18 +05:30
Sameer6305 f170e1bab6 docs(split): align chunking docs with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 4bb7a49e08 docs(parse): align docling installation instructions 2026-06-16 11:18:18 +05:30
Sameer6305 8cebb052c8 docs(parse): improve onboarding and align with implementation 2026-06-16 11:18:18 +05:30
Sameer6305 366add9fa0 docs(semantic_extract): improve onboarding and workflow guidance 2026-06-16 11:18:18 +05:30
Sameer6305 db8d3581aa docs(llms): clarify implementation defaults for reproducibility 2026-06-16 11:18:17 +05:30
Sameer6305 4ed8c34f78 docs(llms): fix config examples and snippet imports 2026-06-16 11:18:17 +05:30
Sameer6305 4561faa254 docs(llms): improve onboarding and provider setup guidance 2026-06-16 11:18:17 +05:30
luffy2208 22382c2cf2 feat(ingest): add public API ingestion support 2026-06-09 06:52:58 +05:30
Mohd Kaif 5d70d0c10d docs: replace Exported Classes import blocks with summary tables (all 25 modules) (#567)
* docs: replace Exported Classes import blocks with summary tables across all 25 modules

* docs: add method/parameter tables to parse, ingest, ontology, normalize, triplet_store, change_management, conflicts, export, graph_store, provenance, and semantic_extract modules
2026-05-24 15:49:58 +05:30
KaifAhmad1 68fcff5b3a docs: add Exported Classes blocks to all remaining reference docs
Adds ## Exported Classes (or equivalent interface block) to:
- change_management.md, conflicts.md, context.md, embeddings.md
- graph_store.md, ingest.md, normalize.md, pipeline.md
- seed.md, split.md, triplet_store.md, vector_store.md
- visualization.md

Adds ## Launch Interface to explorer.md (CLI-only module).
Adds ## Server Interface to mcp_server.md (stdio process, not importable).

All blocks sourced from module __all__ with inline usage hints.
evals.md intentionally skipped (placeholder, __all__ = []).
2026-05-24 14:56:11 +05:30
KaifAhmad1 beacc88b02 fix(ci): replace list[Event] with List[Event] for Python 3.8 compat 2026-05-24 14:46:50 +05:30
KaifAhmad1 37e640e7b4 docs: comprehensive audit and DX overhaul of all reference modules
llms.md:
- Only Groq/OpenAI/LiteLLM/HuggingFaceLLM are exported — remove non-exported
  Anthropic/Ollama/Gemini/DeepSeek/Novita as direct imports
- Rename HuggingFace -> HuggingFaceLLM (correct class name)
- Remove non-existent create_provider() — replace with LiteLLM provider/model pattern
- Add LiteLLM 100+ providers section with provider/model string examples
- Add Exported Classes table (class -> provider -> API key)
- Update Provider Comparison table to show correct import per provider

ontology.md:
- Remove non-existent OntologyManager — replace with OntologyEngine facade
- Remove non-existent start_explorer() — replace with CLI: semantica-explorer
- SHACLValidator -> OntologyValidator (correct exported name)
- OWLExporter -> OWLGenerator (correct exported name)
- Add Exported Classes block with all 15+ exported symbols
- Add LLMOntologyGenerator section, NamespaceManager section
- Add OntologyEvaluator section with coverage/completeness metrics
- Add ingest_ontology() section
- Add versioning moved-to note (change_management module)

kg.md:
- TemporalKnowledgeGraph does not exist — replace with TemporalGraphQuery
- DistanceCalculator does not exist — replace with SimilarityCalculator
- Add Exported Classes block with all 20+ exported symbols
- Fix temporal example to use TemporalGraphQuery + TemporalVersionManager correctly
- Add SimilarityCalculator section with NodeEmbedder integration example

provenance.md:
- ActivityTracker not exported — remove; ProvenanceManager handles tracking
- Fix track_entity() signature: add source_location, source_quote params
- Fix GraphBuilderWithProvenance import: from semantica.kg, not semantica.provenance
- Add Exported Classes block with storage backends and checksum utilities
- Add SourceReference section with DOI/page/quote fields
- Add tamper-evident checksum section (compute_checksum/verify_checksum)
- Add Enable Provenance in Extractors section
- Fix duplicate heading (W3C PROV-O Export appeared twice)

reasoning.md:
- Add Exported Classes block with all engines + data types + explanation types
- Add Quick Start section
- Add Choosing an Engine comparison table
- Add InferenceResult/Explanation/ReasoningStep type annotations in examples
- Add Tip: use DatalogReasoner for recursive rules

semantic_extract.md:
- Add Exported Classes block with NamedEntityRecognizer, EventDetector, Entity,
  Relation, Event, CoreferenceChain, EntityClassifier, TemporalEventProcessor
- Add Quick Start section (one-liner extraction pipeline)
- Rename EventExtractor -> EventDetector (correct exported name)
- Clarify NERExtractor vs NamedEntityRecognizer distinction
- Add return type annotations to EventDetector example

core.md:
- Add Exported Classes block
- Add When to Use Core vs. Individual Modules decision table
- Add Tip: LifecycleManager only for long-running apps
- Fix MethodRegistry example to import build_knowledge_base correctly

parse.md:
- Add Exported Classes block with all format-specific parsers + data types
- Add DoclingParser optional import note

utils.md:
- Add Exported Classes block with logging/validation/progress/helpers/exceptions

deduplication.md:
- Add Exported Classes block with PropertyMergeRule, MergeStrategyManager,
  method_registry, and all convenience functions

export.md:
- Add Exported Classes block with all exporters, NamespaceManager,
  SemanticNetworkYAMLExporter, and all convenience functions
2026-05-24 14:41:57 +05:30
KaifAhmad1 5a7a740185 docs(context): full audit and overhaul of context.md
API fixes:
- retrieve(): top_k= -> max_results= (correct parameter name)
- remove non-existent add_decision_simple() -> use record_decision() on ContextGraph
- remove non-existent analyze_decision_influence() -> get_causal_chain() + trace_decision_explainability()
- find_precedents() returns List[Decision] not Precedent; removed .similarity attribute usage
- ContextRetriever.retrieve(): top_k -> max_results, add use_graph_expansion / min_relevance_score params
- AgentMemory.retrieve(): top_k -> max_results

New constructor params documented:
- retention_days, max_memories, max_expansion_hops, hybrid_alpha

New methods documented:
- batch_store(), forget(), update(), get_memory(), stats(), health()
- save() / load(), export() / import_data()
- conversation(), get_causal_chain(), query_decisions()
- trace_decision_explainability(), get_policy_engine()
- checkpoint(), diff_checkpoints(), flush_checkpoint()
- ContextGraph: add_nodes/add_edges (bulk), find_node, find_nodes, find_active_nodes
- ContextGraph: find_edges, query, stats, density, clear, build_from_conversations
- ContextGraph: link_graph, navigate_to, cross_graph_path, resolve_links

New sections:
- Cross-Graph Navigation with full example
- Checkpoint Methods with example
- Conversation Methods with example
- Persist and Restore real-world tab
- Policy dataclass in Data Structures accordion
- Decision.valid_from / valid_until temporal fields documented
- CausalChainAnalyzer and ContextRetriever added to What You Get cards
- New Tips: max_results param name, checkpoint auditing
2026-05-24 14:28:37 +05:30
KaifAhmad1 daa79ccef3 fix: audit and correct all remaining API mismatches in docs
- llms.md: replace non-exported Anthropic/Ollama imports with LiteLLM provider-prefix pattern; replace ReasoningEngine with Reasoner; replace create_provider with LiteLLM in YAML config example and tip
- concepts.md: replace ReasoningEngine with Reasoner/ReteEngine/GraphReasoner; fix DatalogReasoner.reason() to evaluate()/query(); replace TemporalKnowledgeGraph with TemporalGraphQuery; replace DistanceCalculator with SimilarityCalculator; replace EntityDeduplicator with DuplicateDetector/EntityMerger
- kg.md: replace non-exported build_knowledge_graph with method_registry.execute()
- semantic_extract.md: replace Anthropic import with LiteLLM
- index.md: replace Anthropic/Ollama imports with LiteLLM
- modules.md: fix TemporalKnowledgeGraph, DistanceCalculator, OntologyManager, ReasoningEngine, DatalogEngine, start_explorer, create_provider across code examples and module index table
- triplet_store.md: replace non-exported NamespacePrefixManager with semantica.ontology.NamespaceManager
2026-05-24 14:28:36 +05:30
KaifAhmad1 ce765b6f66 fix: correct docs-to-code mismatches in 8 reference modules
- graph_store: remove create_constraint(), add_nodes_bulk(), add_edges_bulk() → create_nodes(), add_edges()
- deduplication: fix PropertyMergeRule → MergeStrategy enum; add_rule() → add_property_rule(); merge() → merge_entities(); remove non-existent UNION/MAX/MIN/VOTING constants
- conflicts: set_credibility() → set_source_credibility(); group_by_severity/identify_patterns/analyze_sources → analyze_conflicts() dict keys; generate() → generate_guide(); remove time_window= param from analyze_trends()
- reasoning: infer() → forward_chain(); remove apply_transitivity/symmetry/inverse() templates that don't exist; GraphReasoner(kg) → GraphReasoner(); infer(kg) → reason(graph, query)
- split: split_document() (singular) → split_documents([parsed]) throughout
- seed: remove register_source_object(), populate(), inject(), load_from_file(), diff_versions(), get_version(tag=) — replace with register_source() and load_from_csv/json()
- change_management: remove rollback(), get_log_entry(), export_audit_trail(), get_audit_trail() — replace audit section with list_versions() + diff() pattern
- export: export_to_file() → export_to_rdf(); YAMLExporter → SemanticNetworkYAMLExporter
2026-05-24 13:36:14 +05:30
KaifAhmad1 ff43887842 fix: correct remaining API mismatches in pipeline, vector_store, and normalize docs
- pipeline.md: ParallelismManager pool_type="thread"/"process" → use_processes=False/True;
  execute_parallel() returns List[ParallelExecutionResult] not aggregate object
- vector_store.md: remove MetadataStore.add_field() (method is on MetadataSchema, not
  MetadataStore); fix tip to reference MetadataStore.update_metadata() not VectorStore
- normalize.md: Pipeline() orchestrator misuse → PipelineBuilder + ExecutionEngine pattern
2026-05-24 13:14:01 +05:30