diff --git a/docs/architecture.md b/docs/architecture.md index 2dfefaab..aeb31d68 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,7 +4,7 @@ description: "Four-layer, modular architecture designed for independent componen icon: "building" --- -Semantica is built around a four-layer modular architecture. Import only what you need — the framework never forces a full stack. Every component is independently swappable, and every layer communicates through clean interfaces with no hidden coupling. +Semantica is built around a four-layer modular architecture. Import only what you need: the framework never forces a full stack. Every component is independently swappable, and every layer communicates through clean interfaces with no hidden coupling. ## Four-Layer Architecture @@ -13,7 +13,7 @@ Semantica is built around a four-layer modular architecture. Import only what yo - + Loads data from any source into the pipeline as a unified `SourceDocument`. @@ -31,7 +31,7 @@ Loads data from any source into the pipeline as a unified `SourceDocument`. - + Transforms raw text into structured, enriched documents ready for knowledge store ingestion. @@ -45,14 +45,14 @@ Transforms raw text into structured, enriched documents ready for knowledge stor - + Persistent knowledge stores and embedding infrastructure that power retrieval and reasoning. | Component | Module | Description | | :--------- | :------ | :----------- | | Knowledge Graph | `kg` | Graph construction, temporal models, analytics, Distance Intelligence | -| Vector Store | `vector_store` | pgvector, Qdrant, Weaviate, Pinecone — semantic similarity search | +| Vector Store | `vector_store` | pgvector, Qdrant, Weaviate, Pinecone: semantic similarity search | | Ontology | `ontology` | OWL/RDFS modeling, SHACL validation, ontology alignment | | Triplet Store | `triplet_store` | RDF triple storage and SPARQL querying | | Embeddings | `embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE | @@ -60,7 +60,7 @@ Persistent knowledge stores and embedding infrastructure that power retrieval an - + Consumes the knowledge graph and vector stores for downstream use cases. @@ -91,11 +91,11 @@ Every pipeline follows the same linear path from raw source to delivered output: | Layer | Category | Modules | | :----- | :-------- | :------- | -| **Layer 1 — Ingestion** | Sources | `ingest`, `split` | -| **Layer 2 — Processing** | Transform | `parse`, `normalize`, `semantic_extract`, `deduplication`, `conflicts` | -| **Layer 3 — Intelligence** | Stores | `kg`, `vector_store`, `graph_store`, `triplet_store`, `embeddings`, `ontology` | -| **Layer 4 — Application** | Delivery | `context`, `reasoning`, `export`, `visualization`, `explorer`, `pipeline` | -| — | Cross-cutting | `provenance`, `change_management`, `llms`, `mcp_server`, `seed`, `evals`, `core`, `utils` | +| **Layer 1: Ingestion** | Sources | `ingest`, `split` | +| **Layer 2: Processing** | Transform | `parse`, `normalize`, `semantic_extract`, `deduplication`, `conflicts` | +| **Layer 3: Intelligence** | Stores | `kg`, `vector_store`, `graph_store`, `triplet_store`, `embeddings`, `ontology` | +| **Layer 4: Application** | Delivery | `context`, `reasoning`, `export`, `visualization`, `explorer`, `pipeline` | +|: | Cross-cutting | `provenance`, `change_management`, `llms`, `mcp_server`, `seed`, `evals`, `core`, `utils` | ## Extension Points @@ -147,27 +147,27 @@ registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") - + -Every component works standalone. `NERExtractor` runs without a graph store. `VectorStore` runs without decision tracking. The framework never forces a full stack instantiation — you pay only for what you import. +Every component works standalone. `NERExtractor` runs without a graph store. `VectorStore` runs without decision tracking. The framework never forces a full stack instantiation: you pay only for what you import. - + -Custom ingestors, extractors, validators, and exporters follow the same base class pattern. Register them via `PluginRegistry` and they participate in the full pipeline — provenance tracking, retry policies, and parallel execution included — with no changes to core code. +Custom ingestors, extractors, validators, and exporters follow the same base class pattern. Register them via `PluginRegistry` and they participate in the full pipeline: provenance tracking, retry policies, and parallel execution included: with no changes to core code. -Lineage tracking is built into graph construction at the lowest level. Every node and edge carries a `source_id` pointing back to the originating document, extraction method, and timestamp. There's no opt-in required — provenance is always on. +Lineage tracking is built into graph construction at the lowest level. Every node and edge carries a `source_id` pointing back to the originating document, extraction method, and timestamp. There's no opt-in required: provenance is always on. -Centralized `ConfigManager` with environment variable overrides. No magic defaults — all behavior is explicit and overridable. Suitable for multi-environment deployments where dev, staging, and production need different backends. +Centralized `ConfigManager` with environment variable overrides. No magic defaults: all behavior is explicit and overridable. Suitable for multi-environment deployments where dev, staging, and production need different backends. @@ -179,10 +179,10 @@ Centralized `ConfigManager` with environment variable overrides. No magic defaul | Characteristic | Mechanism | | :-------------- | :--------- | | **Parallel execution** | `Pipeline(workers=N)` with configurable workers per stage | -| **Delta processing** | Incremental graph updates — no full recompute on new data | +| **Delta processing** | Incremental graph updates: no full recompute on new data | | **Streaming ingestion** | Process large corpora without loading everything into memory | | **Backend flexibility** | Swap in-memory NetworkX for Neo4j / FalkorDB with no API changes | -| **Deduplication v2** | `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster than v1 | +| **Deduplication v2** | `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7x faster than v1 | | **Indexed search** | Explorer search at 0.004ms on 118k nodes (v0.5.0) | diff --git a/docs/citation.md b/docs/citation.md index 74bfe50d..aff60344 100644 --- a/docs/citation.md +++ b/docs/citation.md @@ -44,7 +44,7 @@ icon: "quote-left" ## Share Your Research -Published research using Semantica? [Let us know](https://github.com/semantica-agi/semantica/issues) — we may feature your work. +Published research using Semantica? [Let us know](https://github.com/semantica-agi/semantica/issues): we may feature your work. ## See Also diff --git a/docs/cli-setup.md b/docs/cli-setup.md index abb9aea9..6edb1987 100644 --- a/docs/cli-setup.md +++ b/docs/cli-setup.md @@ -1,6 +1,6 @@ --- title: "CLI Setup" -description: "The five Semantica executables — what each one does, when to use it, and how to confirm it is working." +description: "The five Semantica executables: what each one does, when to use it, and how to confirm it is working." icon: "terminal" --- @@ -139,7 +139,7 @@ python -c "import semantica; print(semantica.__version__)" See [Explorer Setup](explorer-setup) for the full walkthrough including how to build and save a graph file. - Every command also runs as a Python module — useful when the script directory is not on `PATH`: + Every command also runs as a Python module: useful when the script directory is not on `PATH`: ```bash python -m semantica.mcp_server @@ -201,7 +201,7 @@ If you have multiple Python environments, make sure you are installing into the python -m pip install semantica ``` -### `semantica-explorer` — "uvicorn is required" +### `semantica-explorer`: "uvicorn is required" The Explorer extras are not included in the base install: diff --git a/docs/community-projects.md b/docs/community-projects.md index faf7375f..b37beafd 100644 --- a/docs/community-projects.md +++ b/docs/community-projects.md @@ -5,7 +5,7 @@ icon: "people-group" --- - Building something with Semantica? [Submit it on GitHub](https://github.com/semantica-agi/semantica/issues/new?template=community_project.md) — we'd love to feature it here. + Building something with Semantica? [Submit it on GitHub](https://github.com/semantica-agi/semantica/issues/new?template=community_project.md): we'd love to feature it here. Semantica is used across academia, enterprise, and independent research. Below is a snapshot of the ecosystem being built by the community. @@ -17,27 +17,27 @@ Semantica is used across academia, enterprise, and independent research. Below i Teams in academia are using Semantica to build structured, auditable knowledge from unstructured scientific literature. -- **Academic literature mapping** — citation graph construction across multi-year corpora with temporal provenance -- **Biomedical knowledge graphs** — connecting genes, proteins, drugs, and diseases from PubMed and preprint feeds -- **Social network analysis** — community detection and influence analysis over entity-linked interaction graphs -- **Computational linguistics** — coreference resolution pipelines with entity-linked output for downstream NLP tasks +- **Academic literature mapping**: citation graph construction across multi-year corpora with temporal provenance +- **Biomedical knowledge graphs**: connecting genes, proteins, drugs, and diseases from PubMed and preprint feeds +- **Social network analysis**: community detection and influence analysis over entity-linked interaction graphs +- **Computational linguistics**: coreference resolution pipelines with entity-linked output for downstream NLP tasks ### Enterprise & Industry Production deployments span regulated and high-stakes industries where AI accountability is not optional. -- **Business intelligence** — corporate knowledge bases built from filings, reports, and internal documentation -- **Cybersecurity & threat intelligence** — adversary attribution graphs, CVE-linked threat feeds, incident timelines -- **Healthcare & clinical AI** — patient safety graphs, drug interaction knowledge bases, HIPAA-compliant audit trails -- **Financial services** — fraud detection graphs, regulatory compliance pipelines (SOX/GDPR/MiFID II), risk lineage -- **Legal & compliance** — contract analysis pipelines, regulatory change tracking, evidence-backed research graphs -- **Critical infrastructure** — supply chain risk graphs, energy grid event graphs, logistics provenance +- **Business intelligence**: corporate knowledge bases built from filings, reports, and internal documentation +- **Cybersecurity & threat intelligence**: adversary attribution graphs, CVE-linked threat feeds, incident timelines +- **Healthcare & clinical AI**: patient safety graphs, drug interaction knowledge bases, HIPAA-compliant audit trails +- **Financial services**: fraud detection graphs, regulatory compliance pipelines (SOX/GDPR/MiFID II), risk lineage +- **Legal & compliance**: contract analysis pipelines, regulatory change tracking, evidence-backed research graphs +- **Critical infrastructure**: supply chain risk graphs, energy grid event graphs, logistics provenance ### Independent & Open Source -- **GraphRAG toolkits** — custom retrieval layers built on top of Semantica's `context` + `vector_store` modules -- **Domain-specific extractors** — NER and relation extractors for clinical, legal, and scientific text -- **Temporal graph dashboards** — visual timelines built with Semantica's `TemporalKnowledgeGraph` + custom visualization adapters +- **GraphRAG toolkits**: custom retrieval layers built on top of Semantica's `context` + `vector_store` modules +- **Domain-specific extractors**: NER and relation extractors for clinical, legal, and scientific text +- **Temporal graph dashboards**: visual timelines built with Semantica's `TemporalKnowledgeGraph` + custom visualization adapters ## Supported Integrations @@ -66,12 +66,12 @@ Production deployments span regulated and high-stakes industries where AI accoun | :-------- | :---- | | **OpenAI** | GPT-4o, GPT-4, GPT-3.5 | | **Anthropic** | Claude Opus, Sonnet, Haiku | - | **Google Gemini** | — | - | **Groq** | LLaMA, Mixtral — fast inference | + | **Google Gemini** |: | + | **Groq** | LLaMA, Mixtral: fast inference | | **Ollama** | Fully local, air-gapped | - | **HuggingFace** | — | - | **DeepSeek** | — | - | **Novita AI** | — | + | **HuggingFace** |: | + | **DeepSeek** |: | + | **Novita AI** |: | | **LiteLLM** | 100+ model gateway | @@ -89,11 +89,11 @@ Production deployments span regulated and high-stakes industries where AI accoun The plugin system (`PluginRegistry`) makes it easy to add new capabilities without touching core code. The community has built: -- **Custom entity extractors** — domain-specific NER for clinical entities, legal clause types, and financial instruments -- **Export adapters** — specialized serialization formats for proprietary industry systems -- **Ingestor plugins** — adapters for SharePoint, Notion, Confluence, and custom databases -- **Visualization plugins** — enhanced dashboards with Plotly, D3.js, and custom graph renderers -- **Evaluation harnesses** — domain-specific precision/recall benchmarks using `semantica.evals` +- **Custom entity extractors**: domain-specific NER for clinical entities, legal clause types, and financial instruments +- **Export adapters**: specialized serialization formats for proprietary industry systems +- **Ingestor plugins**: adapters for SharePoint, Notion, Confluence, and custom databases +- **Visualization plugins**: enhanced dashboards with Plotly, D3.js, and custom graph renderers +- **Evaluation harnesses**: domain-specific precision/recall benchmarks using `semantica.evals` ## Build Your Own Extension diff --git a/docs/community.md b/docs/community.md index 5f8512b3..facd9604 100644 --- a/docs/community.md +++ b/docs/community.md @@ -4,7 +4,7 @@ description: "Get help, connect with contributors, and share what you build with icon: "users" --- -Semantica is built in the open, with contributions from researchers, engineers, and practitioners across many domains. Whether you're filing a bug, sharing a project, or reviewing a PR — you're part of the ecosystem. +Semantica is built in the open, with contributions from researchers, engineers, and practitioners across many domains. Whether you're filing a bug, sharing a project, or reviewing a PR: you're part of the ecosystem. ## Get Help @@ -20,7 +20,7 @@ Semantica is built in the open, with contributions from researchers, engineers, Browse open contributions and submit your own. - Report vulnerabilities privately — never in public issues. + Report vulnerabilities privately: never in public issues. @@ -29,10 +29,10 @@ Semantica is built in the open, with contributions from researchers, engineers, We follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). The community is built on four principles: -- **Respect** — treat everyone with kindness and patience, regardless of experience level -- **Inclusion** — welcome all backgrounds, domains, and approaches -- **Collaboration** — work toward better solutions together rather than competing on approaches -- **Learning** — share knowledge openly, ask questions without hesitation, and help others grow +- **Respect**: treat everyone with kindness and patience, regardless of experience level +- **Inclusion**: welcome all backgrounds, domains, and approaches +- **Collaboration**: work toward better solutions together rather than competing on approaches +- **Learning**: share knowledge openly, ask questions without hesitation, and help others grow To report unacceptable behavior, open a GitHub issue with the `[CoC]` prefix. Every report is investigated. @@ -70,10 +70,10 @@ See the [Contributing Guide](contributing-guide) for the full development workfl ## Stay Connected -- **[GitHub](https://github.com/semantica-agi/semantica)** — source code, releases, and the public roadmap -- **[PyPI](https://pypi.org/project/semantica/)** — package releases and download stats -- **[Discord](https://discord.gg/sV34vps5hH)** — real-time community chat -- **[X / Twitter](https://x.com/BuildSemantica)** — announcements and release highlights +- **[GitHub](https://github.com/semantica-agi/semantica)**: source code, releases, and the public roadmap +- **[PyPI](https://pypi.org/project/semantica/)**: package releases and download stats +- **[Discord](https://discord.gg/sV34vps5hH)**: real-time community chat +- **[X / Twitter](https://x.com/BuildSemantica)**: announcements and release highlights ## See Also diff --git a/docs/concepts.md b/docs/concepts.md index 98c048fc..b46f7ca3 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -1,6 +1,6 @@ --- title: "Core Concepts" -description: "The fundamental ideas behind Semantica — knowledge graphs, reasoning, provenance, and temporal intelligence explained." +description: "The fundamental ideas behind Semantica: knowledge graphs, reasoning, provenance, and temporal intelligence explained." icon: "book-open" --- @@ -8,9 +8,9 @@ icon: "book-open" New here? Start with [Getting Started](getting-started) for hands-on examples, then return here for deeper understanding. -Semantica transforms unstructured data — documents, web pages, reports, databases — into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources. +Semantica transforms unstructured data: documents, web pages, reports, databases: into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources. -At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs **grounded**, **traceable**, and **auditable**. +At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider: it makes their outputs **grounded**, **traceable**, and **auditable**. @@ -20,7 +20,7 @@ At its core, Semantica adds a **context and accountability layer** on top of you Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable. - `PluginRegistry` and `MethodRegistry` let you replace or augment any component — ingestors, extractors, reasoning engines, backends — without changing framework code. + `PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code. @@ -31,11 +31,11 @@ At its core, Semantica adds a **context and accountability layer** on top of you The foundation of everything in Semantica. A knowledge graph stores information as three building blocks: -- **Nodes (entities)** — people, companies, locations, events, concepts -- **Edges (relationships)** — `works_for`, `located_in`, `founded_by` -- **Properties** — name, date, confidence score, source URL +- **Nodes (entities)**: people, companies, locations, events, concepts +- **Edges (relationships)**: `works_for`, `located_in`, `founded_by` +- **Properties**: name, date, confidence score, source URL -This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it. +This structure makes knowledge **searchable**, **connectable**, **queryable**, and: critically: **explainable**: every answer can be traced back to the facts and relationships that produced it. ## Entity Extraction (NER) @@ -58,9 +58,9 @@ Each entity gets a type, confidence score, and a link to its source document. Th | Method | Speed | Accuracy | Requirements | | :------ | :----- | :-------- | :------------ | -| `"pattern"` | ⚡ Very fast | Moderate | No API key — regex-based | +| `"pattern"` | ⚡ Very fast | Moderate | No API key: regex-based | | `"ml"` | Fast | High | Local ML model | -| `"llm"` | Medium | Highest | LLM provider — all 9 supported | +| `"llm"` | Medium | Highest | LLM provider: all 9 supported | ## Relationship Extraction @@ -75,12 +75,12 @@ Finding how entities connect to each other: } ``` -Relationships can be extracted via rule-based methods, ML models, or LLMs — each producing typed triplets with confidence scores and source attribution. +Relationships can be extracted via rule-based methods, ML models, or LLMs: each producing typed triplets with confidence scores and source attribution. ## Knowledge Graph vs. Vector Store -Both store information for AI retrieval — but they're built for different jobs. +Both store information for AI retrieval: but they're built for different jobs. @@ -89,7 +89,7 @@ Both store information for AI retrieval — but they're built for different jobs | Strength | Why it matters | | :-------- | :------------- | | **Traversal** | Multi-hop queries: "Who founded companies that Apple alumni later joined?" | - | **Explainability** | Every answer traces back to specific nodes and edges — no black-box retrieval | + | **Explainability** | Every answer traces back to specific nodes and edges: no black-box retrieval | | **Temporal reasoning** | Point-in-time queries, `valid_from`/`valid_until` windows, historical snapshots | | **Conflict detection** | Two sources disagreeing on the same fact is surfaced and resolvable | | **Schema enforcement** | SHACL validation catches constraint violations before they corrupt results | @@ -106,14 +106,14 @@ Both store information for AI retrieval — but they're built for different jobs - Stores **dense embeddings** of text chunks. Answers questions by finding semantically similar passages — useful when the structure of the answer isn't known in advance. + Stores **dense embeddings** of text chunks. Answers questions by finding semantically similar passages: useful when the structure of the answer isn't known in advance. | Strength | Why it matters | | :-------- | :------------- | | **Fuzzy similarity** | Finds relevant content even when exact words don't match | | **Speed** | Sub-millisecond approximate nearest-neighbor search at scale | | **Unstructured text** | Works directly on paragraphs, sentences, and raw documents | - | **Simplicity** | No schema design required — embed and index | + | **Simplicity** | No schema design required: embed and index | **Use when:** you need fast semantic search over large text corpora. @@ -127,7 +127,7 @@ Both store information for AI retrieval — but they're built for different jobs - Semantica combines both — vector search seeds the graph traversal, and the graph provides structure and provenance the vector store cannot. + Semantica combines both: vector search seeds the graph traversal, and the graph provides structure and provenance the vector store cannot. | Step | What happens | | :---- | :----------- | @@ -136,7 +136,7 @@ Both store information for AI retrieval — but they're built for different jobs | **Context assembly** | Facts + relationships are assembled with source attribution for each claim | | **LLM generation** | LLM generates an answer grounded in the retrieved structured context | - **Result:** every claim in the response links back to a specific graph node — no hallucination from training data, full audit trail. + **Result:** every claim in the response links back to a specific graph node: no hallucination from training data, full audit trail. ```python from semantica.context import AgentContext, ContextGraph @@ -154,15 +154,15 @@ Both store information for AI retrieval — but they're built for different jobs ## Embeddings -Embeddings convert text into numerical vectors so AI systems can measure semantic similarity — finding related concepts even when the exact words differ. +Embeddings convert text into numerical vectors so AI systems can measure semantic similarity: finding related concepts even when the exact words differ. Semantica uses embeddings for: -- **Semantic search** — retrieve by meaning, not just keywords -- **Entity resolution** — match the same entity across different sources -- **Precedent search** — find similar past decisions -- **GraphRAG retrieval** — hybrid vector + graph traversal -- **Distance Intelligence** — N×N semantic distance matrices between any node set +- **Semantic search**: retrieve by meaning, not just keywords +- **Entity resolution**: match the same entity across different sources +- **Precedent search**: find similar past decisions +- **GraphRAG retrieval**: hybrid vector + graph traversal +- **Distance Intelligence**: N×N semantic distance matrices between any node set **Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings. @@ -178,13 +178,13 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses The query is embedded and used to seed both vector search and graph traversal simultaneously. - Semantica retrieves relevant graph context — entities, typed relationships, and multi-hop reasoning paths — alongside vector-similar text chunks. + Semantica retrieves relevant graph context: entities, typed relationships, and multi-hop reasoning paths: alongside vector-similar text chunks. Retrieved facts and reasoning paths are assembled into a structured prompt context, each fact tagged with its source node and confidence. - The LLM produces an answer where every claim links back to a source node in the graph — no floating assertions, no hallucinations from training data. + The LLM produces an answer where every claim links back to a source node in the graph: no floating assertions, no hallucinations from training data. @@ -195,7 +195,7 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses ## Ontology -An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply. +An ontology defines the schema and rules for your knowledge: what entity types exist, which relationships are valid, and what constraints apply. ```python ontology = { @@ -239,7 +239,7 @@ Inferred: Steve Jobs has a connection to Cupertino ``` - Efficient pattern matching for large rule sets — the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts. + Efficient pattern matching for large rule sets: the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts. ```python from semantica.reasoning import ReteEngine @@ -250,9 +250,9 @@ Inferred: Steve Jobs has a connection to Cupertino ``` - **Deductive** — classical syllogistic reasoning from premises to guaranteed conclusions. + **Deductive**: classical syllogistic reasoning from premises to guaranteed conclusions. - **Abductive** — infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases. + **Abductive**: infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases. ```python from semantica.reasoning import GraphReasoner @@ -263,7 +263,7 @@ Inferred: Steve Jobs has a connection to Cupertino ``` - Recursive Horn clause rules with fixpoint semantics — handles transitive closure and recursive relationships that forward chaining cannot express. + Recursive Horn clause rules with fixpoint semantics: handles transitive closure and recursive relationships that forward chaining cannot express. ```python from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule @@ -289,7 +289,7 @@ Inferred: Steve Jobs has a connection to Cupertino -All engines produce **explainable inference paths** — not black-box conclusions. Every derived fact includes the rules and premises that produced it. +All engines produce **explainable inference paths**: not black-box conclusions. Every derived fact includes the rules and premises that produced it. ## Temporal Intelligence @@ -313,7 +313,7 @@ snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15 ## Distance Intelligence -Explore the semantic neighborhood of any entity in your graph — useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology. +Explore the semantic neighborhood of any entity in your graph: useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology. ```python from semantica.kg import SimilarityCalculator @@ -329,7 +329,7 @@ The [Visualization module](reference/visualization) renders distance matrices as ## Deduplication & Entity Resolution -Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance. +Real-world data contains the same entity under many names: "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance. @@ -337,7 +337,7 @@ Real-world data contains the same entity under many names — "Apple", "Apple In | Strategy | Algorithm | Best For | | :-------- | :--------- | :-------- | | `v1` | Jaro-Winkler string similarity | Small datasets, fast baseline | - | `blocking_v2` | Candidate blocking + similarity | Large corpora — reduces O(n²) comparisons | + | `blocking_v2` | Candidate blocking + similarity | Large corpora: reduces O(n²) comparisons | | `hybrid_v2` | Blocking + semantic embedding match | Mixed structured/unstructured entity names | | `semantic_v2` | Pure embedding-based resolution | Up to 7× faster than v1; handles abbreviations and aliases | @@ -366,7 +366,7 @@ Every fact in Semantica links back to: - The **reasoning steps** that produced any inferred fact - This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export. + This is W3C PROV-O compliant lineage: suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export. ```python @@ -384,7 +384,7 @@ print(f"Checksum: {lineage.checksum}") ## Decision Intelligence -Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes. +Every agent decision is a first-class object in Semantica: recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes. ```python decision_id = context.record_decision( @@ -403,7 +403,7 @@ influence = context.analyze_decision_influence(decision_id) ``` - **Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply — reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history. + **Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply: reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history. @@ -413,20 +413,20 @@ When multiple sources disagree on the same fact, Semantica flags and resolves th **Resolution strategies:** -- **Recency** — prefer the most recent source -- **Source credibility** — prefer the most reliable source (configurable credibility scores) -- **Majority vote** — aggregate across all sources with ≥ 2 agreeing -- **Manual review** — flag for human arbitration; continue pipeline without blocking +- **Recency**: prefer the most recent source +- **Source credibility**: prefer the most reliable source (configurable credibility scores) +- **Majority vote**: aggregate across all sources with ≥ 2 agreeing +- **Manual review**: flag for human arbitration; continue pipeline without blocking See the [Conflicts reference](reference/conflicts) for `ConflictResolver`, `SourceTracker`, and `InvestigationGuideGenerator`. ## Custom Plugin Development -Semantica is designed for extension. Any component — ingestor, extractor, graph builder, reasoning engine — can be replaced or augmented with a custom implementation registered at runtime. +Semantica is designed for extension. Any component: ingestor, extractor, graph builder, reasoning engine: can be replaced or augmented with a custom implementation registered at runtime. - + `PluginRegistry` provides dynamic plugin discovery, registration, and loading across all modules. Register your own class under a string key; Semantica will use it wherever that key is referenced in config or pipeline steps. @@ -447,7 +447,7 @@ Semantica is designed for extension. Any component — ingestor, extractor, grap plugin = registry.load_plugin("my_sql_ingestor", connection_string="postgresql://...") result = plugin.execute("SELECT * FROM documents") - # Reference by name in pipeline YAML — no code changes needed + # Reference by name in pipeline YAML: no code changes needed ``` ```yaml @@ -461,9 +461,9 @@ Semantica is designed for extension. Any component — ingestor, extractor, grap **Extension points available:** ingestors, parsers, normalizers, extractors, reasoning engines, export formats, vector store backends, graph store backends, visualization renderers. - + - `MethodRegistry` lets you register custom methods on knowledge graph objects by name — useful for adding domain-specific graph operations without subclassing. + `MethodRegistry` lets you register custom methods on knowledge graph objects by name: useful for adding domain-specific graph operations without subclassing. ```python from semantica.kg import MethodRegistry diff --git a/docs/contributing-guide.md b/docs/contributing-guide.md index 6b16979b..59125a7a 100644 --- a/docs/contributing-guide.md +++ b/docs/contributing-guide.md @@ -4,7 +4,7 @@ description: "How to contribute code, documentation, tests, and community suppor icon: "code-pull-request" --- -Contributions of all kinds are welcome — code, documentation, tests, and community support. Every contribution is recognized in release notes and the GitHub contributors list. +Contributions of all kinds are welcome: code, documentation, tests, and community support. Every contribution is recognized in release notes and the GitHub contributors list. ## Quick Start @@ -17,7 +17,7 @@ pip install -e ".[dev]" pytest ``` -New to the project? Start with [`good-first-issue`](https://github.com/semantica-agi/semantica/labels/good-first-issue) labeled tickets — they're scoped to be completable in a few hours without deep codebase knowledge. +New to the project? Start with [`good-first-issue`](https://github.com/semantica-agi/semantica/labels/good-first-issue) labeled tickets: they're scoped to be completable in a few hours without deep codebase knowledge. ## Ways to Contribute @@ -77,16 +77,16 @@ Style conventions: **Black** for formatting, **isort** for imports, **flake8** f Before submitting a PR, confirm: -Tests pass locally — `pytest` +Tests pass locally: `pytest` New features include documentation with working code examples -Code follows project style — Black, isort, flake8 +Code follows project style: Black, isort, flake8 Commit messages are clear and describe the *why*, not just the *what* No unresolved merge conflicts ## Code of Conduct -All contributors are expected to follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive — especially toward newcomers. Report violations by opening an issue with the `[CoC]` prefix. +All contributors are expected to follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive: especially toward newcomers. Report violations by opening an issue with the `[CoC]` prefix. ## Help diff --git a/docs/cookbook.md b/docs/cookbook.md index 9faf9dda..063e97e7 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -6,9 +6,9 @@ icon: "flask" **Where to start:** - - **New to Semantica** — begin with [Core Tutorials](#core-tutorials) - - **Building an application** — see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases) - - **Need installation help** — see the [Installation Guide](installation) + - **New to Semantica**: begin with [Core Tutorials](#core-tutorials) + - **Building an application**: see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases) + - **Need installation help**: see the [Installation Guide](installation) diff --git a/docs/explorer-setup.md b/docs/explorer-setup.md index ccbf4634..63274084 100644 --- a/docs/explorer-setup.md +++ b/docs/explorer-setup.md @@ -4,7 +4,7 @@ description: "Install the Explorer extras, save a ContextGraph to JSON, and laun icon: "map" --- -**`semantica-explorer`** is an **interactive browser dashboard** for knowledge graph exploration. You give it a graph file, it starts a local server, and opens a browser tab where you can search nodes, find paths, inspect provenance, and run analytics — no code required after launch. +**`semantica-explorer`** is an **interactive browser dashboard** for knowledge graph exploration. You give it a graph file, it starts a local server, and opens a browser tab where you can search nodes, find paths, inspect provenance, and run analytics: no code required after launch. This page covers everything needed to go from zero to a running Explorer. For the full REST API reference and endpoint catalogue, see [Explorer Reference](reference/explorer). @@ -60,7 +60,7 @@ curl http://127.0.0.1:8000/api/health ``` -## Step 1 — Build and Save a ContextGraph +## Step 1: Build and Save a ContextGraph Explorer loads a graph from a JSON file on disk. You need to create that file first. @@ -113,7 +113,7 @@ Explorer loads a graph from a JSON file on disk. You need to create that file fi -## Step 2 — Launch Explorer +## Step 2: Launch Explorer ```bash semantica-explorer --graph my_graph.json @@ -122,7 +122,7 @@ semantica-explorer --graph my_graph.json The startup sequence prints: ``` -✓ Graph loaded — 3 nodes, 2 edges +✓ Graph loaded: 3 nodes, 2 edges ╭─ Semantica Explorer · http://127.0.0.1:8000 ─╮ │ API docs http://127.0.0.1:8000/docs │ │ Health http://127.0.0.1:8000/api/health │ @@ -140,15 +140,15 @@ The browser opens automatically at `http://127.0.0.1:8000` shortly after the ser | :---- | :----- | :------- | :----------- | | `--graph` | `-g` | *(**required**)* | Path to a ContextGraph JSON file | | `--port` | `-p` | `8000` | Port to bind the server | -| `--host` | — | `127.0.0.1` | Host to bind the server | -| `--no-browser` | — | off | Do not open a browser tab automatically | +| `--host` |: | `127.0.0.1` | Host to bind the server | +| `--no-browser` |: | off | Do not open a browser tab automatically | There are no flags for authentication, log level, or TLS. Those are not implemented in the CLI. ### Examples ```bash -# Minimal — local only, port 8000, browser opens automatically +# Minimal: local only, port 8000, browser opens automatically semantica-explorer --graph my_graph.json # Short flags @@ -157,7 +157,7 @@ semantica-explorer -g my_graph.json -p 8080 # Expose on the network so other machines can connect semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080 -# Headless — skip the auto-open and navigate manually +# Headless: skip the auto-open and navigate manually semantica-explorer --graph my_graph.json --no-browser ``` @@ -173,8 +173,8 @@ Once the server is running: | URL | What you get | | :--- | :------------ | | `http://127.0.0.1:8000` | Interactive dashboard | -| `http://127.0.0.1:8000/docs` | Swagger UI — every REST endpoint, interactive | -| `http://127.0.0.1:8000/api/health` | Health check — `{"status": "healthy"}` | +| `http://127.0.0.1:8000/docs` | Swagger UI: every REST endpoint, interactive | +| `http://127.0.0.1:8000/api/health` | Health check: `{"status": "healthy"}` | The browser tab opens shortly after startup. If it does not open, navigate to the URL manually or pass `--no-browser` and open it yourself. @@ -241,12 +241,12 @@ This is expected in headless, SSH, and container environments. Add `--no-browser Once running, Explorer exposes a REST API and dashboard for: -- **Node and edge search** — indexed search across all nodes by ID, type, and content -- **Neighborhood expansion** — inspect neighbors up to configurable hop depth -- **Path finding** — BFS shortest path between any two nodes -- **Graph analytics** — centrality, community detection, connectivity -- **Decisions and provenance** — query recorded decisions and their causal chains -- **Import / export** — upload JSON or CSV to extend the graph; download the current state +- **Node and edge search**: indexed search across all nodes by ID, type, and content +- **Neighborhood expansion**: inspect neighbors up to configurable hop depth +- **Path finding**: BFS shortest path between any two nodes +- **Graph analytics**: centrality, community detection, connectivity +- **Decisions and provenance**: query recorded decisions and their causal chains +- **Import / export**: upload JSON or CSV to extend the graph; download the current state The full endpoint catalogue is documented in the Swagger UI at `/docs` and in the reference page below. @@ -258,7 +258,7 @@ The full endpoint catalogue is documented in the Swagger UI at `/docs` and in th All five Semantica executables and when to use each one. - Full documentation for ContextGraph — build, query, save, and load. + Full documentation for ContextGraph: build, query, save, and load. End-to-end pipeline: ingest → extract → build graph → export. diff --git a/docs/faq.md b/docs/faq.md index 1f8c500e..dc9b0239 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,6 +1,6 @@ --- title: "FAQ" -description: "Common questions about Semantica — installation, features, integrations, and troubleshooting." +description: "Common questions about Semantica: installation, features, integrations, and troubleshooting." icon: "circle-question" --- @@ -12,13 +12,13 @@ icon: "circle-question" | Question | Answer | | :-------- | :------ | -| License? | MIT — free forever, no paywalled features | +| License? | MIT: free forever, no paywalled features | | Python version? | 3.8+ (3.11+ recommended) | -| API key required? | Optional — pattern extraction works with no keys | -| Works with LangChain / LlamaIndex? | Yes — Semantica is a layer on top, not a replacement | -| Production-ready? | Yes — 1,000+ tests, v0.5.0 ships with 12 security fixes | +| API key required? | Optional: pattern extraction works with no keys | +| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement | +| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes | | Latest version? | **v0.5.0** (May 2026) | -| Local LLMs? | Yes — Ollama via LiteLLM, HuggingFaceLLM for air-gapped | +| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped | ## General @@ -27,9 +27,9 @@ icon: "circle-question" -Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data — documents, APIs, databases — into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable. +Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data: documents, APIs, databases: into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable. -It's not a replacement for LangChain or LlamaIndex. It's the **accountability layer** that goes on top — recording decisions, tracing facts to sources, and making reasoning transparent. +It's not a replacement for LangChain or LlamaIndex. It's the **accountability layer** that goes on top: recording decisions, tracing facts to sources, and making reasoning transparent. @@ -46,7 +46,7 @@ It's not a replacement for LangChain or LlamaIndex. It's the **accountability la -Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit *why* an AI reached a conclusion — not just what it said. +Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit *why* an AI reached a conclusion: not just what it said. Semantica works alongside these frameworks, not against them. @@ -54,13 +54,13 @@ Semantica works alongside these frameworks, not against them. -Yes — MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source. +Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source. -**v0.5.0** — released May 2026. +**v0.5.0**: released May 2026. Highlights: Ontology Hub, Distance Intelligence, Parquet/XML ingestion, 12 security fixes, Graph Explorer redesign, NER gateway fix. @@ -95,7 +95,7 @@ Python **3.8 or higher**. Python 3.11+ is recommended for best performance and c -This was a known bug — fixed in **v0.5.0**. Upgrade: +This was a known bug: fixed in **v0.5.0**. Upgrade: ```bash pip install --upgrade semantica @@ -141,10 +141,10 @@ If you're on an older version, install extras individually: `pip install "semant Yes. Semantica supports: -- **Custom NER and extraction models** — register via `method_registry` -- **Custom embedding models** — any model with a `.encode()` interface -- **Custom LLM providers** — via LiteLLM (100+ models) or direct provider integration -- **Custom pipeline processors** — register via `PluginRegistry` +- **Custom NER and extraction models**: register via `method_registry` +- **Custom embedding models**: any model with a `.encode()` interface +- **Custom LLM providers**: via LiteLLM (100+ models) or direct provider integration +- **Custom pipeline processors**: register via `PluginRegistry` @@ -162,10 +162,10 @@ This includes PyTorch with CUDA, FAISS GPU, and CuPy. -- **Batching** — process documents in configurable chunks to control memory usage -- **Parallel processing** — `Pipeline(workers=N)` runs extraction steps concurrently -- **Delta processing** — update graphs incrementally without full recompute on new data -- **Persistent backends** — swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs +- **Batching**: process documents in configurable chunks to control memory usage +- **Parallel processing**: `Pipeline(workers=N)` runs extraction steps concurrently +- **Delta processing**: update graphs incrementally without full recompute on new data +- **Persistent backends**: swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs @@ -187,13 +187,13 @@ Available since v0.4.0. -A visual browser UI for the full ontology lifecycle — launched via `semantica.explorer`. Includes: +A visual browser UI for the full ontology lifecycle: launched via `semantica.explorer`. Includes: -- **Visual editor** — create and edit classes, properties, and relationships -- **SHACL Studio** — author, validate, and export SHACL shapes -- **Alignment authoring** — map concepts across ontologies -- **Health dashboard** — coverage, consistency, and constraint violation metrics -- **Version control** — diff and history for ontology changes +- **Visual editor**: create and edit classes, properties, and relationships +- **SHACL Studio**: author, validate, and export SHACL shapes +- **Alignment authoring**: map concepts across ontologies +- **Health dashboard**: coverage, consistency, and constraint violation metrics +- **Version control**: diff and history for ontology changes Available since v0.5.0. @@ -231,11 +231,11 @@ pip install --upgrade semantica -- **Neo4j** — industry standard, Cypher query language -- **FalkorDB** — Redis-protocol, ultra-low latency -- **Apache AGE** — PostgreSQL extension, OpenCypher -- **Amazon Neptune** — managed AWS, SPARQL and Gremlin -- **NetworkX** — in-memory, for development and small graphs +- **Neo4j**: industry standard, Cypher query language +- **FalkorDB**: Redis-protocol, ultra-low latency +- **Apache AGE**: PostgreSQL extension, OpenCypher +- **Amazon Neptune**: managed AWS, SPARQL and Gremlin +- **NetworkX**: in-memory, for development and small graphs @@ -247,7 +247,7 @@ RDF (Turtle, JSON-LD, N-Triples, XML), Apache Parquet, ArangoDB AQL, Apache Arro -FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, and in-memory. All backends share the same `VectorStore` API — swap with one line change. +FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, and in-memory. All backends share the same `VectorStore` API: swap with one line change. diff --git a/docs/getting-started.md b/docs/getting-started.md index 8883c692..0bc4a266 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ --- title: "Getting Started" -description: "The context and intelligence layer for AI — turning raw data into explainable, auditable knowledge graphs." +description: "The context and intelligence layer for AI: turning raw data into explainable, auditable knowledge graphs." icon: "rocket" --- @@ -58,7 +58,7 @@ icon: "rocket" - Pick the track that matches what you're building — each starts with a focused 5-minute example. + Pick the track that matches what you're building: each starts with a focused 5-minute example. | Track | You want to... | Start with | | :----- | :-------------- | :--------- | @@ -70,10 +70,10 @@ icon: "rocket" - The full 6-step pipeline — ingest, parse, extract, build, visualize, export — is in the [Quickstart](quickstart). Takes under 5 minutes with pattern-based extraction (no API key required). + The full 6-step pipeline: ingest, parse, extract, build, visualize, export: is in the [Quickstart](quickstart). Takes under 5 minutes with pattern-based extraction (no API key required). - An LLM API key is **optional** for the quickstart. Pattern-based extraction works out of the box — upgrade to LLM extraction for higher accuracy when you're ready. + An LLM API key is **optional** for the quickstart. Pattern-based extraction works out of the box: upgrade to LLM extraction for higher accuracy when you're ready. @@ -145,7 +145,7 @@ icon: "rocket" - Ground every LLM response in your knowledge graph — no floating assertions. + Ground every LLM response in your knowledge graph: no floating assertions. ```python from semantica.context import AgentContext, ContextGraph @@ -175,7 +175,7 @@ icon: "rocket" - Use Semantica from Claude Desktop, VS Code, Cursor, or any MCP client — no Python code required after setup. + Use Semantica from Claude Desktop, VS Code, Cursor, or any MCP client: no Python code required after setup. ```bash pip install semantica @@ -202,7 +202,7 @@ icon: "rocket" ## Core Architecture -Semantica uses a modular, layered architecture — import only what you need. +Semantica uses a modular, layered architecture: import only what you need. diff --git a/docs/glossary.md b/docs/glossary.md index 70da7d95..264889e1 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -17,22 +17,22 @@ A quick-reference dictionary of every concept, data structure, algorithm, and st An autonomous AI system that perceives its environment, reasons about information, and takes actions to achieve goals. In Semantica, agents use knowledge graphs for structured memory and context, with every decision recorded as a first-class object. **Context Graph** -A persistent, queryable graph of everything an agent knows, decides, and reasons about — entities, relationships, decisions, and their causal links. The core data structure of `semantica.context`. +A persistent, queryable graph of everything an agent knows, decides, and reasons about: entities, relationships, decisions, and their causal links. The core data structure of `semantica.context`. **Decision** A first-class object in Semantica: a recorded agent choice with category, scenario, reasoning, outcome, confidence score, causal chain, and source provenance. Stored and searchable via `context.record_decision()`. **Entity** -A distinct object or concept in the real world — a person, organization, location, event, or abstract concept. Entities are nodes in a knowledge graph, each with typed properties and a source provenance record. +A distinct object or concept in the real world: a person, organization, location, event, or abstract concept. Entities are nodes in a knowledge graph, each with typed properties and a source provenance record. **Knowledge Graph (KG)** -A structured representation of knowledge using entities (nodes) and relationships (edges). Knowledge graphs enable reasoning, querying, semantic search, and traceable inference — unlike flat vector stores. +A structured representation of knowledge using entities (nodes) and relationships (edges). Knowledge graphs enable reasoning, querying, semantic search, and traceable inference: unlike flat vector stores. **Relationship** A directed, typed connection between two entities: e.g., `works_for`, `located_in`, `founded_by`. Relationships carry confidence scores and provenance back to the source document. **Semantic** -Relating to meaning in language or logic. Semantic understanding captures context and intent — going beyond keyword matching to understand what text *means*. +Relating to meaning in language or logic. Semantic understanding captures context and intent: going beyond keyword matching to understand what text *means*. ## Data Processing @@ -41,19 +41,19 @@ Relating to meaning in language or logic. Semantic understanding captures contex Splitting large documents into smaller pieces while preserving semantic context. Semantica supports recursive, semantic boundary, entity-aware, relation-aware, sliding window, structural, and table-aware chunking strategies. **Ingestion** -Loading data from external sources — files, databases, APIs, streams — into the pipeline as a unified `SourceDocument`. The first stage in every Semantica pipeline. +Loading data from external sources: files, databases, APIs, streams: into the pipeline as a unified `SourceDocument`. The first stage in every Semantica pipeline. **Normalization** Standardizing data into a consistent canonical form: converting dates to ISO format, canonicalizing entity names, fixing encoding issues, stripping noise. Ensures downstream extraction works on clean, consistent text. **Parsing** -Extracting structured text, layout, and metadata from unstructured or semi-structured documents — PDFs, Word files, HTML, PPTX. `DoclingParser` additionally handles multi-column layouts, merged-cell tables, and OCR. +Extracting structured text, layout, and metadata from unstructured or semi-structured documents: PDFs, Word files, HTML, PPTX. `DoclingParser` additionally handles multi-column layouts, merged-cell tables, and OCR. ## Artificial Intelligence **Abductive Reasoning** -Inference to the most plausible explanation for observed facts. One of six reasoning engines in `semantica.reasoning` — returns the most likely hypothesis given available evidence. +Inference to the most plausible explanation for observed facts. One of six reasoning engines in `semantica.reasoning`: returns the most likely hypothesis given available evidence. **Datalog** A declarative logic programming language for knowledge base queries. Semantica's `DatalogEngine` supports recursive Horn clause rules with bottom-up semi-naive fixpoint semantics. Added in v0.4.0. @@ -62,7 +62,7 @@ A declarative logic programming language for knowledge base queries. Semantica's An advanced RAG approach that combines vector similarity search with knowledge graph traversal. Every LLM response is grounded in structured graph context, with each claim traceable to a source node. Eliminates hallucination without source attribution. **Inference** -Deriving new facts or conclusions from existing knowledge using logical rules — without the derived facts being explicitly present in the source data. +Deriving new facts or conclusions from existing knowledge using logical rules: without the derived facts being explicitly present in the source data. **LLM (Large Language Model)** An AI model trained on large text corpora, capable of understanding and generating natural language. Semantica integrates with 8+ LLM providers for entity extraction, relation extraction, and reasoning. @@ -74,7 +74,7 @@ A technique that enhances LLM outputs by retrieving relevant context from a know ## Knowledge Graph Components **Allen Interval Algebra** -A system of 13 relations for describing how two time intervals relate — before, after, meets, overlaps, during, starts, finishes, equals, and their inverses. Supported in `TemporalKnowledgeGraph` since v0.4.0. +A system of 13 relations for describing how two time intervals relate: before, after, meets, overlaps, during, starts, finishes, equals, and their inverses. Supported in `TemporalKnowledgeGraph` since v0.4.0. **BiTemporalFact** A fact with two independent time dimensions: *valid time* (when it was true in the world) and *transaction time* (when it was recorded in the system). Enables full audit trails for slowly changing data. @@ -86,43 +86,43 @@ A directed connection between two nodes in a graph, representing a typed relatio A vertex in a knowledge graph representing an entity or concept. Nodes carry typed properties, a confidence score, and provenance linking back to the source document. **Property** -An attribute or characteristic of an entity or relationship — name, date, URI, confidence score, source URL. +An attribute or characteristic of an entity or relationship: name, date, URI, confidence score, source URL. **Temporal Graph** A knowledge graph where nodes and edges carry `valid_from` / `valid_until` time windows, enabling point-in-time queries and historical state reconstruction. **Triplet** -The atomic unit of knowledge: a `(subject, predicate, object)` triple — e.g., `(Apple_Inc, founded_by, Steve_Jobs)`. The building block of RDF and SPARQL-based storage. +The atomic unit of knowledge: a `(subject, predicate, object)` triple: e.g., `(Apple_Inc, founded_by, Steve_Jobs)`. The building block of RDF and SPARQL-based storage. ## Entity Recognition & Extraction **Coreference Resolution** -Determining when multiple expressions in text refer to the same entity — e.g., "Apple" and "the company" both referring to Apple Inc. Handled by `CoreferenceResolver` in `semantica.semantic_extract`. +Determining when multiple expressions in text refer to the same entity: e.g., "Apple" and "the company" both referring to Apple Inc. Handled by `CoreferenceResolver` in `semantica.semantic_extract`. **Entity Resolution** Determining when two entity mentions across different documents refer to the same real-world entity. Also called entity linking or deduplication. Uses similarity scoring, blocking, and semantic embeddings. **Event Detection** -Identifying and classifying events in text — acquisitions, partnerships, product launches, regulatory decisions. Handled by `EventDetector` in `semantica.semantic_extract`. +Identifying and classifying events in text: acquisitions, partnerships, product launches, regulatory decisions. Handled by `EventDetector` in `semantica.semantic_extract`. **Named Entity Recognition (NER)** Identifying and classifying named entities in text into predefined categories: persons, organizations, locations, dates, products, and custom types. Three modes: pattern-based, ML-based, and LLM-based. **Relationship Extraction** -Identifying and extracting typed semantic relationships between entities — e.g., `(Google, acquired, DeepMind)` — from raw text. +Identifying and extracting typed semantic relationships between entities: e.g., `(Google, acquired, DeepMind)`: from raw text. ## Ontology & Schema **Axiom** -A statement accepted as true in an ontology, used to define logical constraints — e.g., "every Person must have a name", "Organization can have at most one CEO at a time". +A statement accepted as true in an ontology, used to define logical constraints: e.g., "every Person must have a name", "Organization can have at most one CEO at a time". **Class** -A category or type of entity in an ontology — `Person`, `Organization`, `Location`. Classes form a hierarchy and carry constraints validated by SHACL. +A category or type of entity in an ontology: `Person`, `Organization`, `Location`. Classes form a hierarchy and carry constraints validated by SHACL. **Ontology** -A formal specification of domain concepts, relationships, and constraints — typically expressed in OWL. Semantica can auto-generate ontologies from knowledge graphs or import existing OWL/RDF/Turtle files. +A formal specification of domain concepts, relationships, and constraints: typically expressed in OWL. Semantica can auto-generate ontologies from knowledge graphs or import existing OWL/RDF/Turtle files. **Ontology Hub** Semantica's v0.5.0 visual browser UI for the full ontology lifecycle: visual class editor, SHACL Studio, alignment authoring, health dashboard, and version-controlled diffs. @@ -140,13 +140,13 @@ A W3C standard for representing controlled vocabularies, taxonomies, and thesaur ## Storage & Retrieval **Embedding** -A dense numerical vector that represents text, images, or other data in a continuous semantic space. Entities with similar meaning produce vectors that are close together — enabling similarity search and semantic matching. +A dense numerical vector that represents text, images, or other data in a continuous semantic space. Entities with similar meaning produce vectors that are close together: enabling similarity search and semantic matching. **Graph Database** A database optimized for storing and querying graph-structured data using node and edge primitives. Semantica supports Neo4j, FalkorDB, Apache AGE, and Amazon Neptune. **Hybrid Search** -A retrieval strategy combining vector similarity search with keyword or metadata filtering — higher accuracy than either approach alone. +A retrieval strategy combining vector similarity search with keyword or metadata filtering: higher accuracy than either approach alone. **Triplet Store** A database designed specifically for storing and querying RDF `(subject, predicate, object)` triples. Semantica supports Blazegraph, Apache Jena, and RDF4J. @@ -161,7 +161,7 @@ A database optimized for storing and searching high-dimensional embedding vector A measure of a node's importance in the graph. Common metrics: PageRank (link-based importance), betweenness centrality (bridge nodes), closeness centrality (average distance to all others). **Community Detection** -Identifying groups of densely connected nodes — clusters that share more internal links than external ones. Used for finding subject communities, fraud rings, and organizational clusters. +Identifying groups of densely connected nodes: clusters that share more internal links than external ones. Used for finding subject communities, fraud rings, and organizational clusters. **Distance Band** A classification of a node's semantic proximity to a target: `near`, `mid`, or `far`, based on embedding distance thresholds. Part of Distance Intelligence (v0.5.0). @@ -170,7 +170,7 @@ A classification of a node's semantic proximity to a target: `near`, `mid`, or ` Semantica's v0.5.0 feature for semantic neighborhood exploration: N×N distance matrices, ego-mode visualization centered on a single entity, and distance band classification across the graph. **PageRank** -An algorithm measuring node importance based on the structure of incoming relationships — originally designed for web pages, applicable to any directed graph. +An algorithm measuring node importance based on the structure of incoming relationships: originally designed for web pages, applicable to any directed graph. ## Query Languages & Standards @@ -194,13 +194,13 @@ The W3C query language for RDF data. Semantica's `SparqlReasoner` uses SPARQL fo Handling contradictory facts from multiple sources in the same knowledge graph. Semantica's `ConflictDetector` surfaces conflicts; resolution strategies include prefer-most-recent, prefer-most-reliable, majority-vote, and flag-for-review. **Data Provenance** -Complete information about the origin, history, and lineage of every fact — source document, extraction method, timestamp, confidence score. W3C PROV-O compliant in Semantica. +Complete information about the origin, history, and lineage of every fact: source document, extraction method, timestamp, confidence score. W3C PROV-O compliant in Semantica. **Deduplication** Identifying and merging duplicate entity records. Semantica v2 strategies (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to 7x faster than v1. **W3C PROV-O** -The W3C provenance ontology standard. Semantica tracks lineage across all modules in PROV-O compliant format — suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 compliance. +The W3C provenance ontology standard. Semantica tracks lineage across all modules in PROV-O compliant format: suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 compliance. ## Security Terms @@ -219,7 +219,7 @@ A vulnerability in XML parsers that allows attackers to read arbitrary files or Deeper explanation of key ideas with code examples. - First working examples — no prior graph experience required. + First working examples: no prior graph experience required. All 27 modules explained with code and pipeline chains. diff --git a/docs/governance.md b/docs/governance.md index 19fc19a8..c1233160 100644 --- a/docs/governance.md +++ b/docs/governance.md @@ -1,6 +1,6 @@ --- title: "Governance" -description: "Project governance model — roles, decision process, release cadence, and code review guidelines." +description: "Project governance model: roles, decision process, release cadence, and code review guidelines." icon: "scale-balanced" --- @@ -11,7 +11,7 @@ icon: "scale-balanced" - Hawksight AI team — review and merge PRs, manage releases and code quality, set project direction and community standards. + Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards. Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md). @@ -65,36 +65,36 @@ Semantica follows **Semantic Versioning** (`MAJOR.MINOR.PATCH`): ## Communication -- **GitHub Issues** — bug reports, feature requests, questions -- **GitHub PRs** — code contributions -- **GitHub Discussions** — community conversation -- **Security Advisories** — [report security issues privately](https://github.com/semantica-agi/semantica/security/advisories/new) +- **GitHub Issues**: bug reports, feature requests, questions +- **GitHub PRs**: code contributions +- **GitHub Discussions**: community conversation +- **Security Advisories**: [report security issues privately](https://github.com/semantica-agi/semantica/security/advisories/new) ## Project Goals - Easy to use and understand — sensible defaults, clear documentation, minimal ceremony. + Easy to use and understand: sensible defaults, clear documentation, minimal ceremony. - Production-ready quality — tested across Python versions, platforms, and real-world workloads. + Production-ready quality: tested across Python versions, platforms, and real-world workloads. - Efficient and scalable — from single-machine notebooks to enterprise graph databases. + Efficient and scalable: from single-machine notebooks to enterprise graph databases. Easy to extend with plugins and custom modules via the `PluginRegistry` pattern. - Welcoming and inclusive — all backgrounds and experience levels contribute and are recognized. + Welcoming and inclusive: all backgrounds and experience levels contribute and are recognized. ## License -MIT License — see [LICENSE](https://github.com/semantica-agi/semantica/blob/main/LICENSE) and the [License page](license). +MIT License: see [LICENSE](https://github.com/semantica-agi/semantica/blob/main/LICENSE) and the [License page](license). ## See Also diff --git a/docs/graph_stores/apache_age.md b/docs/graph_stores/apache_age.md index 778a75e1..ff06e2e1 100644 --- a/docs/graph_stores/apache_age.md +++ b/docs/graph_stores/apache_age.md @@ -29,7 +29,7 @@ pip install psycopg2-binary - Use `GraphStore(backend="age", …)` — the same interface as Neo4j and FalkorDB: + Use `GraphStore(backend="age", …)`: the same interface as Neo4j and FalkorDB: ```python from semantica.graph_store import GraphStore @@ -94,7 +94,7 @@ graph_store_config.set("age_graph_name", "production") ## Connection & Initialization -On `connect()`, the store performs idempotent setup — safe to call repeatedly: +On `connect()`, the store performs idempotent setup: safe to call repeatedly: diff --git a/docs/index.md b/docs/index.md index 349cbf87..11eaddc6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,19 +1,19 @@ --- title: "Semantica" -description: "The Accountability and Context Layer for AI — Context Graphs · Decision Intelligence · Full Provenance" +description: "The Accountability and Context Layer for AI: Context Graphs · Decision Intelligence · Full Provenance" --- - **v0.5.0 is live** — Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. [What's new →](#whats-new) + **v0.5.0 is live**: Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. [What's new →](#whats-new) Your AI agent just made a decision. Now someone needs to explain it. -*What did it know at the time? Which facts shaped the outcome? Where did those facts come from? Has it made the same call before — and did that go well?* +*What did it know at the time? Which facts shaped the outcome? Where did those facts come from? Has it made the same call before: and did that go well?* -If your stack can't answer those questions with a traceable record, you have a gap. Not a capability gap — an **accountability gap**. It's the reason AI hasn't landed at scale in healthcare, finance, legal, and government. And it's why teams building for those markets keep rebuilding the same guardrails from scratch. +If your stack can't answer those questions with a traceable record, you have a gap. Not a capability gap: an **accountability gap**. It's the reason AI hasn't landed at scale in healthcare, finance, legal, and government. And it's why teams building for those markets keep rebuilding the same guardrails from scratch. -**Semantica closes that gap.** It's the context and accountability layer that sits beneath your existing agent framework — not a replacement for LangChain or LlamaIndex, but the infrastructure that makes their outputs trustworthy. +**Semantica closes that gap.** It's the context and accountability layer that sits beneath your existing agent framework: not a replacement for LangChain or LlamaIndex, but the infrastructure that makes their outputs trustworthy. @@ -50,7 +50,7 @@ Powerful agents aren't automatically trustworthy ones. Five structural blind spo Outputs can't be traced to source facts. - - In healthcare, finance, and legal — this is a hard compliance blocker + - In healthcare, finance, and legal: this is a hard compliance blocker - No lineage from inference back to the original document - Impossible to demonstrate what the agent actually relied on @@ -69,7 +69,7 @@ Powerful agents aren't automatically trustworthy ones. Five structural blind spo - These aren't edge cases. They're why enterprise AI pilots stall — and why your compliance team keeps saying *not yet*. + These aren't edge cases. They're why enterprise AI pilots stall: and why your compliance team keeps saying *not yet*. @@ -80,7 +80,7 @@ Semantica gives every agent the infrastructure it needs to be accountable. Drop A structured, queryable graph of everything your agent knows, decides, and reasons about. - - Persistent across agent runs — no context loss between sessions + - Persistent across agent runs: no context loss between sessions - Queryable with SPARQL and full graph algorithms - Temporal model with `valid_from` / `valid_until` on nodes and edges - Point-in-time snapshots of the full knowledge state @@ -100,15 +100,15 @@ Semantica gives every agent the infrastructure it needs to be accountable. Drop - Audit-ready for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 - Explainable reasoning paths — not black boxes. + Explainable reasoning paths: not black boxes. - Forward chaining, Rete, deductive, abductive - SPARQL query-based inference over RDF graphs - Datalog with recursive Horn clause rules - Every conclusion backed by a traceable derivation path - Your graph knows not just *what* — but *when*. - - Allen interval algebra — all 13 temporal relations + Your graph knows not just *what*: but *when*. + - Allen interval algebra: all 13 temporal relations - Point-in-time queries over historical graph states - Temporal provenance stamping on every fact - OWL-Time export for standards-compliant archiving @@ -123,7 +123,7 @@ Semantica gives every agent the infrastructure it needs to be accountable. Drop - Works alongside any LLM provider and any agent framework — add it to an existing stack without changing your architecture. + Works alongside any LLM provider and any agent framework: add it to an existing stack without changing your architecture. Semantica four-layer architecture: Ingestion → Processing → Intelligence → Application @@ -203,7 +203,7 @@ context = AgentContext( llm=LiteLLM(model="ollama/llama3.2", base_url="http://localhost:11434"), ) -# Fully local — no data leaves your infrastructure +# Fully local: no data leaves your infrastructure context.store("Local LLMs enable air-gapped compliance deployments") decision_id = context.record_decision( @@ -292,7 +292,7 @@ Semantica was designed for domains where every decision must be explainable and [Core Concepts](concepts) covers: - - Knowledge graphs vs. vector stores — when to use each + - Knowledge graphs vs. vector stores: when to use each - What GraphRAG is and how Semantica implements it - How provenance and decision tracking work together - The accountability layer architecture @@ -328,15 +328,15 @@ Semantica was designed for domains where every decision must be explainable and - + Released **May 11, 2026** | Area | Highlights | | :------ | :------------ | -| **Ontology Hub** | Visual editor, SHACL Studio, alignment authoring, health dashboard, version control — full ontology lifecycle in the browser | +| **Ontology Hub** | Visual editor, SHACL Studio, alignment authoring, health dashboard, version control: full ontology lifecycle in the browser | | **Distance Intelligence** | Semantic neighborhoods, N×N distance matrices, ego-mode visualization, distance band classification, embedding cache optimization | -| **Parquet Ingestion** | `ParquetIngestor` with PyArrow — single file, partitioned directories, Hive-style discovery, selective column reading | +| **Parquet Ingestion** | `ParquetIngestor` with PyArrow: single file, partitioned directories, Hive-style discovery, selective column reading | | **XML Ingestion** | `XMLIngestor` with XXE-safe lxml backend, XSD/DTD validation, namespace handling, directory scanning | | **Graph Explorer** | Landing page redesign, bidirectional path finding, indexed search (0.004ms on 118k nodes) | | **Security** | 12 vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal | @@ -348,12 +348,12 @@ pip install semantica==0.5.0 - + | Area | Highlights | | :------ | :------------ | | **Temporal Intelligence** | 6-PR system: temporal data model, point-in-time queries, Allen interval algebra (all 13 relations), OWL-Time export | -| **Knowledge Explorer API** | Full FastAPI backend — 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail | +| **Knowledge Explorer API** | Full FastAPI backend: 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail | | **Ontology Foundations** | SHACL generation/validation, SKOS vocabulary, ontology alignment API, diff & migration tooling | | **Datalog Reasoning** | Pure-Python bottom-up semi-naive fixpoint, recursive Horn clause rules, guaranteed termination | | **Agno Integration** | 5 components: graph-backed memory, multi-hop GraphRAG, decision toolkit, KG toolkit, shared team context; 110 tests | @@ -374,7 +374,7 @@ pip install semantica==0.5.0 - Structured, persistent graph of entities, relationships, and decisions - Temporal model with `valid_from` / `valid_until` on every node and edge - Point-in-time queries across historical graph states -- Distance Intelligence — semantic neighborhoods and N×N distance matrices +- Distance Intelligence: semantic neighborhoods and N×N distance matrices ### Decision Tracking @@ -389,16 +389,16 @@ pip install semantica==0.5.0 ### Entity & Relation Extraction -- Named entity recognition — pattern, ML, or LLM methods +- Named entity recognition: pattern, ML, or LLM methods - Typed triplet extraction via LLM or rule-based pipelines - Event extraction with temporal and causal linking ### Ontology & Schema -- Ontology Hub — visual editor, SHACL Studio, alignments, health dashboard -- Deduplication v2 — `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster -- Datalog reasoning — recursive Horn clause rules with fixpoint semantics -- SPARQL reasoning — query-based inference over RDF graphs +- Ontology Hub: visual editor, SHACL Studio, alignments, health dashboard +- Deduplication v2: `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7x faster +- Datalog reasoning: recursive Horn clause rules with fixpoint semantics +- SPARQL reasoning: query-based inference over RDF graphs @@ -406,17 +406,17 @@ pip install semantica==0.5.0 ### Lineage Tracking -- W3C PROV-O lineage across all modules — every fact has a source +- W3C PROV-O lineage across all modules: every fact has a source - `recorded_at` stamping with full OWL-Time export - Change management with SHA-256 checksums and version control - Full audit trails from ingestion event to final inference ### Compliance Infrastructure -- HIPAA — patient data handling with audit-ready provenance chains -- SOX / MiFID II — financial decision records with full traceability -- GDPR — data lineage for subject access and right-to-erasure workflows -- FDA 21 CFR Part 11 — electronic records and signature compliance +- HIPAA: patient data handling with audit-ready provenance chains +- SOX / MiFID II: financial decision records with full traceability +- GDPR: data lineage for subject access and right-to-erasure workflows +- FDA 21 CFR Part 11: electronic records and signature compliance @@ -458,25 +458,25 @@ pip install semantica==0.5.0 | `semantica.reasoning` | Forward chaining, Rete, deductive, abductive, SPARQL, Datalog | | `semantica.ontology` | SHACL, SKOS, alignments, diff/migration, auto-generation, OWL/RDF | | `semantica.explorer` | FastAPI Knowledge Explorer, Ontology Hub, Distance Intelligence, SHACL Studio | -| `semantica.mcp_server` | MCP stdio server — 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline | +| `semantica.mcp_server` | MCP stdio server: 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline | | `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector | | `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune | | `semantica.triplet_store` | In-memory and persistent RDF triple store with SPARQL | | `semantica.ingest` | Files, web, feeds, databases, Snowflake, Parquet, XML, MCP | -| `semantica.parse` | Document parsing — PDF, DOCX, HTML, PPTX, Docling layout analysis | -| `semantica.split` | Text chunking — sentence, paragraph, token, semantic boundary strategies | +| `semantica.parse` | Document parsing: PDF, DOCX, HTML, PPTX, Docling layout analysis | +| `semantica.split` | Text chunking: sentence, paragraph, token, semantic boundary strategies | | `semantica.normalize` | Text normalization, entity canonicalization, whitespace and encoding cleanup | | `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings | | `semantica.pipeline` | Pipeline DSL, parallel workers, retry policies, failure handling | | `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, OWL, Arrow, GraphML, GEXF, DOT | -| `semantica.visualization` | Programmatic graph rendering — force, hierarchical, circular, spring layouts | +| `semantica.visualization` | Programmatic graph rendering: force, hierarchical, circular, spring layouts | | `semantica.deduplication` | Entity deduplication v1/v2, similarity scoring, blocking, merging | | `semantica.conflicts` | Conflict detection and resolution across overlapping knowledge sources | | `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails | | `semantica.change_management` | Version control with SHA-256 checksums, diff, rollback | | `semantica.llms` | Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, HuggingFace | | `semantica.seed` | Foundation graph seeding from CSV, JSON, SQL, API, and RDF sources | -| `semantica.evals` | Evaluation harness — KG quality, extraction F1, pipeline benchmarking, regression tracking | +| `semantica.evals` | Evaluation harness: KG quality, extraction F1, pipeline benchmarking, regression tracking | | `semantica.core` | Orchestration, ConfigManager, LifecycleManager, PluginRegistry, MethodRegistry | | `semantica.utils` | Logging, validation, progress tracking, hash utilities, nested dict helpers | @@ -503,6 +503,6 @@ pip install semantica==0.5.0 - Use `NERExtractor` without a graph store - Use `ContextGraph` without vector storage - Every component independently swappable and testable - - No framework lock-in — works with any agent stack + - No framework lock-in: works with any agent stack diff --git a/docs/installation.md b/docs/installation.md index af7fc75c..8099dbcd 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -5,7 +5,7 @@ icon: "download" --- - **Available on PyPI** — `pip install semantica` and you're ready. + **Available on PyPI**: `pip install semantica` and you're ready. diff --git a/docs/integrations/agno.md b/docs/integrations/agno.md index 59a86999..8ca7bffc 100644 --- a/docs/integrations/agno.md +++ b/docs/integrations/agno.md @@ -26,19 +26,19 @@ pip install "semantica[agno,graph-neo4j,vectorstore-pgvector]" - `AgentMemory(db=…)` — Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent. + `AgentMemory(db=…)`: Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent. - `Agent(knowledge=…)` — Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG. + `Agent(knowledge=…)`: Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG. - `Agent(tools=[…])` — 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history. + `Agent(tools=[…])`: 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history. - `Agent(tools=[…])` — 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph. + `Agent(tools=[…])`: 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph. - Team-level — A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role. + Team-level: A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role. @@ -224,11 +224,11 @@ from integrations.agno import ( AgnoDecisionKit, # Decision intelligence Toolkit AgnoKGToolkit, # Knowledge graph Toolkit AgnoSharedContext, # Team-level shared context - AGNO_AVAILABLE, # bool — True if agno is installed + AGNO_AVAILABLE, # bool: True if agno is installed ) ``` -All five classes are usable without `agno` installed — they carry the full Semantica API and degrade gracefully. +All five classes are usable without `agno` installed: they carry the full Semantica API and degrade gracefully. ## See Also diff --git a/docs/integrations/docling.md b/docs/integrations/docling.md index fb28e14d..29e3b402 100644 --- a/docs/integrations/docling.md +++ b/docs/integrations/docling.md @@ -4,7 +4,7 @@ description: "Native Docling integration for high-fidelity PDF, DOCX, and PPTX p icon: "file-lines" --- -> Parse complex documents — PDFs, DOCX, PPTX, HTML — with high-fidelity table extraction and built-in OCR. +> Parse complex documents: PDFs, DOCX, PPTX, HTML: with high-fidelity table extraction and built-in OCR. ## Overview diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.md index 153dd604..3bd4725a 100644 --- a/docs/integrations/snowflake.md +++ b/docs/integrations/snowflake.md @@ -34,7 +34,7 @@ ingestor = SnowflakeIngestor( ) data = ingestor.ingest_table("CUSTOMERS") -print(f"Retrieved {data.row_count} rows — columns: {data.columns}") +print(f"Retrieved {data.row_count} rows: columns: {data.columns}") ``` @@ -64,7 +64,7 @@ Use environment variables (or a `.env` file with `python-dotenv`) to keep creden warehouse="COMPUTE_WH", ) ``` - Preferred for production — no password stored in config. + Preferred for production: no password stored in config. ```python @@ -165,7 +165,7 @@ from semantica.ingest import SnowflakeConnector connector = SnowflakeConnector(account="myaccount", user="myuser", password="mypassword") if not connector.test_connection(): - print("Connection failed — check credentials and account identifier") + print("Connection failed: check credentials and account identifier") ``` diff --git a/docs/learning-more.md b/docs/learning-more.md index 1eb34d65..b82d1258 100644 --- a/docs/learning-more.md +++ b/docs/learning-more.md @@ -4,7 +4,7 @@ description: "Structured learning paths, configuration reference, troubleshootin icon: "graduation-cap" --- -Whether you're running your first pipeline or deploying Semantica in production, this page gives you a structured path forward — from beginner to enterprise-grade usage. +Whether you're running your first pipeline or deploying Semantica in production, this page gives you a structured path forward: from beginner to enterprise-grade usage. ## Learning Paths @@ -30,19 +30,19 @@ Whether you're running your first pipeline or deploying Semantica in production, - [Installation Guide](installation) — virtual environments, optional extras, platform-specific fixes. + [Installation Guide](installation): virtual environments, optional extras, platform-specific fixes. - [Core Concepts](concepts) — what knowledge graphs are, how embeddings work, what extraction does. + [Core Concepts](concepts): what knowledge graphs are, how embeddings work, what extraction does. - [Getting Started](getting-started) — 5-minute code walkthrough with pattern-based extraction (no API key needed). + [Getting Started](getting-started): 5-minute code walkthrough with pattern-based extraction (no API key needed). - [Quickstart Tutorial](quickstart) — full 6-step pipeline from ingestion to visualization. + [Quickstart Tutorial](quickstart): full 6-step pipeline from ingestion to visualization. - [Welcome to Semantica notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb) — Jupyter walkthrough of every module. + [Welcome to Semantica notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb): Jupyter walkthrough of every module. @@ -51,16 +51,16 @@ Whether you're running your first pipeline or deploying Semantica in production, - [Modules Guide](modules) — all 27 modules with code examples and common pipeline chains. + [Modules Guide](modules): all 27 modules with code examples and common pipeline chains. - [Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) — multi-source, deduplication, conflict resolution. + [Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb): multi-source, deduplication, conflict resolution. - [Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb) — providers, pooling strategies, vector stores. + [Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb): providers, pooling strategies, vector stores. - [GraphRAG Complete notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — hybrid retrieval, reasoning, source attribution. + [GraphRAG Complete notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb): hybrid retrieval, reasoning, source attribution. [Multi-Source Data Integration notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb) and [Use Cases](use-cases) for domain-specific patterns. @@ -72,19 +72,19 @@ Whether you're running your first pipeline or deploying Semantica in production, - [Architecture Guide](architecture) — four-layer design, extension points, and design decisions. + [Architecture Guide](architecture): four-layer design, extension points, and design decisions. - [Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb) — `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries. + [Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb): `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries. - [Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb) — auto-generation, SHACL validation, Ontology Hub (v0.5.0). + [Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb): auto-generation, SHACL validation, Ontology Hub (v0.5.0). - [Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb) — UMAP, t-SNE, community layouts, embedding projections. + [Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb): UMAP, t-SNE, community layouts, embedding projections. - [Multi-Format Export notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb) — RDF with PROV-O, Parquet, Neo4j Cypher, Arrow, OWL. + [Multi-Format Export notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb): RDF with PROV-O, Parquet, Neo4j Cypher, Arrow, OWL. @@ -93,7 +93,7 @@ Whether you're running your first pipeline or deploying Semantica in production, ## Configuration Reference -All settings can be overridden with environment variables — no code changes needed. +All settings can be overridden with environment variables: no code changes needed. | Setting | Environment Variable | Default | | :------- | :-------------------- | :------- | @@ -126,7 +126,7 @@ pip install "semantica[gpu]" # GPU acceleration ### `AuthenticationError` -Set your API key as an environment variable — never hardcode keys in source files: +Set your API key as an environment variable: never hardcode keys in source files: ```bash export OPENAI_API_KEY="sk-..." @@ -189,14 +189,14 @@ set PYTHONIOENCODING=utf-8 | :--------- | :------------------ | :---------------- | | Graph construction | Fast | Moderate | | Query performance | Moderate | Fast | -| Scalability | Low — in-memory only | High — persistent | +| Scalability | Low: in-memory only | High: persistent | | Recommended for | Development, small graphs | Production, large corpora | Use NetworkX for local development and prototyping. Switch to a persistent backend before deploying to production. ### Batch Processing -Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM — a good starting point is 1,000 documents per batch on a 16 GB machine. +Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM: a good starting point is 1,000 documents per batch on a 16 GB machine. ### Deduplication v2 @@ -210,10 +210,10 @@ merged = resolver.resolve(entities, strategy="semantic_v2") # up to 7x faster ## Security Best Practices -- **API keys** — store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule -- **Sensitive data** — use local embedding models (Ollama, HuggingFace) for PII or classified content; avoid sending sensitive data to external APIs without data handling agreements -- **Graph exports** — encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways -- **XML ingestion** — always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser +- **API keys**: store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule +- **Sensitive data**: use local embedding models (Ollama, HuggingFace) for PII or classified content; avoid sending sensitive data to external APIs without data handling agreements +- **Graph exports**: encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways +- **XML ingestion**: always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser diff --git a/docs/modules.md b/docs/modules.md index 54eb0d66..f0c18bb9 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -1,6 +1,6 @@ --- title: "Modules" -description: "Every Semantica module works independently — use only what you need." +description: "Every Semantica module works independently: use only what you need." icon: "puzzle-piece" --- @@ -8,7 +8,7 @@ icon: "puzzle-piece" Looking for a quick reference? Jump to the [Module Index](#module-index) at the bottom. -Semantica is organized into **27 modules** across six logical layers. Each module is independently importable — you never pay for what you don't use. +Semantica is organized into **27 modules** across six logical layers. Each module is independently importable: you never pay for what you don't use. ## Architecture Overview @@ -51,7 +51,7 @@ documents = ingestor.ingest_directory("data/") web_ingestor = WebIngestor() pages = web_ingestor.ingest_urls(["https://example.com"]) -# Parquet — single file, partitioned directory, Hive-style (v0.5.0) +# Parquet: single file, partitioned directory, Hive-style (v0.5.0) parquet = ParquetIngestor() sources = parquet.ingest("data/events.parquet") @@ -69,7 +69,7 @@ Extracts structured text and layout metadata from raw documents. ```python from semantica.parse import DocumentParser, DoclingParser -# Standard parser — all common formats +# Standard parser: all common formats parser = DocumentParser() parsed = parser.parse_document("document.pdf") @@ -183,13 +183,13 @@ engine.apply_transitivity("located_in") engine.apply_symmetry("knows") result = engine.infer() -# Datalog — recursive Horn clause rules (v0.4.0) +# Datalog: recursive Horn clause rules (v0.4.0) datalog = DatalogEngine() datalog.add_rule("ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).") results = datalog.query("ancestor(alice, ?)") ``` -**Engines:** forward chaining, Rete network, deductive, abductive, SPARQL, Datalog — all produce explainable inference paths +**Engines:** forward chaining, Rete network, deductive, abductive, SPARQL, Datalog: all produce explainable inference paths ## Storage @@ -442,11 +442,11 @@ Exposes Semantica as an MCP stdio server for IDE and agent integrations. python -m semantica.mcp_server ``` -**Integrations:** Claude Desktop, VS Code, Cursor, Windsurf, Cline — 12 MCP tools exposed +**Integrations:** Claude Desktop, VS Code, Cursor, Windsurf, Cline: 12 MCP tools exposed ### Seed -Bootstrap knowledge graphs from verified structured sources — fixed-point reference data, controlled vocabularies, and domain anchors. +Bootstrap knowledge graphs from verified structured sources: fixed-point reference data, controlled vocabularies, and domain anchors. ```python from semantica.seed import SeedManager @@ -499,7 +499,7 @@ from semantica.core import Semantica, PluginRegistry, ConfigManager sem = Semantica(config_path="config.yaml") sem.initialize() -# Plugin registry — register custom components +# Plugin registry: register custom components registry = PluginRegistry() registry.register("my_ingestor", MyCustomIngestor) @@ -548,7 +548,7 @@ graph = GraphBuilder(merge_entities=True).build( - Ground every LLM response in a knowledge graph — structured retrieval with source attribution. + Ground every LLM response in a knowledge graph: structured retrieval with source attribution. **Pipeline:** `KG` + `VectorStore` → `AgentContext` → GraphRAG query → grounded answer @@ -604,7 +604,7 @@ precedents = context.find_precedents("model selection", limit=5) - Full provenance from raw data to final inference — W3C PROV-O, SHA-256 checksums, audit trail. + Full provenance from raw data to final inference: W3C PROV-O, SHA-256 checksums, audit trail. **Pipeline:** `Ingest` → `Parse` → `Extract` → `KG` → `Provenance` → `ChangeManagement` → `Export` @@ -654,7 +654,7 @@ for page in pages: - Track how facts change over time — point-in-time queries, snapshots, and versioning. + Track how facts change over time: point-in-time queries, snapshots, and versioning. **Pipeline:** `KG (Temporal)` → `TemporalGraphQuery` → `VersionManager` → `ChangeManagement` diff --git a/docs/project-license.md b/docs/project-license.md index 12d83c54..2eaf2efe 100644 --- a/docs/project-license.md +++ b/docs/project-license.md @@ -1,6 +1,6 @@ --- title: "License" -description: "Semantica is open source under the MIT License — free for personal and commercial use." +description: "Semantica is open source under the MIT License: free for personal and commercial use." icon: "file-contract" --- @@ -38,7 +38,7 @@ SOFTWARE. **You can:** -Use Semantica commercially — free for business use +Use Semantica commercially: free for business use Modify the source code Distribute copies or derivatives Use in proprietary software @@ -48,7 +48,7 @@ SOFTWARE. - Keep the copyright notice in copies - Include the MIT license text in distributions -**No warranty** — the authors are not responsible for damages or liable for how you use the software. +**No warranty**: the authors are not responsible for damages or liable for how you use the software. ## Commercial Use diff --git a/docs/quickstart.md b/docs/quickstart.md index 94baaad0..eab082f6 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -5,10 +5,10 @@ icon: "rocket" --- - **v0.5.0** — Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. [What's new →](index#whats-new) + **v0.5.0**: Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. [What's new →](index#whats-new) -This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional — pattern-based extraction works out of the box. +This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional: pattern-based extraction works out of the box. ## Install @@ -95,7 +95,7 @@ print(parsed.metadata) # title, author, date, source ``` - For PDFs with tables, charts, or multi-column layouts, use `DoclingParser` — it applies advanced layout analysis and returns structured table data alongside text. + For PDFs with tables, charts, or multi-column layouts, use `DoclingParser`: it applies advanced layout analysis and returns structured table data alongside text. ```python @@ -161,7 +161,7 @@ founders = graph.get_neighbors("Apple Inc.", predicate="founded_by") ``` - `merge_entities=True` automatically resolves duplicate entity references — "Apple", "Apple Inc.", "AAPL" — using semantic similarity. No manual deduplication needed. + `merge_entities=True` automatically resolves duplicate entity references: "Apple", "Apple Inc.", "AAPL": using semantic similarity. No manual deduplication needed. @@ -181,7 +181,7 @@ viz = GraphVisualizer( viz.visualize(graph, output="graph.html") ``` -Open `graph.html` in any browser — pan, zoom, click nodes for details, filter by entity type. +Open `graph.html` in any browser: pan, zoom, click nodes for details, filter by entity type. @@ -205,7 +205,7 @@ from semantica.export import ParquetExporter exporter = ParquetExporter() exporter.export(graph, output_dir="output/") -# Writes nodes.parquet + edges.parquet — ready for Spark, BigQuery, Databricks +# Writes nodes.parquet + edges.parquet: ready for Spark, BigQuery, Databricks ``` ```python ArangoDB @@ -225,7 +225,7 @@ aql = exporter.export(graph) ## Add Decision Intelligence -Track every agent decision with full causal chains and provenance — one extra import: +Track every agent decision with full causal chains and provenance: one extra import: ```python from semantica.context import AgentContext, ContextGraph @@ -249,7 +249,7 @@ decision_id = context.record_decision( confidence=0.91, ) -# Retrieve similar past decisions — prevents inconsistent choices +# Retrieve similar past decisions: prevents inconsistent choices precedents = context.find_precedents("model selection reasoning", limit=5) influence = context.analyze_decision_influence(decision_id) ``` @@ -259,7 +259,7 @@ influence = context.analyze_decision_influence(decision_id) - + ```python from semantica.semantic_extract import NERExtractor, RelationExtractor @@ -313,7 +313,7 @@ print(snapshot.get_node("Steve Jobs")) # role: CEO - + ```python from semantica.graph_store import Neo4jStore @@ -327,12 +327,12 @@ store = Neo4jStore( builder = GraphBuilder(merge_entities=True, graph_store=store) graph = builder.build(entities=entities, relationships=relationships) -# Graph persisted to Neo4j — survives process restarts +# Graph persisted to Neo4j: survives process restarts ``` - + ```python from semantica.provenance import ProvenanceTracker @@ -422,7 +422,7 @@ pip install --upgrade semantica - Knowledge graphs, ontologies, reasoning engines — the mental model behind Semantica. + Knowledge graphs, ontologies, reasoning engines: the mental model behind Semantica. Every module explained with key classes and common chains. diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 86f470ab..e31c6fc0 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -6,7 +6,7 @@ icon: "clock-rotate-left" **`semantica.change_management`** provides **enterprise-grade versioning and audit trails** for knowledge graphs and ontologies: -- SHA-256 checksums on every snapshot — tamper detection without external infrastructure +- SHA-256 checksums on every snapshot: tamper detection without external infrastructure - Structural diff between any two versions: nodes added, removed, or modified - Full rollback to any named snapshot - Per-entity mutation history for audit trail queries @@ -23,8 +23,8 @@ icon: "clock-rotate-left" | :--- | :--- | | `TemporalVersionManager` | Snapshot, diff, rollback, and per-node mutation history for KGs | | `OntologyVersionManager` | Schema versioning with structural diff support | -| `InMemoryVersionStorage` | Fast in-memory storage for dev and testing — no persistence | -| `SQLiteVersionStorage` | Production storage — persists to a local SQLite file | +| `InMemoryVersionStorage` | Fast in-memory storage for dev and testing: no persistence | +| `SQLiteVersionStorage` | Production storage: persists to a local SQLite file | | `compute_checksum()` | Returns SHA-256 fingerprint of any dict (graph snapshot, ontology snapshot) | | `verify_checksum()` | Detects tampering by recomputing and comparing the stored checksum inside a snapshot dict | @@ -38,7 +38,7 @@ icon: "clock-rotate-left" Version control for OWL ontologies with diff and schema migration support. - Pluggable backends — `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production. + Pluggable backends: `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production. SHA-256 checksums on every snapshot to detect any unauthorised modification. @@ -74,7 +74,7 @@ icon: "clock-rotate-left" ``` - Run deduplication, conflict resolution, merges, or any graph modification. The version manager tracks nothing automatically — you control when snapshots are taken. + Run deduplication, conflict resolution, merges, or any graph modification. The version manager tracks nothing automatically: you control when snapshots are taken. ```python @@ -82,7 +82,7 @@ icon: "clock-rotate-left" graph=kg, version_label="v2.0", author="user@example.com", - description="After deduplication — 1 342 duplicates merged" + description="After deduplication: 1 342 duplicates merged" ) ``` @@ -99,7 +99,7 @@ icon: "clock-rotate-left" ## TemporalVersionManager -Version control for knowledge graphs — snapshot, diff, and rollback. +Version control for knowledge graphs: snapshot, diff, and rollback. ### Constructor Parameters @@ -110,7 +110,7 @@ Version control for knowledge graphs — snapshot, diff, and rollback. ### List and Retrieve ```python -# List all versions — returns List[Dict] with label, author, timestamp, checksum, entity_count +# List all versions: returns List[Dict] with label, author, timestamp, checksum, entity_count versions = manager.list_versions() for v in versions: print(v["label"], "|", v["author"], "|", v["timestamp"], "|", v["checksum"][:8], "...") @@ -137,7 +137,7 @@ snapshot = manager.get_version("v1.0") ## Diff Analysis -Compare any two snapshots to see exactly what changed — useful for code review, incident investigation, and regulatory audit: +Compare any two snapshots to see exactly what changed: useful for code review, incident investigation, and regulatory audit: ```python diff = manager.diff("v1.0", "v2.0") @@ -186,7 +186,7 @@ for item in diff["entities_modified"]: ## OntologyVersionManager -Version control for ontologies — save, diff, and track schema changes: +Version control for ontologies: save, diff, and track schema changes: ```python from semantica.change_management import OntologyVersionManager @@ -201,7 +201,7 @@ snapshot = manager.create_snapshot( description="Added FHIR alignment mappings" ) -# Diff two ontology versions — returns a plain dict +# Diff two ontology versions: returns a plain dict diff = manager.compare_versions("1.1.0", "1.2.0") print("Classes added: ", diff["classes_added"]) print("Classes removed: ", diff["classes_removed"]) @@ -229,12 +229,12 @@ print("Properties added: ", diff["properties_added"]) manager = TemporalVersionManager() ``` - Fast and zero-setup. Data is **not persisted** — all version history is lost when the process exits. Use this for unit tests and development only. + Fast and zero-setup. Data is **not persisted**: all version history is lost when the process exits. Use this for unit tests and development only. - The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production — otherwise your entire version history disappears on restart. + The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production: otherwise your entire version history disappears on restart. ## Integrity Verification @@ -257,7 +257,7 @@ if not is_valid: ``` - `verify_checksum` takes the full snapshot dict (which contains the stored `"checksum"` key). Pass the dict returned by `create_snapshot` or `get_version` directly — no separate `expected_checksum` argument is needed. + `verify_checksum` takes the full snapshot dict (which contains the stored `"checksum"` key). Pass the dict returned by `create_snapshot` or `get_version` directly: no separate `expected_checksum` argument is needed. ## ChangeLogEntry @@ -334,17 +334,17 @@ for record in history: ### Compliance Coverage - + Use `manager.get_node_history("patient_001")` to retrieve every recorded mutation on a patient entity. Each `MutationRecord` includes `timestamp`, `operation`, `entity_id`, `payload`, and `version_label`. The SHA-256 checksum on each snapshot proves the record has not been altered. - + Use `manager.list_versions()` to enumerate all snapshots and `manager.diff(v1, v2)` to scope the change report to the relevant quarter. The immutable snapshot chain provides the chain of custody required by SOX Section 404. - + After deleting a data subject's entities, snapshot the graph and diff against the pre-deletion snapshot. `diff["entities_removed"]` provides a machine-readable record of exactly what was deleted and when, satisfying Article 17 documentation requirements. - - Every snapshot dict includes `author`, `timestamp`, and `checksum` — the three fields required for a compliant electronic record. `verify_checksum(snapshot)` provides the tamper-evidence required by 21 CFR § 11.10(e). + + Every snapshot dict includes `author`, `timestamp`, and `checksum`: the three fields required for a compliant electronic record. `verify_checksum(snapshot)` provides the tamper-evidence required by 21 CFR § 11.10(e). @@ -359,7 +359,7 @@ for record in history: - **Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"` — use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes. + **Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"`: use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes. diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index 3fef582f..37a24eab 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -1,6 +1,6 @@ --- title: "Conflicts Module" -description: "Multi-source conflict detection and resolution — value, type, temporal, and logical conflicts with investigation guides." +description: "Multi-source conflict detection and resolution: value, type, temporal, and logical conflicts with investigation guides." icon: "triangle-exclamation" --- @@ -10,7 +10,7 @@ icon: "triangle-exclamation" - Seven resolution strategies: voting, credibility-weighted, most-recent, first-seen, highest-confidence, manual review, expert review - `InvestigationGuideGenerator` produces step-by-step investigation instructions for manual resolution - `SourceTracker` maps each property value to its contributing source for full attribution -- Conflicts are surfaced explicitly — never silently corrupting the knowledge graph +- Conflicts are surfaced explicitly: never silently corrupting the knowledge graph ## Why Detect Conflicts? @@ -19,11 +19,11 @@ When you ingest data from multiple sources, contradictions are inevitable. One a Semantica's conflict detection makes disagreements explicit and actionable: -- **Value conflicts** — SEC says revenue is $391B; Reuters says $383B -- **Type conflicts** — "Python" is a `ProgrammingLanguage` in one source, a `Snake` species in another -- **Temporal conflicts** — a CEO had two different employers during overlapping date ranges -- **Logical conflicts** — an entity simultaneously holds two mutually exclusive properties -- **Relationship conflicts** — the same relationship has inconsistent cardinality or properties across sources +- **Value conflicts**: SEC says revenue is $391B; Reuters says $383B +- **Type conflicts**: "Python" is a `ProgrammingLanguage` in one source, a `Snake` species in another +- **Temporal conflicts**: a CEO had two different employers during overlapping date ranges +- **Logical conflicts**: an entity simultaneously holds two mutually exclusive properties +- **Relationship conflicts**: the same relationship has inconsistent cardinality or properties across sources ## Exported Classes @@ -118,7 +118,7 @@ Semantica's conflict detection makes disagreements explicit and actionable: # Auto-resolve low-severity conflicts low_conflicts = severity_details.get("low", []) - # Re-fetch full Conflict objects if needed — severity_details contains dicts + # Re-fetch full Conflict objects if needed: severity_details contains dicts auto_resolved = resolver.resolve_conflicts( conflicts, strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED, @@ -203,7 +203,7 @@ for result in results: - Weights each source's value by its assigned credibility score — favors authoritative sources automatically: + Weights each source's value by its assigned credibility score: favors authoritative sources automatically: ```python from semantica.conflicts import ConflictResolver, SourceTracker, ResolutionStrategy @@ -223,7 +223,7 @@ for result in results: **Best for:** sources with known reliability rankings (SEC > blog). - Majority vote — most common value across sources wins: + Majority vote: most common value across sources wins: ```python results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING) @@ -233,16 +233,16 @@ for result in results: ```python - # Most recent source wins — for fast-changing facts + # Most recent source wins: for fast-changing facts results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MOST_RECENT) - # First seen wins — for stable facts (founding date, original name) + # First seen wins: for stable facts (founding date, original name) results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.FIRST_SEEN) ``` ```python - # Flag for human review — use with InvestigationGuideGenerator + # Flag for human review: use with InvestigationGuideGenerator results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MANUAL_REVIEW) generator = InvestigationGuideGenerator() @@ -265,7 +265,7 @@ for result in results: | First seen | `FIRST_SEEN` | Stable facts: founding date, original name | | Highest confidence | `HIGHEST_CONFIDENCE` | Extraction pipeline outputs confidence scores | | Manual review | `MANUAL_REVIEW` | High-stakes decisions, regulated data | - | Expert review | `EXPERT_REVIEW` | Domain-specific ambiguity — escalate to a specialist | + | Expert review | `EXPERT_REVIEW` | Domain-specific ambiguity: escalate to a specialist | @@ -312,7 +312,7 @@ chain = tracker.get_traceability_chain("apple_inc") **Key behaviours:** - Credibility scores default to 0.50 for any source not explicitly set -- `SourceTracker` stores property-level provenance — so you can trace exactly which source contributed each value +- `SourceTracker` stores property-level provenance: so you can trace exactly which source contributed each value ## ConflictAnalyzer @@ -334,9 +334,9 @@ for t in trends: ``` **Key behaviours:** -- `analyze_conflicts()["patterns"]` returns a list of `ConflictPattern` objects — use `pattern.pattern_type` and `pattern.frequency` to find systemic data quality issues -- `analyze_conflicts()["by_source"]` includes `counts` and `top_sources` — sources appearing in many conflicts may have upstream data quality problems -- `analyze_trends()` returns a list of per-period dicts (`period`, `conflict_count`, `trend`, `trend_direction`) — `trend` is `"increasing"`, `"decreasing"`, or `"stable"` +- `analyze_conflicts()["patterns"]` returns a list of `ConflictPattern` objects: use `pattern.pattern_type` and `pattern.frequency` to find systemic data quality issues +- `analyze_conflicts()["by_source"]` includes `counts` and `top_sources`: sources appearing in many conflicts may have upstream data quality problems +- `analyze_trends()` returns a list of per-period dicts (`period`, `conflict_count`, `trend`, `trend_direction`): `trend` is `"increasing"`, `"decreasing"`, or `"stable"` ## InvestigationGuideGenerator @@ -439,7 +439,7 @@ class InvestigationStep: ## Tips and Common Pitfalls - **Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder — you lose the original source attribution. + **Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder: you lose the original source attribution. @@ -447,7 +447,7 @@ class InvestigationStep: - **Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"` — high severity means the disagreement is large and the stakes of getting it wrong are high. + **Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"`: high severity means the disagreement is large and the stakes of getting it wrong are high. diff --git a/docs/reference/context.md b/docs/reference/context.md index 4d6413e5..da311cea 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -17,7 +17,7 @@ icon: "brain" | Class | Role | | :--- | :--- | -| `AgentContext` | Primary entry point — memory, retrieval, decisions, graph traversal, checkpoints | +| `AgentContext` | Primary entry point: memory, retrieval, decisions, graph traversal, checkpoints | | `ContextGraph` | In-memory knowledge graph with centrality, community detection, and decision tracking | | `AgentMemory` | Vector-backed persistent memory: `store(text)`, `retrieve(query, max_results)` | | `EntityLinker` | Link entity mentions to URIs; create typed edges between entity IDs | @@ -152,7 +152,7 @@ decision_id = context.record_decision( ```python - # Search past decisions — prevents contradictory choices across runs + # Search past decisions: prevents contradictory choices across runs precedents = context.find_precedents("model selection reasoning", limit=5) for p in precedents: print("[{}] {} (confidence: {:.2f})".format(p.category, p.outcome, p.confidence)) @@ -174,13 +174,13 @@ decision_id = context.record_decision( - Fastest setup — no knowledge graph. Best for agents that need semantic search over facts without graph traversal overhead. + Fastest setup: no knowledge graph. Best for agents that need semantic search over facts without graph traversal overhead. ```python from semantica.context import AgentContext from semantica.vector_store import VectorStore - # Zero-graph setup — vector memory only + # Zero-graph setup: vector memory only context = AgentContext( vector_store=VectorStore(backend="faiss", dimension=768), ) @@ -198,7 +198,7 @@ decision_id = context.record_decision( - Production setup — graph + decisions + analytics. Use when you need explainability and contradiction-free decision history. + Production setup: graph + decisions + analytics. Use when you need explainability and contradiction-free decision history. ```python from semantica.context import AgentContext, ContextGraph @@ -314,7 +314,7 @@ decision_id = context.record_decision( | :--------- | :---- | :------- | :----------- | | `vector_store` | `VectorStore` | **required** | Backend for embedding-based memory retrieval | | `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and GraphRAG | -| `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder` — requires `knowledge_graph` to also be set | +| `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder`: requires `knowledge_graph` to also be set | | `retention_days` | `Optional[int]` | `30` | Auto-expire memories older than N days; `None` = keep forever | | `max_memories` | `int` | `10000` | Hard cap before LRU eviction | | `graph_expansion` | `bool` | `True` | Auto-expands graph from stored memories | @@ -332,13 +332,13 @@ decision_id = context.record_decision( | Method | Returns | Description | | :------ | :------- | :----------- | | `store(content, metadata, conversation_id, user_id)` | `str` or `Dict` | Store a fact (str → memory ID) or list of documents (list → stats dict) | -| `batch_store(items)` | `List[str]` | Store multiple items at once — returns list of memory IDs | +| `batch_store(items)` | `List[str]` | Store multiple items at once: returns list of memory IDs | | `retrieve(query, max_results, min_score, use_graph, conversation_id)` | `List[Dict]` | Semantic retrieval; auto-selects GraphRAG if `knowledge_graph` is set | | `forget(memory_id, conversation_id, days_old)` | `int` | Delete memories by ID, conversation, or age | | `update(memory_id, content, metadata)` | `bool` | Update content or metadata of a stored memory | | `get_memory(memory_id)` | `Optional[Dict]` | Fetch a specific memory by ID | | `stats()` | `Dict` | Memory counts, vector store status, graph stats | -| `health()` | `Dict` | System health — all backends, status flags | +| `health()` | `Dict` | System health: all backends, status flags | | `save(path)` | `None` | Persist full context state (memory + graph) to disk | | `load(path)` | `None` | Restore context state from disk | | `export(conversation_id, format)` | `str \| Dict` | Export memories as JSON or dict | @@ -366,7 +366,7 @@ results = context.retrieve( ### Multi-Hop GraphRAG -**Requires `knowledge_graph`** to be set at construction — enables `query_with_reasoning()` for LLM-grounded multi-hop traversal: +**Requires `knowledge_graph`** to be set at construction: enables `query_with_reasoning()` for LLM-grounded multi-hop traversal: ```python import os @@ -393,12 +393,12 @@ print("Sources used: {}".format(result["num_sources"])) | `find_precedents(scenario, category, limit, use_hybrid_search, max_hops, as_of)` | `List[Decision]` | Find similar past decisions by semantic + structural similarity | | `query_decisions(query, max_hops, use_hybrid_search)` | `List[Decision]` | Broad context-aware decision search | | `get_causal_chain(decision_id, direction, max_depth)` | `List[Decision]` | Trace `"upstream"` causes or `"downstream"` effects | -| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability — causes, effects, relationship paths | +| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability: causes, effects, relationship paths | | `get_policy_engine()` | `PolicyEngine` | Access the active `PolicyEngine` instance | ### Checkpoint Methods -**Ideal for auditing reasoning loops** — take a snapshot before and after a pass to see exactly what changed: +**Ideal for auditing reasoning loops**: take a snapshot before and after a pass to see exactly what changed: ```python # Take a named snapshot of the current graph state @@ -456,7 +456,7 @@ print("Nodes: {}, Edges: {}".format(stats["node_count"], stats["edge_count"])) | `community_detection` | `bool` | `True` | Louvain community clustering | | `node_embeddings` | `bool` | `True` | Node2Vec embeddings for structural similarity | -### ContextGraph — Full Method Reference +### ContextGraph: Full Method Reference | Method | Returns | Description | | :------ | :------- | :----------- | @@ -622,7 +622,7 @@ linked = linker.link(text="Apple Inc. was founded by Steve Jobs.", entities=enti for e in linked: print("{} → {} (confidence: {:.2f})".format(e.text, e.uri, e.confidence)) -# Explicitly link two entity IDs (not a list — takes two IDs) +# Explicitly link two entity IDs (not a list: takes two IDs) linker.link_entities( entity1_id="apple_inc", entity2_id="aapl", @@ -792,7 +792,7 @@ class EntityLink: ## Real-World Patterns - + ```python from semantica.context import AgentContext, ContextGraph from semantica.vector_store import VectorStore @@ -804,7 +804,7 @@ class EntityLink: ) health_agent.store("Patient has hypertension, type 2 diabetes") - health_agent.store("Patient allergic to penicillin — verified 2024-01") + health_agent.store("Patient allergic to penicillin: verified 2024-01") decision_id = health_agent.record_decision( category="treatment_plan", @@ -822,7 +822,7 @@ class EntityLink: print("Follow-up decisions triggered: {}".format(len(chain))) ``` - + ```python from semantica.context import AgentContext, ContextGraph, PolicyEngine from semantica.context.decision_models import Policy, Decision @@ -856,7 +856,7 @@ class EntityLink: d = Decision( decision_id="dec_loan_001", category="loan_approval", - scenario="First-time homebuyer — 30yr fixed, 20% down", + scenario="First-time homebuyer: 30yr fixed, 20% down", reasoning="Credit score above threshold, DTI within limits", outcome="approved_300k", confidence=0.94, @@ -894,7 +894,7 @@ class EntityLink: # Persist everything context.save("agent_state/") - # Later — restore and continue + # Later: restore and continue restored = AgentContext( vector_store=VectorStore(backend="faiss", dimension=768), knowledge_graph=ContextGraph(), @@ -911,11 +911,11 @@ class EntityLink: ## Tips and Common Pitfalls - **`decision_tracking=True` silently does nothing without `knowledge_graph`.** Both must be set at construction. Passing only `decision_tracking=True` without a `knowledge_graph` instance leaves the decision backend uninitialised — `record_decision()` will raise `RuntimeError`. + **`decision_tracking=True` silently does nothing without `knowledge_graph`.** Both must be set at construction. Passing only `decision_tracking=True` without a `knowledge_graph` instance leaves the decision backend uninitialised: `record_decision()` will raise `RuntimeError`. - **Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` — without it the FAISS index lives only in memory and is lost on shutdown. + **Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore`: without it the FAISS index lives only in memory and is lost on shutdown. @@ -955,5 +955,5 @@ class EntityLink: ### Cookbooks -- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb) — memory and decision tracking · Intermediate -- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb) — production FAISS + Neo4j setup · Advanced +- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb): memory and decision tracking · Intermediate +- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb): production FAISS + Neo4j setup · Advanced diff --git a/docs/reference/core.md b/docs/reference/core.md index 9660c1b5..86259399 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -20,7 +20,7 @@ icon: "gear" - High-level orchestrator — coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments. + High-level orchestrator: coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments. YAML config with deep-merge, `SEMANTICA_` env var overrides, and dot-notation nested key access. Keeps secrets out of source files. @@ -29,7 +29,7 @@ icon: "gear" Ordered startup/shutdown hooks, health monitoring, and a 6-state machine. Essential for long-running services like FastAPI apps. - Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime — no imports required. + Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime: no imports required. @@ -37,11 +37,11 @@ icon: "gear" | Class | Role | | :--- | :--- | -| `Semantica` | Orchestration entry point — coordinates the full KG construction pipeline | +| `Semantica` | Orchestration entry point: coordinates the full KG construction pipeline | | `ConfigManager` | YAML config loading, deep-merge, validation, and env var overrides | | `LifecycleManager` | Startup/shutdown state machine with health monitoring and lifecycle hooks | | `PluginRegistry` | Dynamic plugin discovery, registration, and loading | -| `method_registry` | Global `MethodRegistry` instance — register and dispatch custom orchestration methods | +| `method_registry` | Global `MethodRegistry` instance: register and dispatch custom orchestration methods | ## Semantica (Orchestration) @@ -77,7 +77,7 @@ finally: | `build_knowledge_base(sources, **kwargs)` | Orchestrate full KG construction pipeline | | `run_pipeline(pipeline, data)` | Execute an existing `Pipeline` instance | | `get_status()` | Return system health and current state | -| `shutdown(graceful=True)` | Graceful shutdown — waits for in-flight operations | +| `shutdown(graceful=True)` | Graceful shutdown: waits for in-flight operations | ## ConfigManager @@ -107,7 +107,7 @@ config.validate() llm_provider: name: openai model: gpt-4o - # Do not put API keys in YAML — use environment variables instead. + # Do not put API keys in YAML: use environment variables instead. # ConfigManager loads YAML with yaml.safe_load(), which does not # interpolate ${...} expressions. Set secrets via env vars (see below). @@ -183,7 +183,7 @@ manager.shutdown(graceful=True) ## PluginRegistry -Register custom components that participate in the full pipeline — provenance tracking, retry policies, and parallel execution included: +Register custom components that participate in the full pipeline: provenance tracking, retry policies, and parallel execution included: ```python from semantica.core import PluginRegistry @@ -214,7 +214,7 @@ from semantica.core import method_registry from semantica.core.methods import build_knowledge_base def fast_kb_builder(sources, **kwargs): - # Custom logic — skip embeddings for speed + # Custom logic: skip embeddings for speed ... method_registry.register("knowledge_base", "fast", fast_kb_builder) diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index b9a86b02..d165174d 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -1,6 +1,6 @@ --- title: "Deduplication Module" -description: "Entity deduplication — similarity scoring, blocking, merging, and cluster-based batch processing." +description: "Entity deduplication: similarity scoring, blocking, merging, and cluster-based batch processing." icon: "copy" --- @@ -10,23 +10,23 @@ icon: "copy" - `ClusterBuilder` uses Union-Find and hierarchical clustering for batch deduplication at scale - `EntityMerger` preserves original source provenance on every merged entity - `MergeStrategyManager` supports per-property rules and conflict resolution -- All workflows operate on plain Python dicts — no ORM or schema required +- All workflows operate on plain Python dicts: no ORM or schema required ## Exported Classes | Class | Role | | :--- | :--- | -| `DuplicateDetector` | Pairwise and batch detection — returns `DuplicateCandidate` or `DuplicateGroup` lists | -| `EntityMerger` | Merge duplicate groups — returns `List[MergeOperation]` | +| `DuplicateDetector` | Pairwise and batch detection: returns `DuplicateCandidate` or `DuplicateGroup` lists | +| `EntityMerger` | Merge duplicate groups: returns `List[MergeOperation]` | | `SimilarityCalculator` | Multi-factor similarity: string, property, relationship, and embedding | | `ClusterBuilder` | Union-Find and hierarchical clustering for large-scale batch deduplication | | `MergeStrategy` | Enum of merge strategies: `KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL` | | `PropertyMergeRule` | Dataclass holding per-property merge rule: `{property_name, strategy, conflict_resolution, priority}` | | `MergeStrategyManager` | Manage and apply named merge strategies; accepts per-property rules | -| `detect_duplicates()` | Convenience function — `detect_duplicates(entities, method="pairwise", similarity_threshold=0.7)` | -| `merge_entities()` | Convenience function — `merge_entities(entities, method="keep_most_complete")` | -| `calculate_similarity()` | Convenience function — `calculate_similarity(entity_a, entity_b, method="multi_factor")` | +| `detect_duplicates()` | Convenience function: `detect_duplicates(entities, method="pairwise", similarity_threshold=0.7)` | +| `merge_entities()` | Convenience function: `merge_entities(entities, method="keep_most_complete")` | +| `calculate_similarity()` | Convenience function: `calculate_similarity(entity_a, entity_b, method="multi_factor")` | ## What You Get @@ -35,19 +35,19 @@ icon: "copy" Pairwise, batch, incremental, and group detection modes. Returns scored candidates with reasons. - Five merge strategies — keep first, last, most complete, highest confidence, or merge all fields. + Five merge strategies: keep first, last, most complete, highest confidence, or merge all fields. Multi-factor scoring across string edit distance, property overlap, relationship overlap, and embeddings. - Union-Find and hierarchical clustering for batch deduplication at scale — handles 100k+ entity sets. + Union-Find and hierarchical clustering for batch deduplication at scale: handles 100k+ entity sets. Per-property merge rules with conflict resolution priorities. Apply different strategies to different fields. - `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7× faster than v1 for large entity sets. + `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7× faster than v1 for large entity sets. @@ -63,13 +63,13 @@ entities = [ {"id": "3", "name": "Microsoft", "type": "Company"}, ] -# 1. Detect duplicates — returns List[DuplicateCandidate] +# 1. Detect duplicates: returns List[DuplicateCandidate] detector = DuplicateDetector(similarity_threshold=0.7) candidates = detector.detect_duplicates(entities) for dup in candidates: print( - "{} vs {} — sim: {:.2f}, confidence: {:.2f}".format( + "{} vs {}: sim: {:.2f}, confidence: {:.2f}".format( dup.entity1.get("name"), dup.entity2.get("name"), dup.similarity_score, @@ -77,7 +77,7 @@ for dup in candidates: ) ) -# 2. Merge duplicates — returns List[MergeOperation] +# 2. Merge duplicates: returns List[MergeOperation] merger = EntityMerger() operations = merger.merge_duplicates(entities, strategy="keep_most_complete") @@ -95,11 +95,11 @@ Find duplicate entity pairs: from semantica.deduplication import DuplicateDetector detector = DuplicateDetector( - similarity_threshold=0.7, # default 0.7 — minimum score to include a candidate - confidence_threshold=0.6, # default 0.6 — minimum confidence to include a candidate - max_results=100, # optional — hard cap on total candidates returned - top_k_per_entity=3, # optional — max candidates per entity - min_similarity=0.75, # optional — additional floor applied after sorting + similarity_threshold=0.7, # default 0.7: minimum score to include a candidate + confidence_threshold=0.6, # default 0.6: minimum confidence to include a candidate + max_results=100, # optional: hard cap on total candidates returned + top_k_per_entity=3, # optional: max candidates per entity + min_similarity=0.75, # optional: additional floor applied after sorting sort_by="confidence", # "confidence" (default) | "similarity_score" ) @@ -115,11 +115,11 @@ for c in candidates: # Detect duplicate groups with union-find (returns List[DuplicateGroup]) groups = detector.detect_duplicate_groups(entities) for g in groups: - print("Group of {} — confidence: {:.2f} — representative: {}".format( + print("Group of {}: confidence: {:.2f}: representative: {}".format( len(g.entities), g.confidence, g.representative and g.representative.get("name") )) -# Incremental — compare new entities against existing (returns List[DuplicateCandidate]) +# Incremental: compare new entities against existing (returns List[DuplicateCandidate]) new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}] candidates = detector.incremental_detect(new_entities, entities) ``` @@ -198,7 +198,7 @@ Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group() | `"keep_last"` | Keep the most recently seen entity | | `"keep_most_complete"` | Keep the entity with the most non-null properties + relationships | | `"keep_highest_confidence"` | Keep the entity with the highest `.confidence` value | -| `"merge_all"` | Combine all properties — conflicts resolved to lists | +| `"merge_all"` | Combine all properties: conflicts resolved to lists | ### Per-property merge rules @@ -262,7 +262,7 @@ print(result.components["property"]) # property overlap component print(result.components["relationship"]) # relationship jaccard component # result.components["embedding"] is present only when embeddings are supplied -# String similarity methods — method= accepts "levenshtein", "jaro_winkler", "cosine" +# String similarity methods: method= accepts "levenshtein", "jaro_winkler", "cosine" lev = calc.calculate_string_similarity("Apple Inc.", "Apple Inc", method="levenshtein") jaro = calc.calculate_string_similarity("Steve Jobs", "Steven Jobs", method="jaro_winkler") cos = calc.calculate_string_similarity("apple", "apples", method="cosine") @@ -301,7 +301,7 @@ result = builder.build_clusters(entities) print("Clusters found:", len(result.clusters)) for cluster in result.clusters: - print(" [{}] {} entities — quality: {:.2f}".format( + print(" [{}] {} entities: quality: {:.2f}".format( cluster.cluster_id, len(cluster.entities), cluster.quality_score, @@ -327,7 +327,7 @@ print("Quality metrics:", result.quality_metrics) ```python from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity -# Detect — method= accepts "pairwise" (default), "batch", "incremental", "group" +# Detect: method= accepts "pairwise" (default), "batch", "incremental", "group" candidates = detect_duplicates( entities, method="pairwise", @@ -335,11 +335,11 @@ candidates = detect_duplicates( confidence_threshold=0.6, ) -# Merge — method= accepts the strategy strings, same as EntityMerger +# Merge: method= accepts the strategy strings, same as EntityMerger operations = merge_entities(entities, method="keep_most_complete", preserve_provenance=True) # Returns List[MergeOperation]; access .merged_entity on each -# Similarity — method= accepts "exact", "levenshtein", "jaro_winkler", "cosine", +# Similarity: method= accepts "exact", "levenshtein", "jaro_winkler", "cosine", # "property", "relationship", "embedding", "multi_factor" (default) result = calculate_similarity(entity_a, entity_b, method="multi_factor") print(result.score) @@ -402,7 +402,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name") ```python from semantica.deduplication import ClusterBuilder, EntityMerger - # Build clusters first — more efficient for large entity sets + # Build clusters first: more efficient for large entity sets builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2) result = builder.build_clusters(entities) @@ -430,7 +430,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name") ## Tips and Common Pitfalls - **`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score` — not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`. + **`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score`: not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`. @@ -446,7 +446,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name") - **Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters — if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity. + **Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity. diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index 7d085cd5..b5d1691f 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -1,6 +1,6 @@ --- title: "Embeddings Module" -description: "Text and graph embedding generation — FastEmbed, Sentence-Transformers, OpenAI, BGE — with pooling strategies and provider-agnostic API." +description: "Text and graph embedding generation: FastEmbed, Sentence-Transformers, OpenAI, BGE: with pooling strategies and provider-agnostic API." icon: "vector-square" --- @@ -15,41 +15,41 @@ icon: "vector-square" ## Why Embeddings Matter -Raw text can't be compared mathematically. Embeddings translate meaning into geometry — two semantically similar sentences produce vectors that are close together in high-dimensional space, even when they share no words. +Raw text can't be compared mathematically. Embeddings translate meaning into geometry: two semantically similar sentences produce vectors that are close together in high-dimensional space, even when they share no words. Semantica uses embeddings for: -- **Semantic search** — find knowledge graph nodes by meaning, not just keywords -- **Entity resolution** — detect that "Apple Inc." and "Apple Computer" refer to the same entity -- **Deduplication** — `semantic_v2` strategy measures entity similarity via embedding distance -- **GraphRAG retrieval** — hybrid vector + graph traversal for grounded LLM answers -- **Semantic chunking** — detect topic shift boundaries in `TextSplitter(method="semantic_transformer")` +- **Semantic search**: find knowledge graph nodes by meaning, not just keywords +- **Entity resolution**: detect that "Apple Inc." and "Apple Computer" refer to the same entity +- **Deduplication**: `semantic_v2` strategy measures entity similarity via embedding distance +- **GraphRAG retrieval**: hybrid vector + graph traversal for grounded LLM answers +- **Semantic chunking**: detect topic shift boundaries in `TextSplitter(method="semantic_transformer")` ## Exported Classes | Class | Role | | :--- | :--- | -| `EmbeddingGenerator` | Provider-agnostic entry point — handles batching and provider selection | +| `EmbeddingGenerator` | Provider-agnostic entry point: handles batching and provider selection | | `TextEmbedder` | Text embedding with automatic batch processing; default uses FastEmbed | | `GraphEmbeddingManager` | Embed KG nodes and edges for GraphRAG and graph databases | | `VectorEmbeddingManager` | Prepare and format embeddings for vector database backends | | `OpenAIStore` | OpenAI `text-embedding-3-small` / `text-embedding-3-large` provider | | `BGEStore` | BAAI/bge models via `sentence-transformers` | -| `FastEmbedStore` | ONNX-accelerated local embeddings — no CUDA **required** | -| `LlamaStore` | Placeholder store — not production-ready; do not use for embeddings | -| `MeanPooling` | Default pooling strategy — best for retrieval and clustering | +| `FastEmbedStore` | ONNX-accelerated local embeddings: no CUDA **required** | +| `LlamaStore` | Placeholder store: not production-ready; do not use for embeddings | +| `MeanPooling` | Default pooling strategy: best for retrieval and clustering | ## What You Get - Main entry point — provider-agnostic, handles batching automatically across all backends. + Main entry point: provider-agnostic, handles batching automatically across all backends. Text-specific with automatic batching and progress tracking. Default method is FastEmbed. - Node and edge embeddings for graph databases — Neo4j, NetworkX, FalkorDB. + Node and edge embeddings for graph databases: Neo4j, NetworkX, FalkorDB. Prepare, normalize, and format embeddings for FAISS, Weaviate, Qdrant, and Milvus. @@ -58,7 +58,7 @@ Semantica uses embeddings for: `OpenAIStore`, `BGEStore`, `FastEmbedStore`, and `ProviderStoreFactory`. - Mean, Max, CLS, Attention, and Hierarchical — control token-to-vector aggregation. + Mean, Max, CLS, Attention, and Hierarchical: control token-to-vector aggregation. @@ -75,7 +75,7 @@ Semantica uses embeddings for: ```python from semantica.embeddings import EmbeddingGenerator - # FastEmbed is the default — no config needed + # FastEmbed is the default: no config needed generator = EmbeddingGenerator() embedding = generator.generate_embeddings("Text about AI") ``` @@ -159,12 +159,12 @@ providers = check_available_providers() ## Getting Started -`EmbeddingGenerator` is the fastest path to embeddings — the default method is FastEmbed (ONNX, no GPU needed): +`EmbeddingGenerator` is the fastest path to embeddings: the default method is FastEmbed (ONNX, no GPU needed): ```python from semantica.embeddings import EmbeddingGenerator -# Default — FastEmbed with BAAI/bge-small-en-v1.5 +# Default: FastEmbed with BAAI/bge-small-en-v1.5 generator = EmbeddingGenerator() # Embed a single text @@ -173,7 +173,7 @@ embedding = generator.generate_embeddings("Text about AI") # Embed a batch embeddings = generator.generate_embeddings(["Text about AI", "Machine learning concepts"]) -# Compare two embeddings (cosine similarity — 0.0 to 1.0) +# Compare two embeddings (cosine similarity: 0.0 to 1.0) score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosine") print(f"Similarity: {score:.3f}") ``` @@ -195,7 +195,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") ```python from semantica.embeddings import EmbeddingGenerator - # Default — FastEmbed, free, runs locally with no GPU + # Default: FastEmbed, free, runs locally with no GPU generator = EmbeddingGenerator() # Use sentence-transformers instead @@ -213,7 +213,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") ```python - # Cosine similarity — 0.0 (unrelated) to 1.0 (identical meaning) + # Cosine similarity: 0.0 (unrelated) to 1.0 (identical meaning) score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosine") print(f"Similarity: {score:.3f}") ``` @@ -240,7 +240,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") | Provider | Model | Dimension | Speed | Best For | | :-------- | :----- | :--------- | :----- | :-------- | -| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | **Default** — CPU-optimised, no GPU **required** | +| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | **Default**: CPU-optimised, no GPU **required** | | `sentence_transformers` | `all-MiniLM-L6-v2` | 384 | Fast | Good balance of speed and quality | | `sentence_transformers` | `all-mpnet-base-v2` | 768 | Medium | Higher retrieval quality | | `sentence_transformers` | `BAAI/bge-large-en-v1.5` | 1024 | Medium | State-of-the-art retrieval accuracy | @@ -254,13 +254,13 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") ```python from semantica.embeddings import EmbeddingGenerator - # Default — FastEmbed with BAAI/bge-small-en-v1.5 + # Default: FastEmbed with BAAI/bge-small-en-v1.5 generator = EmbeddingGenerator() embeddings = generator.generate_embeddings(texts) similarity = generator.compare_embeddings(embeddings[0], embeddings[1]) ``` - **Best for:** CPU-only production, lowest latency without GPU. Default — works out of the box. + **Best for:** CPU-only production, lowest latency without GPU. Default: works out of the box. ```python @@ -315,7 +315,7 @@ Direct text embedding with batch processing: ```python from semantica.embeddings import TextEmbedder -# Default — FastEmbed with BAAI/bge-small-en-v1.5 +# Default: FastEmbed with BAAI/bge-small-en-v1.5 embedder = TextEmbedder() # Single text → 1D array @@ -341,7 +341,7 @@ dim = embedder.get_embedding_dimension() | `normalize` | `bool` | `True` | L2-normalize output vectors | **Key behaviours:** -- If FastEmbed or sentence-transformers is unavailable, falls back to a 128-dimensional hash-based embedding. Hash embeddings are deterministic but not semantic — do not use in production. +- If FastEmbed or sentence-transformers is unavailable, falls back to a 128-dimensional hash-based embedding. Hash embeddings are deterministic but not semantic: do not use in production. - Large batches are chunked internally by the underlying library to avoid OOM. ## Provider Stores @@ -359,28 +359,28 @@ import os store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small") embedding = store.embed("Hello world") -# BGE (Sentence-Transformers wrapper) — pass model_name= not model= +# BGE (Sentence-Transformers wrapper): pass model_name= not model= store = BGEStore(model_name="BAAI/bge-large-en-v1.5") embedding = store.embed("Hello world") -# FastEmbed — ONNX runtime, no CUDA required +# FastEmbed: ONNX runtime, no CUDA required store = FastEmbedStore(model_name="BAAI/bge-small-en-v1.5") embedding = store.embed("Hello world") # FastEmbedStore also has an efficient batch method embeddings = store.embed_batch(["text1", "text2", "text3"]) -# Auto-select from a name string — useful in config-driven pipelines +# Auto-select from a name string: useful in config-driven pipelines # Supported providers: "openai", "bge", "fastembed" store = ProviderStoreFactory.create(provider="bge", model_name="BAAI/bge-large-en-v1.5") ``` - `LlamaStore` exists in the module but is a placeholder — it does not connect to Ollama and always raises `ProcessingError` at embed time. Do not use it in production. + `LlamaStore` exists in the module but is a placeholder: it does not connect to Ollama and always raises `ProcessingError` at embed time. Do not use it in production. ## Pooling Strategies -Pooling aggregates a set of embeddings into a single vector — useful when you have multiple chunk embeddings to combine: +Pooling aggregates a set of embeddings into a single vector: useful when you have multiple chunk embeddings to combine: @@ -391,7 +391,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings) # shape: (hidden_dim,) ``` - **Best for:** retrieval, semantic search, and clustering — averages all contributions. + **Best for:** retrieval, semantic search, and clustering: averages all contributions. ```python @@ -401,7 +401,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings) ``` - **Best for:** capturing the presence of any feature — takes the max activation per dimension. + **Best for:** capturing the presence of any feature: takes the max activation per dimension. ```python @@ -422,7 +422,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings, chunk_size=10) ``` - **Best for:** long documents — chunk-level mean pooling, then global mean pooling across chunks. + **Best for:** long documents: chunk-level mean pooling, then global mean pooling across chunks. @@ -518,7 +518,7 @@ combined = manager.batch_prepare([embeddings_a, embeddings_b], backend="qdrant") "Amazon was started by Jeff Bezos.", ] - # All at once — more efficient than calling embed_text() per item + # All at once: more efficient than calling embed_text() per item embeddings = embedder.embed_batch(texts) print(f"Shape: {embeddings.shape}") # (3, 384) ``` @@ -579,7 +579,7 @@ combined = manager.batch_prepare([embeddings_a, embeddings_b], backend="qdrant") ```python from semantica.embeddings import calculate_similarity -# Cosine similarity — direction only, not magnitude; most common for text +# Cosine similarity: direction only, not magnitude; most common for text score = calculate_similarity(embedding_a, embedding_b, method="cosine") # → 0.0 (orthogonal / unrelated) to 1.0 (identical direction) @@ -595,7 +595,7 @@ from semantica.embeddings import ( pool_embeddings, check_available_providers, ) -# Single text — fastest path +# Single text: fastest path emb = embed_text("Hello world", method="sentence_transformers") # Batch @@ -624,15 +624,15 @@ providers = check_available_providers() - **FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers — passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control. + **FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers: passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control. - **Always use the same model for indexing and querying.** Vectors from different models are not comparable — they live in different vector spaces. Switching models requires re-embedding your entire corpus. + **Always use the same model for indexing and querying.** Vectors from different models are not comparable: they live in different vector spaces. Switching models requires re-embedding your entire corpus. - **Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()` — if it returns `"fallback"`, install your intended provider. + **Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()`: if it returns `"fallback"`, install your intended provider. diff --git a/docs/reference/evals.md b/docs/reference/evals.md index dc4c756a..0f6303a2 100644 --- a/docs/reference/evals.md +++ b/docs/reference/evals.md @@ -1,6 +1,6 @@ --- title: "Evals Module" -description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance — coming soon." +description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance: coming soon." icon: "chart-line" --- diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index cd60f773..555e5aab 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -6,11 +6,11 @@ icon: "map" **`semantica.explorer`** is a **browser-based dashboard** for exploring knowledge graphs, managing ontologies, and running visual analyses: -- Indexed search: 0.004ms on 118k nodes — no full scans +- Indexed search: 0.004ms on 118k nodes: no full scans - Ontology Hub: visual editor, SHACL Studio, alignment authoring, and health dashboard - Bidirectional path finding between any two nodes - WebSocket progress streaming for live pipeline monitoring -- No code required after launch — full graph exploration in the browser +- No code required after launch: full graph exploration in the browser ## Getting Started @@ -37,7 +37,7 @@ graph.save_to_file("my_graph.json") # 2. Launch the Explorer semantica-explorer --graph my_graph.json # → Loading graph... -# → Graph loaded — 2 nodes, 1 edges +# → Graph loaded: 2 nodes, 1 edges # → Semantica Explorer · http://127.0.0.1:8000 # API docs http://127.0.0.1:8000/docs # Health http://127.0.0.1:8000/api/health @@ -96,8 +96,8 @@ The `semantica-explorer` command accepts exactly four flags: | :---- | :----- | :------- | :----------- | | `--graph` | `-g` | *(**required**)* | Path to a ContextGraph JSON file to load | | `--port` | `-p` | `8000` | Port to bind the server | -| `--host` | — | `127.0.0.1` | Host to bind the server — use `0.0.0.0` to expose on the network | -| `--no-browser` | — | off | Skip auto-opening the browser tab | +| `--host` |: | `127.0.0.1` | Host to bind the server: use `0.0.0.0` to expose on the network | +| `--no-browser` |: | off | Skip auto-opening the browser tab | There are no flags for authentication, CORS, or log level in the CLI. CORS allowed origins are configured via the `EXPLORER_CORS_ORIGINS` environment variable (comma-separated, default: `http://localhost:5173,http://127.0.0.1:5173`). @@ -122,7 +122,7 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ Degree centrality, community detection, connectivity analysis, graph validation, and distance matrices. - All features available as a REST API — fully documented at `/docs`. + All features available as a REST API: fully documented at `/docs`. Real-time graph mutation events streamed over WebSocket at `/ws/graph-updates`. @@ -138,42 +138,42 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ Core dashboard for navigating knowledge graphs: - - **Indexed search** — POST to `/api/graph/search` with a query; 0.004ms on 118k-node graphs - - **Path finding** — BFS or Dijkstra between any two nodes via `GET /api/graph/path?source=&target=` - - **Neighbor expansion** — `GET /api/graph/node/{id}/neighbors?depth=2` - - **Filter by entity type** — `GET /api/graph/nodes?type=Person` - - **Semantic neighborhood** — `GET /api/graph/semantic-neighborhood?node_id=&top_k=20` - - **Distance matrix** — `POST /api/graph/distance-matrix` + - **Indexed search**: POST to `/api/graph/search` with a query; 0.004ms on 118k-node graphs + - **Path finding**: BFS or Dijkstra between any two nodes via `GET /api/graph/path?source=&target=` + - **Neighbor expansion**: `GET /api/graph/node/{id}/neighbors?depth=2` + - **Filter by entity type**: `GET /api/graph/nodes?type=Person` + - **Semantic neighborhood**: `GET /api/graph/semantic-neighborhood?node_id=&top_k=20` + - **Distance matrix**: `POST /api/graph/distance-matrix` Ontology lifecycle management in the browser: - - **Registry** — `GET /api/ontology/registry` — list loaded ontologies - - **SKOS vocabularies** — `GET /api/ontology/skos/schemes`, `GET /api/ontology/skos/concept/{uri}` - - **SHACL** — `POST /api/ontology/shacl/generate`, `POST /api/ontology/shacl/validate` - - **Alignments** — `GET/POST /api/ontology/alignments`, `POST /api/ontology/suggest-alignments` - - **Proposals & versioning** — `POST /api/ontology/propose`, `GET /api/ontology/versions/{uri}` - - **Health** — `GET /api/ontology/health` + - **Registry**: `GET /api/ontology/registry`: list loaded ontologies + - **SKOS vocabularies**: `GET /api/ontology/skos/schemes`, `GET /api/ontology/skos/concept/{uri}` + - **SHACL**: `POST /api/ontology/shacl/generate`, `POST /api/ontology/shacl/validate` + - **Alignments**: `GET/POST /api/ontology/alignments`, `POST /api/ontology/suggest-alignments` + - **Proposals & versioning**: `POST /api/ontology/propose`, `GET /api/ontology/versions/{uri}` + - **Health**: `GET /api/ontology/health` Graph metrics running against the loaded graph: - - **Combined metrics** — `GET /api/analytics?metrics=centrality,community,connectivity` - - **Graph validation** — `GET /api/analytics/validation` - - **Enrich: link prediction** — `POST /api/enrich/links` - - **Enrich: deduplication** — `POST /api/enrich/dedup` - - **Enrich: entity extraction** — `POST /api/enrich/extract` - - **Temporal** — `GET /api/temporal/snapshot`, `GET /api/temporal/diff`, `GET /api/temporal/bounds` + - **Combined metrics**: `GET /api/analytics?metrics=centrality,community,connectivity` + - **Graph validation**: `GET /api/analytics/validation` + - **Enrich: link prediction**: `POST /api/enrich/links` + - **Enrich: deduplication**: `POST /api/enrich/dedup` + - **Enrich: entity extraction**: `POST /api/enrich/extract` + - **Temporal**: `GET /api/temporal/snapshot`, `GET /api/temporal/diff`, `GET /api/temporal/bounds` Decision tracking and provenance queries: - - **Decisions** — `GET /api/decisions`, `GET /api/decisions/{id}`, `GET /api/decisions/{id}/chain` - - **Precedents** — `GET /api/decisions/{id}/precedents` - - **Causal distance** — `GET /api/decisions/causal-distance?source=&target=` - - **Compliance** — `GET /api/decisions/{id}/compliance` - - **Provenance** — `GET /api/provenance?node_id=`, `GET /api/provenance/report?node_id=` - - **Annotations** — `GET/POST /api/annotations`, `DELETE /api/annotations/{id}` + - **Decisions**: `GET /api/decisions`, `GET /api/decisions/{id}`, `GET /api/decisions/{id}/chain` + - **Precedents**: `GET /api/decisions/{id}/precedents` + - **Causal distance**: `GET /api/decisions/causal-distance?source=&target=` + - **Compliance**: `GET /api/decisions/{id}/compliance` + - **Provenance**: `GET /api/provenance?node_id=`, `GET /api/provenance/report?node_id=` + - **Annotations**: `GET/POST /api/annotations`, `DELETE /api/annotations/{id}` @@ -187,14 +187,14 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | | :-------- | :------ | :----------- | | `/api/graph/stats` | `GET` | Node count, edge count, entity type distribution | - | `/api/graph/nodes` | `GET` | List nodes — `?type=&search=&skip=&limit=&cursor=&bbox=` | + | `/api/graph/nodes` | `GET` | List nodes: `?type=&search=&skip=&limit=&cursor=&bbox=` | | `/api/graph/node/{id}` | `GET` | Fetch a single node with all properties | - | `/api/graph/node/{id}/neighbors` | `GET` | Neighbors of a node — `?depth=1` (1–5) | - | `/api/graph/edges` | `GET` | List edges — `?type=&source=&target=&skip=&limit=&cursor=` | - | `/api/graph/path` | `GET` | Shortest path — `?source=&target=&algorithm=bfs&directed=true` | - | `/api/graph/search` | `POST` | Indexed search — body: `{query, limit, filters, anchor_node}` | - | `/api/graph/distance-matrix` | `POST` | Pairwise distances — body: `{node_ids, metric}` (max 50 nodes) | - | `/api/graph/semantic-neighborhood` | `GET` | Semantic neighbors — `?node_id=&top_k=20&min_similarity=0.0` | + | `/api/graph/node/{id}/neighbors` | `GET` | Neighbors of a node: `?depth=1` (1–5) | + | `/api/graph/edges` | `GET` | List edges: `?type=&source=&target=&skip=&limit=&cursor=` | + | `/api/graph/path` | `GET` | Shortest path: `?source=&target=&algorithm=bfs&directed=true` | + | `/api/graph/search` | `POST` | Indexed search: body: `{query, limit, filters, anchor_node}` | + | `/api/graph/distance-matrix` | `POST` | Pairwise distances: body: `{node_ids, metric}` (max 50 nodes) | + | `/api/graph/semantic-neighborhood` | `GET` | Semantic neighbors: `?node_id=&top_k=20&min_similarity=0.0` | @@ -203,7 +203,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | | :-------- | :------ | :----------- | - | `/api/analytics` | `GET` | Graph metrics — `?metrics=centrality,community,connectivity` | + | `/api/analytics` | `GET` | Graph metrics: `?metrics=centrality,community,connectivity` | | `/api/analytics/validation` | `GET` | Graph validation report | **Enrich:** @@ -221,10 +221,10 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | | :-------- | :------ | :----------- | | `/api/temporal/snapshot` | `GET` | Graph snapshot at `?at=ISO8601` (defaults to now) | - | `/api/temporal/diff` | `GET` | Diff between two times — `?from_time=&to_time=` | + | `/api/temporal/diff` | `GET` | Diff between two times: `?from_time=&to_time=` | | `/api/temporal/patterns` | `GET` | Temporal activity patterns | | `/api/temporal/bounds` | `GET` | Earliest and latest temporal bounds in graph | - | `/api/temporal/distance-history` | `GET` | Distance history — `?source=&target=` | + | `/api/temporal/distance-history` | `GET` | Distance history: `?source=&target=` | @@ -236,7 +236,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | `/api/ontology/registry` | `GET` | List loaded ontologies | | `/api/ontology/load` | `POST` | Load an ontology from URL or content | | `/api/ontology/create` | `POST` | Create a new ontology | - | `/api/ontology/search` | `GET` | Search ontology entities — `?q=term` | + | `/api/ontology/search` | `GET` | Search ontology entities: `?q=term` | | `/api/ontology/health` | `GET` | Ontology health and coverage metrics | | `/api/ontology/alignments` | `GET/POST` | List or create ontology alignments | | `/api/ontology/suggest-alignments` | `POST` | AI-suggested alignments | @@ -252,7 +252,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | | :-------- | :------ | :----------- | | `/api/vocabulary/schemes` | `GET` | SKOS schemes via TripletStore | - | `/api/vocabulary/concepts` | `GET` | Concepts in a scheme — `?scheme=URI` | + | `/api/vocabulary/concepts` | `GET` | Concepts in a scheme: `?scheme=URI` | | `/api/vocabulary/hierarchy` | `GET` | Concept hierarchy tree | | `/api/vocabulary/import` | `POST` | Import SKOS/RDF vocabulary file | @@ -274,20 +274,20 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | `/api/decisions/{id}/chain` | `GET` | Causal chain for a decision | | `/api/decisions/{id}/precedents` | `GET` | Similar past decisions | | `/api/decisions/{id}/compliance` | `GET` | Policy compliance check | - | `/api/decisions/causal-distance` | `GET` | Causal distance — `?source=&target=` | + | `/api/decisions/causal-distance` | `GET` | Causal distance: `?source=&target=` | **Provenance:** | Endpoint | Method | Description | | :-------- | :------ | :----------- | - | `/api/provenance` | `GET` | Entity provenance lineage — `?node_id=` | - | `/api/provenance/report` | `GET` | Provenance export report — `?node_id=` | + | `/api/provenance` | `GET` | Entity provenance lineage: `?node_id=` | + | `/api/provenance/report` | `GET` | Provenance export report: `?node_id=` | **Annotations:** | Endpoint | Method | Description | | :-------- | :------ | :----------- | - | `/api/annotations` | `GET` | List annotations — `?node_id=` (optional) | + | `/api/annotations` | `GET` | List annotations: `?node_id=` (optional) | | `/api/annotations` | `POST` | Create annotation (returns 201) | | `/api/annotations/{id}` | `DELETE` | Delete annotation (returns 204) | @@ -295,7 +295,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | | :-------- | :------ | :----------- | - | `/api/export` | `POST` | Export graph as JSON or CSV — body: `{format, node_ids}` | + | `/api/export` | `POST` | Export graph as JSON or CSV: body: `{format, node_ids}` | | `/api/export/distance-enriched` | `POST` | Export pairwise distances as CSV or JSONL | | `/api/import` | `POST` | Import nodes/edges from `.json` or `.csv` file (max 50 MB) | @@ -306,7 +306,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | :-------- | :------ | :----------- | | `/api/health` | `GET` | Returns `{"status": "healthy"}` | | `/api/info` | `GET` | Server name, version, status | - | `/docs` | `GET` | Interactive Swagger UI — all endpoints | + | `/docs` | `GET` | Interactive Swagger UI: all endpoints | @@ -395,7 +395,7 @@ Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot ## Tips and Common Pitfalls - **Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting — the force-directed layout becomes unusable on very large graphs. + **Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting: the force-directed layout becomes unusable on very large graphs. @@ -403,7 +403,7 @@ Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot - **Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API — pipe them into scripts to automate batch annotation, SPARQL querying, or exports. + **Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API: pipe them into scripts to automate batch annotation, SPARQL querying, or exports. diff --git a/docs/reference/export.md b/docs/reference/export.md index e78e081c..9c1547d9 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -6,7 +6,7 @@ icon: "file-export" **`semantica.export`** serializes knowledge graphs to **every downstream format**: -- RDF: Turtle, JSON-LD, N-Triples, RDF/XML — with optional W3C PROV-O provenance inline +- RDF: Turtle, JSON-LD, N-Triples, RDF/XML: with optional W3C PROV-O provenance inline - Analytics: Apache Parquet and Arrow for Spark, BigQuery, Databricks - Graph databases: Cypher `CREATE` statements for Neo4j; AQL `INSERT` for ArangoDB - Standard formats: GraphML, GEXF, Graphviz DOT, CSV, OWL 2.0 @@ -28,7 +28,7 @@ icon: "file-export" | `ArrowExporter` | Apache Arrow IPC | Requires `pyarrow`; zero-copy transfer | | `DistanceExporter` | CSV, JSONL | Pairwise distance metrics; takes a `graph` arg | | `ReportGenerator` | HTML, Markdown, JSON, plain text | Analytics reports | -| `NamespaceManager` | — | RDF namespace extraction and declaration generation | +| `NamespaceManager` |: | RDF namespace extraction and declaration generation | ## Getting Started @@ -91,7 +91,7 @@ export_lpg(graph, "import.cypher", method="cypher") - Export to W3C RDF formats — Turtle, JSON-LD, N-Triples, and RDF/XML. + Export to W3C RDF formats: Turtle, JSON-LD, N-Triples, and RDF/XML. **`export_to_rdf()` returns a string; `export()` writes to a file:** @@ -235,7 +235,7 @@ export_lpg(graph, "import.cypher", method="cypher") ``` - **VectorExporter** — takes `(vectors, file_path, format=)`: + **VectorExporter**: takes `(vectors, file_path, format=)`: ```python from semantica.export import VectorExporter @@ -248,7 +248,7 @@ export_lpg(graph, "import.cypher", method="cypher") exporter.export(vectors, "vectors.faiss", format="faiss") ``` - **ArrowExporter** — requires `pyarrow`: + **ArrowExporter**: requires `pyarrow`: ```python from semantica.export import ArrowExporter @@ -257,7 +257,7 @@ export_lpg(graph, "import.cypher", method="cypher") exporter.export(graph, "graph.arrow") ``` - **DistanceExporter** — takes a `graph` argument at construction: + **DistanceExporter**: takes a `graph` argument at construction: ```python from semantica.export import DistanceExporter @@ -347,12 +347,12 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p | `"numpy"` | `numpy` | `VectorExporter` | `.npz` | NumPy arrays from embeddings | | `"binary"` | `binary` | `VectorExporter` | `.bin` | Raw float32 binary | | `"faiss"` | `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files | -| `"html"` / `"markdown"` / `"json"` / `"text"` | — | `ReportGenerator` | `.html` / `.md` / `.json` / `.txt` | Analytics reports | +| `"html"` / `"markdown"` / `"json"` / `"text"` |: | `ReportGenerator` | `.html` / `.md` / `.json` / `.txt` | Analytics reports | ## Tips and Common Pitfalls - **`export_to_rdf()` returns a string — it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk. + **`export_to_rdf()` returns a string: it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk. @@ -372,7 +372,7 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p - **Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented — one triple per line — making it safe to stream, concatenate, and process with standard Unix tools. + **Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented: one triple per line: making it safe to stream, concatenate, and process with standard Unix tools. diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index b998f942..44f433ed 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -6,9 +6,9 @@ icon: "server" **`semantica.graph_store`** provides a **single unified API** for persisting and querying knowledge graphs in production graph databases: -- Swap backends with a one-line change — Neo4j, FalkorDB, Apache AGE, Amazon Neptune +- Swap backends with a one-line change: Neo4j, FalkorDB, Apache AGE, Amazon Neptune - Parameterized Cypher execution with optional result caching via `QueryEngine` -- Batch node and edge loading — faster than individual writes +- Batch node and edge loading: faster than individual writes - `GraphAnalytics` for degree centrality, connected components, shortest path, neighbor traversal - Context manager support: `with GraphStore(...) as store:` closes connection automatically @@ -20,10 +20,10 @@ icon: "server" | `GraphStore` | Unified interface: `create_node`, `create_relationship`, `query`, `get_neighbors`, `shortest_path` | | `QueryEngine` | Parameterized Cypher execution with result caching | | `GraphAnalytics` | `degree_centrality`, `connected_components`, `shortest_path`, `get_neighbors` | -| `Neo4jStore` | Production workloads via Bolt — supports APOC and GDS plugins | -| `ApacheAgeStore` | PostgreSQL + AGE extension — no separate graph server needed | -| `AmazonNeptuneStore` | AWS Neptune — OpenCypher via Bolt protocol | -| `FalkorDBStore` | Redis-based — sub-millisecond latency for real-time applications | +| `Neo4jStore` | Production workloads via Bolt: supports APOC and GDS plugins | +| `ApacheAgeStore` | PostgreSQL + AGE extension: no separate graph server needed | +| `AmazonNeptuneStore` | AWS Neptune: OpenCypher via Bolt protocol | +| `FalkorDBStore` | Redis-based: sub-millisecond latency for real-time applications | ## What You Get @@ -32,7 +32,7 @@ icon: "server" - Unified API across Neo4j, FalkorDB, Apache AGE, Amazon Neptune - Context manager support for automatic connection cleanup - - `create_nodes()` for bulk loading — faster than individual calls + - `create_nodes()` for bulk loading: faster than individual calls - Parameterized Cypher construction prevents injection attacks @@ -45,13 +45,13 @@ icon: "server" - Shortest path between nodes, neighbor traversal up to N hops - - `create_nodes(list)` — one round-trip for many nodes + - `create_nodes(list)`: one round-trip for many nodes - `create_relationship()` with typed properties - `delete_node(detach=True)` removes all connected relationships - - `create_index(label, property_name=)` — makes MATCH queries orders-of-magnitude faster - - `get_stats()` — node counts, edge counts, type breakdown + - `create_index(label, property_name=)`: makes MATCH queries orders-of-magnitude faster + - `get_stats()`: node counts, edge counts, type breakdown - Create indexes before bulk loading for best performance @@ -132,7 +132,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass ```python - # Batch creation — list of dicts with "labels" and "properties" keys + # Batch creation: list of dicts with "labels" and "properties" keys store.create_nodes([ {"labels": ["Person"], "properties": {"name": "Alice"}}, {"labels": ["Organization"], "properties": {"name": "Acme Corp"}}, @@ -187,7 +187,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass uri="bolt://localhost:7687", user="neo4j", password="password", - database="neo4j", # optional — targets default database + database="neo4j", # optional: targets default database ) store.connect() ``` @@ -248,7 +248,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass ) ``` - **Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries — the same query API used for Neo4j. + **Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries: the same query API used for Neo4j. @@ -300,7 +300,7 @@ neighbors = store.get_neighbors( depth=1, ) -# Shortest path — returns {"length", "nodes", "relationships"} or None +# Shortest path: returns {"length", "nodes", "relationships"} or None path = store.shortest_path( start_node_id=jobs_id, end_node_id=cook_id, @@ -331,7 +331,7 @@ result = engine.execute( ) # result is {"success": True, "records": [...], "keys": [...], "metadata": {...}} -# Execute with caching — repeated identical calls return cached result +# Execute with caching: repeated identical calls return cached result result = engine.execute( "MATCH (p:Person) RETURN count(p) as total", use_cache=True, @@ -368,7 +368,7 @@ store.connect() # GraphAnalytics takes the backend store, not the GraphStore facade analytics = GraphAnalytics(store._store_backend) -# Degree centrality — returns list of {"id", "degree"} dicts ordered by degree DESC +# Degree centrality: returns list of {"id", "degree"} dicts ordered by degree DESC scores = analytics.degree_centrality( labels=["Person"], rel_type="KNOWS", @@ -380,7 +380,7 @@ for entry in scores[:5]: # Connected components (requires GDS for Neo4j, NetworkX for in-process) components = analytics.connected_components(labels=["Person"]) -# Shortest path — returns {"length", "nodes", "relationships"} or None +# Shortest path: returns {"length", "nodes", "relationships"} or None path = analytics.shortest_path( start_node_id=alice_id, end_node_id=charlie_id, @@ -401,7 +401,7 @@ neighbors = analytics.get_neighbors( | Method | Returns | Description | | :------ | :------- | :----------- | -| `degree_centrality(labels, rel_type, direction)` | `List[dict]` | Degree-based node importance — records ordered by degree DESC | +| `degree_centrality(labels, rel_type, direction)` | `List[dict]` | Degree-based node importance: records ordered by degree DESC | | `connected_components(labels)` | `List[dict]` | Connected component assignment (requires GDS on Neo4j) | | `shortest_path(start_node_id, end_node_id, rel_type, max_depth)` | `dict \| None` | Path with `length`, `nodes`, `relationships` or `None` | | `get_neighbors(node_id, rel_type, direction, depth)` | `List[dict]` | Neighbor nodes up to `depth` hops | @@ -414,7 +414,7 @@ neighbors = analytics.get_neighbors( ## Schema Management ```python -# Index for fast label-property lookups — use property_name= not property= +# Index for fast label-property lookups: use property_name= not property= store.create_index(label="Person", property_name="name") store.create_index(label="Organization", property_name="id") @@ -481,7 +481,7 @@ stats = store.get_stats() ``` - AGE supports one primary label per vertex. If you pass multiple labels, the first is used as the AGE label and the rest are stored in a `labels` property array. Parameterized queries use literal inlining internally (AGE does not support `$param` binding inside `cypher()` calls) — the store handles escaping automatically. + AGE supports one primary label per vertex. If you pass multiple labels, the first is used as the AGE label and the rest are stored in a `labels` property array. Parameterized queries use literal inlining internally (AGE does not support `$param` binding inside `cypher()` calls): the store handles escaping automatically. @@ -504,7 +504,7 @@ stats = store.get_stats() - **`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")` — using `property=` will be silently ignored. + **`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")`: using `property=` will be silently ignored. diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 5d124e28..70564bf7 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -9,7 +9,7 @@ icon: "database" - 15+ ingestion adapters: files, web, SQL, Snowflake, Kafka, MCP, Git repos, email - PyArrow Parquet with column selection and partitioned dataset support - XXE-safe lxml XML with optional XSD schema validation -- `ingest()` unified dispatcher — auto-detects source type from path or URL +- `ingest()` unified dispatcher: auto-detects source type from path or URL - Each ingestor returns its own typed object (`FileObject`, `WebContent`, `TableData`, etc.) @@ -17,26 +17,26 @@ icon: "database" | Class | Role | | :--- | :--- | -| `FileIngestor` | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR — type auto-detected from extension | +| `FileIngestor` | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR: type auto-detected from extension | | `CloudStorageIngestor` | Unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage | | `WebIngestor` | Web scraping and crawling with `ingest_url`, `crawl_sitemap`, `crawl_domain` | | `RESTIngestor` | Generic REST API ingestion with headers, params, retries, and pagination | | `PublicAPIIngestor` | No-auth public API ingestion with pre-configured examples and rate limiting | | `FeedIngestor` | RSS/Atom feed ingestion with live monitoring via `FeedMonitor` | | `StreamIngestor` | Real-time ingestion from Kafka, RabbitMQ, AWS Kinesis, and Apache Pulsar | -| `RepoIngestor` | Git repositories — source files, commit history, and metadata | -| `DBIngestor` | SQL databases via SQLAlchemy — tables, views, and custom queries | +| `RepoIngestor` | Git repositories: source files, commit history, and metadata | +| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries | | `SnowflakeIngestor` | Snowflake data warehouse queries and table exports | | `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection | | `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation | | `EmailIngestor` | IMAP/POP3 email ingestion with attachment extraction | | `OntologyIngestor` | OWL/RDF/Turtle ontology file ingestion | | `MCPIngestor` | Model Context Protocol (MCP) resource ingestion | -| `ingest()` | Unified dispatcher — detects source type automatically from path or URL | +| `ingest()` | Unified dispatcher: detects source type automatically from path or URL | ## Getting Started -Use **`FileIngestor`** for local files — it **auto-detects format** from the file extension and handles archives: +Use **`FileIngestor`** for local files: it **auto-detects format** from the file extension and handles archives: ```python from semantica.ingest import FileIngestor @@ -64,13 +64,13 @@ from semantica.ingest import WebIngestor wc = WebIngestor(delay=1.0, respect_robots=True).ingest_url("https://example.com") print(wc.title, wc.text) -# Database — constructor takes no required args; pass connection_string to methods +# Database: constructor takes no required args; pass connection_string to methods from semantica.ingest import DBIngestor db = DBIngestor() result = db.ingest_database("postgresql://user:pass@localhost/db") # result["tables"]["documents"]["rows"] contains the rows -# Unified dispatcher — auto-detects source type +# Unified dispatcher: auto-detects source type from semantica.ingest import ingest result = ingest("data/report.pdf") # -> {"files": [FileObject]} result = ingest("https://example.com") # -> {"content": WebContent} @@ -87,7 +87,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} ingestor = FileIngestor() - # Single file — type auto-detected from extension + # Single file: type auto-detected from extension file_obj = ingestor.ingest_file("data/report.pdf") # Recursive directory scan @@ -157,7 +157,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} # Single file file_obj = ingestor.ingest_file("data/report.pdf") - # Directory — returns List[FileObject] + # Directory: returns List[FileObject] files = ingestor.ingest_directory("data/", recursive=True) # ingest() dispatches to ingest_file or ingest_directory automatically @@ -185,7 +185,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} # Partitioned directory (year=2024/month=01/...) data = ingestor.ingest_directory("data/partitioned/") - # Load only specific columns — pass as kwarg + # Load only specific columns: pass as kwarg from semantica.ingest import ingest_parquet data = ingest_parquet("data/events.parquet", columns=["id", "text", "timestamp"]) @@ -206,7 +206,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} ingestor = XMLIngestor() data = ingestor.ingest_file("data/records.xml") - # With XSD validation — pass schema_path as kwarg + # With XSD validation: pass schema_path as kwarg from semantica.ingest import ingest_xml data = ingest_xml("data/records.xml", schema_path="schema.xsd") @@ -295,7 +295,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} ### RepoIngestor - Ingest Git repositories — source code, commit history, and dependency graphs: + Ingest Git repositories: source code, commit history, and dependency graphs: ```python from semantica.ingest import RepoIngestor @@ -347,7 +347,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} from semantica.ingest import CloudStorageIngestor import os - # AWS S3 — list and download objects + # AWS S3: list and download objects ingestor = CloudStorageIngestor( provider="s3", access_key_id=os.getenv("AWS_ACCESS_KEY_ID"), @@ -424,7 +424,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData} ### StreamIngestor - Real-time ingestion from message brokers — each method returns a typed processor: + Real-time ingestion from message brokers: each method returns a typed processor: ```python from semantica.ingest import StreamIngestor @@ -604,7 +604,7 @@ result = ingest_file("source_path", method="my_format") ## Tips and Common Pitfalls - **`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument — not to `DBIngestor()` itself. + **`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself. @@ -612,11 +612,11 @@ result = ingest_file("source_path", method="my_format") - **Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns — critical for wide tables with hundreds of columns. + **Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns. - **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica — it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML. + **`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML. diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 64945d16..c1532dfd 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -17,11 +17,11 @@ icon: "diagram-project" | Class | Role | | :--- | :--- | -| `KnowledgeGraph` | Core graph data structure — nodes, edges, properties, temporal validity | +| `KnowledgeGraph` | Core graph data structure: nodes, edges, properties, temporal validity | | `GraphBuilder` | Construct from entities + relationships; pass `merge_entities=True` to enable deduplication | | `GraphBuilderWithProvenance` | Wraps `GraphBuilder` with optional provenance tracking; pass `provenance=True` to enable | | `EntityResolver` | Entity deduplication and merging during graph construction | -| `GraphAnalyzer` | Unified analytics wrapper — runs centrality, community detection, and connectivity in one call | +| `GraphAnalyzer` | Unified analytics wrapper: runs centrality, community detection, and connectivity in one call | | `ConnectivityAnalyzer` | Connected component detection, bridge identification, density, and degree statistics | | `TemporalGraphQuery` | Point-in-time snapshots, temporal diffs, and all 13 Allen interval queries | | `CentralityCalculator` | PageRank, degree, betweenness, closeness, eigenvector centrality | @@ -41,7 +41,7 @@ icon: "diagram-project" ## GraphBuilder -**`GraphBuilder`** constructs knowledge graphs from extracted entities and relationships. `merge_entities` defaults to `False` — pass **`True`** to enable entity deduplication during construction: +**`GraphBuilder`** constructs knowledge graphs from extracted entities and relationships. `merge_entities` defaults to `False`: pass **`True`** to enable entity deduplication during construction: ```python from semantica.kg import GraphBuilder @@ -83,7 +83,7 @@ kg = builder.build(sources=[ } ]) -# Point-in-time snapshot — TemporalGraphQuery takes no positional graph arg; +# Point-in-time snapshot: TemporalGraphQuery takes no positional graph arg; # pass the graph into each query method instead. query = TemporalGraphQuery() snapshot_2021 = query.reconstruct_at_time(kg, "2021-06-15") @@ -93,7 +93,7 @@ snapshot_2023 = query.reconstruct_at_time(kg, "2023-01-01") range_result = query.query_time_range(kg, "", "2020-01-01", "2023-01-01") print(f"Relationships in range: {range_result['num_relationships']}") -# Versioned snapshots — author and description are required +# Versioned snapshots: author and description are required versioner = TemporalVersionManager() versioner.create_snapshot(kg, version_label="2024-Q1", author="user@example.com", @@ -125,7 +125,7 @@ calc = SimilarityCalculator() score = calc.cosine_similarity(embeddings["Apple Inc."], embeddings["Google"]) print(f"Apple–Google structural similarity: {score:.3f}") -# Find structurally similar nodes — returns List[str] of node IDs +# Find structurally similar nodes: returns List[str] of node IDs similar = embedder.find_similar_nodes(kg, "Apple Inc.", top_k=5) for node_id in similar: print(node_id) @@ -171,10 +171,10 @@ for node_id in similar: detector = CommunityDetector() - # Louvain — fast, high quality (default) + # Louvain: fast, high quality (default) communities = detector.detect_communities(graph, algorithm="louvain") - # Leiden — higher quality, slower + # Leiden: higher quality, slower communities = detector.detect_communities_leiden(graph, resolution=1.2) # Evaluate community quality @@ -185,10 +185,10 @@ for node_id in similar: | Algorithm | Strength | | :--------- | :-------- | - | Louvain | Fast, good modularity — use for large graphs | - | Leiden | Best modularity — use when quality matters more than speed | - | Label Propagation | Near-linear time — use for very large graphs | - | K-Clique | Overlapping communities — nodes can belong to multiple groups | + | Louvain | Fast, good modularity: use for large graphs | + | Leiden | Best modularity: use when quality matters more than speed | + | Label Propagation | Near-linear time: use for very large graphs | + | K-Clique | Overlapping communities: nodes can belong to multiple groups | Find shortest paths and route alternatives between any two nodes. @@ -211,9 +211,9 @@ for node_id in similar: | Algorithm | Use case | | :--------- | :-------- | - | Dijkstra | Weighted shortest path — standard routing | - | A\* | Heuristic-guided search — faster on large sparse graphs | - | BFS | Unweighted shortest path — hop count only | + | Dijkstra | Weighted shortest path: standard routing | + | A\* | Heuristic-guided search: faster on large sparse graphs | + | BFS | Unweighted shortest path: hop count only | | K-Shortest | Multiple alternative routes | @@ -239,7 +239,7 @@ for node_id in similar: | Common Neighbors | Nodes with shared connections | | Jaccard | Normalized common-neighbor overlap | | Adamic-Adar | Weighted common neighbors (penalizes hubs) | - | Resource Allocation | Conservative — ignores high-degree intermediaries | + | Resource Allocation | Conservative: ignores high-degree intermediaries | Compute structural embeddings with Node2Vec, then find similar nodes or feed into downstream ML. @@ -265,7 +265,7 @@ for node_id in similar: ``` - `find_similar_nodes` returns `List[str]` — a list of node IDs, not node objects. Look up full node data via `graph["nodes"]`. + `find_similar_nodes` returns `List[str]`: a list of node IDs, not node objects. Look up full node data via `graph["nodes"]`. @@ -286,7 +286,7 @@ for node_id in similar: ## GraphValidator -Validates graph structure — checks required fields, duplicate IDs, dangling edges, and optionally detects cycles and orphan nodes: +Validates graph structure: checks required fields, duplicate IDs, dangling edges, and optionally detects cycles and orphan nodes: ```python from semantica.kg import GraphValidator @@ -333,8 +333,8 @@ kg: ### Cookbooks -- [Building Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) — fundamentals of KG construction · Beginner -- [Your First Knowledge Graph](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb) — entity extraction to visualization · Beginner -- [Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb) — centrality and community detection · Intermediate -- [Advanced Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb) — PageRank, Louvain, shortest path · Advanced -- [Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb) — temporal logic and graph evolution · Advanced +- [Building Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb): fundamentals of KG construction · Beginner +- [Your First Knowledge Graph](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb): entity extraction to visualization · Beginner +- [Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb): centrality and community detection · Intermediate +- [Advanced Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb): PageRank, Louvain, shortest path · Advanced +- [Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb): temporal logic and graph evolution · Advanced diff --git a/docs/reference/llms.md b/docs/reference/llms.md index ce67b3d9..0101d43e 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -8,7 +8,7 @@ icon: "microchip" - Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoners, and agents - `LiteLLM` routes to 100+ providers with a single class and model-string prefixes -- `HuggingFaceLLM` runs fully on-premise — no API key, no network calls +- `HuggingFaceLLM` runs fully on-premise: no API key, no network calls - Structured output via `generate_with_schema()` for JSON extraction from any provider - Streaming, tool use, and `generate_batch()` for bulk inference @@ -32,21 +32,21 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM ## What You Get -- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application code changes -- **`LiteLLM`** — single class for 100+ providers using model-string routing -- **Local models** — `HuggingFaceLLM` runs fully on-premise, no API key -- **Streaming** — token-by-token output for low-latency UX -- **Custom gateways** — point `OpenAI` at any OpenAI-compatible endpoint via `base_url` +- **Unified `LLMProvider` interface**: swap providers with a one-line change, no application code changes +- **`LiteLLM`**: single class for 100+ providers using model-string routing +- **Local models**: `HuggingFaceLLM` runs fully on-premise, no API key +- **Streaming**: token-by-token output for low-latency UX +- **Custom gateways**: point `OpenAI` at any OpenAI-compatible endpoint via `base_url` ## Choosing a Provider - + Free tier, fastest inference, zero setup friction. Best for development and high-throughput extraction pipelines. | | | | :-- | :-- | - | **Speed** | Very fast — 100+ tok/s | + | **Speed** | Very fast: 100+ tok/s | | **Cost** | Free tier available | | **Context** | 128k | | **Best for** | Development, high-throughput extraction | @@ -64,7 +64,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM Get your free key at [console.groq.com](https://console.groq.com). - + Highest accuracy, best JSON mode and function calling. Use for production pipelines where extraction quality matters. | | | @@ -86,8 +86,8 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM ) ``` - - Fully on-premise — no API key, no data leaves your infrastructure. Required for air-gapped deployments. + + Fully on-premise: no API key, no data leaves your infrastructure. Required for air-gapped deployments. | | | | :-- | :-- | @@ -114,7 +114,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM No API key required. Ensure the Ollama server is running (`ollama serve`) before creating the `LiteLLM` instance. - + Largest context window, best multi-hop reasoning, highest safety bar. Use for complex analysis and long-document extraction. | | | @@ -135,7 +135,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM ) ``` - + Lowest cost per token for high-volume workloads. Strong on coding and structured data extraction. | | | @@ -257,7 +257,7 @@ llm = LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTH # Google Gemini llm = LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY")) -# Ollama (local — no API key) +# Ollama (local: no API key) llm = LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434") # DeepSeek @@ -282,12 +282,12 @@ llm = HuggingFaceLLM( max_new_tokens=512, temperature=0.1, ) -# Bring your own model — full local control, no API key +# Bring your own model: full local control, no API key ``` -## LiteLLM — 100+ Providers +## LiteLLM: 100+ Providers `LiteLLM` is the recommended way to access any provider not directly exported by `semantica.llms`. Use the `provider/model` string format: @@ -317,7 +317,7 @@ response = providers["Anthropic"].generate("Explain GraphRAG in one paragraph.") ## Custom / Enterprise Gateways -Any OpenAI-compatible endpoint — internal routing layers, Qwen proxies, or private LLaMA deployments: +Any OpenAI-compatible endpoint: internal routing layers, Qwen proxies, or private LLaMA deployments: ```python import os diff --git a/docs/reference/mcp_server.md b/docs/reference/mcp_server.md index ff84af0a..53bc7ad9 100644 --- a/docs/reference/mcp_server.md +++ b/docs/reference/mcp_server.md @@ -1,13 +1,13 @@ --- title: "MCP Server" -description: "Model Context Protocol server — expose Semantica's full capability set to Claude Desktop, VS Code, Cursor, and any MCP-aware tool." +description: "Model Context Protocol server: expose Semantica's full capability set to Claude Desktop, VS Code, Cursor, and any MCP-aware tool." icon: "plug" --- **`semantica.mcp_server`** exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) **server over stdio**: - 12 MCP tools exposed: extract entities, query graph, record decisions, run reasoning, export results -- No Python code required after launch — configure once, use from any MCP-aware client +- No Python code required after launch: configure once, use from any MCP-aware client - Compatible with Claude Desktop, Windsurf, Cline, Continue, VS Code, Roo Code, Cursor @@ -32,7 +32,7 @@ python -m semantica.mcp_server ``` - `semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes — all interaction happens through MCP tool calls from a connected AI client. + `semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes: all interaction happens through MCP tool calls from a connected AI client. ## What You Get @@ -42,10 +42,10 @@ python -m semantica.mcp_server Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export graph. - Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info — readable by any MCP client. + Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info: readable by any MCP client. - Runs over stdio — no server, no port, no Docker required. One config block to activate in any MCP client. + Runs over stdio: no server, no port, no Docker required. One config block to activate in any MCP client. Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup. @@ -64,7 +64,7 @@ python -m semantica.mcp_server pip install semantica ``` -The MCP server is included in the base install — no extras required. +The MCP server is included in the base install: no extras required. ## Configuration @@ -153,7 +153,7 @@ The MCP server is included in the base install — no extras required. | Variable | Default | Description | | :-------- | :------- | :----------- | -| `SEMANTICA_KG_PATH` | *(none — in-memory graph)* | Path to a persisted graph file to load on startup | +| `SEMANTICA_KG_PATH` | *(none: in-memory graph)* | Path to a persisted graph file to load on startup | | `SEMANTICA_LOG_LEVEL` | `WARNING` | Log verbosity: `DEBUG`, `INFO`, `WARNING` | ## Tools @@ -162,7 +162,7 @@ The MCP server exposes 12 tools that any connected AI assistant can call: | Tool | Category | Description | | :---- | :-------- | :----------- | -| `extract_entities` | Extraction | NER — find people, places, organisations, concepts | +| `extract_entities` | Extraction | NER: find people, places, organisations, concepts | | `extract_relations` | Extraction | Typed relation and triplet extraction | | `record_decision` | Decision Intelligence | Save a decision with reasoning and outcome | | `query_decisions` | Decision Intelligence | Search recorded decisions by natural language or category | @@ -173,7 +173,7 @@ The MCP server exposes 12 tools that any connected AI assistant can call: | `get_graph_summary` | Graph Operations | Node count, decision count, graph status | | `get_graph_analytics` | Graph Operations | PageRank centrality and community detection | | `run_reasoning` | Reasoning | Forward-chain IF/THEN rules over facts | -| `export_graph` | Reasoning & Export | Serialise the graph (`turtle`/`ttl` — RDF Turtle aliases, `nt`, `xml`, `json-ld`, `json`) | +| `export_graph` | Reasoning & Export | Serialise the graph (`turtle`/`ttl`: RDF Turtle aliases, `nt`, `xml`, `json-ld`, `json`) | ### Knowledge Extraction @@ -447,15 +447,15 @@ The MCP server exposes three readable resources: - **Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path — it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs. + **Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path: it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs. - **Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently — the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works. + **Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently: the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works. - **The server communicates over stdio — don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable. + **The server communicates over stdio: don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable. diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index dc224e3f..52dad4f6 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -1,6 +1,6 @@ --- title: "Normalize Module" -description: "Text cleaning, entity canonicalization, date normalization, number conversion, language detection, and encoding repair — before extraction runs." +description: "Text cleaning, entity canonicalization, date normalization, number conversion, language detection, and encoding repair: before extraction runs." icon: "broom" --- @@ -19,10 +19,10 @@ All normalizers expose convenience functions (one-liners) and stateful class ins Unstructured data is inconsistent by nature. Without normalization, the same real-world entity appears as dozens of variants in your graph: -- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."` — multiple nodes, one company -- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"` — three formats, one date -- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"` — three strings, one number -- `"Hello World"` vs `"Hello\u00a0World"` — a non-breaking space that breaks string matching +- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."`: multiple nodes, one company +- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"`: three formats, one date +- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"`: three strings, one number +- `"Hello World"` vs `"Hello\u00a0World"`: a non-breaking space that breaks string matching Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data. @@ -49,7 +49,7 @@ from semantica.normalize import ( EncodingHandler, ) -# Text — normalize unicode, collapse whitespace, replace smart quotes +# Text: normalize unicode, collapse whitespace, replace smart quotes normalizer = TextNormalizer() clean = normalizer.normalize_text(" Hello,\u00a0 World\u2026 ") # → "Hello, World..." @@ -64,12 +64,12 @@ num_norm = NumberNormalizer() num = num_norm.normalize_number("$1.2B") # → 1200000000.0 -# Language — returns a language code string +# Language: returns a language code string detector = LanguageDetector() lang = detector.detect("Bonjour le monde") # → "fr" -# Encoding — returns (encoding_name, confidence) tuple +# Encoding: returns (encoding_name, confidence) tuple handler = EncodingHandler() encoding, confidence = handler.detect(raw_bytes) utf8_text = handler.convert_to_utf8(raw_bytes) @@ -78,7 +78,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes) ## Recommended Processing Order - + Broken bytes corrupt everything downstream. Always run this before anything else. ```python @@ -91,7 +91,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes) utf8_text = handler.convert_to_utf8(raw_bytes) ``` - + ```python from semantica.normalize import TextNormalizer @@ -104,7 +104,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes) ) ``` - + ```python from semantica.normalize import EntityNormalizer @@ -119,7 +119,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes) # → "Apple Inc." (if the alias_map contains it, else title-cased input) ``` - + ```python from semantica.normalize import DateNormalizer, NumberNormalizer @@ -134,7 +134,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes) # → 1200000000.0 ``` - + ```python from semantica.normalize import LanguageDetector @@ -153,7 +153,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes) ## Convenience Functions -The fastest path — one import, one call: +The fastest path: one import, one call: ```python from semantica.normalize import ( @@ -205,7 +205,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") line_break_type="unix", # "unix" | "windows" ) - # HTML stripping and text cleaning — separate clean_text() method + # HTML stripping and text cleaning: separate clean_text() method cleaned = normalizer.clean_text(html_text, remove_html=True) # Batch normalization @@ -231,9 +231,9 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") | Form | Use When | | :---- | :-------- | - | `NFC` | Default — best for storage and display | - | `NFKC` | Search indexing — normalises ligatures, fullwidth chars, and fractions | - | `NFD` | Stripping diacritics — split é → e + combining accent, then strip accents | + | `NFC` | Default: best for storage and display | + | `NFKC` | Search indexing: normalises ligatures, fullwidth chars, and fractions | + | `NFD` | Stripping diacritics: split é → e + combining accent, then strip accents | | `NFKD` | Same as NFD but also decomposes compatibility characters | **Sub-normalizers for fine-grained control:** @@ -262,7 +262,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") ```python from semantica.normalize import EntityNormalizer - # With alias map — resolves exact matches (lowercase key lookup) + # With alias map: resolves exact matches (lowercase key lookup) normalizer = EntityNormalizer(alias_map={ "apple computer inc.": "Apple Inc.", "ms": "Microsoft", @@ -272,21 +272,21 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") normalizer.normalize_entity("Apple Computer Inc.", entity_type="Organization") # → "Apple Inc." - # Without alias map — only whitespace/format cleanup + # Without alias map: only whitespace/format cleanup normalizer2 = EntityNormalizer() normalizer2.normalize_entity("apple inc", entity_type="Organization") # → "apple inc" (no built-in suffix expansion) - # Person — title-cased + # Person: title-cased normalizer2.normalize_entity("john doe", entity_type="Person") # → "John Doe" ``` **Key behaviours:** - - Alias map uses **lowercase key lookup** — register aliases in lowercase + - Alias map uses **lowercase key lookup**: register aliases in lowercase - `entity_type="Person"` activates `title()` casing on the name - There is no built-in corporate suffix normalization (Inc → Incorporated etc.) - — add these mappings to `alias_map` manually if needed + : add these mappings to `alias_map` manually if needed **Sub-normalizers:** @@ -350,7 +350,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") utc_dt = tz_norm.convert_to_utc(dt_naive) tz_dt = tz_norm.normalize_timezone(dt_naive, target_timezone="America/New_York") - # RelativeDateProcessor — reference_date is passed to process_relative_expression(), + # RelativeDateProcessor: reference_date is passed to process_relative_expression(), # not to the constructor processor = RelativeDateProcessor() ref = datetime(2025, 1, 15) @@ -416,7 +416,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") results = detector.detect_multiple("This might be mixed", top_n=3) # → [("en", 0.85), ...] - # Batch — returns List[str] + # Batch: returns List[str] codes = detector.detect_batch(["Hello", "Hola", "Bonjour", "Ciao"]) # Check specific language @@ -452,17 +452,17 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") ``` **Key behaviours:** - - `detect()` uses `chardet` internally — accuracy improves with longer input + - `detect()` uses `chardet` internally: accuracy improves with longer input - `convert_to_utf8()` auto-detects encoding if `source_encoding` is not provided, then falls back through `latin-1`, `cp1252`, `iso-8859-1` - - Always run `EncodingHandler` first — broken bytes cause cascading failures + - Always run `EncodingHandler` first: broken bytes cause cascading failures in every downstream normalizer ## DataCleaner -Cleans structured record sets — useful before loading into a vector store or graph: +Cleans structured record sets: useful before loading into a vector store or graph: ```python from semantica.normalize import DataCleaner, DataValidator, DuplicateDetector diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index fcd44f76..cee512ac 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -21,7 +21,7 @@ icon: "sitemap" | `OntologyGenerator` | Auto-generate ontologies from KG data (5-stage pipeline) | | `LLMOntologyGenerator` | LLM-powered ontology generation for complex domains | | `SHACLGenerator` | Generate SHACL shapes from an ontology or KG schema | -| `OntologyValidator` | Validate any graph against SHACL shapes — returns `SHACLValidationReport` | +| `OntologyValidator` | Validate any graph against SHACL shapes: returns `SHACLValidationReport` | | `OWLGenerator` | Serialize ontologies to Turtle, RDF/XML, JSON-LD | | `NamespaceManager` | IRI generation, prefix management, and namespace binding | | `OntologyEvaluator` | Coverage, completeness, and granularity quality metrics | @@ -55,7 +55,7 @@ engine.export_owl(ontology, "ontology.ttl", format="turtle") ## OntologyEngine (Unified Facade) -**`OntologyEngine`** orchestrates the full ontology lifecycle — **generation, validation, export, and evaluation**: +**`OntologyEngine`** orchestrates the full ontology lifecycle: **generation, validation, export, and evaluation**: ```python from semantica.ontology import OntologyEngine @@ -109,7 +109,7 @@ ontology = generator.generate_ontology({ Map definitions to OWL constructs: `owl:Class`, `owl:ObjectProperty`, `owl:DatatypeProperty`. - Build taxonomy trees using transitive closure and cycle detection — produces `rdfs:subClassOf` chains. + Build taxonomy trees using transitive closure and cycle detection: produces `rdfs:subClassOf` chains. Serialize the final ontology to Turtle format using `rdflib`. Also available: RDF/XML and JSON-LD. diff --git a/docs/reference/parse.md b/docs/reference/parse.md index 73813750..4c7fced6 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -1,13 +1,13 @@ --- title: "Parse Module" -description: "Document parsing and text extraction — DocumentParser for standard formats and DoclingParser for complex layouts." +description: "Document parsing and text extraction: DocumentParser for standard formats and DoclingParser for complex layouts." icon: "file-lines" --- **`semantica.parse`** extracts **structured text, layout, tables, and metadata** from unstructured documents: -- `DocumentParser` — broad format support (PDF, DOCX, HTML, JSON, CSV, PPTX, XLSX), no extra dependencies -- `DoclingParser` — complex layouts, merged-cell tables, multi-column PDFs, OCR (`pip install docling`) +- `DocumentParser`: broad format support (PDF, DOCX, HTML, JSON, CSV, PPTX, XLSX), no extra dependencies +- `DoclingParser`: complex layouts, merged-cell tables, multi-column PDFs, OCR (`pip install docling`) - Both return a consistent `dict` with `full_text`, `metadata`, `pages`, and `tables` keys - `parse_batch()` processes multiple files in parallel with configurable error handling @@ -60,14 +60,14 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa ## Parser Selection Guide - + Zero extra dependencies. Use for clean PDFs, Word docs, HTML, and structured formats. | | | | :-- | :-- | | **Formats** | PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX | | **Speed** | Fast | - | **Setup** | None — included in base install | + | **Setup** | None: included in base install | | **Best for** | Clean documents, broad format support, production pipelines | ```python @@ -81,7 +81,7 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa print(len(result.get("pages", []))) # Per-page breakdown ``` - + Superior table extraction, OCR, multi-column PDFs. Requires `pip install docling`. | | | @@ -130,7 +130,7 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa print(f"{item['file_path']}: {len(item['result']['full_text'])} chars") for item in results["failed"]: - print(f"FAILED: {item['file_path']} — {item['error']}") + print(f"FAILED: {item['file_path']}: {item['error']}") ``` @@ -143,7 +143,7 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa | Class | Role | | :--- | :--- | -| `DocumentParser` | Auto-detects format — delegates to format-specific parser (PDF, DOCX, HTML, JSON, CSV, ...) | +| `DocumentParser` | Auto-detects format: delegates to format-specific parser (PDF, DOCX, HTML, JSON, CSV, ...) | | `DoclingParser` | Complex layouts, merged-cell tables, multi-column PDFs, and OCR (`pip install docling`) | | `DoclingMetadata` | Document metadata from Docling parsing | | `PDFParser` | PDF text and metadata extraction | @@ -171,7 +171,7 @@ Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX. ## DoclingParser -Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot: +Advanced parser using the Docling backend: handles layouts that `DocumentParser` cannot: ```bash pip install docling @@ -275,7 +275,7 @@ metadata = { ## Integration with FileIngestor -The most common pattern — ingest a directory then parse each source: +The most common pattern: ingest a directory then parse each source: ```python from semantica.ingest import FileIngestor diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index f2b0bfd9..2419ec4e 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -7,10 +7,10 @@ icon: "gear" **`semantica.pipeline`** lets you chain Semantica components into **reproducible, fault-tolerant workflows**: - Per-step failure strategies: `skip`, `retry`, `abort`, or `fallback` -- Parallel workers via `ParallelismManager` — thread or process pool +- Parallel workers via `ParallelismManager`: thread or process pool - `PipelineValidator` catches cycles, missing handlers, and config errors before running - Pre-built templates: `"document_processing"`, `"rag_pipeline"`, `"kg_construction"`, `"ontology_generation"` -- Pipelines are serializable to YAML — save and reload in any environment +- Pipelines are serializable to YAML: save and reload in any environment ## Exported Classes @@ -19,7 +19,7 @@ icon: "gear" | :--- | :--- | | `PipelineBuilder` | DSL for wiring steps: `add_step`, `connect_steps`, `set_parallel`, `build` | | `ExecutionEngine` | Runs a built pipeline: `execute_pipeline(pipeline, data)` → `ExecutionResult` | -| `ExecutionResult` | `{success, output, metadata, metrics, errors}` — full run summary | +| `ExecutionResult` | `{success, output, metadata, metrics, errors}`: full run summary | | `FailureHandler` | Per-step strategy: `skip`, `retry`, `abort`, or `fallback` on failure | | `ParallelismManager` | Thread or process pool for concurrent step execution with configurable workers | | `PipelineValidator` | Catches dependency cycles, missing handlers, and config errors before running | @@ -154,7 +154,7 @@ result = engine.execute_pipeline(pipeline, data="data/") result = engine.execute_pipeline(pipeline, data="data/") ``` - Best for transient API errors and rate limits — waits longer with each retry, giving upstream services time to recover. + Best for transient API errors and rate limits: waits longer with each retry, giving upstream services time to recover. ```python @@ -167,7 +167,7 @@ result = engine.execute_pipeline(pipeline, data="data/") ) ``` - Use when the delay between retries should grow predictably — e.g., waiting for a database lock to release. + Use when the delay between retries should grow predictably: e.g., waiting for a database lock to release. ```python @@ -188,8 +188,8 @@ result = engine.execute_pipeline(pipeline, data="data/") | Strategy | Behaviour | When to Use | | :-------- | :--------- | :----------- | -| `"skip"` | Log failure, continue to next document | Production — one bad doc shouldn't stop 10k | -| `"stop"` | Raise exception immediately | Development — surface errors fast | +| `"skip"` | Log failure, continue to next document | Production: one bad doc shouldn't stop 10k | +| `"stop"` | Raise exception immediately | Development: surface errors fast | | `"retry"` | Retry via `RetryPolicy`, then skip | When failures are likely transient | @@ -227,7 +227,7 @@ result = engine.execute_pipeline(pipeline, data="data/") while t.is_alive(): progress = engine.get_progress(pipeline.name) if progress: - print(f" {progress['completed_steps']}/{progress['total_steps']} steps — {progress['status']}") + print(f" {progress['completed_steps']}/{progress['total_steps']} steps: {progress['status']}") time.sleep(2) ``` @@ -244,7 +244,7 @@ from semantica.pipeline import PipelineBuilder, ExecutionEngine builder = PipelineBuilder() -# Add steps — step_type is a string label, handler is the callable invoked at runtime +# Add steps: step_type is a string label, handler is the callable invoked at runtime builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file) builder.add_step("parse", "document_parse", handler=parser.parse) builder.add_step("normalize", "text_normalize", handler=normalizer.normalize) @@ -291,12 +291,12 @@ result = ExecutionEngine().execute_pipeline(restored, data="data/") ``` - Serialized pipelines capture step names, types, and config — but not handler functions (callables can't be serialized). Re-register handlers on the restored steps before executing. + Serialized pipelines capture step names, types, and config: but not handler functions (callables can't be serialized). Re-register handlers on the restored steps before executing. ## Pre-Built Templates -`PipelineTemplateManager` wires common workflows with the correct step order — no manual wiring required: +`PipelineTemplateManager` wires common workflows with the correct step order: no manual wiring required: ```python from semantica.pipeline import PipelineTemplateManager @@ -320,7 +320,7 @@ The `create_pipeline_from_template(name)` method returns a configured `PipelineB **Ingest → Chunk → Embed → Store Vectors** - RAG pipeline for question answering — builds a vector-indexed store. + RAG pipeline for question answering: builds a vector-indexed store. ```python builder = manager.create_pipeline_from_template("rag_pipeline") @@ -351,7 +351,7 @@ The `create_pipeline_from_template(name)` method returns a configured `PipelineB ## ExecutionEngine -Fine-grained control over pipeline execution — pause, resume, cancel, and inspect live progress: +Fine-grained control over pipeline execution: pause, resume, cancel, and inspect live progress: ```python from semantica.pipeline import ExecutionEngine @@ -394,7 +394,7 @@ validator = PipelineValidator() result = validator.validate_pipeline(pipeline) if result.valid: - print("Pipeline is valid — safe to run") + print("Pipeline is valid: safe to run") else: for error in result.errors: # errors is List[str] print(f"Error: {error}") @@ -403,10 +403,10 @@ else: ``` Checks performed: -- **Dependency cycle detection** — A depends on B, B depends on A -- **Step type validation** — each step type must be registered -- **Connection integrity** — referenced step names must exist -- **Configuration completeness** — required parameters must be present +- **Dependency cycle detection**: A depends on B, B depends on A +- **Step type validation**: each step type must be registered +- **Connection integrity**: referenced step names must exist +- **Configuration completeness**: required parameters must be present ## ParallelismManager @@ -556,7 +556,7 @@ from semantica.pipeline import StepStatus StepStatus.PENDING # Not yet started StepStatus.RUNNING # Currently executing StepStatus.COMPLETED # Finished successfully -StepStatus.FAILED # Error occurred — check step.error +StepStatus.FAILED # Error occurred: check step.error StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy ``` @@ -578,7 +578,7 @@ StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy - **Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order — saving you from common mistakes like deduplicating before normalizing. + **Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index 311bee6d..c4002c1d 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -4,9 +4,9 @@ description: "W3C PROV-O lineage tracking, source attribution, tamper-evident ch icon: "link" --- -`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, chunking, and relationship building: +`semantica.provenance` tracks the full lineage of every fact: from raw ingestion through extraction, chunking, and relationship building: -- W3C PROV-O compliant — suitable for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 audit trails +- W3C PROV-O compliant: suitable for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 audit trails - SHA-256 checksums for tamper detection on every stored `ProvenanceEntry` - `SQLiteStorage` for persistence across restarts; `InMemoryStorage` for development - `ProvenanceManager` provides `track_entity`, `track_relationship`, `track_chunk`, and `get_lineage` @@ -21,8 +21,8 @@ icon: "link" | `ProvenanceEntry` | Single lineage record: `{entity_id, entity_type, activity_id, source_document, confidence, checksum, ...}` | | `SourceReference` | Rich source pointer: `{document, page, section, line, confidence, metadata}` | | `ProvenanceStorage` | Abstract storage interface | -| `InMemoryStorage` | Default backend — fast, not persisted across restarts | -| `SQLiteStorage` | Persistent backend — persists to a local SQLite file | +| `InMemoryStorage` | Default backend: fast, not persisted across restarts | +| `SQLiteStorage` | Persistent backend: persists to a local SQLite file | | `compute_checksum` | Returns SHA-256 fingerprint of a `ProvenanceEntry` | | `verify_checksum` | Detects tampering by comparing stored vs recomputed hash | @@ -30,7 +30,7 @@ icon: "link" - Zero configuration — fast, no disk writes. Use for notebooks, testing, and single-run scripts. + Zero configuration: fast, no disk writes. Use for notebooks, testing, and single-run scripts. ```python from semantica.provenance import ProvenanceManager, compute_checksum, verify_checksum @@ -46,7 +46,7 @@ icon: "link" ) print(entry.checksum) # SHA-256 hex auto-computed - print(verify_checksum(entry)) # True — tamper detection + print(verify_checksum(entry)) # True: tamper detection ``` @@ -62,7 +62,7 @@ icon: "link" # Option 1: explicit storage instance manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) - # Option 2: shorthand — equivalent to above + # Option 2: shorthand: equivalent to above manager = ProvenanceManager(storage_path="provenance.db") entry = manager.track_entity( @@ -72,7 +72,7 @@ icon: "link" confidence=0.98, ) - # Retrieve lineage after restart — entries persist in provenance.db + # Retrieve lineage after restart: entries persist in provenance.db lineage = manager.get_lineage("apple_inc") print(f"{len(lineage)} provenance entries for apple_inc") ``` @@ -92,7 +92,7 @@ icon: "link" ```python ProvenanceManager( storage=None, # ProvenanceStorage instance; defaults to InMemoryStorage - storage_path=None, # str path — creates SQLiteStorage if provided + storage_path=None, # str path: creates SQLiteStorage if provided ) ``` @@ -169,7 +169,7 @@ count = manager.track_chunks_batch(chunks, source_document="doc_1") ### Retrieving Lineage ```python -# get_lineage returns a dict — not a ProvenanceEntry +# get_lineage returns a dict: not a ProvenanceEntry lineage = manager.get_lineage("apple_inc") print(lineage["entity_id"]) # "apple_inc" @@ -233,23 +233,23 @@ from semantica.provenance import ProvenanceEntry # All fields with their types and defaults entry = ProvenanceEntry( - entity_id="entity_001", # str — required - entity_type="entity", # str — required (entity, chunk, relationship, property) - activity_id="ner_extraction", # str — required - agent_id="semantica", # str — default "semantica" - source_document="report.pdf", # str — default "" - source_location="Page 4", # Optional[str] — default None - source_quote="Relevant text...", # Optional[str] — default None - timestamp="2024-01-01T12:00:00", # str — auto-set to utcnow() - first_seen=None, # Optional[str] — ISO timestamp - last_updated=None, # Optional[str] — ISO timestamp - confidence=0.9, # float — default 1.0 - checksum=None, # Optional[str] — set by compute_checksum() - parent_entity_id=None, # Optional[str] — prov:wasDerivedFrom - used_entities=[], # List[str] — prov:used - start_index=None, # Optional[int] — for chunks - end_index=None, # Optional[int] — for chunks - credibility=None, # Optional[float] — source credibility + entity_id="entity_001", # str: required + entity_type="entity", # str: required (entity, chunk, relationship, property) + activity_id="ner_extraction", # str: required + agent_id="semantica", # str: default "semantica" + source_document="report.pdf", # str: default "" + source_location="Page 4", # Optional[str]: default None + source_quote="Relevant text...", # Optional[str]: default None + timestamp="2024-01-01T12:00:00", # str: auto-set to utcnow() + first_seen=None, # Optional[str]: ISO timestamp + last_updated=None, # Optional[str]: ISO timestamp + confidence=0.9, # float: default 1.0 + checksum=None, # Optional[str]: set by compute_checksum() + parent_entity_id=None, # Optional[str]: prov:wasDerivedFrom + used_entities=[], # List[str]: prov:used + start_index=None, # Optional[int]: for chunks + end_index=None, # Optional[int]: for chunks + credibility=None, # Optional[float]: source credibility metadata={}, # Dict[str, Any] version="1.0", # str ) @@ -269,12 +269,12 @@ entry2 = ProvenanceEntry.from_dict(d) from semantica.provenance import SourceReference ref = SourceReference( - document="DOI:10.1038/s41586-021-03371-z", # str — required (DOI, URL, file path) + document="DOI:10.1038/s41586-021-03371-z", # str: required (DOI, URL, file path) page=4, # Optional[int] section="Table S4", # Optional[str] line=None, # Optional[int] timestamp=None, # Optional[datetime] - confidence=0.92, # float — default 1.0 + confidence=0.92, # float: default 1.0 metadata={"credibility": "peer-reviewed"}, # Dict[str, Any] ) @@ -387,7 +387,7 @@ prov_manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) builder = GraphBuilderWithProvenance(provenance_manager=prov_manager) kg = builder.build_single_source(graph_data) -# Retrieve lineage — get_lineage returns a dict +# Retrieve lineage: get_lineage returns a dict lineage = prov_manager.get_lineage("apple_inc") print(lineage["source_documents"]) # list of source document IDs print(lineage["first_seen"]) # ISO timestamp diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 779093fe..29f64fbc 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -7,7 +7,7 @@ icon: "microchip" `semantica.reasoning` derives new knowledge from existing facts using logical rules: - Six reasoning engines: forward chaining, Rete, SPARQL, Datalog, temporal, and LLM-powered GraphReasoner -- Every engine produces explainable inference paths — traceable chains of rules and facts +- Every engine produces explainable inference paths: traceable chains of rules and facts - `DatalogReasoner` guarantees termination via semi-naive fixpoint evaluation - `TemporalReasoningEngine` implements all 13 Allen interval algebra relations - `ExplanationGenerator` produces step-by-step natural-language justifications @@ -18,11 +18,11 @@ icon: "microchip" | Class | Role | | :--- | :--- | | `Reasoner` | Forward-chaining inference: `add_fact`, `add_rule`, `forward_chain`, `backward_chain`, `infer_facts` | -| `GraphReasoner` | LLM-powered reasoning over a KG dict — answers natural language queries via `reason(graph, query)` | +| `GraphReasoner` | LLM-powered reasoning over a KG dict: answers natural language queries via `reason(graph, query)` | | `ReteEngine` | Rete pattern matching: `build_network`, `add_fact`, `match_patterns`, `execute_matches` | | `SPARQLReasoner` | Rule-based SPARQL query expansion via `execute_query`, `expand_query`, `infer_results` | -| `DatalogReasoner` | Recursive Horn clause rules with semi-naive fixpoint — `add_fact`, `add_rule`, `derive_all`, `query` | -| `TemporalReasoningEngine` | All 13 Allen interval algebra relations — `relation(a, b)`, `overlaps`, `contains`, `active_at` | +| `DatalogReasoner` | Recursive Horn clause rules with semi-naive fixpoint: `add_fact`, `add_rule`, `derive_all`, `query` | +| `TemporalReasoningEngine` | All 13 Allen interval algebra relations: `relation(a, b)`, `overlaps`, `contains`, `active_at` | | `ExplanationGenerator` | Step-by-step explanations via `generate_explanation(inference_result)` | | `Rule` | IF/THEN rule: `{rule_id, name, conditions, conclusion, rule_type, confidence, priority}` | | `Fact` | Working-memory fact: `{fact_id, predicate, arguments}` | @@ -33,10 +33,10 @@ icon: "microchip" - IF/THEN rules, forward and backward chaining. **Start here** — covers 90% of use cases. No query language required. + IF/THEN rules, forward and backward chaining. **Start here**: covers 90% of use cases. No query language required. - Natural language queries over a knowledge graph via LLM. No SPARQL or rules — just ask a question. + Natural language queries over a knowledge graph via LLM. No SPARQL or rules: just ask a question. Recursive Horn clause rules with guaranteed termination. Use for complex multi-hop transitive rules. @@ -69,7 +69,7 @@ reasoner.add_fact("Employee(Alice)") # Add an IF-THEN rule using the string form reasoner.add_rule("IF Manager(?x) THEN HasAuthority(?x)") -# Run forward chaining — returns List[InferenceResult] +# Run forward chaining: returns List[InferenceResult] results = reasoner.forward_chain() for r in results: print(r.conclusion) # "HasAuthority(Alice)" @@ -97,7 +97,7 @@ reasoner.add_rule(rule) ## Reasoner (Forward/Backward Chaining) -**`Reasoner`** is the unified entry point for rule-based inference — iterates facts and rules to a **fixpoint**, then optionally proves a specific goal via backward chaining: +**`Reasoner`** is the unified entry point for rule-based inference: iterates facts and rules to a **fixpoint**, then optionally proves a specific goal via backward chaining: ```python from semantica.reasoning import Reasoner, Rule, RuleType, InferenceResult @@ -111,14 +111,14 @@ reasoner.add_fact("Employee(John)") # IF-THEN string form reasoner.add_rule("IF Manager(?x) AND Employee(?x) THEN SeniorStaff(?x)") -# Forward chaining — iterates until fixpoint +# Forward chaining: iterates until fixpoint results = reasoner.forward_chain() for r in results: print(r.conclusion) # e.g. "SeniorStaff(John)" print(r.premises) # list of premise strings matched print(r.confidence) # float -# Backward chaining — prove a specific goal +# Backward chaining: prove a specific goal result = reasoner.backward_chain("SeniorStaff(John)", max_depth=10) if result: print(f"Proven: {result.conclusion}") @@ -149,7 +149,7 @@ conclusions = reasoner.infer_facts( ```python from semantica.reasoning import Rule, Fact, RuleType -# Rule — all fields +# Rule: all fields rule = Rule( rule_id="rule_001", # required: unique identifier name="manager_authority", # required: display name @@ -160,7 +160,7 @@ rule = Rule( priority=0, # higher priority rules run first ) -# Fact — for working with the Rete engine directly +# Fact: for working with the Rete engine directly from semantica.reasoning import Fact fact = Fact( fact_id="f001", # required: unique identifier @@ -173,12 +173,12 @@ fact = Fact( ## GraphReasoner -**`GraphReasoner`** uses an LLM to answer **natural language queries** over a knowledge graph dict — no SPARQL or rule authoring required: +**`GraphReasoner`** uses an LLM to answer **natural language queries** over a knowledge graph dict: no SPARQL or rule authoring required: ```python from semantica.reasoning import GraphReasoner -# Initialize — uses openai by default; override via kwargs +# Initialize: uses openai by default; override via kwargs reasoner = GraphReasoner(provider="openai", model="gpt-4o-mini") kg = { @@ -250,7 +250,7 @@ engine.reset() ## SPARQLReasoner -**`SPARQLReasoner`** extends SPARQL with **inference rule expansion** — add IF-THEN rules and they are automatically woven into queries before execution: +**`SPARQLReasoner`** extends SPARQL with **inference rule expansion**: add IF-THEN rules and they are automatically woven into queries before execution: ```python from semantica.reasoning import SPARQLReasoner @@ -260,7 +260,7 @@ reasoner = SPARQLReasoner() # Add an inference rule (IF-THEN string form) reasoner.add_inference_rule("IF is_a(?x, Manager) THEN has_authority(?x)") -# Execute a query — returns SPARQLQueryResult +# Execute a query: returns SPARQLQueryResult result = reasoner.execute_query(""" PREFIX ex: SELECT ?person ?company WHERE { @@ -296,14 +296,14 @@ SPARQLReasoner( ## DatalogReasoner -Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops: +Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed**: the engine detects fixpoint convergence and stops: ```python from semantica.reasoning import DatalogReasoner, DatalogFact datalog = DatalogReasoner() -# Add base facts — string form is the simplest +# Add base facts: string form is the simplest datalog.add_fact("parent(alice, bob)") datalog.add_fact("parent(bob, charlie)") @@ -314,11 +314,11 @@ datalog.add_fact(DatalogFact(predicate="parent", args=("charlie", "dave"))) datalog.add_rule("ancestor(X, Y) :- parent(X, Y).") datalog.add_rule("ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).") -# Evaluate to fixpoint — returns all derived fact strings +# Evaluate to fixpoint: returns all derived fact strings all_facts = datalog.derive_all() # e.g. ["parent(alice, bob)", "parent(bob, charlie)", ..., "ancestor(alice, bob)", ...] -# Query with variable pattern — variables start with uppercase or ? +# Query with variable pattern: variables start with uppercase or ? results = datalog.query("ancestor(alice, ?Z)") # → [{"Z": "bob"}, {"Z": "charlie"}, {"Z": "dave"}] @@ -331,11 +331,11 @@ datalog.clear() ```python from semantica.reasoning import DatalogFact, DatalogRule -# DatalogFact — ground fact; args must all be constants (lowercase start) +# DatalogFact: ground fact; args must all be constants (lowercase start) fact = DatalogFact(predicate="parent", args=("alice", "bob")) -# DatalogRule — parsed from string; head and body are set by the parser -# Use add_rule("head(X, Y) :- body(X, Z), body2(Z, Y).") — do not construct directly +# DatalogRule: parsed from string; head and body are set by the parser +# Use add_rule("head(X, Y) :- body(X, Z), body2(Z, Y)."): do not construct directly ``` ### DatalogReasoner Methods @@ -345,14 +345,14 @@ fact = DatalogFact(predicate="parent", args=("alice", "bob")) | `add_fact(fact)` | `None` | Add string, dict, or `DatalogFact`; constants must be lowercase-starting | | `add_rule(rule_str)` | `None` | Parse and add a Horn clause string like `"ancestor(X,Y) :- parent(X,Y)."` | | `derive_all()` | `List[str]` | Run semi-naive fixpoint evaluation; returns all facts as strings | -| `query(pattern)` | `List[dict]` | Query derived facts — auto-runs `derive_all()` if needed | +| `query(pattern)` | `List[dict]` | Query derived facts: auto-runs `derive_all()` if needed | | `load_from_graph(graph)` | `int` | Load a ContextGraph's nodes/edges as Datalog facts; returns count added | | `clear()` | `None` | Clear all facts and rules | ## TemporalReasoningEngine -Pure-Python Allen interval algebra — all 13 relations, no LLM calls: +Pure-Python Allen interval algebra: all 13 relations, no LLM calls: ```python from datetime import datetime @@ -363,7 +363,7 @@ engine = TemporalReasoningEngine() ceo_tenure = TemporalInterval(start=datetime(1997, 9, 16), end=datetime(2011, 8, 24)) board_member = TemporalInterval(start=datetime(2000, 1, 1), end=datetime(2012, 6, 1)) -# Compute Allen relation — method is relation(), not get_relation() +# Compute Allen relation: method is relation(), not get_relation() rel = engine.relation(ceo_tenure, board_member) # → IntervalRelation.DURING (ceo_tenure is fully inside board_member) @@ -413,7 +413,7 @@ results = reasoner.forward_chain() # ExplanationGenerator takes no positional args generator = ExplanationGenerator() -# Pass an InferenceResult object — not a dict +# Pass an InferenceResult object: not a dict explanation = generator.generate_explanation(results[0]) print(f"Type: {explanation.explanation_type}") # "inference" @@ -472,7 +472,7 @@ step.confidence # float | `TemporalReasoningEngine` | Time interval relationships | Always | `relation(a, b)` | - For recursive rules (e.g. ancestor, reachability, transitivity), use `DatalogReasoner` — it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner.forward_chain()` has a `max_iterations` cap (default 50) and will silently stop early with deep recursion. + For recursive rules (e.g. ancestor, reachability, transitivity), use `DatalogReasoner`: it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner.forward_chain()` has a `max_iterations` cap (default 50) and will silently stop early with deep recursion. diff --git a/docs/reference/seed.md b/docs/reference/seed.md index 844750b0..a00e8a66 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -1,12 +1,12 @@ --- title: "Seed Module" -description: "Bootstrap Knowledge Graphs from verified, structured sources — taxonomies, reference tables, product catalogs, and domain anchors." +description: "Bootstrap Knowledge Graphs from verified, structured sources: taxonomies, reference tables, product catalogs, and domain anchors." icon: "database" --- **`semantica.seed`** gives your knowledge graph a **reliable, verified starting point**: -- Load verified reference data first — ISO codes, employee rosters, product catalogs, domain taxonomies +- Load verified reference data first: ISO codes, employee rosters, product catalogs, domain taxonomies - `SeedDataManager` merges freshly extracted data onto foundation nodes without creating duplicates - Supports JSON, CSV, and programmatic registration of seed sources - Deterministic test graph generation from structured seed data @@ -80,7 +80,7 @@ icon: "database" for warning in report["warnings"]: print(f"Warning: {warning}") else: - print(f"Validated {report['metrics']['entity_count']} entities — no issues found") + print(f"Validated {report['metrics']['entity_count']} entities: no issues found") ``` @@ -190,7 +190,7 @@ Different strategies for resolving conflicts during `integrate_with_extracted()` - **Seed data wins conflicts** — preserves curated relationships over extracted ones. + **Seed data wins conflicts**: preserves curated relationships over extracted ones. ```python final_kg = manager.integrate_with_extracted( @@ -203,7 +203,7 @@ Different strategies for resolving conflicts during `integrate_with_extracted()` Use when seed data is high-confidence and extraction is exploratory. - **Extracted data wins conflicts** — overwrites seed with fresh information. + **Extracted data wins conflicts**: overwrites seed with fresh information. ```python final_kg = manager.integrate_with_extracted( @@ -216,7 +216,7 @@ Different strategies for resolving conflicts during `integrate_with_extracted()` Use for rapid prototyping when extraction quality is known to be good. - **Intelligent conflict resolution** — merges complementary attributes, deduplicates entities. + **Intelligent conflict resolution**: merges complementary attributes, deduplicates entities. ```python final_kg = manager.integrate_with_extracted( @@ -282,7 +282,7 @@ manager.export_seed_data("output/enriched_kg.json", format="json") ## YAML Configuration -Define sources in YAML for production deployments — no code changes needed to switch environments: +Define sources in YAML for production deployments: no code changes needed to switch environments: ```yaml seed: @@ -318,7 +318,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first ## Tips and Common Pitfalls - **Load seed data before extracted data.** Seed data is your ground truth — normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values. + **Load seed data before extracted data.** Seed data is your ground truth: normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values. @@ -326,7 +326,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first - **Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast — always run it first. + **Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast: always run it first. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 3a1d27ca..b81e093e 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -4,12 +4,12 @@ description: "Named entity recognition, relation extraction, event detection, an icon: "magnifying-glass-chart" --- -`semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica: +`semantica.semantic_extract` extracts structured information from unstructured text: the foundation of every knowledge graph in Semantica: -- `NERExtractor` — named entity recognition with confidence scores and source attribution -- `RelationExtractor` — typed relationship extraction (`founded_by`, `located_in`, and custom types) -- `TripletExtractor` — direct `(subject, predicate, object)` triplet generation for RDF output -- `EventDetector` — event detection with participants, temporal context, and confidence +- `NERExtractor`: named entity recognition with confidence scores and source attribution +- `RelationExtractor`: typed relationship extraction (`founded_by`, `located_in`, and custom types) +- `TripletExtractor`: direct `(subject, predicate, object)` triplet generation for RDF output +- `EventDetector`: event detection with participants, temporal context, and confidence - Three extraction modes on every extractor: `"pattern"` (no API key), `"huggingface"`, `"llm"` @@ -67,7 +67,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") | Class | Role | | :--- | :--- | | `NamedEntityRecognizer` | High-level NER with confidence thresholding and overlap merging | -| `NERExtractor` | Core NER implementation — use directly for simplicity | +| `NERExtractor` | Core NER implementation: use directly for simplicity | | `RelationExtractor` | Typed relationship extraction (`founded_by`, `located_in`, ...) | | `TripletExtractor` | Direct `(subject, predicate, object)` triplet generation for RDF output | | `EventDetector` | Event detection with participants, temporal context, and confidence scores | @@ -79,12 +79,12 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") ## Method Selection Guide - + Zero dependencies, no API key required. Uses spaCy rules and regex to match standard entity types. | | | | :-- | :-- | - | **Setup** | None — works out of the box | + | **Setup** | None: works out of the box | | **Cost** | Free | | **Accuracy** | Good for standard entity types | | **Best for** | Quick prototyping, batch processing, air-gapped systems | @@ -99,7 +99,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") relationships = rel.extract(text, entities=entities) ``` - + Use any pre-trained or fine-tuned transformer model. Free inference, runs locally. | | | @@ -121,14 +121,14 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") entities = ner.extract(text, model="d4data/biomedical-ner-all") ``` - + Highest accuracy for complex schemas and custom entity types. Requires an LLM API key. | | | | :-- | :-- | | **Setup** | `pip install semantica[llm-groq]` + API key | | **Cost** | Depends on provider | - | **Accuracy** | Highest — handles complex types and context | + | **Accuracy** | Highest: handles complex types and context | | **Best for** | Production, custom entity types, complex relation schemas | ```python @@ -148,7 +148,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") ``` - Try methods in priority order — guarantees non-empty results even when the preferred method is unavailable. + Try methods in priority order: guarantees non-empty results even when the preferred method is unavailable. ```python from semantica.semantic_extract import NERExtractor, RelationExtractor @@ -157,7 +157,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") ner = NERExtractor(method=["llm", "pattern"]) rel = RelationExtractor(method=["llm", "pattern"]) - # Always returns results — safe for production pipelines + # Always returns results: safe for production pipelines entities = ner.extract(text) relationships = rel.extract(text, entities=entities) ``` @@ -225,15 +225,15 @@ from semantica.semantic_extract import NERExtractor from semantica.llms import Groq import os -# Pattern-based — fast, no API key, good for standard entity types +# Pattern-based: fast, no API key, good for standard entity types ner = NERExtractor(method="pattern") entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino.") -# HuggingFace-based — custom models, no API cost +# HuggingFace-based: custom models, no API cost ner = NERExtractor(method="huggingface") entities = ner.extract(text, model="dslim/bert-base-NER", device="cpu") -# LLM-based — best accuracy, handles complex schemas and custom types +# LLM-based: best accuracy, handles complex schemas and custom types llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) entities = ner.extract(text) @@ -285,11 +285,11 @@ Output format: Available methods: -- `"pattern"` — rule-based pattern matching -- `"dependency"` — spaCy dependency parsing -- `"cooccurrence"` — proximity-based co-occurrence -- `"huggingface"` — custom models -- `"llm"` — highest accuracy, requires API key +- `"pattern"`: rule-based pattern matching +- `"dependency"`: spaCy dependency parsing +- `"cooccurrence"`: proximity-based co-occurrence +- `"huggingface"`: custom models +- `"llm"`: highest accuracy, requires API key ## TripletExtractor @@ -327,11 +327,11 @@ for event in events: Output fields per event: -- `type` — event category (e.g. `"founding"`, `"acquisition"`) -- `participants` — list of entities with roles -- `temporal` — date or time reference -- `location` — location entity (when present) -- `confidence` — extraction confidence score +- `type`: event category (e.g. `"founding"`, `"acquisition"`) +- `participants`: list of entities with roles +- `temporal`: date or time reference +- `location`: location entity (when present) +- `confidence`: extraction confidence score ## CoreferenceResolver @@ -384,7 +384,7 @@ results = ner.extract(documents) # Entities include document_id in metadata ## Using All Extractors Together -The standard extraction pipeline — entities → relationships → triplets: +The standard extraction pipeline: entities → relationships → triplets: ```python from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor diff --git a/docs/reference/split.md b/docs/reference/split.md index 6b380cec..29deeaf2 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -27,9 +27,9 @@ Semantica's chunking methods are designed to avoid these failure modes. | Class | Role | | :--- | :--- | -| `TextSplitter` | Unified entry point — swap `method=` without changing downstream code | +| `TextSplitter` | Unified entry point: swap `method=` without changing downstream code | | `Chunk` | `{text, start_index, end_index, metadata, id}` | -| `SemanticChunker` | Embedding-based topic-shift detection — splits only when content actually changes | +| `SemanticChunker` | Embedding-based topic-shift detection: splits only when content actually changes | | `StructuralChunker` | Heading/section-based splits using structural text analysis | | `EntityAwareChunker` | Prevents named entity mentions from being split across chunk boundaries | | `RelationAwareChunker` | Keeps subject-predicate-object triplets intact within a single chunk | @@ -39,7 +39,7 @@ Semantica's chunking methods are designed to avoid these failure modes. | Method | Best for | | :--- | :--- | -| `recursive` | General text — splits on paragraphs, sentences, words in order | +| `recursive` | General text: splits on paragraphs, sentences, words in order | | `sentence` | Conversational text, QA | | `paragraph` | Long-form text where paragraph integrity matters | | `token` | LLM context window enforcement | @@ -53,13 +53,13 @@ Semantica's chunking methods are designed to avoid these failure modes. - Unified interface for 11 chunking strategies — swap methods without changing downstream code. + Unified interface for 11 chunking strategies: swap methods without changing downstream code. - Embedding-based topic shift detection — splits only when the topic actually changes. + Embedding-based topic shift detection: splits only when the topic actually changes. - Entity spans never cross chunk boundaries — guaranteed by boundary adjustment. + Entity spans never cross chunk boundaries: guaranteed by boundary adjustment. Subject–predicate–object triplets kept within a single chunk for KG pipelines. @@ -96,7 +96,7 @@ Semantica's chunking methods are designed to avoid these failure modes. ```python # split_documents() accepts any object with a .text attribute, - # or a plain string — no specific document class required. + # or a plain string: no specific document class required. class Doc: def __init__(self, text, metadata=None): self.text = text @@ -127,7 +127,7 @@ Semantica's chunking methods are designed to avoid these failure modes. | Method | How It Splits | Best For | | :------ | :------------- | :-------- | | `recursive` | Paragraph → sentence → word (cascading fallback) | General-purpose default | -| `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG — topic coherence matters | +| `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG: topic coherence matters | | `entity_aware` | Adjusts boundaries so entity spans are never cut | NER pipelines | | `relation_aware` | Keeps subject–predicate–object triplets within one chunk | KG construction | | `sentence` | Sentence boundary detection (regex, NLTK, spaCy) | Short documents, Q&A | @@ -158,7 +158,7 @@ Use this decision tree before picking a method: from semantica.split import TextSplitter splitter = TextSplitter( - method="semantic_transformer", # chunking strategy — see Splitting Methods table + method="semantic_transformer", # chunking strategy: see Splitting Methods table chunk_size=1000, # target size in characters chunk_overlap=200, # character overlap between adjacent chunks similarity_threshold=0.7, # cosine similarity cutoff (semantic_transformer only) @@ -171,19 +171,19 @@ splitter = TextSplitter( | Parameter | Type | Default | Description | | :--------- | :---- | :------- | :----------- | | `method` | `str \| list[str]` | `"recursive"` | Chunking strategy, or list of methods as fallback chain | -| `chunk_size` | `int` | `1000` | Target size in **characters** (not tokens — if you were using token-based sizing before, multiply by ~4 to approximate the same boundary) | +| `chunk_size` | `int` | `1000` | Target size in **characters** (not tokens: if you were using token-based sizing before, multiply by ~4 to approximate the same boundary) | | `chunk_overlap` | `int` | `200` | Character overlap between adjacent chunks | -| `similarity_threshold` | `float` | `0.7` | Cosine similarity cutoff for `semantic_transformer` — lower = more splits | +| `similarity_threshold` | `float` | `0.7` | Cosine similarity cutoff for `semantic_transformer`: lower = more splits | | `model` | `str` | `"all-MiniLM-L6-v2"` | Sentence-transformers model name for `semantic_transformer` | | `ner_method` | `str` | `"ml"` | NER method for `entity_aware`: `"pattern"` \| `"regex"` \| `"ml"` \| `"huggingface"` \| `"llm"` | | `relation_method` | `str` | `"ml"` | Relation extraction method for `relation_aware`: `"ml"` \| `"llm"` \| `"huggingface"` | -| `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method — unrecognised names fall back to `cl100k_base` | +| `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method: unrecognised names fall back to `cl100k_base` | ## Splitting Method Details - Tries paragraph breaks first, then sentence boundaries, then word boundaries — falling back only when the chunk exceeds `chunk_size`: + Tries paragraph breaks first, then sentence boundaries, then word boundaries: falling back only when the chunk exceeds `chunk_size`: ```python splitter = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200) @@ -192,7 +192,7 @@ splitter = TextSplitter( **Key behaviours:** - Preserves paragraph and sentence structure wherever possible - - Falls back gracefully — never produces chunks larger than `chunk_size` + - Falls back gracefully: never produces chunks larger than `chunk_size` - Overlap ensures context continuity across chunk boundaries - Good starting point when you're unsure which method to use @@ -207,14 +207,14 @@ splitter = TextSplitter( model="all-MiniLM-L6-v2", # any sentence-transformers model name similarity_threshold=0.7, # 0.6 = more splits, 0.8 = fewer splits chunk_size=800, - chunk_overlap=0, # not needed — chunks are already coherent + chunk_overlap=0, # not needed: chunks are already coherent ) chunks = splitter.split(text) ``` **Key behaviours:** - - Requires the `sentence-transformers` package — uses `all-MiniLM-L6-v2` by default, configurable via `model=` - - Produces variable-length chunks — some topics are short, others long + - Requires the `sentence-transformers` package: uses `all-MiniLM-L6-v2` by default, configurable via `model=` + - Produces variable-length chunks: some topics are short, others long - Falls back to sentence splitting if `sentence-transformers` is not installed - Slower than `recursive` due to embedding computation; cache embeddings for repeated splits @@ -241,13 +241,13 @@ splitter = TextSplitter( ``` **Key behaviours:** - - NER is run internally — entity extraction happens automatically inside the splitter + - NER is run internally: entity extraction happens automatically inside the splitter - Entity objects are available in `chunk.metadata["entities"]` for each chunk - - Chunk sizes vary slightly from `chunk_size` — boundary adjustments are ≤ one sentence + - Chunk sizes vary slightly from `chunk_size`: boundary adjustments are ≤ one sentence - Works with all entity types: PERSON, ORGANIZATION, LOCATION, DATE, custom types - Keeps subject–predicate–object triplets within the same chunk — critical for KG pipelines: + Keeps subject–predicate–object triplets within the same chunk: critical for KG pipelines: ```python from semantica.split import TextSplitter @@ -268,9 +268,9 @@ splitter = TextSplitter( ``` **Key behaviours:** - - Relation extraction is run internally — no pre-computed entities or triplets needed + - Relation extraction is run internally: no pre-computed entities or triplets needed - Relation objects are available in `chunk.metadata["relationships"]` for each chunk - - Implies entity-aware behaviour — both entities in a triplet are kept whole too + - Implies entity-aware behaviour: both entities in a triplet are kept whole too - Best used as the split step in a `Parse → Split → Extract → Build KG` pipeline @@ -288,7 +288,7 @@ splitter = TextSplitter( ``` **Key behaviours:** - - Operates on plain text — no structural document format required + - Operates on plain text: no structural document format required - Respects heading hierarchy (lines starting with `#` or all-caps headings) and paragraph breaks - Uses `max_chunk_size=` parameter instead of the standard `chunk_size=` for maximum size control - Falls back to `recursive` if `StructuralChunker` is unavailable @@ -306,7 +306,7 @@ class Chunk: text: str # the chunk's text content start_index: int # character offset of start in source text end_index: int # character offset of end in source text - metadata: Dict[str, Any] # method-specific fields — see table below + metadata: Dict[str, Any] # method-specific fields: see table below id: Optional[str] = None # optional chunk identifier ``` @@ -322,7 +322,7 @@ Metadata keys vary by method. Only keys that are actually set by the implementat | `sentence_count` | `int` | `sentence`, `semantic_transformer`, spaCy path | Number of sentences in this chunk | | `paragraph_count` | `int` | `paragraph` | Number of paragraphs in this chunk | | `word_count` | `int` | `word` | Number of words in this chunk | -| `token_count` | `int` | `token`; `sentence`/`semantic_transformer` when spaCy is available | Token count — not always present | +| `token_count` | `int` | `token`; `sentence`/`semantic_transformer` when spaCy is available | Token count: not always present | | `entity_count` | `int` | `entity_aware` | Number of entities whose boundaries fall in this chunk | | `entities` | `list` | `entity_aware` | Entity objects whose boundaries fall in this chunk | | `relation_count` | `int` | `relation_aware` | Number of relation triplets in this chunk | @@ -348,7 +348,7 @@ If `tiktoken` is not installed, the `token` method falls back to splitting by wh ## Pipeline Integration -`TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern — parse a file, split the text, then extract entities from each chunk: +`TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern: parse a file, split the text, then extract entities from each chunk: ```python from semantica.parse import DocumentParser @@ -376,7 +376,7 @@ For the full pipeline orchestration API, see the [Pipeline reference](pipeline). ## Tips and Common Pitfalls - **`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum — for `chunk_size=1000`, set `chunk_overlap=100` to `200`. + **`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`. @@ -384,12 +384,12 @@ For the full pipeline orchestration API, see the [Pipeline reference](pipeline). - **Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting — use `recursive` instead. + **Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead. - Parse documents before chunking — produces sections and metadata. + Parse documents before chunking: produces sections and metadata. Embed chunks for vector search and semantic chunking. diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index 937613d9..02cb3d46 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -1,6 +1,6 @@ --- title: "Triplet Store Module" -description: "RDF triple storage with SPARQL queries and bulk loading — Blazegraph, Apache Jena, and RDF4J." +description: "RDF triple storage with SPARQL queries and bulk loading: Blazegraph, Apache Jena, and RDF4J." icon: "table" --- @@ -13,15 +13,15 @@ icon: "table" | `TripletStore` | Unified interface: `add_triplet`, `add_triplets`, `get_triplets`, `delete_triplet`, `execute_query` | | `QueryEngine` | **SPARQL 1.1** execution with query optimization and result caching | | `BulkLoader` | High-volume RDF loading with batching, retries, and progress tracking | -| `BlazegraphStore` | Blazegraph REST API — SPARQL 1.1 Update, namespace management | -| `JenaStore` | Apache Jena — rdflib-backed, SPARQL read support via remote endpoint | -| `RDF4JStore` | Eclipse RDF4J — REST API, transaction support | +| `BlazegraphStore` | Blazegraph REST API: SPARQL 1.1 Update, namespace management | +| `JenaStore` | Apache Jena: rdflib-backed, SPARQL read support via remote endpoint | +| `RDF4JStore` | Eclipse RDF4J: REST API, transaction support | ## What You Get - Unified interface across Blazegraph, Apache Jena, and RDF4J — swap backends with one parameter. + Unified interface across Blazegraph, Apache Jena, and RDF4J: swap backends with one parameter. Full SPARQL SELECT, ASK, CONSTRUCT, and UPDATE query support via `execute_query()`. @@ -61,7 +61,7 @@ t = Triplet( ) store.add_triplet(t) -# Query with SPARQL — returns a QueryResult with a .bindings list +# Query with SPARQL: returns a QueryResult with a .bindings list result = store.execute_query(""" PREFIX ex: SELECT ?person ?company WHERE { @@ -105,7 +105,7 @@ for row in result.bindings: ```python - # execute_query returns a QueryResult — iterate result.bindings + # execute_query returns a QueryResult: iterate result.bindings result = store.execute_query(""" PREFIX ex: SELECT ?person ?company WHERE { @@ -224,7 +224,7 @@ t = Triplet( | `get_triplets(subject, predicate, object)` | `List[Triplet]` | Retrieve triplets matching subject/predicate/object filters | | `delete_triplet(triplet)` | `dict` | Delete a `Triplet` from the store | | `update_triplet(old_triplet, new_triplet)` | `dict` | Atomic delete + add | -| `execute_query(query, parameters, graph, graphs)` | `QueryResult` | Execute a SPARQL query — returns `QueryResult` with `.bindings`, `.variables`, `.execution_time` | +| `execute_query(query, parameters, graph, graphs)` | `QueryResult` | Execute a SPARQL query: returns `QueryResult` with `.bindings`, `.variables`, `.execution_time` | | `store(knowledge_graph, ontology)` | `dict` | Convert a KG + ontology dict to RDF triples and bulk-load them | | `add_skos_concept(concept_uri, scheme_uri, pref_label, ...)` | `dict` | Add a SKOS concept with optional alt labels, broader/narrower/related | | `get_skos_concepts(scheme_uri)` | `List[dict]` | Retrieve all SKOS concepts, optionally filtered by scheme URI | @@ -233,14 +233,14 @@ t = Triplet( ## SPARQL Queries -`execute_query()` is the single entry point for all SPARQL operations. It returns a `QueryResult` — access results via `.bindings`: +`execute_query()` is the single entry point for all SPARQL operations. It returns a `QueryResult`: access results via `.bindings`: ```python from semantica.triplet_store import TripletStore store = TripletStore(backend="blazegraph", endpoint="http://localhost:9999/blazegraph/sparql") -# SELECT — iterate result.bindings +# SELECT: iterate result.bindings result = store.execute_query(""" PREFIX ex: SELECT ?person ?company WHERE { @@ -252,7 +252,7 @@ for row in result.bindings: print(row.get("person", {}).get("value")) print(row.get("company", {}).get("value")) -# ASK, CONSTRUCT, UPDATE — same method, different SPARQL form +# ASK, CONSTRUCT, UPDATE: same method, different SPARQL form result = store.execute_query(""" PREFIX ex: ASK { ex:apple_inc ex:founded_by ex:steve_jobs . } @@ -304,7 +304,7 @@ while True: Blazegraph and RDF4J support named graphs. Scope `execute_query()` to a named graph with the `graph=` parameter: ```python -# Add a triplet — named graph stored in metadata or backend-specific API +# Add a triplet: named graph stored in metadata or backend-specific API from semantica.semantic_extract.types import Triplet t = Triplet( @@ -471,7 +471,7 @@ for row in result.bindings: ## Tips and Common Pitfalls - **Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory — no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`. + **Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`. @@ -479,7 +479,7 @@ for row in result.bindings: - **`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object — not `subject=`, `predicate=`, `obj=` to `add_triplet`. + **`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object: not `subject=`, `predicate=`, `obj=` to `add_triplet`. diff --git a/docs/reference/utils.md b/docs/reference/utils.md index b0b871ef..60402e99 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -11,19 +11,19 @@ icon: "wrench" - Progress tracking: `ProgressTracker` class and `track_progress()` iterable wrapper with ETA - Typed exceptions: `SemanticaError`, `ValidationError`, `ProcessingError`, `ConfigurationError`, `QualityError` -Most users won't call utils directly — it's the **shared foundation** for all modules. +Most users won't call utils directly: it's the **shared foundation** for all modules. ## Exported Classes | Name | Type | Role | | :--- | :--- | :--- | -| `setup_logging` | function | Configure the `semantica` root logger — accepts `level`, `file`, `console`, `rotation` kwargs | +| `setup_logging` | function | Configure the `semantica` root logger: accepts `level`, `file`, `console`, `rotation` kwargs | | `get_logger` | function | Get a named logger instance (`semantica.`) | -| `log_execution_time` | decorator | Wraps a function — logs name, execution time, and success/failure | +| `log_execution_time` | decorator | Wraps a function: logs name, execution time, and success/failure | | `log_performance` | function | Log pre-collected performance metrics: `log_performance(func_name, execution_time, **metrics)` | -| `validate_entity` | function | Validate entity dict — returns `(bool, Optional[str])`; does not raise | -| `validate_config` | function | Validate config dict — returns `(bool, Optional[str])`; does not raise | +| `validate_entity` | function | Validate entity dict: returns `(bool, Optional[str])`; does not raise | +| `validate_config` | function | Validate config dict: returns `(bool, Optional[str])`; does not raise | | `ProgressTracker` | class | Class-based progress tracker with ETA and step callbacks | | `track_progress` | function | Wrap any iterable with a live progress bar | | `clean_text` | function | Normalize whitespace and strip zero-width control characters | @@ -44,16 +44,16 @@ Most users won't call utils directly — it's the **shared foundation** for all `validate_entity` and `validate_config` with a typed `ValidationError` carrying field and value context. - `track_progress` wraps any iterable — auto-detects console vs Jupyter for the right renderer. + `track_progress` wraps any iterable: auto-detects console vs Jupyter for the right renderer. `clean_text`, `hash_data`, `safe_filename`, and nested dict utilities used throughout the framework. - `SemanticaError` → `ValidationError`, `ProcessingError` — typed exceptions for targeted recovery. + `SemanticaError` → `ValidationError`, `ProcessingError`: typed exceptions for targeted recovery. - `read_json_file` raises `FileNotFoundError` or `json.JSONDecodeError` on failure — no boilerplate try/except around JSON I/O. + `read_json_file` raises `FileNotFoundError` or `json.JSONDecodeError` on failure: no boilerplate try/except around JSON I/O. @@ -113,15 +113,15 @@ if not is_valid: ```python from semantica.utils import track_progress -# Wraps any iterable — auto-detects console vs Jupyter +# Wraps any iterable: auto-detects console vs Jupyter for item in track_progress(items, desc="Processing documents"): process(item) ``` Supports: -- **Console** — tqdm progress bar with ETA -- **Jupyter** — notebook-compatible widget (auto-detected) -- **File** — write progress to a log file +- **Console**: tqdm progress bar with ETA +- **Jupyter**: notebook-compatible widget (auto-detected) +- **File**: write progress to a log file ## Helper Functions @@ -140,7 +140,7 @@ fname = safe_filename("My File?.txt") # -> "My_File.txt" ## Nested Dict Utilities -Helper functions for deep configuration access — used extensively inside `Config` and `ConfigManager`: +Helper functions for deep configuration access: used extensively inside `Config` and `ConfigManager`: ```python from semantica.utils import get_nested_value, set_nested_value, merge_dicts @@ -150,14 +150,14 @@ config = { "llm": {"provider": "groq", "model": "llama-3.3-70b-versatile"}, } -# Dot-notation read — returns default if key path is absent +# Dot-notation read: returns default if key path is absent batch = get_nested_value(config, "processing.batch_size", default=16) # -> 32 # Dot-notation write set_nested_value(config, "processing.batch_size", 64) -# Deep merge — nested keys are merged recursively (deep=True by default) +# Deep merge: nested keys are merged recursively (deep=True by default) base = {"a": {"x": 1, "y": 2}, "b": 3} overrides = {"a": {"y": 99, "z": 4}, "c": 5} merged = merge_dicts(base, overrides) @@ -187,7 +187,7 @@ except SemanticaError as e: | Exception | When Raised | Key Attributes | | :--------- | :----------- | :-------------- | -| `SemanticaError` | Base class — all framework errors inherit from this | `.message`, `.context`, `.error_code` | +| `SemanticaError` | Base class: all framework errors inherit from this | `.message`, `.context`, `.error_code` | | `ValidationError` | Input data failed schema or type validation | `.field`, `.value`, `.constraint` | | `ProcessingError` | Failure during extraction, graph build, or pipeline step | `.stage`, `.input_data`, `.output_data` | | `ConfigurationError` | Configuration key missing or wrong type | `.config_key`, `.config_value`, `.expected_type` | @@ -201,7 +201,7 @@ except SemanticaError as e: ```python from semantica.utils import read_json_file -# Read and parse a JSON file — raises FileNotFoundError or json.JSONDecodeError on failure +# Read and parse a JSON file: raises FileNotFoundError or json.JSONDecodeError on failure config = read_json_file("config.json") ``` @@ -212,11 +212,11 @@ config = read_json_file("config.json") - **`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected — it is not a decorator. + **`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected: it is not a decorator. - **`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string — suitable as a cache key or idempotency token in pipeline steps. + **`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string: suitable as a cache key or idempotency token in pipeline steps. @@ -224,7 +224,7 @@ config = read_json_file("config.json") - **`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment — the same call works in both. + **`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment: the same call works in both. diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 004ed114..69b052d8 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -6,7 +6,7 @@ icon: "database" `semantica.vector_store` provides a unified API for storing and searching vector embeddings across all major backends: -- Swap backends with a one-line change — no application code changes needed +- Swap backends with a one-line change: no application code changes needed - `HybridSearch` fuses dense vector similarity with metadata filtering via RRF or weighted average - `NamespaceManager` for multi-tenant structural isolation - `FAISSStore` with flat, ivf, hnsw, and pq index types @@ -20,13 +20,13 @@ icon: "database" | `VectorStore` | Unified interface: `store_vectors`, `search_vectors`, `update_vectors`, `delete_vectors` | | `HybridSearch` | Fuses dense vector similarity with metadata filtering via RRF or weighted average | | `MetadataFilter` | Chainable filter builder: `.eq("type", "person").gt("year", 2020).in_list("tag", [...])` | -| `NamespaceManager` | Multi-tenant isolation — separate index namespaces per project or user | -| `FAISSStore` | Local disk or in-memory — flat, ivf, hnsw, and pq index types | +| `NamespaceManager` | Multi-tenant isolation: separate index namespaces per project or user | +| `FAISSStore` | Local disk or in-memory: flat, ivf, hnsw, and pq index types | | `WeaviateStore` | Cloud or self-hosted, schema-aware | | `QdrantStore` | Cloud or self-hosted with payload-based filtering | | `PineconeStore` | Managed cloud vector database with serverless and pod modes | | `MilvusStore` | Scalable self-hosted vector database | -| `PgVectorStore` | PostgreSQL with `pgvector` extension — no extra infrastructure | +| `PgVectorStore` | PostgreSQL with `pgvector` extension: no extra infrastructure | | `MetadataStore` | Standalone metadata indexing and querying | | `SearchRanker` | RRF and weighted-average result fusion | @@ -35,7 +35,7 @@ icon: "database" - Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector - - One-line backend swap — no application code changes + - One-line backend swap: no application code changes - `add_documents()` auto-embeds; `store_vectors()` for pre-computed embeddings @@ -73,7 +73,7 @@ icon: "database" ```python from semantica.vector_store import VectorStore -# In-memory (development / testing — no persistence) +# In-memory (development / testing: no persistence) store = VectorStore(backend="inmemory", dimension=384) # FAISS (local, persists to disk via save/load) @@ -88,7 +88,7 @@ ids = store.add_documents( # Search by text query (auto-embedded) results = store.search("technology company founders", limit=5) for r in results: - print(f"{r['id']} — score: {r['score']:.3f}") + print(f"{r['id']}: score: {r['score']:.3f}") ``` ## Quick Start @@ -129,7 +129,7 @@ for r in results: results = store.search_vectors(query_vector, k=10) for r in results: - print(f"{r['id']} — score: {r['score']:.3f}") + print(f"{r['id']}: score: {r['score']:.3f}") ``` @@ -138,7 +138,7 @@ for r in results: mf = MetadataFilter().eq("category", "research").gt("year", 2022) - # Pass vector_store to the constructor — search() resolves vectors automatically + # Pass vector_store to the constructor: search() resolves vectors automatically search = HybridSearch(vector_store=store) results = search.search(query=query_vector, k=10, metadata_filter=mf) ``` @@ -151,10 +151,10 @@ for r in results: ```python -# In-memory — no persistence, for development and testing +# In-memory: no persistence, for development and testing store = VectorStore(backend="inmemory", dimension=384) -# FAISS — local disk persistence via save() / load() +# FAISS: local disk persistence via save() / load() store = VectorStore(backend="faiss", dimension=384) store.save("./my_store") # save to directory store.load("./my_store") # restore from directory @@ -228,7 +228,7 @@ store = VectorStore( ) ``` -`connection_string` is required — the store raises `ValueError` at construction if it is absent. +`connection_string` is required: the store raises `ValueError` at construction if it is absent. @@ -269,7 +269,7 @@ store = VectorStore( ```python from semantica.vector_store import HybridSearch, MetadataFilter -# With vector_store — search() pulls vectors from the store automatically +# With vector_store: search() pulls vectors from the store automatically search = HybridSearch(vector_store=store) mf = MetadataFilter().eq("category", "research").gt("year", 2022) @@ -280,7 +280,7 @@ results = search.search( ) for r in results: - print(f"{r['id']} — score: {r['score']:.3f} metadata: {r['metadata']}") + print(f"{r['id']}: score: {r['score']:.3f} metadata: {r['metadata']}") ``` Without a `vector_store`, pass vectors explicitly: @@ -309,7 +309,7 @@ fused = search.multi_source_search(query_vector, sources, k=10) ## Metadata Filtering -`MetadataFilter` supports chained conditions — all conditions are ANDed: +`MetadataFilter` supports chained conditions: all conditions are ANDed: ```python from semantica.vector_store import MetadataFilter @@ -320,7 +320,7 @@ mf = MetadataFilter().gt("year", 2022).lte("year", 2024) # range mf = MetadataFilter().in_list("tag", ["ai", "ml"]) # set membership mf = MetadataFilter().contains("title", "neural") # substring / list contains -# Multiple conditions — all must match (AND) +# Multiple conditions: all must match (AND) mf = ( MetadataFilter() .eq("category", "research") @@ -349,19 +349,19 @@ mf = ( ```python from semantica.vector_store import SearchRanker -# Reciprocal Rank Fusion — robust to score scale differences +# Reciprocal Rank Fusion: robust to score scale differences ranker = SearchRanker(strategy="reciprocal_rank_fusion") fused = ranker.rank([results_list_1, results_list_2]) -# Weighted average — requires normalised scores on the same scale +# Weighted average: requires normalised scores on the same scale ranker = SearchRanker(strategy="weighted_average") fused = ranker.rank([results_list_1, results_list_2], weights=[0.7, 0.3]) ``` | Fusion strategy | Description | | :--------------- | :----------- | -| `reciprocal_rank_fusion` | Rank-based combination via RRF — robust to score scale differences (default) | -| `weighted_average` | Weighted sum of scores — pass `weights=[...]` to `rank()` | +| `reciprocal_rank_fusion` | Rank-based combination via RRF: robust to score scale differences (default) | +| `weighted_average` | Weighted sum of scores: pass `weights=[...]` to `rank()` | ## Namespace Isolation @@ -397,7 +397,7 @@ ns_manager.delete_namespace("tenant_a") ## Batch Operations ```python -# Batch add text documents — parallel embedding with configurable workers +# Batch add text documents: parallel embedding with configurable workers ids = store.add_documents( documents=large_doc_list, metadata=large_meta_list, @@ -424,7 +424,7 @@ store.update_vectors( store = VectorStore(backend="faiss", dimension=384) store.add_documents(documents=docs, metadata=meta) -# Save to a directory — creates index.bin and store_data.pkl +# Save to a directory: creates index.bin and store_data.pkl store.save("./vector_store_backup") # Restore in a new process @@ -449,7 +449,7 @@ meta_store = MetadataStore() meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"}) meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"}) -# Query — returns List[str] of matching vector IDs +# Query: returns List[str] of matching vector IDs ids = meta_store.query_metadata({"category": "research", "year": 2024}) # OR query @@ -476,16 +476,16 @@ from semantica.vector_store import FAISSStore store = FAISSStore(dimension=384) -# flat — brute-force exact search +# flat: brute-force exact search store.create_index(index_type="flat", metric="L2") -# ivf — inverted file index +# ivf: inverted file index store.create_index(index_type="ivf", metric="L2", nlist=100) -# hnsw — hierarchical navigable small world graph +# hnsw: hierarchical navigable small world graph store.create_index(index_type="hnsw", metric="L2", M=32) -# pq — product quantization for memory efficiency +# pq: product quantization for memory efficiency store.create_index(index_type="pq", metric="L2", m=8) ``` @@ -577,11 +577,11 @@ store.create_index(index_type="pq", metric="L2", m=8) ## Tips and Common Pitfalls - **Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size — `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time. + **Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time. - **FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"` — not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`. + **FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`. @@ -589,7 +589,7 @@ store.create_index(index_type="pq", metric="L2", m=8) - **Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically — you only need to pass the query and filter. + **Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter. diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index 35d59320..bcf32608 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -4,12 +4,12 @@ description: "Interactive and static knowledge graph, ontology, embedding, and t icon: "chart-bar" --- -**`semantica.visualization`** renders knowledge graphs, ontologies, embedding spaces, and temporal data as **interactive HTML or static images** — without launching the full Explorer server: +**`semantica.visualization`** renders knowledge graphs, ontologies, embedding spaces, and temporal data as **interactive HTML or static images**: without launching the full Explorer server: -- `KGVisualizer` — interactive network with force, hierarchical, and circular layouts -- `EmbeddingVisualizer` — 2D/3D UMAP or t-SNE projections with cluster labels -- `TemporalVisualizer` — timeline views and graph evolution across snapshots -- `AnalyticsVisualizer` — centrality scores, community structure, degree distribution charts +- `KGVisualizer`: interactive network with force, hierarchical, and circular layouts +- `EmbeddingVisualizer`: 2D/3D UMAP or t-SNE projections with cluster labels +- `TemporalVisualizer`: timeline views and graph evolution across snapshots +- `AnalyticsVisualizer`: centrality scores, community structure, degree distribution charts Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` or `graphviz`. @@ -34,7 +34,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o viz = KGVisualizer(layout="force", color_scheme="default") - # Interactive — opens in browser, supports hover and click + # Interactive: opens in browser, supports hover and click viz.visualize_network(graph, output="interactive") ``` @@ -52,10 +52,10 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o ```python - # Static PNG — for reports and embedding in documents + # Static PNG: for reports and embedding in documents viz.visualize_network(graph, output="png", file_path="graph.png") - # Vector SVG — for publications and scalable diagrams + # Vector SVG: for publications and scalable diagrams viz.visualize_network(graph, output="svg", file_path="graph.svg") ``` @@ -72,7 +72,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o viz = KGVisualizer(layout="force", color_scheme="default") - # Interactive — opens in browser + # Interactive: opens in browser viz.visualize_network(graph, output="interactive") # Save as HTML file @@ -98,7 +98,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o | Layout | Description | Best For | | :------ | :----------- | :-------- | - | `force` | Physics simulation — clusters emerge naturally | General graphs | + | `force` | Physics simulation: clusters emerge naturally | General graphs | | `hierarchical` | Top-down tree layout | Taxonomies, org charts | | `circular` | Nodes on a circle, edges as chords | Small dense graphs | @@ -160,7 +160,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o # Timeline of entity/relationship changes viz.visualize_timeline(temporal_data, output="interactive") - # Animated network evolution — one frame per time step + # Animated network evolution: one frame per time step viz.visualize_network_evolution(temporal_kg, output="html", file_path="evolution.html") # Side-by-side snapshot comparison @@ -171,7 +171,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o } viz.visualize_snapshot_comparison(snapshots, output="html", file_path="diff.html") - # Temporal patterns — pass a list of pattern dicts + # Temporal patterns: pass a list of pattern dicts viz.visualize_temporal_patterns(patterns, output="html", file_path="patterns.html") # Metrics evolution over time @@ -179,7 +179,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o ``` - Visualize graph analytics results — centrality, communities, and degree distribution: + Visualize graph analytics results: centrality, communities, and degree distribution: ```python from semantica.visualization import AnalyticsVisualizer diff --git a/docs/use-cases.md b/docs/use-cases.md index 8e224bf5..686b5458 100644 --- a/docs/use-cases.md +++ b/docs/use-cases.md @@ -18,8 +18,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `ingest` (PubMed RSS), `semantic_extract`, `kg`, `deduplication`, `context` **Notebooks:** - - [Drug Discovery Pipeline](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — Intermediate - - [Genomic Variant Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — Intermediate + - [Drug Discovery Pipeline](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb): Intermediate + - [Genomic Variant Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb): Intermediate Ground LLM answers in structured scientific literature with hybrid retrieval, logical inference, and source attribution on every claim. @@ -27,8 +27,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `context`, `vector_store`, `kg`, `reasoning`, `llms` **Notebooks:** - - [GraphRAG Complete](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — Advanced - - [RAG vs. GraphRAG Comparison](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — Advanced + - [GraphRAG Complete](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb): Advanced + - [RAG vs. GraphRAG Comparison](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb): Advanced @@ -38,12 +38,12 @@ Semantica is purpose-built for environments where AI outputs must be explainable - Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph — with conflict detection when sources disagree. + Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph: with conflict detection when sources disagree. **Key modules:** `ingest` (API, MCP, stream), `normalize`, `kg`, `conflicts`, `provenance` **Notebooks:** - - [Financial Data Integration (MCP)](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Intermediate + - [Financial Data Integration (MCP)](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb): Intermediate Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data. Temporal edges let you query: "what connections existed during this window?" @@ -51,7 +51,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `kg` (temporal), `conflicts`, `reasoning`, `visualization` **Notebooks:** - - [Fraud Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — Advanced + - [Fraud Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb): Advanced Map transaction flows, analyze DeFi protocols, and detect illicit activity. Graph algorithms (centrality, community detection) surface high-risk actors that linear transaction analysis misses. @@ -59,8 +59,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `kg`, `reasoning`, `visualization` **Notebooks:** - - [DeFi Protocol Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) — Intermediate - - [Transaction Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) — Intermediate + - [DeFi Protocol Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb): Intermediate + - [Transaction Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb): Intermediate @@ -75,8 +75,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `ingest` (stream, feed), `kg` (temporal), `context`, `reasoning`, `export` **Notebooks:** - - [Real-Time Anomaly Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) — Advanced - - [Threat Intelligence Hybrid RAG](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) — Advanced + - [Real-Time Anomaly Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb): Advanced + - [Threat Intelligence Hybrid RAG](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb): Advanced Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns. Network centrality surfaces actors text search alone would miss. @@ -84,7 +84,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `ingest`, `semantic_extract`, `kg`, `visualization` (community detection) **Notebooks:** - - [Criminal Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) — Intermediate + - [Criminal Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb): Intermediate Process multiple intelligence sources in parallel with an orchestrator-worker pipeline. Multi-source conflict detection flags disagreements rather than silently discarding minority reports. @@ -92,7 +92,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `pipeline`, `ingest`, `conflicts`, `provenance`, `export` **Notebooks:** - - [Intelligence Analysis Orchestrator-Worker](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) — Intermediate + - [Intelligence Analysis Orchestrator-Worker](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb): Intermediate @@ -107,7 +107,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `ingest`, `kg`, `reasoning`, `visualization`, `export` (Parquet for analytics) **Notebooks:** - - [Supply Chain Data Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) — Intermediate + - [Supply Chain Data Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb): Intermediate Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations. Temporal graphs track asset states over time and correlate maintenance events with performance degradation. @@ -115,7 +115,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable **Key modules:** `ingest` (stream, API), `kg` (temporal), `reasoning`, `visualization` **Notebooks:** - - [Energy Market Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) — Intermediate + - [Energy Market Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb): Intermediate @@ -133,8 +133,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable | Semantica capability | Compliance role | | :-------------------- | :-------------- | -| W3C PROV-O provenance | Full lineage from raw data to inference — required for FDA audit trails | -| SHA-256 checksums | Tamper detection on every snapshot — supports electronic record integrity | +| W3C PROV-O provenance | Full lineage from raw data to inference: required for FDA audit trails | +| SHA-256 checksums | Tamper detection on every snapshot: supports electronic record integrity | | Decision tracking | Every AI-assisted recommendation is recorded with causal chain and confidence | | Temporal graphs | Point-in-time queries for retrospective safety analysis | | SHACL validation | Schema enforcement before data enters the knowledge graph | @@ -147,10 +147,10 @@ Semantica is purpose-built for environments where AI outputs must be explainable | Semantica capability | Compliance role | | :-------------------- | :-------------- | -| Decision audit trail | Full record of model decisions with reasoning — required for model risk management | +| Decision audit trail | Full record of model decisions with reasoning: required for model risk management | | Conflict detection | Flags when two sources disagree on a valuation or risk figure | -| Version control | SHA-256 snapshot history — supports point-in-time reconstruction for audits | -| Provenance export | RDF with PROV-O inline — submittable to regulatory bodies as structured evidence | +| Version control | SHA-256 snapshot history: supports point-in-time reconstruction for audits | +| Provenance export | RDF with PROV-O inline: submittable to regulatory bodies as structured evidence | @@ -160,7 +160,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable | Semantica capability | Operational role | | :-------------------- | :--------------- | -| Local LLM support | `HuggingFaceLLM` and Ollama via LiteLLM — fully air-gapped deployments | +| Local LLM support | `HuggingFaceLLM` and Ollama via LiteLLM: fully air-gapped deployments | | Provenance chains | Every intelligence claim traceable to source document and extraction event | | Conflict resolution | Multiple-source disagreement resolved with auditable strategy | | Temporal intelligence | Historical queries over evolving intelligence graphs | diff --git a/docs/vector_stores/pgvector.md b/docs/vector_stores/pgvector.md index 942e3b41..21443f01 100644 --- a/docs/vector_stores/pgvector.md +++ b/docs/vector_stores/pgvector.md @@ -1,10 +1,10 @@ --- title: "pgvector Store" -description: "PostgreSQL with pgvector extension — cosine, L2, and inner product similarity search with IVFFlat and HNSW indexing." +description: "PostgreSQL with pgvector extension: cosine, L2, and inner product similarity search with IVFFlat and HNSW indexing." icon: "database" --- -**`PgVectorStore`** adds PostgreSQL-native vector storage and similarity search to Semantica — no dedicated vector database required. +**`PgVectorStore`** adds PostgreSQL-native vector storage and similarity search to Semantica: no dedicated vector database required. ## Overview @@ -18,7 +18,7 @@ icon: "database" JSONB metadata storage with filtering support Connection pooling via psycopg3/psycopg2 Batch insert, update, and delete -Idempotent index creation — safe to call multiple times +Idempotent index creation: safe to call multiple times ## Setup @@ -196,7 +196,7 @@ store.create_index( ) ``` -Index creation is idempotent — calling multiple times is safe. +Index creation is idempotent: calling multiple times is safe. ### Statistics @@ -226,7 +226,7 @@ stats = store.get_stats() - **Hierarchical Navigable Small World** — best for high-dimensional vectors with high recall requirements. + **Hierarchical Navigable Small World**: best for high-dimensional vectors with high recall requirements. | | | | :-- | :-- | @@ -241,7 +241,7 @@ stats = store.get_stats() ``` - **Inverted File with Flat Index** — best for large datasets in memory-constrained environments. + **Inverted File with Flat Index**: best for large datasets in memory-constrained environments. | | | | :-- | :-- |