mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
chore: remove legacy MkDocs files and orphan docs pages
Deleted MkDocs infrastructure: - mkdocs.yml, mkdocs_local.yml, requirements-docs.txt, setup_docs.py - docs/netlify.toml, docs/DOCS_README.md, docs/css/custom.css Deleted orphan docs not wired into Mintlify nav: - docs/LIBS_README.md, docs/MIGRATION_V2.md, docs/CodeExamples.md - docs/arrow_exporter.md, docs/deep-dive.md, docs/examples.md - docs/vector_store_usage.md Updated docs.json and broken See Also hrefs to match removed pages
This commit is contained in:
KaifAhmad1
File diff suppressed because it is too large
Load Diff
@@ -1,114 +0,0 @@
|
||||
# Semantica Documentation
|
||||
|
||||
This documentation is built with [MkDocs](https://www.mkdocs.org/) - a fast, simple static site generator for project documentation.
|
||||
|
||||
## Features
|
||||
|
||||
- **Great themes available** - Using Material theme with beautiful design
|
||||
- **Easy to customize** - Custom CSS and theme configuration
|
||||
- **Preview as you work** - Built-in dev server with auto-reload
|
||||
- **Host anywhere** - Static HTML that works on GitHub Pages, Netlify, etc.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements-docs.txt
|
||||
```
|
||||
|
||||
### 2. Preview Locally
|
||||
|
||||
```bash
|
||||
mkdocs serve
|
||||
```
|
||||
|
||||
Then open `http://127.0.0.1:8000` in your browser.
|
||||
|
||||
### 3. Build for Production
|
||||
|
||||
```bash
|
||||
mkdocs build
|
||||
```
|
||||
|
||||
This creates a `site/` directory with static HTML files ready to deploy.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
semantica/
|
||||
├── mkdocs.yml # MkDocs configuration
|
||||
├── requirements-docs.txt # Python dependencies
|
||||
├── docs/ # Documentation source files
|
||||
│ ├── index.md # Homepage
|
||||
│ ├── *.md # Documentation pages
|
||||
│ ├── css/
|
||||
│ │ └── custom.css # Custom styling
|
||||
│ └── assets/
|
||||
│ └── img/
|
||||
│ └── Semantica Logo.png
|
||||
└── site/ # Generated site (created by mkdocs build)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Main configuration is in `mkdocs.yml`:
|
||||
- Site metadata
|
||||
- Theme settings (Material theme)
|
||||
- Navigation structure
|
||||
- Markdown extensions
|
||||
- Plugins
|
||||
|
||||
## Adding New Pages
|
||||
|
||||
1. Create a new `.md` file in `docs/`
|
||||
2. Add it to `nav:` section in `mkdocs.yml`
|
||||
3. Run `mkdocs serve` to preview
|
||||
|
||||
## Customization
|
||||
|
||||
### Theme
|
||||
|
||||
Edit `mkdocs.yml` under `theme:` section to customize:
|
||||
- Color scheme
|
||||
- Logo
|
||||
- Features enabled
|
||||
- Icons
|
||||
|
||||
### Styling
|
||||
|
||||
Edit `docs/css/custom.css` for custom styles.
|
||||
|
||||
## Deployment
|
||||
|
||||
### GitHub Pages
|
||||
|
||||
```bash
|
||||
mkdocs gh-deploy
|
||||
```
|
||||
|
||||
### Netlify/Vercel
|
||||
|
||||
1. Build: `mkdocs build`
|
||||
2. Deploy the `site/` directory
|
||||
|
||||
### Manual
|
||||
|
||||
1. Run `mkdocs build`
|
||||
2. Upload `site/` folder contents to your web server
|
||||
|
||||
## Development Workflow
|
||||
|
||||
1. Edit markdown files in `docs/`
|
||||
2. Run `mkdocs serve` to preview
|
||||
3. Changes auto-reload in browser
|
||||
4. When ready, build with `mkdocs build`
|
||||
|
||||
## Benefits
|
||||
|
||||
- ✅ Python-based (fits with your Python project)
|
||||
- ✅ Beautiful Material theme
|
||||
- ✅ Fast and lightweight
|
||||
- ✅ Easy to customize
|
||||
- ✅ Great search functionality
|
||||
- ✅ Mobile responsive
|
||||
-1454
File diff suppressed because it is too large
Load Diff
@@ -1,158 +0,0 @@
|
||||
---
|
||||
title: "Deduplication V2 Migration"
|
||||
description: "Migration guide for the Deduplication V2 engine with blocking_v2 and semantic_v2 strategies."
|
||||
icon: "arrows-rotate"
|
||||
---
|
||||
|
||||
## Semantica Deduplication V2: Migration & Performance Guide
|
||||
|
||||
Welcome to the Deduplication V2 engine!! This release specifically targets severe CI delays and production bottlenecks caused by massive knowledge graph deduplication workloads. By introducing smarter candidate generation, fast-fail prefilters, and semantic triplet canonicalization, we have reduced worst-case execution times by up to **80%**.
|
||||
|
||||
**Note:** This upgrade is **100% backward compatible.** All existing scripts, tests, and API signatures will continue to work exactly as they did before.
|
||||
|
||||
|
||||
|
||||
To utilize this new addition, you must explicitly **opt-in** using the new configuration keys detailed below.
|
||||
|
||||
---
|
||||
|
||||
### 1. Candidate Generation V2 (Beating the $O(N^2)$ Pair Explosion)
|
||||
|
||||
**The Problem:** The legacy engine relied on a naive first-character blocking strategy. If your dataset contained 5,000 companies starting with letter "A", the engine generated nearly 12.5 million candidate pairs.
|
||||
|
||||
**The V2 Solution:** Multi-key token blocking, prefix matching, and deterministic candidate budgeting.
|
||||
|
||||
|
||||
|
||||
**How to Opt-In**
|
||||
|
||||
Pass the keys into the `similarity`configuration dictionary when initializing the `DuplicateDetector`:
|
||||
|
||||
|
||||
|
||||
```python
|
||||
from semantica.deduplication import DuplicateDetector
|
||||
|
||||
detector = DuplicateDetector(
|
||||
similarity_threshold=0.8,
|
||||
similarity = {
|
||||
# Switches from legacy to v2
|
||||
"candidate_strategy": "blocking_v2",
|
||||
|
||||
# Highly recommended: Limits the max number of comparisons
|
||||
# per entity to prevent adversarial latency spikes.
|
||||
"max_candidates_per_entity": 50,
|
||||
|
||||
# Optional: Generates blocks using Soundex algorithm to catch
|
||||
# phonetic misspellings (e.g, "Jon" vs "John")
|
||||
"enable_phonetic_blocking": True
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 2. Two-Stage scoring (The Fast Prefilter)
|
||||
|
||||
**The Problem**: Calculating multi-factor semantic scores (Levenshtein, Jaro-Winkler, property intersections, and Embeddings) is computationally expensive. Running these
|
||||
|
||||
calculations on two entities that share absolutely zero words or have vastly different string lengths is a waste of resources.
|
||||
|
||||
**The V2 Solution:** A lightning-fast prefilter gate that instantly drops obvious non-matches before they ever reach the heavy semantic scorers.
|
||||
|
||||
|
||||
|
||||
**How to Opt-In**
|
||||
|
||||
Enable the prefilter and define your rejection thresholds:
|
||||
|
||||
|
||||
|
||||
```python
|
||||
from semantica.deduplication import DuplicateDetector
|
||||
|
||||
detector = DuplicateDetector(
|
||||
similarity_threshold=0.8,
|
||||
similarity={
|
||||
"candidate_strategy": "blocking_v2",
|
||||
|
||||
# Enable prefilter
|
||||
"prefilter_enabled": True,
|
||||
|
||||
"prefilter_thresholds": {
|
||||
# Rejects pairs if shortest string is less than 40% the length
|
||||
# of the longest
|
||||
"min_length_ratio": 0.4,
|
||||
|
||||
# Instantly rejects pairs if they don't share at least one
|
||||
# valid word token
|
||||
"required_shared_token": True
|
||||
},
|
||||
# Optional Explainability: Injects a 'score_breakdown' dict into
|
||||
# the candidate metadata so you can see exactly how the string,
|
||||
# property, and relationships scores contributed.
|
||||
|
||||
"score_breakdown_enabled": True
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 3. Semantic Relationship & Triplet Deduplication
|
||||
|
||||
**The problem:** The legacy relationship deduplication relied on exact `(Subject, Predicate, Object)` string matches. It couldn't recognize that `(Person, "works_for", Company)` is semantically identical to `(Person, "employed_by", Company)` .
|
||||
|
||||
**The V2 Solution:** A new `semantic_v2` mode that introduces predicate synonym mapping, literal normalization (cleaning up rogue spaces/casing), and a highly optimized $O(1)$ canonical hash path for fast matching.
|
||||
|
||||
|
||||
|
||||
**How to Opt-In**
|
||||
|
||||
When calling relationship-specific dedup methods, pass the new configuration keys:
|
||||
|
||||
```python
|
||||
from semantica.deduplication import DuplicateDetector
|
||||
from semantica.deduplication.methods import dedup_triplets
|
||||
|
||||
|
||||
# Approach A: Using the Detector explicitly
|
||||
detector = DuplicateDetector()
|
||||
duplicates = detector.detect_relationship_duplicates(
|
||||
relationship_list,
|
||||
relationship_dedup_mode="semantic_v2",
|
||||
|
||||
# Cleans up messy object strings
|
||||
# (e.g., " Apple Inc. " -> "apple inc.")
|
||||
literal_normalization_enabled=True,
|
||||
|
||||
# Maps various synonyms to a single canonical predicate
|
||||
# before hashing
|
||||
predicate_synonym_map={
|
||||
"works_for": "employed_by",
|
||||
"is_employee_of": "employed_by",
|
||||
"has_employer": "employed_by"
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Approach B: Using the new simplified wrapper in methods.py
|
||||
duplicates = dedup_triplets(
|
||||
relationships_list,
|
||||
mode="semantic_v2",
|
||||
literal_normalization_enabled=True,
|
||||
predicate_synonym_map={"works_for": "employed_by"}
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
###### Note on Merge Strategies
|
||||
|
||||
When using `semantic_v2` for relationships, the `MergeStrategyManager` will now automatically respect your canonicalized keys. If two entities share a relationship that differs only by a mapped synonym, the engine will correctly identify them as the same relationship and prevent duplicate graph edges during the merge phase.
|
||||
|
||||
|
||||
|
||||
### Need Help?
|
||||
|
||||
If you experience any unexpected behavior when switching from `legacy` to `blocking_v2` or `semantic_v2`, please check the explainability metadata (by setting `"score_breakdown_enabled": True`) to audit the exact scoring process, or open an issue on GitHub.
|
||||
@@ -193,7 +193,7 @@ Centralized `ConfigManager` with environment variable overrides. No magic defaul
|
||||
<Card title="Modules" icon="cubes" href="modules">
|
||||
Full module documentation with code examples.
|
||||
</Card>
|
||||
<Card title="Deep Dive" icon="microscope" href="deep-dive">
|
||||
<Card title="Learning More" icon="microscope" href="learning-more">
|
||||
Internals, algorithms, and advanced extension patterns.
|
||||
</Card>
|
||||
<Card title="Pipeline Reference" icon="gear" href="reference/pipeline">
|
||||
|
||||
@@ -1,273 +0,0 @@
|
||||
---
|
||||
title: "Apache Arrow Exporter"
|
||||
description: "High-performance columnar export for knowledge graphs, entities, and relationships using Apache Arrow IPC format."
|
||||
icon: "file-arrow-down"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Apache Arrow exporter provides high-performance columnar data export for Semantica's knowledge graphs, entities, and relationships. It uses explicit schemas (no inference) and writes Arrow IPC files (.arrow) that are compatible with Pandas and DuckDB.
|
||||
|
||||
## Features
|
||||
|
||||
- **Explicit Schemas**: Pre-defined schemas for entities and relationships (no inference)
|
||||
- **Columnar Format**: Efficient storage and fast analytics
|
||||
- **Metadata Support**: Converts metadata dictionaries to Arrow struct fields
|
||||
- **Field Normalization**: Handles various entity and relationship field name variations
|
||||
- **Progress Tracking**: Integrated progress monitoring
|
||||
- **Error Handling**: Structured error handling with detailed logging
|
||||
- **Pandas/DuckDB Compatible**: Direct conversion to DataFrames and SQL queries
|
||||
|
||||
## Installation
|
||||
|
||||
The Arrow exporter requires PyArrow:
|
||||
|
||||
```bash
|
||||
pip install pyarrow
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from semantica.export import ArrowExporter
|
||||
|
||||
# Initialize exporter
|
||||
exporter = ArrowExporter()
|
||||
|
||||
# Export entities
|
||||
entities = [
|
||||
{"id": "e1", "text": "Alice", "type": "Person", "confidence": 0.95},
|
||||
{"id": "e2", "text": "Acme Corp", "type": "Organization", "confidence": 0.88}
|
||||
]
|
||||
exporter.export_entities(entities, "entities.arrow")
|
||||
|
||||
# Export relationships
|
||||
relationships = [
|
||||
{"id": "r1", "source_id": "e1", "target_id": "e2", "type": "WORKS_FOR"}
|
||||
]
|
||||
exporter.export_relationships(relationships, "relationships.arrow")
|
||||
|
||||
# Export knowledge graph
|
||||
knowledge_graph = {
|
||||
"entities": entities,
|
||||
"relationships": relationships
|
||||
}
|
||||
exporter.export_knowledge_graph(knowledge_graph, "kg_base")
|
||||
# Creates: kg_base_entities.arrow, kg_base_relationships.arrow
|
||||
```
|
||||
|
||||
### Using Convenience Function
|
||||
|
||||
```python
|
||||
from semantica.export import export_arrow
|
||||
|
||||
# Simple export
|
||||
export_arrow(entities, "entities.arrow")
|
||||
|
||||
# Export multiple types
|
||||
data = {
|
||||
"entities": entities,
|
||||
"relationships": relationships
|
||||
}
|
||||
export_arrow(data, "output_base")
|
||||
```
|
||||
|
||||
### With Compression
|
||||
|
||||
```python
|
||||
# Use LZ4 compression
|
||||
exporter = ArrowExporter(compression="lz4")
|
||||
exporter.export_entities(entities, "entities_compressed.arrow")
|
||||
```
|
||||
|
||||
## Schemas
|
||||
|
||||
### Entity Schema
|
||||
|
||||
```python
|
||||
ENTITY_SCHEMA = pa.schema([
|
||||
pa.field("id", pa.string(), nullable=False),
|
||||
pa.field("text", pa.string(), nullable=True),
|
||||
pa.field("type", pa.string(), nullable=True),
|
||||
pa.field("confidence", pa.float64(), nullable=True),
|
||||
pa.field("start", pa.int64(), nullable=True),
|
||||
pa.field("end", pa.int64(), nullable=True),
|
||||
pa.field("metadata", pa.struct([
|
||||
pa.field("keys", pa.list_(pa.string())),
|
||||
pa.field("values", pa.list_(pa.string()))
|
||||
]), nullable=True),
|
||||
])
|
||||
```
|
||||
|
||||
### Relationship Schema
|
||||
|
||||
```python
|
||||
RELATIONSHIP_SCHEMA = pa.schema([
|
||||
pa.field("id", pa.string(), nullable=False),
|
||||
pa.field("source_id", pa.string(), nullable=False),
|
||||
pa.field("target_id", pa.string(), nullable=False),
|
||||
pa.field("type", pa.string(), nullable=True),
|
||||
pa.field("confidence", pa.float64(), nullable=True),
|
||||
pa.field("metadata", pa.struct([
|
||||
pa.field("keys", pa.list_(pa.string())),
|
||||
pa.field("values", pa.list_(pa.string()))
|
||||
]), nullable=True),
|
||||
])
|
||||
```
|
||||
|
||||
## Field Normalization
|
||||
|
||||
The exporter automatically normalizes field names:
|
||||
|
||||
**Entities:**
|
||||
- `text`, `label`, `name` → `text`
|
||||
- `type`, `entity_type` → `type`
|
||||
- `id`, `entity_id` → `id`
|
||||
- `start`, `start_offset` → `start`
|
||||
- `end`, `end_offset` → `end`
|
||||
|
||||
**Relationships:**
|
||||
- `source`, `source_id` → `source_id`
|
||||
- `target`, `target_id` → `target_id`
|
||||
- `type`, `relationship_type` → `type`
|
||||
|
||||
## Reading Arrow Files
|
||||
|
||||
### With PyArrow
|
||||
|
||||
```python
|
||||
import pyarrow as pa
|
||||
import pyarrow.ipc as ipc
|
||||
|
||||
with pa.OSFile("entities.arrow", 'rb') as source:
|
||||
with ipc.open_file(source) as reader:
|
||||
table = reader.read_all()
|
||||
print(table.schema)
|
||||
print(table.to_pandas())
|
||||
```
|
||||
|
||||
### With Pandas
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
import pyarrow.ipc as ipc
|
||||
|
||||
with ipc.open_file("entities.arrow") as reader:
|
||||
df = reader.read_all().to_pandas()
|
||||
print(df)
|
||||
```
|
||||
|
||||
### With DuckDB
|
||||
|
||||
```python
|
||||
import duckdb
|
||||
|
||||
# Query Arrow file directly
|
||||
result = duckdb.query("SELECT * FROM 'entities.arrow' WHERE type = 'Person'")
|
||||
print(result.df())
|
||||
```
|
||||
|
||||
## Methods
|
||||
|
||||
### `export(data, file_path, schema=None, **options)`
|
||||
|
||||
Generic export method that handles both single and multiple files.
|
||||
|
||||
**Parameters:**
|
||||
- `data`: List of dicts or dict with list values
|
||||
- `file_path`: Output file path (base path for dict exports)
|
||||
- `schema`: Optional Arrow schema (auto-detected if not provided)
|
||||
- `**options`: Additional options
|
||||
|
||||
### `export_entities(entities, file_path, **options)`
|
||||
|
||||
Export entities to Arrow IPC file with normalization.
|
||||
|
||||
**Parameters:**
|
||||
- `entities`: List of entity dictionaries
|
||||
- `file_path`: Output Arrow file path
|
||||
- `**options`: Additional options
|
||||
|
||||
### `export_relationships(relationships, file_path, **options)`
|
||||
|
||||
Export relationships to Arrow IPC file with normalization.
|
||||
|
||||
**Parameters:**
|
||||
- `relationships`: List of relationship dictionaries
|
||||
- `file_path`: Output Arrow file path
|
||||
- `**options`: Additional options
|
||||
|
||||
### `export_knowledge_graph(knowledge_graph, base_path, **options)`
|
||||
|
||||
Export knowledge graph to multiple Arrow files.
|
||||
|
||||
**Parameters:**
|
||||
- `knowledge_graph`: Knowledge graph dictionary with 'entities' and 'relationships'
|
||||
- `base_path`: Base path for output files (without extension)
|
||||
- `**options`: Additional options
|
||||
|
||||
## Examples
|
||||
|
||||
See `examples/arrow_export_example.py` for comprehensive usage examples.
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test suite:
|
||||
|
||||
```bash
|
||||
# All Arrow exporter tests
|
||||
pytest tests/test_arrow_exporter.py -v
|
||||
|
||||
# Integration tests
|
||||
pytest tests/test_export_module.py::TestExportModule::test_arrow_exporter -v
|
||||
```
|
||||
|
||||
## Performance Benefits
|
||||
|
||||
- **Columnar Storage**: Faster analytics on specific columns
|
||||
- **Compression**: Smaller file sizes (especially with LZ4/ZSTD)
|
||||
- **Zero-Copy**: Memory-efficient data transfer
|
||||
- **Cross-Language**: Works with Python, R, Julia, JavaScript, and more
|
||||
- **SQL Queries**: Direct querying with DuckDB without loading into memory
|
||||
|
||||
## Comparison with Other Formats
|
||||
|
||||
| Feature | Arrow | CSV | JSON |
|
||||
|---------|-------|-----|------|
|
||||
| Type Safety | ✓ | ✗ | ✗ |
|
||||
| Compression | ✓ | ✗ | ✗ |
|
||||
| Schema Validation | ✓ | ✗ | ✗ |
|
||||
| Pandas Compatible | ✓ | ✓ | ✓ |
|
||||
| DuckDB Native | ✓ | ✓ | ✗ |
|
||||
| Binary Format | ✓ | ✗ | ✗ |
|
||||
| Human Readable | ✗ | ✓ | ✓ |
|
||||
|
||||
## Architecture
|
||||
|
||||
The Arrow exporter follows Semantica's export architecture:
|
||||
|
||||
1. **Normalization**: Field names are normalized to consistent format
|
||||
2. **Schema Application**: Explicit schemas ensure type safety
|
||||
3. **Metadata Conversion**: Dicts converted to Arrow struct fields
|
||||
4. **Progress Tracking**: Integrated with Semantica's progress tracker
|
||||
5. **Error Handling**: Structured exceptions with detailed messages
|
||||
|
||||
## Contributing
|
||||
|
||||
When contributing to the Arrow exporter:
|
||||
|
||||
1. Maintain explicit schemas (no inference)
|
||||
2. Follow existing code style and patterns
|
||||
3. Add comprehensive tests for new features
|
||||
4. Update this documentation
|
||||
5. Ensure Pandas/DuckDB compatibility
|
||||
|
||||
## License
|
||||
|
||||
MIT License - See LICENSE file for details.
|
||||
|
||||
## Author
|
||||
|
||||
Semantica Contributors
|
||||
@@ -1,315 +0,0 @@
|
||||
/* Semantica Documentation - Monochrome Pro Theme */
|
||||
|
||||
/* Smooth scrolling */
|
||||
html {
|
||||
scroll-behavior: smooth;
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Color Variables - Monochrome Pro
|
||||
Primary: #212121 (Grey 900)
|
||||
Accent: #2962FF (Electric Blue)
|
||||
==========================================================================
|
||||
*/
|
||||
:root {
|
||||
/* Light Mode */
|
||||
--md-default-bg-color: #FFFFFF;
|
||||
--md-default-fg-color: #212121;
|
||||
--md-default-fg-color--light: #616161;
|
||||
--md-default-fg-color--lighter: #9E9E9E;
|
||||
--md-default-fg-color--lightest: #E0E0E0;
|
||||
|
||||
--md-primary-fg-color: #212121;
|
||||
--md-primary-fg-color--light: #484848;
|
||||
--md-primary-fg-color--dark: #000000;
|
||||
|
||||
--md-accent-fg-color: #2962FF;
|
||||
|
||||
/* Code blocks */
|
||||
--md-code-bg-color: #F5F5F5;
|
||||
--md-code-fg-color: #212121;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] {
|
||||
/* Dark Mode */
|
||||
--md-default-bg-color: #0F1115;
|
||||
--md-default-fg-color: #E0E0E0;
|
||||
|
||||
--md-primary-fg-color: #0F1115;
|
||||
--md-primary-fg-color--light: #212121;
|
||||
--md-primary-fg-color--dark: #000000;
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Typography
|
||||
==========================================================================
|
||||
*/
|
||||
.md-typeset h2 {
|
||||
font-weight: 700;
|
||||
letter-spacing: -0.01em;
|
||||
margin-top: 2rem;
|
||||
margin-bottom: 0.75rem;
|
||||
}
|
||||
|
||||
.md-typeset p {
|
||||
line-height: 1.6;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
/* Links */
|
||||
.md-typeset a {
|
||||
color: var(--md-accent-fg-color);
|
||||
text-decoration: none;
|
||||
font-weight: 500;
|
||||
background-color: #F1F8F5;
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Admonitions
|
||||
==========================================================================
|
||||
*/
|
||||
/* Tip */
|
||||
.md-typeset .admonition.tip .admonition-title {
|
||||
color: #00C853;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset .admonition.tip {
|
||||
border-color: #2E303E;
|
||||
border-left-color: #69F0AE;
|
||||
background-color: #0E1B14;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset .admonition.tip .admonition-title {
|
||||
color: #69F0AE;
|
||||
}
|
||||
|
||||
/* Warning */
|
||||
.md-typeset .admonition.warning {
|
||||
border-color: #E0E0E0;
|
||||
border-left-color: #FFAB00;
|
||||
background-color: #FFF8E1;
|
||||
}
|
||||
|
||||
.md-typeset .admonition.warning .admonition-title {
|
||||
color: #FFAB00;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset .admonition.warning {
|
||||
border-color: #2E303E;
|
||||
border-left-color: #FFD740;
|
||||
background-color: #1F1B0E;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset .admonition.warning .admonition-title {
|
||||
color: #FFD740;
|
||||
}
|
||||
|
||||
/* Danger */
|
||||
.md-typeset .admonition.danger {
|
||||
border-color: #E0E0E0;
|
||||
border-left-color: #FF1744;
|
||||
background-color: #FFEBEE;
|
||||
}
|
||||
|
||||
.md-typeset .admonition.danger .admonition-title {
|
||||
color: #FF1744;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset .admonition.danger {
|
||||
border-color: #2E303E;
|
||||
border-left-color: #FF5252;
|
||||
background-color: #241214;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset .admonition.danger .admonition-title {
|
||||
color: #FF5252;
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Code Blocks
|
||||
==========================================================================
|
||||
*/
|
||||
.md-typeset pre {
|
||||
background-color: var(--md-code-bg-color);
|
||||
border: 1px solid rgba(0, 0, 0, 0.05);
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-typeset pre {
|
||||
border-color: rgba(255, 255, 255, 0.05);
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Scrollbars
|
||||
==========================================================================
|
||||
*/
|
||||
::-webkit-scrollbar {
|
||||
width: 6px;
|
||||
height: 6px;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-thumb {
|
||||
background-color: rgba(0, 0, 0, 0.2);
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] ::-webkit-scrollbar-thumb {
|
||||
background-color: #2962FF;
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Footer Attribution - Keep MkDocs Credit Visible
|
||||
==========================================================================
|
||||
*/
|
||||
.md-footer-meta__inner {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.md-footer-copyright {
|
||||
opacity: 1 !important;
|
||||
color: var(--md-default-fg-color--light) !important;
|
||||
}
|
||||
|
||||
.md-footer-copyright__highlight {
|
||||
opacity: 1 !important;
|
||||
color: var(--md-default-fg-color) !important;
|
||||
font-weight: 500 !important;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-footer-copyright {
|
||||
color: rgba(255, 255, 255, 0.7) !important;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"] .md-footer-copyright__highlight {
|
||||
color: rgba(255, 255, 255, 0.9) !important;
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Active Link Highlighting
|
||||
==========================================================================
|
||||
*/
|
||||
/* Left Sidebar (Navigation) - Active Link */
|
||||
.md-nav__link--active {
|
||||
color: var(--md-accent-fg-color) !important;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
/* Right Sidebar (Table of Contents) - Active Link */
|
||||
.md-nav__item--active > .md-nav__link {
|
||||
color: var(--md-accent-fg-color) !important;
|
||||
border-left: 2px solid var(--md-accent-fg-color);
|
||||
padding-left: 0.5rem;
|
||||
}
|
||||
|
||||
/* Ensure nested items in TOC don't inherit the border unless active themselves */
|
||||
.md-nav__item .md-nav__item--active > .md-nav__link {
|
||||
border-left: 2px solid var(--md-accent-fg-color);
|
||||
}
|
||||
|
||||
/*
|
||||
==========================================================================
|
||||
Layout Optimization
|
||||
==========================================================================
|
||||
*/
|
||||
|
||||
/* Widen the overall grid */
|
||||
.md-grid {
|
||||
max-width: 1440px;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
padding-left: 0.5rem;
|
||||
}
|
||||
|
||||
/* Narrow left sidebar to give content more room */
|
||||
.md-sidebar--primary {
|
||||
width: 11rem;
|
||||
padding-right: 0.25rem;
|
||||
padding-left: 0.25rem;
|
||||
}
|
||||
|
||||
/* Right TOC sidebar */
|
||||
.md-sidebar--secondary {
|
||||
width: 11rem;
|
||||
padding-left: 0.5rem;
|
||||
padding-right: 0;
|
||||
margin-left: 0;
|
||||
}
|
||||
|
||||
.md-sidebar--secondary .md-nav {
|
||||
width: 11rem;
|
||||
}
|
||||
|
||||
/* Tighten TOC list spacing */
|
||||
.md-sidebar--secondary .md-nav__list {
|
||||
padding-bottom: 1.5rem;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.md-sidebar--secondary .md-nav__item {
|
||||
padding: 0;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.md-sidebar--secondary .md-nav__link {
|
||||
white-space: normal;
|
||||
word-break: break-word;
|
||||
overflow: visible;
|
||||
text-overflow: unset;
|
||||
padding-top: 0.15rem;
|
||||
padding-bottom: 0.15rem;
|
||||
line-height: 1.4;
|
||||
font-size: 0.7rem;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
/* Nested TOC items (h3, h4) */
|
||||
.md-sidebar--secondary .md-nav__item .md-nav__item .md-nav__link {
|
||||
padding-left: 0.6rem;
|
||||
font-size: 0.68rem;
|
||||
}
|
||||
|
||||
/* Remove extra gap between TOC title and first item */
|
||||
.md-sidebar--secondary .md-nav__title {
|
||||
margin-bottom: 0.25rem;
|
||||
padding-bottom: 0.25rem;
|
||||
}
|
||||
|
||||
/* Give the main content area maximum available width */
|
||||
.md-content {
|
||||
max-width: none;
|
||||
padding-left: 1rem;
|
||||
padding-right: 1rem;
|
||||
}
|
||||
|
||||
.md-content__inner {
|
||||
max-width: none;
|
||||
padding-left: 1rem;
|
||||
padding-right: 1rem;
|
||||
margin-left: 0;
|
||||
margin-right: 0;
|
||||
}
|
||||
|
||||
.md-main__inner {
|
||||
margin-left: 0;
|
||||
margin-right: 0;
|
||||
}
|
||||
|
||||
/* Ensure text content is left-aligned by default */
|
||||
.md-typeset {
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
/* Keep hero section centered */
|
||||
.md-typeset > div[align="center"] {
|
||||
text-align: center;
|
||||
}
|
||||
@@ -1,208 +0,0 @@
|
||||
---
|
||||
title: "Deep Dive"
|
||||
description: "Internals, advanced concepts, and extension points for contributors and power users."
|
||||
icon: "microscope"
|
||||
---
|
||||
|
||||
> Internals, advanced concepts, and extension points for contributors and power users.
|
||||
|
||||
<Tip>
|
||||
New to Semantica? Read the [Architecture](architecture) overview first for a higher-level picture.
|
||||
</Tip>
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Internals
|
||||
|
||||
Full data flow through a Semantica pipeline:
|
||||
|
||||
```text
|
||||
Data Sources
|
||||
└─ Ingestion Layer (FileIngestor, WebIngestor, SnowflakeIngestor, StreamIngestor)
|
||||
└─ Parsing Layer (DocumentParser, DoclingParser, OCR)
|
||||
└─ Extraction (NER → Entity Linking → Validation)
|
||||
└─ Normalization
|
||||
└─ Conflict Resolution
|
||||
└─ Knowledge Graph Builder
|
||||
└─ Embedding Generator
|
||||
└─ Export Layer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System Components
|
||||
|
||||
### Ingestion Layer
|
||||
|
||||
- **FileIngestor** — PDF, DOCX, HTML, JSON, CSV, TXT, Parquet (v0.5.0), XML (v0.5.0), archives
|
||||
- **WebIngestor** — URL crawling and scraping
|
||||
- **SnowflakeIngestor** — SQL databases and cloud warehouses
|
||||
- **StreamIngestor** — Kafka and real-time feeds
|
||||
|
||||
### Parsing Layer
|
||||
|
||||
- Text and metadata extraction from documents
|
||||
- OCR for scanned content
|
||||
- Layout analysis via Docling (tables, columns, headers)
|
||||
|
||||
### Extraction Layer
|
||||
|
||||
```text
|
||||
text → Tokenization → NER → Entity Linking → Entity Validation
|
||||
```
|
||||
|
||||
Components: Named Entity Recognition, Relationship Extraction, Triplet Extraction, Coreference Resolution.
|
||||
|
||||
### Normalization Layer
|
||||
|
||||
Standardizes entity names, date formats, numbers, encodings, and language. Includes the v0.5.0 cp1252 encoding fix for Windows environments.
|
||||
|
||||
### Conflict Resolution
|
||||
|
||||
Multiple source facts that contradict each other are resolved using one of four strategies:
|
||||
|
||||
| Strategy | Behavior |
|
||||
|----------|----------|
|
||||
| `voting` | Most common value wins |
|
||||
| `credibility_weighted` | Higher-credibility source wins |
|
||||
| `most_recent` | Latest timestamp wins |
|
||||
| `highest_confidence` | Highest extraction confidence wins |
|
||||
|
||||
### Knowledge Graph Builder
|
||||
|
||||
- Entity resolution across sources
|
||||
- Edge creation with typed relationships
|
||||
- Property assignment with confidence scores
|
||||
- Graph validation and quality checks
|
||||
|
||||
### Embedding Generator
|
||||
|
||||
- Text embeddings: Sentence-Transformers, FastEmbed, OpenAI, BGE
|
||||
- Graph embeddings: Node2Vec, GraphSAGE
|
||||
- Distance caching for Distance Intelligence (v0.5.0)
|
||||
|
||||
---
|
||||
|
||||
## Advanced Concepts
|
||||
|
||||
### Entity Resolution
|
||||
|
||||
```python
|
||||
def resolve_entities(entities, threshold=0.85):
|
||||
clusters = []
|
||||
for entity in entities:
|
||||
matched = False
|
||||
for cluster in clusters:
|
||||
if similarity(entity, cluster.representative) > threshold:
|
||||
cluster.add(entity)
|
||||
matched = True
|
||||
break
|
||||
if not matched:
|
||||
clusters.append(EntityCluster(entity))
|
||||
return clusters
|
||||
```
|
||||
|
||||
### Relationship Inference
|
||||
|
||||
Semantica's reasoning engines derive implicit relationships:
|
||||
|
||||
- **Transitive** — if A→B and B→C, infer A→C
|
||||
- **Temporal** — before/after/during from timestamped facts (Allen Interval Algebra)
|
||||
- **Causal** — IF/THEN rules via `Reasoner`
|
||||
- **Hierarchical** — subclass/instance inference via `OntologyReasoner`
|
||||
- **Datalog** — recursive rules with termination guarantee (v0.4.0)
|
||||
|
||||
### Batch Processing for Large Datasets
|
||||
|
||||
```python
|
||||
def process_large_dataset(sources, batch_size=100):
|
||||
for i in range(0, len(sources), batch_size):
|
||||
batch = sources[i : i + batch_size]
|
||||
result = semantica.build_knowledge_base(batch)
|
||||
save_result(result)
|
||||
del result
|
||||
gc.collect()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Extension Points
|
||||
|
||||
### Custom Plugin
|
||||
|
||||
```python
|
||||
from semantica.core import Plugin
|
||||
|
||||
class CustomPlugin(Plugin):
|
||||
def initialize(self):
|
||||
...
|
||||
|
||||
def process(self, data):
|
||||
return processed_data
|
||||
```
|
||||
|
||||
### Custom Extractor
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import BaseExtractor
|
||||
|
||||
class DomainSpecificExtractor(BaseExtractor):
|
||||
def extract(self, text):
|
||||
# Domain-specific entity extraction logic
|
||||
return entities
|
||||
```
|
||||
|
||||
### Custom Ingestor
|
||||
|
||||
```python
|
||||
from semantica.ingest import BaseIngestor
|
||||
|
||||
class CustomIngestor(BaseIngestor):
|
||||
def ingest(self, source):
|
||||
# Load and return document dicts
|
||||
return documents
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Internal APIs
|
||||
|
||||
| API | Purpose |
|
||||
|-----|---------|
|
||||
| `Semantica.build_knowledge_base()` | Main orchestration entry point |
|
||||
| `GraphBuilder.build()` | Graph construction |
|
||||
| `ConflictResolver.resolve()` | Conflict resolution |
|
||||
| `EmbeddingGenerator.generate()` | Embedding generation |
|
||||
|
||||
Extension hooks: plugin registration, custom extractor registration, custom exporter registration, event hooks.
|
||||
|
||||
---
|
||||
|
||||
## Design Decisions
|
||||
|
||||
**Why modular architecture?** Each component is independently testable and swappable. You can use `NERExtractor` alone without pulling in graph storage or pipelines.
|
||||
|
||||
**Why built-in conflict resolution?** Multi-source data always has contradictions. Ignoring them produces low-quality graphs. Explicit strategies give you control over data quality.
|
||||
|
||||
**Why W3C PROV-O for provenance?** It's an industry standard with broad tooling support. A custom format would make lineage data non-portable.
|
||||
|
||||
**Why multiple reasoning engines?** Different problems need different reasoning: forward chaining for rule application, SPARQL for graph queries, abductive for hypothesis generation, Datalog for recursive rules.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Modules" icon="cubes" href="modules">
|
||||
Every module with code examples.
|
||||
</Card>
|
||||
<Card title="Core Module" icon="gear" href="reference/core">
|
||||
Framework orchestration internals.
|
||||
</Card>
|
||||
<Card title="Pipeline" icon="arrows-turn-to-dots" href="reference/pipeline">
|
||||
Pipeline DSL and execution model.
|
||||
</Card>
|
||||
<Card title="Contributing" icon="code-pull-request" href="contributing">
|
||||
How to extend the framework.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -68,9 +68,7 @@
|
||||
{
|
||||
"group": "Guides",
|
||||
"pages": [
|
||||
"examples",
|
||||
"architecture",
|
||||
"deep-dive",
|
||||
"learning-more"
|
||||
]
|
||||
},
|
||||
@@ -91,7 +89,6 @@
|
||||
{
|
||||
"group": "Vector Stores",
|
||||
"pages": [
|
||||
"vector_store_usage",
|
||||
"vector_stores/pgvector"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,253 +0,0 @@
|
||||
---
|
||||
title: "Examples"
|
||||
description: "Code examples organized by complexity — beginner through production."
|
||||
icon: "code"
|
||||
---
|
||||
|
||||
> Code examples organized by complexity. For interactive notebooks, see the [Cookbook](cookbook).
|
||||
|
||||
---
|
||||
|
||||
## Beginner
|
||||
|
||||
### Basic Knowledge Graph
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
ingestor = FileIngestor()
|
||||
parser = DocumentParser()
|
||||
ner = NERExtractor()
|
||||
rel = RelationExtractor()
|
||||
|
||||
sources = ingestor.ingest("data/sample.pdf")
|
||||
parsed = parser.parse(sources[0])
|
||||
|
||||
entities = ner.extract(parsed)
|
||||
relationships = rel.extract(parsed, entities=entities)
|
||||
|
||||
kg = GraphBuilder(merge_entities=True).build(
|
||||
entities=entities, relationships=relationships
|
||||
)
|
||||
print(f"{len(kg.nodes)} nodes, {len(kg.edges)} edges")
|
||||
```
|
||||
|
||||
### Entity Extraction from Text
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import NERExtractor
|
||||
|
||||
ner = NERExtractor()
|
||||
entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
|
||||
|
||||
for entity in entities:
|
||||
print(f"{entity['text']}: {entity['type']}")
|
||||
# Apple Inc.: ORGANIZATION
|
||||
# Steve Jobs: PERSON
|
||||
# 1976: DATE
|
||||
```
|
||||
|
||||
### Custom NER with LLM
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import NERExtractor
|
||||
from semantica.llms import OpenAI
|
||||
|
||||
llm = OpenAI(model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY"))
|
||||
ner = NERExtractor(method="llm", llm_provider=llm, confidence_threshold=0.8)
|
||||
entities = ner.extract("Your document text here...")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Intermediate
|
||||
|
||||
### Multi-Source Integration
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
ingestor = FileIngestor()
|
||||
parser = DocumentParser()
|
||||
ner = NERExtractor()
|
||||
rel = RelationExtractor()
|
||||
builder = GraphBuilder(merge_entities=True)
|
||||
|
||||
all_entities, all_rels = [], []
|
||||
|
||||
for path in ["source1.pdf", "source2.pdf", "source3.pdf"]:
|
||||
sources = ingestor.ingest(path)
|
||||
parsed = parser.parse(sources[0])
|
||||
all_entities.extend(ner.extract(parsed))
|
||||
all_rels.extend(rel.extract(parsed, entities=all_entities))
|
||||
|
||||
kg = builder.build(entities=all_entities, relationships=all_rels)
|
||||
print(f"Unified graph: {len(kg.nodes)} nodes, {len(kg.edges)} edges")
|
||||
```
|
||||
|
||||
### Conflict Detection and Resolution
|
||||
|
||||
```python
|
||||
from semantica.conflicts import ConflictDetector, ConflictResolver
|
||||
|
||||
detector = ConflictDetector()
|
||||
conflicts = detector.detect_conflicts(all_entities)
|
||||
|
||||
resolver = ConflictResolver(default_strategy="voting")
|
||||
resolved = resolver.resolve_conflicts(conflicts)
|
||||
|
||||
print(f"Detected {len(conflicts)} conflicts, resolved {len(resolved)}")
|
||||
```
|
||||
|
||||
### Parquet and XML Ingestion (v0.5.0)
|
||||
|
||||
```python
|
||||
from semantica.ingest import ParquetIngestor, XMLIngestor
|
||||
|
||||
parquet_data = ParquetIngestor().ingest("data/records.parquet")
|
||||
xml_data = XMLIngestor(safe_mode=True).ingest("data/feed.xml")
|
||||
```
|
||||
|
||||
### Persistent Storage — Neo4j
|
||||
|
||||
```python
|
||||
from semantica.graph_store import GraphStore
|
||||
|
||||
store = GraphStore(
|
||||
backend="neo4j",
|
||||
uri="bolt://localhost:7687",
|
||||
user="neo4j",
|
||||
password="password",
|
||||
)
|
||||
store.connect()
|
||||
|
||||
apple = store.create_node(labels=["Company"], properties={"name": "Apple Inc."})
|
||||
tim = store.create_node(labels=["Person"], properties={"name": "Tim Cook"})
|
||||
store.create_relationship(
|
||||
start_node_id=tim["id"],
|
||||
end_node_id=apple["id"],
|
||||
rel_type="CEO_OF",
|
||||
)
|
||||
store.close()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced
|
||||
|
||||
### GraphRAG with Reasoning
|
||||
|
||||
```python
|
||||
from semantica.context import AgentContext
|
||||
from semantica.reasoning import Reasoner
|
||||
|
||||
context = AgentContext(
|
||||
vector_store=vs,
|
||||
knowledge_graph=kg,
|
||||
graph_expansion=True,
|
||||
hybrid_alpha=0.7,
|
||||
)
|
||||
|
||||
reasoner = Reasoner()
|
||||
reasoner.add_rule("IF Library(?x) AND Language(?y) THEN TechStackItem(?x)")
|
||||
inferred = reasoner.infer_facts(kg.get_all_triplets())
|
||||
|
||||
for fact in inferred:
|
||||
kg.add_fact_from_string(fact)
|
||||
|
||||
results = context.retrieve("What technologies are used in this project?")
|
||||
```
|
||||
|
||||
### Temporal Knowledge Graph (v0.4.0)
|
||||
|
||||
```python
|
||||
from semantica.kg import TemporalKnowledgeGraph
|
||||
|
||||
tkg = TemporalKnowledgeGraph()
|
||||
tkg.add_temporal_fact("Apple", "CEO", "Tim Cook", valid_from="2011-08-24")
|
||||
tkg.add_temporal_fact("Apple", "CEO", "Steve Jobs", valid_from="1997-09-16", valid_to="2011-08-24")
|
||||
|
||||
ceo_2005 = tkg.query_at("Apple", "CEO", timestamp="2005-01-01")
|
||||
```
|
||||
|
||||
### Distance Intelligence (v0.5.0)
|
||||
|
||||
```python
|
||||
from semantica.kg import DistanceCalculator
|
||||
|
||||
calc = DistanceCalculator(kg)
|
||||
dist = calc.calculate("Apple Inc.", "Microsoft")
|
||||
|
||||
print(f"Distance: {dist.score:.3f} — Band: {dist.band}")
|
||||
similar = calc.find_similar("Apple Inc.", radius=0.3)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Production
|
||||
|
||||
### Batch Processing (Large Datasets)
|
||||
|
||||
```python
|
||||
from semantica.pipeline import Pipeline
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.semantic_extract import NERExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
pipeline = Pipeline(workers=4)
|
||||
pipeline.add_step("ingest", FileIngestor())
|
||||
pipeline.add_step("parse", DocumentParser())
|
||||
pipeline.add_step("extract", NERExtractor(), parallel=True, batch_size=50)
|
||||
pipeline.add_step("build", GraphBuilder())
|
||||
|
||||
result = pipeline.run("data/")
|
||||
print(f"Processed: {result.processed_count}, Failed: {result.failed_count}")
|
||||
```
|
||||
|
||||
### Real-Time Streaming
|
||||
|
||||
```python
|
||||
from semantica.ingest import StreamIngestor
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
stream = StreamIngestor(stream_uri="kafka://localhost:9092/topic")
|
||||
ner = NERExtractor()
|
||||
rel = RelationExtractor()
|
||||
builder = GraphBuilder()
|
||||
|
||||
for batch in stream.stream(batch_size=100):
|
||||
all_entities, all_rels = [], []
|
||||
for item in batch:
|
||||
text = str(item)
|
||||
all_entities.extend(ner.extract(text))
|
||||
all_rels.extend(rel.extract(text, entities=all_entities))
|
||||
kg = builder.build(entities=all_entities, relationships=all_rels)
|
||||
print(f"Processed batch: {len(kg.nodes)} nodes")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Quickstart" icon="play" href="quickstart">
|
||||
Step-by-step first pipeline tutorial.
|
||||
</Card>
|
||||
<Card title="Cookbook" icon="book-open" href="cookbook">
|
||||
Interactive Jupyter notebook tutorials.
|
||||
</Card>
|
||||
<Card title="Use Cases" icon="briefcase" href="use-cases">
|
||||
Domain-specific examples.
|
||||
</Card>
|
||||
<Card title="API Reference" icon="code" href="reference/core">
|
||||
Complete API documentation.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,12 +0,0 @@
|
||||
[build]
|
||||
command = "pip install -r requirements-docs.txt && mkdocs build"
|
||||
publish = "site"
|
||||
|
||||
[build.environment]
|
||||
PYTHON_VERSION = "3.11"
|
||||
|
||||
[[redirects]]
|
||||
from = "/*"
|
||||
to = "/index.html"
|
||||
status = 200
|
||||
|
||||
+1
-1
@@ -428,7 +428,7 @@ pip install --upgrade semantica
|
||||
<Card title="Core Concepts" icon="book-open" href="concepts">
|
||||
Knowledge graphs, ontologies, reasoning engines — the mental model behind Semantica.
|
||||
</Card>
|
||||
<Card title="Examples" icon="code" href="examples">
|
||||
<Card title="Cookbook" icon="code" href="cookbook">
|
||||
15+ copy-paste examples for healthcare, finance, legal, and cybersecurity.
|
||||
</Card>
|
||||
<Card title="API Reference" icon="rectangle-terminal" href="reference/context">
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
---
|
||||
title: "Vector Store: High-Performance Usage"
|
||||
description: "Parallel ingestion, batch processing, and performance tuning for the Semantica Vector Store."
|
||||
icon: "bolt"
|
||||
---
|
||||
|
||||
> High-performance batch ingestion with parallel embedding generation — 3–10× faster than sequential processing.
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Parallel ingestion** — multi-threaded embedding generation and storage
|
||||
- **Batch processing** — minimizes overhead by grouping documents into chunks
|
||||
- **Unified API** — `add_documents` handles embedding generation and storage in one call
|
||||
|
||||
---
|
||||
|
||||
## Quick Start: Parallel Ingestion
|
||||
|
||||
```python
|
||||
from semantica.vector_store import VectorStore
|
||||
import time
|
||||
|
||||
store = VectorStore(backend="faiss", dimension=768)
|
||||
|
||||
documents = [f"This is document number {i} with some content." for i in range(1000)]
|
||||
metadata = [{"source": "generated", "id": i} for i in range(1000)]
|
||||
|
||||
start = time.time()
|
||||
ids = store.add_documents(
|
||||
documents=documents,
|
||||
metadata=metadata,
|
||||
batch_size=64,
|
||||
parallel=True, # default: True
|
||||
)
|
||||
print(f"Ingested {len(ids)} documents in {time.time() - start:.2f}s")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Comparison
|
||||
|
||||
**Old method (sequential loop)** — slower due to per-item overhead:
|
||||
|
||||
```python
|
||||
for doc in documents:
|
||||
emb = embedder.generate(doc)
|
||||
store.store_vectors([emb], [{"text": doc}])
|
||||
```
|
||||
|
||||
**New method (parallel batching)** — 3–10× faster:
|
||||
|
||||
```python
|
||||
store.add_documents(documents, parallel=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration and Tuning
|
||||
|
||||
### `max_workers`
|
||||
|
||||
Number of concurrent threads for embedding generation.
|
||||
|
||||
- **Default**: 6 (optimized for most systems)
|
||||
- Override only if you have very high core counts or specific throughput needs
|
||||
|
||||
```python
|
||||
store = VectorStore(max_workers=16)
|
||||
```
|
||||
|
||||
### `batch_size`
|
||||
|
||||
Number of documents processed in a single chunk.
|
||||
|
||||
- **Default**: 32
|
||||
- **Local models**: 32–64 works well
|
||||
- **API models (OpenAI, etc.)**: 100–200 reduces network latency overhead
|
||||
|
||||
```python
|
||||
store.add_documents(documents, batch_size=100)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manual Batch Embedding
|
||||
|
||||
If you need embeddings without immediately storing them:
|
||||
|
||||
```python
|
||||
vectors = store.embed_batch(texts=documents[:100])
|
||||
print(f"Generated {len(vectors)} vectors")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
<Tip>
|
||||
- **Metadata consistency** — ensure your `metadata` list is the same length as `documents`.
|
||||
- **Error handling** — `add_documents` propagates exceptions if embedding fails; validate your data first.
|
||||
- **Memory usage** — very large `batch_size` combined with high `max_workers` increases RAM usage. Monitor system resources for large corpora.
|
||||
</Tip>
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Vector Store Reference" icon="vector-square" href="reference/vector_store">
|
||||
Full VectorStore API with all backends.
|
||||
</Card>
|
||||
<Card title="Embeddings" icon="brain" href="reference/embeddings">
|
||||
Embedding providers and GPU acceleration.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
+233
@@ -0,0 +1,233 @@
|
||||
"""Docs integrity checker for PR #561."""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import glob
|
||||
|
||||
DOCS_DIR = "docs"
|
||||
|
||||
results = {"pass": [], "fail": []}
|
||||
|
||||
|
||||
def ok(msg):
|
||||
results["pass"].append(msg)
|
||||
print(f" PASS {msg}")
|
||||
|
||||
|
||||
def fail(msg):
|
||||
results["fail"].append(msg)
|
||||
print(f" FAIL {msg}")
|
||||
|
||||
|
||||
# ── 1. docs.json valid JSON ───────────────────────────────────────────────────
|
||||
print("\n[1] docs.json validity")
|
||||
try:
|
||||
with open(os.path.join(DOCS_DIR, "docs.json"), encoding="utf-8") as f:
|
||||
cfg = json.load(f)
|
||||
ok("docs.json is valid JSON")
|
||||
except Exception as e:
|
||||
fail(f"docs.json parse error: {e}")
|
||||
cfg = {}
|
||||
|
||||
|
||||
# ── 2. All nav pages exist on disk ────────────────────────────────────────────
|
||||
print("\n[2] Nav pages exist on disk")
|
||||
|
||||
|
||||
def collect_pages(obj):
|
||||
pages = []
|
||||
if isinstance(obj, dict):
|
||||
if "pages" in obj:
|
||||
for p in obj["pages"]:
|
||||
if isinstance(p, str):
|
||||
pages.append(p)
|
||||
else:
|
||||
pages.extend(collect_pages(p))
|
||||
for v in obj.values():
|
||||
if isinstance(v, (dict, list)):
|
||||
pages.extend(collect_pages(v))
|
||||
elif isinstance(obj, list):
|
||||
for item in obj:
|
||||
pages.extend(collect_pages(item))
|
||||
return pages
|
||||
|
||||
|
||||
nav_pages = [p for p in set(collect_pages(cfg)) if not p.startswith("http")]
|
||||
missing_pages = []
|
||||
for p in sorted(nav_pages):
|
||||
path = os.path.join(DOCS_DIR, p + ".md")
|
||||
if not os.path.exists(path):
|
||||
missing_pages.append(p)
|
||||
|
||||
if missing_pages:
|
||||
for m in missing_pages:
|
||||
fail(f"Nav page missing: {m}")
|
||||
else:
|
||||
ok(f"All {len(nav_pages)} nav pages exist on disk")
|
||||
|
||||
|
||||
# ── 3. Internal Card hrefs resolve (page-relative) ───────────────────────────
|
||||
print("\n[3] Internal Card hrefs")
|
||||
broken_hrefs = []
|
||||
|
||||
for fpath in glob.glob(DOCS_DIR + "/**/*.md", recursive=True):
|
||||
file_dir = os.path.dirname(fpath)
|
||||
with open(fpath, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
for m in re.finditer(r'href=["\'](?!http)([^"\'#]+)["\']', content):
|
||||
href = m.group(1).strip()
|
||||
if not href:
|
||||
continue
|
||||
# Resolve relative to the file's directory (mirrors browser URL resolution)
|
||||
resolved = os.path.normpath(os.path.join(file_dir, href))
|
||||
target_md = resolved + ".md"
|
||||
if not os.path.exists(target_md):
|
||||
rel = fpath.replace("\\", "/")
|
||||
broken_hrefs.append(f"{rel}: href '{href}' -> {target_md}")
|
||||
|
||||
if broken_hrefs:
|
||||
for b in broken_hrefs[:20]:
|
||||
fail(f"Broken href: {b}")
|
||||
if len(broken_hrefs) > 20:
|
||||
fail(f"...and {len(broken_hrefs) - 20} more broken hrefs")
|
||||
else:
|
||||
ok("All internal Card hrefs resolve to real files")
|
||||
|
||||
|
||||
# ── 4. No old repo URLs ───────────────────────────────────────────────────────
|
||||
print("\n[4] Repo URL consistency")
|
||||
old_patterns = ["Hawksight-AI/semantica", "semantica-dev/semantica"]
|
||||
old_url_hits = []
|
||||
for fpath in glob.glob(DOCS_DIR + "/**/*.md", recursive=True) + [
|
||||
os.path.join(DOCS_DIR, "docs.json")
|
||||
]:
|
||||
with open(fpath, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
for pat in old_patterns:
|
||||
if pat in content:
|
||||
old_url_hits.append(f"{fpath}: contains '{pat}'")
|
||||
|
||||
if old_url_hits:
|
||||
for h in old_url_hits:
|
||||
fail(h)
|
||||
else:
|
||||
ok("No old repo URLs found (Hawksight-AI, semantica-dev)")
|
||||
|
||||
|
||||
# ── 5. All reference .md files have frontmatter ───────────────────────────────
|
||||
print("\n[5] Reference page frontmatter")
|
||||
ref_pages = list(glob.glob(DOCS_DIR + "/reference/*.md"))
|
||||
no_frontmatter = []
|
||||
for fpath in ref_pages:
|
||||
with open(fpath, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
if not content.startswith("---"):
|
||||
no_frontmatter.append(os.path.basename(fpath))
|
||||
|
||||
if no_frontmatter:
|
||||
for f in no_frontmatter:
|
||||
fail(f"Missing frontmatter: {f}")
|
||||
else:
|
||||
ok(f"All {len(ref_pages)} reference pages have frontmatter")
|
||||
|
||||
|
||||
# ── 6. Code examples: no non-existent class names (exact word match) ──────────
|
||||
print("\n[6] Known-wrong class names")
|
||||
# (symbol, file, exclude_pattern) — exclude_pattern avoids substring false positives
|
||||
banned = [
|
||||
("BaseIngestor", "docs/architecture.md", None),
|
||||
("BaseExtractor", "docs/architecture.md", None),
|
||||
("BasePlugin", "docs/architecture.md", None),
|
||||
(r"PluginRegistry\.register\(", "docs/architecture.md", r"register_plugin"),
|
||||
("start_explorer", "docs/reference/explorer.md", None),
|
||||
(r"graph\.save\(", "docs/reference/explorer.md", None),
|
||||
(r"\bDataNormalizer\b", "docs/reference/normalize.md", None),
|
||||
(r"\bEntityResolver\b", "docs/reference/deduplication.md", None),
|
||||
# ReasoningEngine: only flag as exact word, not as part of TemporalReasoningEngine
|
||||
(r"(?<!Temporal)(?<!Graph)\bReasoningEngine\b", "docs/reference/reasoning.md", None),
|
||||
(r"\bDeductiveEngine\b", "docs/reference/reasoning.md", None),
|
||||
(r"\bAbductiveEngine\b", "docs/reference/reasoning.md", None),
|
||||
(r"\bArangoExporter\b", "docs/reference/export.md", r"ArangoAQLExporter"),
|
||||
(r"\bGraphMLExporter\b", "docs/reference/export.md", None),
|
||||
]
|
||||
wrong_hits = []
|
||||
for item in banned:
|
||||
symbol, fpath, exclude = item[0], item[1], item[2]
|
||||
if not os.path.exists(fpath):
|
||||
continue
|
||||
with open(fpath, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
matches = re.findall(symbol, content)
|
||||
if matches:
|
||||
# Apply exclusion filter
|
||||
if exclude:
|
||||
matches = [m for m in re.finditer(symbol, content)
|
||||
if exclude not in content[max(0, m.start()-30):m.end()+30]]
|
||||
if not matches:
|
||||
continue
|
||||
wrong_hits.append(f"{fpath}: contains pattern '{symbol}'")
|
||||
|
||||
if wrong_hits:
|
||||
for w in wrong_hits:
|
||||
fail(w)
|
||||
else:
|
||||
ok("No known-wrong class names in fixed reference pages")
|
||||
|
||||
|
||||
# ── 7. Python 3.8 compat in docs snippets ─────────────────────────────────────
|
||||
print("\n[7] Python 3.8 typing compatibility in docs code blocks")
|
||||
# Only check inside ```python code blocks
|
||||
py39_hits = []
|
||||
in_code_block = False
|
||||
for fpath in glob.glob(DOCS_DIR + "/**/*.md", recursive=True):
|
||||
with open(fpath, encoding="utf-8") as f:
|
||||
lines = f.readlines()
|
||||
in_block = False
|
||||
for i, line in enumerate(lines, 1):
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("```"):
|
||||
in_block = not in_block
|
||||
if in_block and re.search(r':\s*(list|dict|tuple|set)\[', line):
|
||||
rel = fpath.replace("\\", "/")
|
||||
py39_hits.append(f"{rel}:{i}: {line.rstrip()}")
|
||||
|
||||
if py39_hits:
|
||||
for h in py39_hits[:10]:
|
||||
fail(f"Py3.9+ syntax: {h}")
|
||||
if len(py39_hits) > 10:
|
||||
fail(f"...and {len(py39_hits) - 10} more")
|
||||
else:
|
||||
ok("No Python 3.9+ lowercase generic type hints in doc code blocks")
|
||||
|
||||
|
||||
# ── 8. Module table covers all 27 modules ─────────────────────────────────────
|
||||
print("\n[8] Module table coverage in index.md")
|
||||
expected_modules = [
|
||||
"semantica.ingest", "semantica.parse", "semantica.split", "semantica.normalize",
|
||||
"semantica.semantic_extract", "semantica.kg", "semantica.ontology", "semantica.reasoning",
|
||||
"semantica.embeddings", "semantica.vector_store", "semantica.graph_store", "semantica.triplet_store",
|
||||
"semantica.context", "semantica.provenance", "semantica.change_management",
|
||||
"semantica.deduplication", "semantica.conflicts", "semantica.export", "semantica.visualization",
|
||||
"semantica.pipeline", "semantica.seed", "semantica.llms", "semantica.mcp_server",
|
||||
"semantica.explorer", "semantica.evals", "semantica.utils", "semantica.core",
|
||||
]
|
||||
index_path = os.path.join(DOCS_DIR, "index.md")
|
||||
with open(index_path, encoding="utf-8") as f:
|
||||
index_content = f.read()
|
||||
|
||||
missing_modules = [m for m in expected_modules if m not in index_content]
|
||||
if missing_modules:
|
||||
for m in missing_modules:
|
||||
fail(f"Module missing from index table: {m}")
|
||||
else:
|
||||
ok(f"All {len(expected_modules)} modules present in index.md table")
|
||||
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────────────────
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Results: {len(results['pass'])} passed, {len(results['fail'])} failed")
|
||||
if results["fail"]:
|
||||
print("STATUS: FAIL")
|
||||
raise SystemExit(1)
|
||||
else:
|
||||
print("STATUS: ALL CHECKS PASSED")
|
||||
-161
@@ -1,161 +0,0 @@
|
||||
# Deployment Trigger: Public GitHub Pages
|
||||
site_name: Semantica
|
||||
site_description: Open Source Framework for Semantic Intelligence & Knowledge Engineering
|
||||
site_url: https://hawksight-ai.github.io/semantica/
|
||||
repo_url: https://github.com/Hawksight-AI/semantica
|
||||
repo_name: Hawksight-AI/semantica
|
||||
edit_uri: edit/main/docs/
|
||||
|
||||
# Copyright
|
||||
copyright: Copyright © 2026 Hawksight AI
|
||||
|
||||
# Theme Configuration
|
||||
theme:
|
||||
name: material
|
||||
palette:
|
||||
# Light mode
|
||||
- scheme: default
|
||||
primary: custom
|
||||
accent: custom
|
||||
toggle:
|
||||
icon: material/brightness-7
|
||||
name: Switch to dark mode
|
||||
# Dark mode
|
||||
- scheme: slate
|
||||
primary: custom
|
||||
accent: custom
|
||||
toggle:
|
||||
icon: material/brightness-4
|
||||
name: Switch to light mode
|
||||
features:
|
||||
- navigation.tabs
|
||||
- navigation.sections
|
||||
- navigation.expand
|
||||
- navigation.top
|
||||
- navigation.indexes
|
||||
- navigation.tracking
|
||||
- search.suggest
|
||||
- search.highlight
|
||||
- search.share
|
||||
- content.code.copy
|
||||
- content.code.annotate
|
||||
- content.tooltips
|
||||
icon:
|
||||
logo: material/brain
|
||||
repo: fontawesome/brands/github
|
||||
|
||||
# Extensions
|
||||
markdown_extensions:
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
line_spans: __span
|
||||
pygments_lang_class: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.snippets:
|
||||
base_path: ["."]
|
||||
- pymdownx.superfences:
|
||||
custom_fences:
|
||||
- name: mermaid
|
||||
class: mermaid
|
||||
format: !!python/name:pymdownx.superfences.fence_code_format
|
||||
- pymdownx.emoji:
|
||||
emoji_index: !!python/name:material.extensions.emoji.twemoji
|
||||
emoji_generator: !!python/name:material.extensions.emoji.to_svg
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- pymdownx.tasklist:
|
||||
custom_checkbox: true
|
||||
- admonition
|
||||
- pymdownx.details
|
||||
- attr_list
|
||||
- md_in_html
|
||||
- tables
|
||||
- toc:
|
||||
permalink: true
|
||||
toc_depth: 6
|
||||
|
||||
# Plugins
|
||||
plugins:
|
||||
- search:
|
||||
lang: en
|
||||
- minify:
|
||||
minify_html: true
|
||||
- mkdocstrings:
|
||||
handlers:
|
||||
python:
|
||||
options:
|
||||
docstring_style: google
|
||||
show_source: true
|
||||
show_root_heading: true
|
||||
show_category_heading: true
|
||||
- mkdocs-jupyter:
|
||||
include_source: true
|
||||
|
||||
# Custom CSS
|
||||
extra_css:
|
||||
- css/custom.css
|
||||
|
||||
# Custom JavaScript
|
||||
extra_javascript: []
|
||||
|
||||
# Navigation
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Getting Started:
|
||||
- Overview: getting-started.md
|
||||
- installation.md
|
||||
- quickstart.md
|
||||
- Docs:
|
||||
- Change Management: reference/change_management.md
|
||||
- Conflicts: reference/conflicts.md
|
||||
- Context: reference/context.md
|
||||
- Core: reference/core.md
|
||||
- Deduplication: reference/deduplication.md
|
||||
- Embeddings: reference/embeddings.md
|
||||
- Evals: reference/evals.md
|
||||
- Export: reference/export.md
|
||||
- Graph Store: reference/graph_store.md
|
||||
- Ingest: reference/ingest.md
|
||||
- Knowledge Graph: reference/kg.md
|
||||
- LLMs: reference/llms.md
|
||||
- Normalize: reference/normalize.md
|
||||
- Ontology: reference/ontology.md
|
||||
- Parse: reference/parse.md
|
||||
- Pipeline: reference/pipeline.md
|
||||
- Provenance: reference/provenance.md
|
||||
- Reasoning: reference/reasoning.md
|
||||
- Seed: reference/seed.md
|
||||
- Semantic Extract: reference/semantic_extract.md
|
||||
- Split: reference/split.md
|
||||
- Triplet Store: reference/triplet_store.md
|
||||
- Utils: reference/utils.md
|
||||
- Vector Store: reference/vector_store.md
|
||||
- Visualization: reference/visualization.md
|
||||
- Guides:
|
||||
- concepts.md
|
||||
- modules.md
|
||||
- use-cases.md
|
||||
- examples.md
|
||||
- glossary.md
|
||||
- Integrations:
|
||||
- Agno: integrations/agno.md
|
||||
- Docling: integrations/docling.md
|
||||
- Snowflake: integrations/snowflake.md
|
||||
- Cookbook: cookbook.md
|
||||
- Resources:
|
||||
- community.md
|
||||
- contributing.md
|
||||
- faq.md
|
||||
- license.md
|
||||
|
||||
# Extra
|
||||
extra:
|
||||
social:
|
||||
- icon: fontawesome/brands/github
|
||||
link: https://github.com/Hawksight-AI/semantica
|
||||
- icon: fontawesome/brands/python
|
||||
link: https://pypi.org/project/semantica/
|
||||
version:
|
||||
provider: mike
|
||||
generator: true
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
INHERIT: mkdocs.yml
|
||||
plugins:
|
||||
- search:
|
||||
lang: en
|
||||
- minify:
|
||||
minify_html: true
|
||||
- mkdocstrings:
|
||||
handlers:
|
||||
python:
|
||||
options:
|
||||
docstring_style: google
|
||||
show_source: true
|
||||
show_root_heading: true
|
||||
show_category_heading: true
|
||||
@@ -1,9 +0,0 @@
|
||||
mkdocs>=1.6.1
|
||||
mkdocs-material>=9.7.6
|
||||
mkdocs-minify-plugin>=0.7.0
|
||||
mkdocs-mermaid2-plugin>=1.2.3
|
||||
pymdown-extensions>=10.21.2
|
||||
|
||||
mkdocstrings[python]>=0.24.0
|
||||
mkdocs-jupyter>=0.26.3
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
import os
|
||||
import shutil
|
||||
|
||||
# Create directories
|
||||
os.makedirs("docs/reference", exist_ok=True)
|
||||
os.makedirs("docs/cookbook", exist_ok=True)
|
||||
|
||||
# Modules to generate docs for
|
||||
modules = {
|
||||
"core": "semantica.core",
|
||||
"ingest": "semantica.ingest",
|
||||
"parse": "semantica.parse",
|
||||
"normalize": "semantica.normalize",
|
||||
"semantic_extract": "semantica.semantic_extract",
|
||||
"kg": "semantica.kg",
|
||||
"embeddings": "semantica.embeddings",
|
||||
"vector_store": "semantica.vector_store",
|
||||
"triplet_store": "semantica.triplet_store",
|
||||
"ontology": "semantica.ontology",
|
||||
"reasoning": "semantica.reasoning",
|
||||
"pipeline": "semantica.pipeline",
|
||||
"export": "semantica.export",
|
||||
"visualization": "semantica.visualization",
|
||||
"utils": "semantica.utils"
|
||||
}
|
||||
|
||||
# Generate reference markdown files
|
||||
for name, package in modules.items():
|
||||
content = f"# {name.replace('_', ' ').title()}\n\n::: {package}\n"
|
||||
with open(f"docs/reference/{name}.md", "w") as f:
|
||||
f.write(content)
|
||||
print(f"Created docs/reference/{name}.md")
|
||||
|
||||
# Copy cookbook directory
|
||||
if os.path.exists("cookbook"):
|
||||
if os.path.exists("docs/cookbook"):
|
||||
shutil.rmtree("docs/cookbook")
|
||||
shutil.copytree("cookbook", "docs/cookbook")
|
||||
print("Copied cookbook to docs/cookbook")
|
||||
|
||||
# Remove old files
|
||||
files_to_remove = ["docs/MODULES_DOCUMENTATION.md", "docs/cookbook.md", "docs/api.md"]
|
||||
for f in files_to_remove:
|
||||
if os.path.exists(f):
|
||||
os.remove(f)
|
||||
print(f"Removed {f}")
|
||||
Reference in New Issue
Block a user