mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-15 04:00:33 +00:00
fix: correct remaining API mismatches in pipeline, vector_store, and normalize docs
- pipeline.md: ParallelismManager pool_type="thread"/"process" → use_processes=False/True; execute_parallel() returns List[ParallelExecutionResult] not aggregate object - vector_store.md: remove MetadataStore.add_field() (method is on MetadataSchema, not MetadataStore); fix tip to reference MetadataStore.update_metadata() not VectorStore - normalize.md: Pipeline() orchestrator misuse → PipelineBuilder + ExecutionEngine pattern
This commit is contained in:
@@ -360,21 +360,27 @@ print(f"Invalid: {result.error_count}")
|
||||
## Pipeline Integration
|
||||
|
||||
```python
|
||||
from semantica.pipeline import Pipeline
|
||||
from semantica.pipeline import PipelineBuilder, ExecutionEngine
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.normalize import TextNormalizer
|
||||
from semantica.semantic_extract import NERExtractor
|
||||
from semantica.llms import Groq
|
||||
import os
|
||||
|
||||
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
|
||||
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
|
||||
ingestor = FileIngestor()
|
||||
normalizer = TextNormalizer(strip_html=True, normalize_unicode=True)
|
||||
extractor = NERExtractor(method="llm", llm_provider=llm)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_step("ingest", FileIngestor())
|
||||
pipeline.add_step("normalize", TextNormalizer(strip_html=True, normalize_unicode=True))
|
||||
pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm))
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest)
|
||||
builder.add_step("normalize", "text_normalize", handler=normalizer.normalize)
|
||||
builder.add_step("extract", "ner_extract", handler=extractor.extract)
|
||||
builder.connect_steps("ingest", "normalize")
|
||||
builder.connect_steps("normalize", "extract")
|
||||
|
||||
result = pipeline.run("data/documents/")
|
||||
pipeline = builder.build("normalize_pipeline")
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data="data/documents/")
|
||||
```
|
||||
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
@@ -405,25 +405,29 @@ Checks performed:
|
||||
```python
|
||||
from semantica.pipeline import ParallelismManager
|
||||
|
||||
manager = ParallelismManager(max_workers=8, pool_type="thread")
|
||||
# use_processes=False (default) → thread pool for I/O-bound tasks
|
||||
manager = ParallelismManager(max_workers=8, use_processes=False)
|
||||
|
||||
tasks = [{"fn": ner.extract, "args": [text]} for text in texts]
|
||||
result = manager.execute_parallel(tasks, timeout=60)
|
||||
tasks = [{"fn": ner.extract, "args": [text]} for text in texts]
|
||||
results = manager.execute_parallel(tasks, timeout=60)
|
||||
# returns List[ParallelExecutionResult]
|
||||
|
||||
print(f"Successful: {result.success_count}, Failed: {result.failure_count}")
|
||||
successes = [r for r in results if r.success]
|
||||
failures = [r for r in results if not r.success]
|
||||
```
|
||||
|
||||
Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls. Threads share memory and context-switch cheaply between waiting operations.
|
||||
Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls.
|
||||
</Tab>
|
||||
<Tab title="Process pool (CPU-bound)">
|
||||
```python
|
||||
manager = ParallelismManager(max_workers=4, pool_type="process")
|
||||
# use_processes=True → process pool, bypasses Python GIL
|
||||
manager = ParallelismManager(max_workers=4, use_processes=True)
|
||||
|
||||
tasks = [{"fn": embedder.embed, "args": [chunk]} for chunk in chunks]
|
||||
result = manager.execute_parallel(tasks, timeout=120)
|
||||
tasks = [{"fn": embedder.generate_embeddings, "args": [chunk]} for chunk in chunks]
|
||||
results = manager.execute_parallel(tasks, timeout=120)
|
||||
```
|
||||
|
||||
Use process pools for **CPU-bound** steps: embedding computation, OCR, large NER batches. Processes bypass Python's GIL for true multi-core parallelism.
|
||||
Use process pools for **CPU-bound** steps: embedding, OCR, large NER batches.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
|
||||
@@ -317,12 +317,6 @@ from semantica.vector_store import MetadataStore
|
||||
|
||||
meta_store = MetadataStore()
|
||||
|
||||
# Define schema fields
|
||||
meta_store.add_field("author", str, required=True)
|
||||
meta_store.add_field("year", int, required=True)
|
||||
meta_store.add_field("category", str)
|
||||
meta_store.add_field("score", float, default=0.0)
|
||||
|
||||
# Store and retrieve metadata
|
||||
meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"})
|
||||
meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"})
|
||||
@@ -423,7 +417,7 @@ store = VectorStore(backend="faiss", dimension=768, metric="cosine")
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Update metadata without re-embedding.** `store.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
|
||||
**Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
|
||||
</Tip>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
|
||||
Reference in New Issue
Block a user