fix: correct remaining API mismatches in pipeline, vector_store, and normalize docs

- pipeline.md: ParallelismManager pool_type="thread"/"process" → use_processes=False/True;
  execute_parallel() returns List[ParallelExecutionResult] not aggregate object
- vector_store.md: remove MetadataStore.add_field() (method is on MetadataSchema, not
  MetadataStore); fix tip to reference MetadataStore.update_metadata() not VectorStore
- normalize.md: Pipeline() orchestrator misuse → PipelineBuilder + ExecutionEngine pattern
This commit is contained in:
KaifAhmad1
2026-05-24 13:14:01 +05:30
parent 6f726c708f
commit ff43887842
3 changed files with 27 additions and 23 deletions
+13 -7
View File
@@ -360,21 +360,27 @@ print(f"Invalid: {result.error_count}")
## Pipeline Integration
```python
from semantica.pipeline import Pipeline
from semantica.pipeline import PipelineBuilder, ExecutionEngine
from semantica.ingest import FileIngestor
from semantica.normalize import TextNormalizer
from semantica.semantic_extract import NERExtractor
from semantica.llms import Groq
import os
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
ingestor = FileIngestor()
normalizer = TextNormalizer(strip_html=True, normalize_unicode=True)
extractor = NERExtractor(method="llm", llm_provider=llm)
pipeline = Pipeline()
pipeline.add_step("ingest", FileIngestor())
pipeline.add_step("normalize", TextNormalizer(strip_html=True, normalize_unicode=True))
pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm))
builder = PipelineBuilder()
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest)
builder.add_step("normalize", "text_normalize", handler=normalizer.normalize)
builder.add_step("extract", "ner_extract", handler=extractor.extract)
builder.connect_steps("ingest", "normalize")
builder.connect_steps("normalize", "extract")
result = pipeline.run("data/documents/")
pipeline = builder.build("normalize_pipeline")
result = ExecutionEngine().execute_pipeline(pipeline, data="data/documents/")
```
## Tips and Common Pitfalls
+13 -9
View File
@@ -405,25 +405,29 @@ Checks performed:
```python
from semantica.pipeline import ParallelismManager
manager = ParallelismManager(max_workers=8, pool_type="thread")
# use_processes=False (default) → thread pool for I/O-bound tasks
manager = ParallelismManager(max_workers=8, use_processes=False)
tasks = [{"fn": ner.extract, "args": [text]} for text in texts]
result = manager.execute_parallel(tasks, timeout=60)
tasks = [{"fn": ner.extract, "args": [text]} for text in texts]
results = manager.execute_parallel(tasks, timeout=60)
# returns List[ParallelExecutionResult]
print(f"Successful: {result.success_count}, Failed: {result.failure_count}")
successes = [r for r in results if r.success]
failures = [r for r in results if not r.success]
```
Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls. Threads share memory and context-switch cheaply between waiting operations.
Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls.
</Tab>
<Tab title="Process pool (CPU-bound)">
```python
manager = ParallelismManager(max_workers=4, pool_type="process")
# use_processes=True → process pool, bypasses Python GIL
manager = ParallelismManager(max_workers=4, use_processes=True)
tasks = [{"fn": embedder.embed, "args": [chunk]} for chunk in chunks]
result = manager.execute_parallel(tasks, timeout=120)
tasks = [{"fn": embedder.generate_embeddings, "args": [chunk]} for chunk in chunks]
results = manager.execute_parallel(tasks, timeout=120)
```
Use process pools for **CPU-bound** steps: embedding computation, OCR, large NER batches. Processes bypass Python's GIL for true multi-core parallelism.
Use process pools for **CPU-bound** steps: embedding, OCR, large NER batches.
</Tab>
</Tabs>
+1 -7
View File
@@ -317,12 +317,6 @@ from semantica.vector_store import MetadataStore
meta_store = MetadataStore()
# Define schema fields
meta_store.add_field("author", str, required=True)
meta_store.add_field("year", int, required=True)
meta_store.add_field("category", str)
meta_store.add_field("score", float, default=0.0)
# Store and retrieve metadata
meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"})
meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"})
@@ -423,7 +417,7 @@ store = VectorStore(backend="faiss", dimension=768, metric="cosine")
</Warning>
<Tip>
**Update metadata without re-embedding.** `store.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
**Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
</Tip>
<CardGroup cols={2}>