semantica/docs/reference/vector_store.md

title, description, icon

title	description	icon
Vector Store Module	Unified interface for FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector with hybrid search.	database

semantica.vector_store provides a unified API for storing and searching vector embeddings across all major backends. Swap backends with a one-line change — no application code changes needed.

What You Get

Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector. Combine dense vector similarity with sparse keyword/BM25 filtering and configurable fusion strategies. Rich metadata indexing and schema management — query by field values without a vector. Multi-tenant namespace isolation — structural separation, not just metadata filters. Bulk add, delete, and metadata updates — automatically chunked for memory efficiency. Flat, IVF, HNSW, and PQ index types with full configuration control.

Quick Start

```python from semantica.vector_store import VectorStore

# In-memory (development)
store = VectorStore(backend="inmemory", dimension=768)

# FAISS (local production — persists to disk)
store = VectorStore(backend="faiss", dimension=768, index_path="store.faiss")
```

```python store.add_vectors( embeddings=embeddings, ids=["doc1", "doc2"], metadata=[{"title": "Document 1"}, {"title": "Document 2"}] ) ``` ```python results = store.search(query_vector, top_k=10) for r in results: print(f"{r['id']} — score: {r['score']:.3f}") print(f" metadata: {r['metadata']}") ``` ```python # Equality, range, and set filters results = store.search(query_vector, filters={ "$and": [ {"category": "research"}, {"year": {"$gte": 2022}} ] }) ```

Backends

store = VectorStore(
    backend="faiss",
    dimension=768,
    index_type="IVF",       # "Flat" | "IVF" | "HNSW" | "PQ"
    index_path="store.faiss"
)

Best for: local development, on-premise production with no external services. No API key required.

pip install "semantica[pinecone]"

store = VectorStore(
    backend="pinecone",
    dimension=768,
    api_key=os.getenv("PINECONE_API_KEY"),
    index_name="semantica-index",
    environment="us-east-1-aws"
)

pip install "semantica[weaviate]"

store = VectorStore(
    backend="weaviate",
    dimension=768,
    url="http://localhost:8080",
    class_name="Document"
)

pip install "semantica[qdrant]"

store = VectorStore(
    backend="qdrant",
    dimension=768,
    url="http://localhost:6333",
    collection_name="semantica"
)

pip install "semantica[pgvector]"

store = VectorStore(
    backend="pgvector",
    dimension=768,
    connection_string="postgresql://user:pass@localhost/db",
    table_name="embeddings"
)

See the PgVector Guide for full setup.

Hybrid Search

Combine vector similarity with keyword/metadata filters for higher precision:

results = store.hybrid_search(
    query_vector=query_embedding,
    query_text="machine learning",  # keyword component
    top_k=10,
    alpha=0.7,                      # 0.0 = keyword only, 1.0 = vector only
    filters={"category": "research", "year": {"$gte": 2022}}
)

Metadata Filtering

# Equality
results = store.search(query_vector, filters={"author": "John Smith"})

# Range
results = store.search(query_vector, filters={"date": {"$gte": "2023-01-01"}})

# Set membership
results = store.search(query_vector, filters={"tag": {"$in": ["ai", "ml"]}})

# Compound AND
results = store.search(query_vector, filters={
    "$and": [
        {"category": "research"},
        {"year": {"$gte": 2022}}
    ]
})

Namespace Isolation

Isolate vectors per tenant, project, or use case:

store = VectorStore(backend="faiss", dimension=768)

# Write to separate namespaces
store.add_vectors(embeddings_a, ids_a, namespace="tenant_a")
store.add_vectors(embeddings_b, ids_b, namespace="tenant_b")

# Search is scoped to the specified namespace
results = store.search(query_vector, namespace="tenant_a")

Batch Operations

# Batch add — automatically chunked for memory efficiency
store.add_vectors_batch(embeddings_list, ids_list, batch_size=1000)

# Batch delete
store.delete_vectors(ids=["doc1", "doc2", "doc3"])

# Update metadata without re-embedding
store.update_metadata("doc1", {"status": "archived", "reviewed": True})

Backend Comparison

Backend	Deployment	API Key	Hybrid Search	Best For
FAISS	Local	No	No	On-premise, offline
Pinecone	Cloud	Yes	Yes	Managed cloud, serverless
Weaviate	Self-hosted / Cloud	Optional	Yes	Rich metadata filtering
Qdrant	Self-hosted / Cloud	Optional	Yes	High-performance filtering
Milvus	Self-hosted	No	Yes	Large-scale production
PgVector	PostgreSQL	No	Limited	Postgres-native integration
In-memory	Process	No	No	Development, testing

HybridSearch

HybridSearch is the low-level class behind store.hybrid_search() — use it directly when you need custom result fusion logic:

from semantica.vector_store import HybridSearch, VectorStore

store  = VectorStore(backend="faiss", dimension=768)
hybrid = HybridSearch(vector_store=store)

results = hybrid.search(
    query_vector=query_embedding,
    query_text="machine learning frameworks",
    top_k=20,
    vector_weight=0.7,       # weight for vector similarity leg
    keyword_weight=0.3,      # weight for BM25/keyword leg
    fusion="rrf",            # "rrf" (Reciprocal Rank Fusion) | "weighted_avg"
    filters={"category": "research", "year": {"$gte": 2022}},
    deduplicate=True,
)

for r in results:
    print(f"{r['id']}  vector_score={r['vector_score']:.3f}  final_score={r['score']:.3f}")

Fusion strategy	Description
rrf	Reciprocal Rank Fusion — rank-based combination, robust to score scale differences
weighted_avg	Weighted average of normalised scores — requires vector_weight + keyword_weight = 1.0

MetadataStore

MetadataStore manages structured metadata attached to vectors — query by field values without a vector:

from semantica.vector_store import MetadataStore

meta_store = MetadataStore()

meta_store.register_schema({
    "author":   "str",
    "year":     "int",
    "category": "str",
    "score":    "float",
})

meta_store.add("doc1", {"author": "Alice", "year": 2024, "category": "research"})
meta_store.add("doc2", {"author": "Bob",   "year": 2023, "category": "review"})

results = meta_store.filter({"category": "research", "year": {"$gte": 2023}})
meta    = meta_store.get("doc1")
meta_store.update("doc1", {"score": 0.92})

NamespaceManager

Isolates vector collections per tenant, project, or model version:

from semantica.vector_store import NamespaceManager, VectorStore

base_store = VectorStore(backend="faiss", dimension=768)
ns_manager  = NamespaceManager(vector_store=base_store)

ns_manager.create_namespace("tenant_a", description="Customer A data")
ns_manager.create_namespace("tenant_b", description="Customer B data")

ns_manager.add_vectors("tenant_a", embeddings_a, ids_a, metadata_a)
ns_manager.add_vectors("tenant_b", embeddings_b, ids_b, metadata_b)

# Search is scoped — tenant_a never sees tenant_b's data
results = ns_manager.search("tenant_a", query_vector, top_k=10)

for ns in ns_manager.list_namespaces():
    print(f"{ns['name']}: {ns['vector_count']} vectors")

ns_manager.delete_namespace("tenant_a")

FAISS Index Type Reference

Index	Memory	Speed	Accuracy	When to Use
Flat	High	Slow	Exact (100%)	< 100K vectors, correctness critical
IVF	Medium	Fast	~95–98%	100K–10M vectors, good balance
HNSW	Medium-High	Very fast	~97–99%	Low latency, production retrieval
PQ	Low	Fast	~90–95%	Millions of vectors, memory-constrained

# Flat — brute-force exact search
store = VectorStore(backend="faiss", dimension=768, index_type="Flat")

# IVF — inverted file index with nlist clusters
store = VectorStore(backend="faiss", dimension=768, index_type="IVF", nlist=100)

# HNSW — hierarchical navigable small world graph
store = VectorStore(backend="faiss", dimension=768, index_type="HNSW", M=16, ef_construction=200)

# PQ — product quantization for memory efficiency
store = VectorStore(backend="faiss", dimension=768, index_type="PQ", m=8)

Similarity Metrics

Metric	Constructor arg	Distance → Similarity	Best For
Cosine	metric="cosine"	1 - cosine_distance	Text, embeddings
L2 (Euclidean)	metric="l2"	1 / (1 + distance)	Image features
Inner Product	metric="ip"	raw dot product	Recommendation systems

store = VectorStore(backend="faiss", dimension=768, metric="cosine")

Tips and Common Pitfalls

**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size — `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time, not at store creation. **Use `Flat` index only for small datasets.** Flat (brute-force) search has perfect recall but O(n) query time. At 500K+ vectors, switch to `IVF` or `HNSW` — they sacrifice less than 5% recall for 100–1000x speedup. **Don't search without normalizing first.** If you disabled `normalize=True` in `EmbeddingGenerator`, compute cosine similarity with `metric="cosine"` (which normalizes internally). Raw dot product on un-normalized vectors produces incorrect similarity rankings. **Use `hybrid_search` for precision-sensitive workloads.** Pure vector search finds semantically similar results but may miss keyword matches important to the user. Hybrid search (vector + BM25) combines both signals — especially valuable for domain-specific terminology. **Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and leaks data if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation). **Persist FAISS indexes to disk.** `VectorStore(backend="faiss", index_path="store.faiss")` saves the index to disk on each write. Without a path, the index is in-memory only and is lost on process exit. **Update metadata without re-embedding.** `store.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content. Generate the vectors stored here. AgentContext uses VectorStore for memory retrieval. PostgreSQL vector storage with pgvector extension. Ingest documents before embedding and storing.