semantica/docs/reference/vector_store.md

Vector Store

Unified vector database interface supporting FAISS, Pinecone, Weaviate, Qdrant, and Milvus with Hybrid Search.

---

🎯 Overview

• :material-database:{ .lg .middle } Multi-Backend Support

  ---

  Seamlessly switch between FAISS (Local), Pinecone, Weaviate, Qdrant, and Milvus

• :material-magnify-plus:{ .lg .middle } Hybrid Search

  ---

  Combine dense vector similarity with sparse keyword/metadata filtering

• :material-filter:{ .lg .middle } Metadata Filtering

  ---

  Rich filtering capabilities (eq, ne, gt, lt, in, contains)

• :material-layers-triple:{ .lg .middle } Namespace Isolation

  ---

  Multi-tenant support via isolated namespaces

• :material-flash:{ .lg .middle } Performance

  ---

  Batch operations, index optimization, and caching

• :material-cloud-upload:{ .lg .middle } Cloud & Local

  ---

  Support for both embedded (local) and cloud-native deployments

!!! tip "When to Use" - Semantic Search: Finding documents similar to a query - RAG: Retrieving context for LLM generation - Memory: Storing agent memories as embeddings - Recommendation: Finding similar items based on vector proximity

---

⚙️ Algorithms Used

Similarity Metrics

• Cosine Similarity: A · B / ||A|| ||B|| (Default for semantic search)
• Euclidean Distance (L2): ||A - B||
• Dot Product: A · B (Faster, requires normalized vectors)

Indexing (FAISS)

• Flat: Exact search (brute force). High accuracy, slow for large datasets.
• IVF (Inverted File): Partitions space into Voronoi cells. Faster search.
• HNSW: Hierarchical Navigable Small World graphs. Best trade-off for speed/accuracy.
• PQ (Product Quantization): Compresses vectors for memory efficiency.

Hybrid Search

• Reciprocal Rank Fusion (RRF): Combines ranked lists from vector search and keyword search. Score = 1 / (k + rank_vector) + 1 / (k + rank_keyword)
• Pre-filtering: Apply metadata filters before vector search (supported by most backends).

---

Main Classes

VectorStore

The main facade for all vector operations.

Methods:

Method	Description
store_vectors(vectors, metadata)	Store embeddings
search(query, k)	Semantic search
delete(ids)	Remove vectors

Example:

from semantica.vector_store import VectorStore

# Initialize (defaults to FAISS)
store = VectorStore(backend="faiss", dimension=1536)

# Store
ids = store.store_vectors(
    vectors=[[0.1, 0.2, ...], ...],
    metadata=[{"text": "Hello"}, ...]
)

# Search
results = store.search(query_vector=[0.1, 0.2, ...], k=5)

HybridSearch

Combines vector and metadata search.

Methods:

Method	Description
search(query_vec, filter)	Execute hybrid query

Example:

from semantica.vector_store import HybridSearch, MetadataFilter

searcher = HybridSearch(store)
filters = MetadataFilter().eq("category", "news").gt("date", "2023-01-01")

results = searcher.search(
    query_vector=emb,
    filter=filters,
    k=10
)

Adapters

Backend-specific implementations:

• FAISSAdapter: Local, in-memory/disk.
• PineconeAdapter: Managed cloud service.
• WeaviateAdapter: Schema-aware vector DB.
• QdrantAdapter: Rust-based high-performance DB.
• MilvusAdapter: Scalable cloud-native DB.

---

Convenience Functions

from semantica.vector_store import store_vectors, search_vectors

# Quick usage (uses default configured backend)
store_vectors(embeddings, metadata)
results = search_vectors(query_embedding)

---

Configuration

Environment Variables

export VECTOR_STORE_BACKEND=pinecone
export PINECONE_API_KEY=sk-...
export PINECONE_ENV=us-west1-gcp

YAML Configuration

vector_store:
  backend: faiss # or pinecone, weaviate, etc.
  dimension: 1536
  metric: cosine
  
  faiss:
    index_type: HNSW
    
  pinecone:
    environment: us-west1-gcp
    index_name: my-index

---

Integration Examples

RAG Retrieval

from semantica.embeddings import EmbeddingGenerator
from semantica.vector_store import VectorStore

# 1. Embed Query
embedder = EmbeddingGenerator()
query_vec = embedder.generate("What is the capital of France?")

# 2. Search
store = VectorStore()
results = store.search(query_vec, k=3)

# 3. Use Context
context = "\n".join([r.metadata['text'] for r in results])
print(f"Context: {context}")

---

Best Practices

1. Normalize Vectors: Always normalize vectors if using Cosine Similarity or Dot Product.
2. Use HNSW: For FAISS, HNSW is usually the best default index type for performance/recall balance.
3. Batch Operations: Use store_vectors with batches (e.g., 100 items) rather than one by one.
4. Filter First: In hybrid search, restrictive filters significantly improve performance.

---

Troubleshooting

Issue: DimensionMismatchError Solution: Ensure your embedding model dimension (e.g., 1536 for OpenAI) matches the VectorStore dimension.

Issue: FAISS index not saved. Solution: Call store.save("index.faiss") explicitly for local FAISS indices, or use a persistent backend like Pinecone/Qdrant.

---

See Also

• Embeddings Module - Generates the vectors
• Context Module - Uses vector store for memory
• Ingest Module - Source of data