semantica/docs/reference/vector_store.md

Vector Store Module

Store and search vector embeddings with lightning-fast approximate nearest neighbor search.

---

🎯 Overview

• :material-database:{ .lg .middle } Vector Storage

  ---

  Efficient storage of high-dimensional vectors with multiple backend support

• :material-magnify:{ .lg .middle } Similarity Search

  ---

  Fast ANN search with HNSW, IVF, and PQ algorithms

• :material-merge:{ .lg .middle } Hybrid Search

  ---

  Combine vector search with keyword/metadata filtering using RRF

• :material-cog:{ .lg .middle } Index Management

  ---

  Multiple index types optimized for different use cases

• :material-flash:{ .lg .middle } Batch Operations

  ---

  Efficient bulk insert and search with parallel processing

• :material-cloud:{ .lg .middle } Multiple Backends

  ---

  FAISS, Pinecone, Qdrant, Weaviate, Milvus support

!!! tip "Choosing the Right Index" - Small datasets (<10K): Use Flat for exact search - Medium (10K-1M): Use IVF or HNSW
- Large (>1M): Use HNSW with GPU - Memory constrained: Use PQ for compression

---

⚙️ Algorithms Used

Indexing Algorithms

• Flat (Exact Search): Brute-force linear scan, O(n*d) complexity
• IVF (Inverted File Index): Clustering-based search with k-means, O(√n*d) complexity
• HNSW (Hierarchical Navigable Small World): Graph-based ANN, O(log n) search complexity
• PQ (Product Quantization): Compression-based search, reduces memory footprint
• LSH (Locality Sensitive Hashing): Hash-based approximate search

Similarity Metrics

• Cosine Similarity: cos(θ) = (A·B) / (||A|| * ||B||)
• Euclidean Distance: d = √(Σ(ai - bi)²)
• Dot Product: A·B = Σ(ai * bi)
• Manhattan Distance: d = Σ|ai - bi|

Hybrid Search Algorithms

• RRF (Reciprocal Rank Fusion): score = Σ(1/(k + rank_i)) where k=60
• Weighted Combination: score = α*vector_score + (1-α)*keyword_score
• Cascade Filtering: Vector search → metadata filtering → reranking

---

Main Classes

VectorStore

Methods:

Method	Description	Algorithm
store(embeddings, documents, metadata)	Store vectors with metadata	Batch insertion with index building
search(query_vector, top_k, filters)	Search similar vectors	ANN search with optional filtering
delete(ids)	Delete vectors by ID	Index update with tombstoning
update(id, vector, metadata)	Update vector/metadata	In-place update or delete+insert
create_index(index_type, params)	Create search index	Index-specific construction algorithm
rebuild_index()	Rebuild index from scratch	Full index reconstruction

Supported Backends:

Backend	Index Types	Best For
FAISS	Flat, IVF, HNSW, PQ	High performance, local deployment
Pinecone	Proprietary	Managed cloud service
Qdrant	HNSW	Production-ready, filtering support
Weaviate	HNSW	GraphQL API, hybrid search
Milvus	IVF, HNSW	Distributed, large-scale

Example:

from semantica.vector_store import VectorStore

# Initialize with FAISS backend
store = VectorStore(
    backend="faiss",
    index_type="HNSW",  # Flat, IVF, HNSW, PQ
    metric="cosine",  # cosine, euclidean, dot_product
    dimension=1536
)

# Store embeddings
store.store(
    embeddings=embeddings,
    documents=documents,
    metadata=[{"source": "doc1.pdf", "page": 1}, ...]
)

# Search
results = store.search(
    query_vector=query_embedding,
    top_k=10,
    filters={"source": "doc1.pdf"}
)

for result in results:
    print(f"Score: {result.score:.3f}, Doc: {result.document}")

---

HybridSearch

Methods:

Method	Description	Algorithm
search(query, top_k)	Hybrid search	Vector + keyword search with RRF fusion
vector_search(query_vector, top_k)	Vector-only search	ANN search
keyword_search(query_text, top_k)	Keyword-only search	BM25 or TF-IDF
combine_results(vector_results, keyword_results)	Merge results	RRF or weighted combination
rerank(results, query)	Rerank results	Cross-encoder reranking

Reciprocal Rank Fusion (RRF):

RRF_score(d) = Σ(1/(k + rank_i(d)))
where k = 60 (constant), rank_i(d) = rank of document d in result set i

Example:

from semantica.vector_store import HybridSearch

hybrid = HybridSearch(
    vector_store=store,
    keyword_index=keyword_index,
    fusion_method="rrf",  # rrf, weighted, cascade
    vector_weight=0.7  # for weighted fusion
)

results = hybrid.search(
    query="machine learning applications",
    top_k=10
)

---

VectorRetriever

Methods:

Method	Description	Algorithm
retrieve(query, top_k)	Retrieve relevant documents	Vector search + document fetching
retrieve_batch(queries, top_k)	Batch retrieval	Parallel vector search
retrieve_with_context(query, context_size)	Retrieve with context	Sliding window context expansion
filter_by_metadata(results, filters)	Filter by metadata	Post-search filtering

---

Index Types Comparison

FAISS Index Types

Index Type	Build Time	Search Time	Memory	Recall	Use Case
Flat	O(1)	O(n*d)	High	1.00	Small datasets, exact search
IVF	O(n*d)	O(√n*d)	Medium	0.95	Medium datasets, good balance
HNSW	O(nlog nd)	O(log n*d)	High	0.98	Large datasets, fast search
PQ	O(n*d)	O(n)	Low	0.85	Memory-constrained, compression

Index Parameters

IVF Parameters:

• nlist: Number of clusters (√n to 4√n recommended)
• nprobe: Number of clusters to search (1-nlist)

HNSW Parameters:

• M: Number of connections per layer (4-64, default: 16)
• efConstruction: Construction time accuracy (100-500)
• efSearch: Search time accuracy (efConstruction to 2*efConstruction)

PQ Parameters:

• m: Number of subquantizers (dimension/m should be divisible)
• nbits: Bits per subquantizer (8 is standard)

---

Configuration

# config.yaml - Vector Store Configuration

vector_store:
  backend: faiss  # faiss, pinecone, qdrant, weaviate, milvus
  
  faiss:
    index_type: HNSW  # Flat, IVF, HNSW, PQ
    metric: cosine  # cosine, euclidean, dot_product
    dimension: 1536
    
    # HNSW parameters
    hnsw_m: 16
    hnsw_ef_construction: 200
    hnsw_ef_search: 100
    
    # IVF parameters
    ivf_nlist: 100
    ivf_nprobe: 10
    
  hybrid_search:
    fusion_method: rrf  # rrf, weighted, cascade
    vector_weight: 0.7
    keyword_weight: 0.3
    enable_reranking: true
    
  batch_operations:
    batch_size: 1000
    parallel_workers: 4

---

Performance Characteristics

Search Complexity

Index Type	Build	Search	Memory
Flat	O(1)	O(n*d)	O(n*d)
IVF	O(ndk)	O(√n*d)	O(n*d)
HNSW	O(nlog nd)	O(log n*d)	O(ndM)
PQ	O(n*d)	O(n*m)	O(n*m)

where n = vectors, d = dimensions, k = clusters, M = HNSW connections, m = subquantizers

Scalability

• Small (<10K vectors): Use Flat for exact search
• Medium (10K-1M vectors): Use IVF or HNSW
• Large (>1M vectors): Use HNSW with GPU or distributed systems
• Very Large (>10M vectors): Use distributed backends (Milvus, Weaviate)

---

See Also

• Embeddings Module - Generate vector embeddings
• Knowledge Graph Module - Graph-based retrieval
• Core Module - Framework orchestration