semantica/docs/reference/vector_store.md

title, description, icon

title	description	icon
Vector Store Module	Unified interface for FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector with hybrid search.	database

semantica.vector_store provides a unified API for storing and searching vector embeddings across all major backends:

• Swap backends with a one-line change: no application code changes needed
• HybridSearch fuses dense vector similarity with metadata filtering via RRF or weighted average
• NamespaceManager for multi-tenant structural isolation
• FAISSStore with flat, ivf, hnsw, and pq index types
• Batch embed and store with parallel workers; metadata update without re-embedding

Exported Classes

Class	Role
VectorStore	Unified interface: store_vectors, search_vectors, update_vectors, delete_vectors
HybridSearch	Fuses dense vector similarity with metadata filtering via RRF or weighted average
MetadataFilter	Chainable filter builder: .eq("type", "person").gt("year", 2020).in_list("tag", [...])
NamespaceManager	Multi-tenant isolation: separate index namespaces per project or user
FAISSStore	Local disk or in-memory: flat, ivf, hnsw, and pq index types
WeaviateStore	Cloud or self-hosted, schema-aware
QdrantStore	Cloud or self-hosted with payload-based filtering
PineconeStore	Managed cloud vector database with serverless and pod modes
MilvusStore	Scalable self-hosted vector database
PgVectorStore	PostgreSQL with pgvector extension: no extra infrastructure
MetadataStore	Standalone metadata indexing and querying
SearchRanker	RRF and weighted-average result fusion

What You Get

• VectorStore — Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector
  • One-line backend swap: no application code changes
  • add_documents() auto-embeds; store_vectors() for pre-computed embeddings
• HybridSearch — Dense vector similarity with metadata filtering
  • RRF or weighted-average fusion strategies
  • Multi-source fusion across separate collections
• MetadataStore — Rich metadata indexing by field values
  • Update metadata fields without re-embedding
  • OR and AND query operators
• NamespaceManager — Structural per-tenant namespace isolation
  • Faster queries (smaller search space per tenant)
  • Safer than metadata-filter-only separation
• Batch Operations — Bulk add, delete, and metadata updates
  • Parallel embedding with configurable batch_size and workers
  • In-place vector updates without full re-indexing
• FAISS Index Types — flat, ivf, hnsw, and pq index types
  • Full configuration control via FAISSStore.create_index()
  • save() / load() for disk persistence

Getting Started

VectorStore is the main entry point. Use "inmemory" for development and "faiss" for local production:

from semantica.vector_store import VectorStore

# In-memory (development / testing: no persistence)
store = VectorStore(backend="inmemory", dimension=384)

# FAISS (local, persists to disk via save/load)
store = VectorStore(backend="faiss", dimension=384)

# Add plain text documents (auto-embedded)
ids = store.add_documents(
    documents=["Apple was founded by Steve Jobs.", "Microsoft was co-founded by Bill Gates."],
    metadata=[{"source": "wiki"}, {"source": "wiki"}]
)

# Search by text query (auto-embedded)
results = store.search("technology company founders", limit=5)
for r in results:
    print(f"{r['id']}: score: {r['score']:.3f}")

**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time. **Use `add_documents()` for text, `store_vectors()` for pre-computed embeddings.** `add_documents()` auto-embeds in parallel batches. If your embeddings are already computed (e.g. from a fine-tuned model), use `store_vectors()` directly to skip re-embedding.

Quick Start

```python from semantica.vector_store import VectorStore

# In-memory (development)
store = VectorStore(backend="inmemory", dimension=384)

# FAISS (local production)
store = VectorStore(backend="faiss", dimension=384)
```

```python # Add text documents (auto-embedded in batches) ids = store.add_documents( documents=["text one", "text two"], metadata=[{"title": "Document 1"}, {"title": "Document 2"}] )

# Add pre-computed vectors directly
ids = store.store_vectors(
    vectors=[embedding1, embedding2],
    metadata=[{"title": "Document 1"}, {"title": "Document 2"}]
)
```

```python # Search by text query (auto-embeds the query) results = store.search("machine learning", limit=10)

# Search by pre-computed vector
results = store.search_vectors(query_vector, k=10)

for r in results:
    print(f"{r['id']}: score: {r['score']:.3f}")
```

```python from semantica.vector_store import HybridSearch, MetadataFilter

mf = MetadataFilter().eq("category", "research").gt("year", 2022)

# Pass vector_store to the constructor: search() resolves vectors automatically
search  = HybridSearch(vector_store=store)
results = search.search(query=query_vector, k=10, metadata_filter=mf)
```

Backends

# In-memory: no persistence, for development and testing
store = VectorStore(backend="inmemory", dimension=384)

# FAISS: local disk persistence via save() / load()
store = VectorStore(backend="faiss", dimension=384)
store.save("./my_store")   # save to directory
store.load("./my_store")   # restore from directory

No installation or API key required. FAISS requires pip install faiss-cpu.

pip install "semantica[pinecone]"

import os
store = VectorStore(
    backend="pinecone",
    dimension=768,
    api_key=os.getenv("PINECONE_API_KEY"),
    index_name="semantica-index",
    environment="us-east-1-aws"
)

pip install "semantica[weaviate]"

store = VectorStore(
    backend="weaviate",
    dimension=768,
    url="http://localhost:8080",
    class_name="Document"
)

pip install "semantica[qdrant]"

store = VectorStore(
    backend="qdrant",
    dimension=768,
    url="http://localhost:6333",
    collection_name="semantica"
)

pip install "semantica[pgvector]"

store = VectorStore(
    backend="pgvector",
    dimension=768,
    connection_string="postgresql://user:pass@localhost/db",
    table_name="embeddings"
)

connection_string is required: the store raises ValueError at construction if it is absent.

pip install pymilvus

store = VectorStore(
    backend="milvus",
    dimension=768,
    host="localhost",
    port=19530,
    collection_name="semantica"
)

Backend Selection Guide

Backend	Deployment	API Key	Persistence	Best For
inmemory	Process	No	No	Development, unit tests
faiss	Local	No	Via save()/load()	On-premise, offline production
pinecone	Cloud	Yes	Managed	Managed cloud, serverless
weaviate	Self-hosted / Cloud	Optional	Managed	Rich metadata filtering
qdrant	Self-hosted / Cloud	Optional	Managed	High-performance filtering
milvus	Self-hosted	No	Managed	Large-scale production
pgvector	PostgreSQL	No	Managed	Postgres-native integration

HybridSearch

HybridSearch combines vector similarity with metadata filtering. Pass vector_store at construction to avoid supplying raw vectors on every call:

from semantica.vector_store import HybridSearch, MetadataFilter

# With vector_store: search() pulls vectors from the store automatically
search = HybridSearch(vector_store=store)
mf     = MetadataFilter().eq("category", "research").gt("year", 2022)

results = search.search(
    query=query_vector,   # np.ndarray or query string (auto-embedded)
    k=10,
    metadata_filter=mf
)

for r in results:
    print(f"{r['id']}: score: {r['score']:.3f}  metadata: {r['metadata']}")

Without a vector_store, pass vectors explicitly:

search = HybridSearch()
results = search.search(
    query=query_vector,
    vectors=my_vectors,
    metadata=my_metadata,
    vector_ids=my_ids,
    k=10,
    metadata_filter=mf,
)

Multi-source fusion across separate collections:

sources = [
    {"vectors": v1, "metadata": m1, "ids": ids1},
    {"vectors": v2, "metadata": m2, "ids": ids2},
]
fused = search.multi_source_search(query_vector, sources, k=10)

**Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter.

Metadata Filtering

MetadataFilter supports chained conditions: all conditions are ANDed:

from semantica.vector_store import MetadataFilter

mf = MetadataFilter().eq("author", "John Smith")          # equality
mf = MetadataFilter().ne("status", "archived")            # not equal
mf = MetadataFilter().gt("year", 2022).lte("year", 2024)  # range
mf = MetadataFilter().in_list("tag", ["ai", "ml"])        # set membership
mf = MetadataFilter().contains("title", "neural")         # substring / list contains

# Multiple conditions: all must match (AND)
mf = (
    MetadataFilter()
    .eq("category", "research")
    .gt("year", 2022)
    .contains("title", "language model")
)

MetadataFilter Methods

Method	Operator	Description
.eq(field, value)	==	Exact equality
.ne(field, value)	!=	Not equal
.gt(field, value)	>	Greater than
.gte(field, value)	>=	Greater than or equal
.lt(field, value)	<	Less than
.lte(field, value)	<=	Less than or equal
.in_list(field, values)	in	Field value is in list
.contains(field, value)	substring	String contains or list contains

SearchRanker

SearchRanker fuses results from multiple ranked lists:

from semantica.vector_store import SearchRanker

# Reciprocal Rank Fusion: robust to score scale differences
ranker  = SearchRanker(strategy="reciprocal_rank_fusion")
fused   = ranker.rank([results_list_1, results_list_2])

# Weighted average: requires normalised scores on the same scale
ranker  = SearchRanker(strategy="weighted_average")
fused   = ranker.rank([results_list_1, results_list_2], weights=[0.7, 0.3])

Fusion strategy	Description
reciprocal_rank_fusion	Rank-based combination via RRF: robust to score scale differences (default)
weighted_average	Weighted sum of scores: pass weights=[...] to rank()

Namespace Isolation

Use NamespaceManager to assign vectors to named namespaces for multi-tenant isolation:

from semantica.vector_store import NamespaceManager, VectorStore

store      = VectorStore(backend="inmemory", dimension=384)
ns_manager = NamespaceManager()

ns_manager.create_namespace("tenant_a", description="Customer A data")
ns_manager.create_namespace("tenant_b", description="Customer B data")

# Store vectors, then assign them to a namespace
ids_a = store.store_vectors(embeddings_a, metadata=metadata_a)
for vid in ids_a:
    ns_manager.add_vector_to_namespace(vid, "tenant_a")

# List all namespace names
for name in ns_manager.list_namespaces():     # returns List[str]
    print(name)

# Get all vectors in a namespace
vectors_in_a = ns_manager.get_namespace_vectors("tenant_a")

# Look up which namespace a vector belongs to
ns = ns_manager.get_vector_namespace("vec_0")

ns_manager.delete_namespace("tenant_a")

**Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and risks data leakage if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation).

Batch Operations

# Batch add text documents: parallel embedding with configurable workers
ids = store.add_documents(
    documents=large_doc_list,
    metadata=large_meta_list,
    batch_size=32,     # texts per embedding batch
    parallel=True,     # use ThreadPoolExecutor
)

# Batch add pre-computed vectors
ids = store.store_vectors(vectors=embeddings_list, metadata=meta_list)

# Delete by vector ID list
store.delete_vectors(vector_ids=["vec_0", "vec_1", "vec_2"])

# Update vectors in-place (rebuilds index for inmemory backend)
store.update_vectors(
    vector_ids=["vec_0"],
    new_vectors=[new_embedding]
)

Persistence (FAISS and in-memory)

store = VectorStore(backend="faiss", dimension=384)
store.add_documents(documents=docs, metadata=meta)

# Save to a directory: creates index.bin and store_data.pkl
store.save("./vector_store_backup")

# Restore in a new process
store2 = VectorStore(backend="faiss", dimension=384)
store2.load("./vector_store_backup")

Cloud backends (Pinecone, Weaviate, Qdrant, Milvus, PgVector) manage persistence themselves. `save()`/`load()` are for the in-memory and FAISS backends only. **inmemory and faiss backends lose data on process exit without `save()`.** Call `store.save(path)` after adding vectors. Cloud backends (Pinecone, Qdrant, Weaviate, Milvus, PgVector) persist automatically.

MetadataStore

MetadataStore indexes structured metadata and lets you query by field values without a vector:

from semantica.vector_store import MetadataStore

meta_store = MetadataStore()

# Store and retrieve metadata
meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"})
meta_store.store_metadata("doc2", {"author": "Bob",   "year": 2023, "category": "review"})

# Query: returns List[str] of matching vector IDs
ids = meta_store.query_metadata({"category": "research", "year": 2024})

# OR query
ids = meta_store.query_metadata({"category": "research"}, operator="OR")

# Get and update metadata for a specific vector
meta = meta_store.get_metadata("doc1")
meta_store.update_metadata("doc1", {"score": 0.92})

# Get all unique values for a field
years = meta_store.get_field_values("year")

# Statistics
stats = meta_store.get_stats()
# {"total_vectors": 2, "indexed_fields": 3, "field_counts": {...}}

**Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.

FAISS Index Type Reference

FAISS index type is configured by creating a FAISSStore directly and calling create_index(). Use lowercase type names:

from semantica.vector_store import FAISSStore

store = FAISSStore(dimension=384)

# flat: brute-force exact search
store.create_index(index_type="flat", metric="L2")

# ivf: inverted file index
store.create_index(index_type="ivf", metric="L2", nlist=100)

# hnsw: hierarchical navigable small world graph
store.create_index(index_type="hnsw", metric="L2", M=32)

# pq: product quantization for memory efficiency
store.create_index(index_type="pq", metric="L2", m=8)

Index	Memory	Speed	Accuracy	When to Use
flat	High	Slow	Exact (100%)	< 100K vectors, correctness critical
ivf	Medium	Fast	~95–98%	100K–10M vectors, good balance
hnsw	Medium-High	Very fast	~97–99%	Low latency, production retrieval
pq	Low	Fast	~90–95%	Millions of vectors, memory-constrained

**FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`. When using `VectorStore(backend="faiss")`, the underlying `FAISSStore` is initialised with a flat index by default. To use ivf/hnsw/pq, construct `FAISSStore` directly and call `create_index()` with the desired type.

Common Workflows

```python from semantica.vector_store import VectorStore

store = VectorStore(backend="faiss", dimension=384)

# Index documents
store.add_documents(
    documents=corpus_texts,
    metadata=[{"source": src} for src in sources],
    batch_size=64,
)

# Persist
store.save("./corpus_index")

# Query
results = store.search("What is knowledge graph construction?", limit=5)
for r in results:
    print(f"[{r['score']:.3f}] {r['metadata']['source']}")
```

```python from semantica.vector_store import VectorStore, HybridSearch, MetadataFilter

store  = VectorStore(backend="inmemory", dimension=384)
search = HybridSearch(vector_store=store)

# Only return results from 2023+ with category "research"
mf = MetadataFilter().gte("year", 2023).eq("category", "research")
results = search.search(query=query_vector, k=10, metadata_filter=mf)
```

```python from semantica.vector_store import HybridSearch, SearchRanker

search = HybridSearch()
sources = [
    {"vectors": v1, "metadata": m1, "ids": ids1},
    {"vectors": v2, "metadata": m2, "ids": ids2},
]
fused = search.multi_source_search(query_vector, sources, k=10)

# Custom fusion weights
ranker = SearchRanker(strategy="weighted_average")
fused  = ranker.rank([results_a, results_b], weights=[0.7, 0.3])
```

```python from semantica.vector_store import VectorStore, NamespaceManager

store = VectorStore(backend="inmemory", dimension=384)
ns    = NamespaceManager()

ns.create_namespace("project_a")
ns.create_namespace("project_b")

ids = store.add_documents(docs_a, metadata=meta_a)
for vid in ids:
    ns.add_vector_to_namespace(vid, "project_a")

# Check ownership
print(ns.get_vector_namespace(ids[0]))  # "project_a"
print(ns.get_namespace_stats("project_a"))
```

• Embeddings — Generate the vectors stored here.
• Context — AgentContext uses VectorStore for memory retrieval.
• Split — Chunk documents before embedding and storing.
• Ingest — Ingest documents before embedding and storing.