# Vector Store Module Usage Guide

This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.

## Table of Contents

1. [Basic Usage](#basic-usage)
2. [Vector Storage Operations](#vector-storage-operations)
3. [Similarity Search](#similarity-search)
4. [Index Management](#index-management)
5. [Hybrid Search](#hybrid-search)
6. [Metadata Management](#metadata-management)
7. [Namespace Management](#namespace-management)
8. [Store Backends](#store-adapters)
9. [Algorithms and Methods](#algorithms-and-methods)
10. [Configuration](#configuration)
11. [Advanced Examples](#advanced-examples)

## Basic Usage

### Using VectorStore

```python
from semantica.vector_store import VectorStore
import numpy as np

# Create vector store
store = VectorStore(backend="faiss", dimension=768)

# Store vectors
vectors = [np.random.rand(768) for _ in range(100)]
metadata = [{"category": "science", "year": 2023} for _ in range(100)]
vector_ids = store.store_vectors(vectors, metadata=metadata)

# Search vectors
query_vector = np.random.rand(768)
results = store.search_vectors(query_vector, k=10)

print(f"Stored {len(vector_ids)} vectors")
print(f"Found {len(results)} similar vectors")
```

### Using Convenience Functions

```python
from semantica.vector_store import store_vectors, search_vectors, hybrid_search
from semantica.vector_store import MetadataFilter
import numpy as np

# Store vectors
vectors = [np.random.rand(768) for _ in range(100)]
metadata = [{"category": "science"} for _ in range(100)]
vector_ids = store_vectors(vectors, metadata=metadata, method="default")

# Search vectors
query_vector = np.random.rand(768)
results = search_vectors(query_vector, vectors, vector_ids, k=10, method="default")

# Hybrid search with metadata filter
filter = MetadataFilter().eq("category", "science")
hybrid_results = hybrid_search(
    query_vector, vectors, metadata, vector_ids,
    filter=filter, k=10, method="default"
)

print(f"Found {len(results)} results")
print(f"Found {len(hybrid_results)} hybrid results")
```

### Using HybridSearch

```python
from semantica.vector_store import HybridSearch, MetadataFilter
import numpy as np

# Create hybrid search
search = HybridSearch()

# Create metadata filter
filter = MetadataFilter().eq("category", "science").gt("year", 2020)

# Perform hybrid search
query_vector = np.random.rand(768)
vectors = [np.random.rand(768) for _ in range(1000)]
metadata = [{"category": "science", "year": 2023} for _ in range(1000)]
vector_ids = [f"vec_{i}" for i in range(1000)]

results = search.search(
    query_vector, vectors, metadata, vector_ids,
    filter=filter, k=10
)

print(f"Found {len(results)} results")
for result in results[:5]:
    print(f"ID: {result['id']}, Score: {result['score']:.3f}")
```

## Vector Storage Operations

### Storing Vectors

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768)

# Store single batch of vectors
vectors = [np.random.rand(768) for _ in range(1000)]
metadata = [{"category": "science", "year": 2023} for _ in range(1000)]
vector_ids = store.store_vectors(vectors, metadata=metadata)

print(f"Stored {len(vector_ids)} vectors")
print(f"First vector ID: {vector_ids[0]}")
```

### Batch Storage

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768)

# Store in batches
batch_size = 100
total_vectors = 10000

for i in range(0, total_vectors, batch_size):
    batch_vectors = [np.random.rand(768) for _ in range(batch_size)]
    batch_metadata = [{"batch": i // batch_size} for _ in range(batch_size)]
    vector_ids = store.store_vectors(batch_vectors, metadata=batch_metadata)
    print(f"Stored batch {i // batch_size + 1}: {len(vector_ids)} vectors")
```

### Updating Vectors

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768)

# Store initial vectors
vectors = [np.random.rand(768) for _ in range(100)]
vector_ids = store.store_vectors(vectors)

# Update vectors
new_vectors = [np.random.rand(768) for _ in range(100)]
success = store.update_vectors(vector_ids, new_vectors)

print(f"Updated {len(vector_ids)} vectors: {success}")
```

### Deleting Vectors

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768)

# Store vectors
vectors = [np.random.rand(768) for _ in range(100)]
vector_ids = store.store_vectors(vectors)

# Delete some vectors
vectors_to_delete = vector_ids[:10]
success = store.delete_vectors(vectors_to_delete)

print(f"Deleted {len(vectors_to_delete)} vectors: {success}")
```

## Similarity Search

### Basic Similarity Search

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768)

# Store vectors
vectors = [np.random.rand(768) for _ in range(1000)]
vector_ids = store.store_vectors(vectors)

# Search for similar vectors
query_vector = np.random.rand(768)
results = store.search_vectors(query_vector, k=10)

print(f"Found {len(results)} similar vectors")
for result in results:
    print(f"ID: {result['id']}, Score: {result['score']:.3f}")
```

### Using VectorRetriever

```python
from semantica.vector_store import VectorRetriever
import numpy as np

retriever = VectorRetriever(backend="faiss")

# Prepare vectors
vectors = [np.random.rand(768) for _ in range(1000)]
vector_ids = [f"vec_{i}" for i in range(1000)]

# Search
query_vector = np.random.rand(768)
results = retriever.search_similar(query_vector, vectors, vector_ids, k=10)

print(f"Found {len(results)} results")
```

### Using VectorManager

```python
from semantica.vector_store import VectorManager

# Create manager
manager = VectorManager()

# Create and register stores
faiss_store = manager.create_store("faiss", {"dimension": 768})
manager.register_store("main_store", "faiss", {"dimension": 768})

# List all stores
stores = manager.list_stores()
print(f"Active stores: {stores}")

# Get store by ID
store = manager.get_store("main_store")

# Get store statistics
stats = manager.get_store_stats("main_store")
print(f"Store stats: {stats}")
```

### Cosine Similarity Search

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768, metric="cosine")

# Store normalized vectors
vectors = [np.random.rand(768) for _ in range(1000)]
# Normalize vectors
vectors = [v / np.linalg.norm(v) for v in vectors]
vector_ids = store.store_vectors(vectors)

# Search
query_vector = np.random.rand(768)
query_vector = query_vector / np.linalg.norm(query_vector)
results = store.search_vectors(query_vector, k=10)

print(f"Cosine similarity results: {len(results)}")
```

### L2 Distance Search

```python
from semantica.vector_store import VectorStore
import numpy as np

store = VectorStore(backend="faiss", dimension=768, metric="l2")

# Store vectors
vectors = [np.random.rand(768) for _ in range(1000)]
vector_ids = store.store_vectors(vectors)

# Search
query_vector = np.random.rand(768)
results = store.search_vectors(query_vector, k=10)

print(f"L2 distance results: {len(results)}")
```

## Index Management

### Creating Indexes

```python
from semantica.vector_store import VectorIndexer
import numpy as np

# Create indexer
indexer = VectorIndexer(backend="faiss", dimension=768)

# Create index
vectors = [np.random.rand(768) for _ in range(1000)]
vector_ids = [f"vec_{i}" for i in range(1000)]
index = indexer.create_index(vectors, vector_ids)

print(f"Created index with {len(vector_ids)} vectors")
```

### FAISS Index Types

```python
from semantica.vector_store import FAISSStore
import numpy as np

adapter = FAISSStore(dimension=768)

# Create Flat index (exact search)
flat_index = adapter.create_index(index_type="flat", metric="L2")

# Create IVF index (approximate search)
ivf_index = adapter.create_index(
    index_type="ivf",
    metric="L2",
    nlist=100  # Number of clusters
)

# Create HNSW index (approximate search)
hnsw_index = adapter.create_index(
    index_type="hnsw",
    metric="L2",
    m=16,  # Number of connections
    ef_construction=200
)

# Create PQ index (compressed)
pq_index = adapter.create_index(
    index_type="pq",
    metric="L2",
    m=8  # Number of sub-vectors
)
```

### Index Training

```python
from semantica.vector_store import FAISSStore
import numpy as np

adapter = FAISSStore(dimension=768)

# Training vectors
training_vectors = np.random.rand(10000, 768).astype('float32')

# Create and train IVF index
index = adapter.create_index(
    index_type="ivf",
    metric="L2",
    nlist=100
)

# Train index
adapter.train_index(index, training_vectors)

# Add vectors
vectors = np.random.rand(1000, 768).astype('float32')
adapter.add_vectors(index, vectors, ids=[f"vec_{i}" for i in range(1000)])
```

### Index Persistence

```python
from semantica.vector_store import FAISSStore
import numpy as np

adapter = FAISSStore(dimension=768)

# Create and populate index
index = adapter.create_index(index_type="flat", metric="L2")
vectors = np.random.rand(1000, 768).astype('float32')
adapter.add_vectors(index, vectors, ids=[f"vec_{i}" for i in range(1000)])

# Save index
adapter.save_index(index, "index.faiss")

# Load index
loaded_index = adapter.load_index("index.faiss", dimension=768, index_type="flat")
```

## Hybrid Search

### Basic Hybrid Search

```python
from semantica.vector_store import HybridSearch, MetadataFilter
import numpy as np

search = HybridSearch()

# Prepare data
query_vector = np.random.rand(768)
vectors = [np.random.rand(768) for _ in range(1000)]
metadata = [
    {"category": "science" if i % 2 == 0 else "technology", "year": 2020 + (i % 4)}
    for i in range(1000)
]
vector_ids = [f"vec_{i}" for i in range(1000)]

# Create metadata filter
filter = MetadataFilter().eq("category", "science").gt("year", 2021)

# Perform hybrid search
results = search.search(
    query_vector, vectors, metadata, vector_ids,
    filter=filter, k=10
)

print(f"Found {len(results)} results")
for result in results:
    print(f"ID: {result['id']}, Score: {result['score']:.3f}, Metadata: {result.get('metadata', {})}")
```

### Metadata Filtering

```python
from semantica.vector_store import MetadataFilter

# Create filter with multiple conditions
filter = MetadataFilter() \
    .eq("category", "science") \
    .gt("year", 2020) \
    .in_list("tags", ["AI", "ML"])

# Test metadata
metadata1 = {"category": "science", "year": 2023, "tags": ["AI", "ML"]}
metadata2 = {"category": "technology", "year": 2023, "tags": ["AI"]}

print(f"Metadata1 matches: {filter.matches(metadata1)}")  # True
print(f"Metadata2 matches: {filter.matches(metadata2)}")  # False
```

### Result Fusion

```python
from semantica.vector_store import SearchRanker
import numpy as np

ranker = SearchRanker(strategy="reciprocal_rank_fusion")

# Multiple result lists
results1 = [
    {"id": "vec_1", "score": 0.9},
    {"id": "vec_2", "score": 0.8},
    {"id": "vec_3", "score": 0.7}
]
results2 = [
    {"id": "vec_2", "score": 0.85},
    {"id": "vec_4", "score": 0.75},
    {"id": "vec_1", "score": 0.7}
]

# Fuse results using RRF
fused = ranker.rank([results1, results2], k=60)

print(f"Fused {len(fused)} results")
for result in fused:
    print(f"ID: {result['id']}, Score: {result['score']:.3f}")
```

### Multi-Source Search

```python
from semantica.vector_store import HybridSearch
import numpy as np

search = HybridSearch()

# Multiple sources
sources = [
    {
        "vectors": [np.random.rand(768) for _ in range(500)],
        "metadata": [{"source": "db1"} for _ in range(500)],
        "ids": [f"db1_vec_{i}" for i in range(500)]
    },
    {
        "vectors": [np.random.rand(768) for _ in range(500)],
        "metadata": [{"source": "db2"} for _ in range(500)],
        "ids": [f"db2_vec_{i}" for i in range(500)]
    }
]

# Search across all sources
query_vector = np.random.rand(768)
results = search.multi_source_search(query_vector, sources, k=10)

print(f"Found {len(results)} results from multiple sources")
```

## Metadata Management

### Storing Metadata

```python
from semantica.vector_store import MetadataStore

store = MetadataStore()

# Store metadata
store.store_metadata("vec_1", {"category": "science", "year": 2023, "tags": ["AI"]})
store.store_metadata("vec_2", {"category": "technology", "year": 2022, "tags": ["ML"]})

# Get metadata
metadata = store.get_metadata("vec_1")
print(f"Metadata: {metadata}")
```

### Metadata Indexing

```python
from semantica.vector_store import MetadataIndex

index = MetadataIndex()

# Index metadata
index.index_metadata("vec_1", {"category": "science", "year": 2023})
index.index_metadata("vec_2", {"category": "science", "year": 2022})
index.index_metadata("vec_3", {"category": "technology", "year": 2023})

# Query by metadata
matching_ids = index.query({"category": "science"}, operator="AND")
print(f"Found {len(matching_ids)} vectors with category='science'")

# Query with multiple conditions
matching_ids = index.query({"category": "science", "year": 2023}, operator="AND")
print(f"Found {len(matching_ids)} vectors matching both conditions")
```

### Metadata Schema

```python
from semantica.vector_store import MetadataSchema

# Define schema
schema = MetadataSchema({
    "category": {"type": str, "required": True},
    "year": {"type": int, "required": True},
    "tags": {"type": list, "required": False}
})

# Validate metadata
metadata1 = {"category": "science", "year": 2023, "tags": ["AI"]}
metadata2 = {"category": "science"}  # Missing required field

print(f"Metadata1 valid: {schema.validate(metadata1)}")  # True
try:
    schema.validate(metadata2)  # Raises ValidationError
except Exception as e:
    print(f"Metadata2 invalid: {e}")
```

### Metadata Querying

```python
from semantica.vector_store import MetadataStore

store = MetadataStore()

# Store metadata
for i in range(100):
    store.store_metadata(
        f"vec_{i}",
        {
            "category": "science" if i % 2 == 0 else "technology",
            "year": 2020 + (i % 4),
            "tags": ["AI"] if i % 3 == 0 else ["ML"]
        }
    )

# Query metadata
vector_ids = store.query_metadata({"category": "science"}, operator="AND")
print(f"Found {len(vector_ids)} vectors with category='science'")

# Query with OR operator
vector_ids = store.query_metadata({"category": "science", "tags": "AI"}, operator="OR")
print(f"Found {len(vector_ids)} vectors matching any condition")
```

## Namespace Management

### Creating Namespaces

```python
from semantica.vector_store import NamespaceManager

manager = NamespaceManager()

# Create namespace
namespace = manager.create_namespace(
    "user1",
    description="User 1 vectors",
    metadata={"owner": "user1", "created": "2024-01-01"}
)

print(f"Created namespace: {namespace.name}")
print(f"Description: {namespace.description}")
```

### Adding Vectors to Namespaces

```python
from semantica.vector_store import NamespaceManager

manager = NamespaceManager()

# Create namespace
manager.create_namespace("user1", description="User 1 vectors")

# Add vectors to namespace
for i in range(100):
    manager.add_vector_to_namespace(f"vec_{i}", "user1")

# Get namespace vectors
vectors = manager.get_namespace_vectors("user1")
print(f"Namespace 'user1' has {len(vectors)} vectors")
```

### Access Control

```python
from semantica.vector_store import NamespaceManager

manager = NamespaceManager()

# Create namespace
namespace = manager.create_namespace("docs", description="Document vectors")

# Set access control
namespace.set_access_control("user1", ["read", "write"])
namespace.set_access_control("user2", ["read"])

# Check permissions
has_read = namespace.has_permission("user1", "read")
has_write = namespace.has_permission("user2", "write")

print(f"User1 has read: {has_read}, write: {has_write}")
print(f"User2 has read: {namespace.has_permission('user2', 'read')}")
print(f"User2 has write: {namespace.has_permission('user2', 'write')}")
```

### Multi-Tenant Support

```python
from semantica.vector_store import NamespaceManager

manager = NamespaceManager()

# Create namespaces for different tenants
manager.create_namespace("tenant1", description="Tenant 1 vectors")
manager.create_namespace("tenant2", description="Tenant 2 vectors")

# Add vectors to each tenant
for i in range(50):
    manager.add_vector_to_namespace(f"t1_vec_{i}", "tenant1")
    manager.add_vector_to_namespace(f"t2_vec_{i}", "tenant2")

# Get tenant-specific vectors
tenant1_vectors = manager.get_namespace_vectors("tenant1")
tenant2_vectors = manager.get_namespace_vectors("tenant2")

print(f"Tenant1: {len(tenant1_vectors)} vectors")
print(f"Tenant2: {len(tenant2_vectors)} vectors")
```

## Store Backends

### FAISS Store

```python
from semantica.vector_store import FAISSStore
import numpy as np

# Create FAISS store
store = FAISSStore(dimension=768)

# Create index
store.create_index(index_type="flat", metric="L2")

# Add vectors
vectors = np.random.rand(1000, 768).astype('float32')
ids = [f"vec_{i}" for i in range(1000)]
store.add_vectors(vectors, ids=ids)

# Search
query_vector = np.random.rand(768).astype('float32')
results = store.search_similar(query_vector, k=10)

print(f"Found {len(results)} similar vectors")
```

### Weaviate Store

```python
from semantica.vector_store import WeaviateStore
import numpy as np

# Create Weaviate store
store = WeaviateStore(url="http://localhost:8080")

# Connect
store.connect()

# Create schema
store.create_schema(
    "Document",
    properties=[{"name": "text", "dataType": "text"}]
)

# Add objects with vectors
objects = [{"text": f"Document {i}"} for i in range(100)]
vectors = [np.random.rand(768).tolist() for _ in range(100)]
object_ids = store.add_objects(objects, vectors=vectors)

# Query
query_vector = np.random.rand(768).tolist()
results = store.query_vectors(
    query_vector,
    limit=10,
    where={"category": "science"}
)

print(f"Found {len(results)} results")
```

### Qdrant Store

```python
from semantica.vector_store import QdrantStore
import numpy as np

# Create Qdrant store
store = QdrantStore(url="http://localhost:6333")

# Connect
store.connect()

# Create collection
collection = store.create_collection("my-collection", dimension=768)

# Upsert vectors
vectors = [np.random.rand(768).tolist() for _ in range(100)]
ids = [f"vec_{i}" for i in range(100)]
payloads = [{"category": "science"} for _ in range(100)]
store.upsert_vectors(collection, vectors, ids, payloads)

# Search
query_vector = np.random.rand(768).tolist()
results = store.search(
    collection,
    query_vector,
    top=10,
    query_filter={"must": [{"key": "category", "match": {"value": "science"}}]}
)

print(f"Found {len(results)} results")
```

### Pinecone Store

```python
from semantica.vector_store import PineconeStore
import numpy as np

# Create Pinecone store
store = PineconeStore(api_key="your-api-key")

# Connect
store.connect()

# Create index
store.create_index("my-index", dimension=768, metric="cosine")

# Upsert vectors
vectors = [np.random.rand(768).tolist() for _ in range(100)]
ids = [f"vec_{i}" for i in range(100)]
metadata = [{"category": "science"} for _ in range(100)]
store.upsert_vectors(vectors, ids, metadata=metadata, namespace="my-namespace")

# Search
query_vector = np.random.rand(768).tolist()
results = store.search_vectors(
    query_vector,
    k=10,
    filter={"category": {"$eq": "science"}},
    namespace="my-namespace"
)

print(f"Found {len(results)} results")
```

### Milvus Store

```python
from semantica.vector_store import MilvusStore
import numpy as np

# Create Milvus store
store = MilvusStore(host="localhost", port="19530")

# Connect
store.connect()

# Create collection
collection = store.create_collection(
    "my-collection",
    dimension=768,
    metric_type="L2"
)

# Insert vectors
vectors = np.random.rand(100, 768).astype('float32')
ids = [f"vec_{i}" for i in range(100)]
store.insert_vectors(collection, vectors, ids)

# Search
query_vector = np.random.rand(768).astype('float32')
results = store.search(collection, query_vector, top_k=10)

print(f"Found {len(results)} results")
```

## Algorithms and Methods

### Vector Storage Algorithms

#### Vector Storage
**Algorithm**: ID generation and vector normalization

1. **ID Generation**: Generate unique IDs (sequential: vec_0, vec_1, ... or UUID-based)
2. **Vector Normalization**: Optional L2 normalization (v / ||v||) for cosine similarity
3. **Metadata Association**: Map vector IDs to metadata dictionaries
4. **Index Update**: Update vector index after storage

**Time Complexity**: O(n) where n = number of vectors
**Space Complexity**: O(n × d) where d = vector dimension

```python
# Vector storage
vector_ids = store.store_vectors(vectors, metadata=metadata_list)
```

#### Batch Storage
**Algorithm**: Chunking algorithm for large datasets

1. **Batch Creation**: Divide vectors into fixed-size batches
2. **Batch Processing**: Process each batch sequentially
3. **Progress Tracking**: Track stored count, progress percentage
4. **Error Handling**: Handle errors per batch, continue on failure

**Time Complexity**: O(n) where n = total vectors
**Space Complexity**: O(b × d) where b = batch size, d = dimension

### Similarity Search Algorithms

#### Cosine Similarity
**Algorithm**: Normalized dot product calculation

1. **Vector Normalization**: Normalize query and database vectors (L2 norm)
2. **Dot Product**: Calculate dot(v1, v2) for all vectors
3. **Similarity Score**: Similarity = dot(v1, v2) / (||v1|| × ||v2||)
4. **Top-k Selection**: Argsort similarities in descending order, select top k

**Time Complexity**: O(n × d) where n = vectors, d = dimension
**Space Complexity**: O(n) for similarities

```python
# Cosine similarity search
results = store.search_vectors(query_vector, k=10)
```

#### L2 Distance
**Algorithm**: Euclidean distance calculation

1. **Distance Calculation**: distance = sqrt(sum((v1 - v2)²))
2. **Distance-to-Similarity**: similarity = 1 / (1 + distance)
3. **Top-k Selection**: Argsort distances in ascending order, select top k

**Time Complexity**: O(n × d) where n = vectors, d = dimension
**Space Complexity**: O(n) for distances

#### Approximate Nearest Neighbor (ANN)
**Algorithm**: FAISS approximate search algorithms

**FAISS IVF**:
1. **k-means Clustering**: Cluster vectors into nlist cells
2. **Inverted File**: Build inverted file mapping cells to vectors
3. **Approximate Search**: Search nprobe cells, compute exact distances within cells

**FAISS HNSW**:
1. **Graph Construction**: Build multi-layer graph with connections
2. **Greedy Search**: Start from entry point, traverse graph to nearest neighbors
3. **Approximate Search**: Configurable ef_search for accuracy/speed tradeoff

**Time Complexity**: O(log n) to O(n) depending on index type and parameters
**Space Complexity**: O(n × d) for vectors, O(n × m) for HNSW graph

```python
# ANN search with FAISS
adapter = FAISSStore(dimension=768)
index = adapter.create_index(index_type="ivf", nlist=100)
results = adapter.search(index, query_vector, k=10)
```

### Index Construction Algorithms

#### FAISS Flat Index
**Algorithm**: Brute force exact search

1. **No Training**: Direct vector storage, no preprocessing
2. **Exact Search**: Compute all distances, select top k
3. **Full Storage**: Store all vectors in memory

**Time Complexity**: O(n × d) for search
**Space Complexity**: O(n × d) for storage

#### FAISS IVF Index
**Algorithm**: Inverted file index with k-means

1. **Training**: k-means clustering on training vectors (nlist clusters)
2. **Cell Assignment**: Assign each vector to nearest cluster center
3. **Inverted File**: Build mapping from cluster to vectors
4. **Search**: Search nprobe clusters, compute exact distances

**Time Complexity**: O(n × d × log(nlist)) for training, O(nprobe × (n/nlist) × d) for search
**Space Complexity**: O(n × d) for vectors, O(nlist × d) for cluster centers

#### FAISS HNSW Index
**Algorithm**: Hierarchical navigable small world graph

1. **Graph Construction**: Build multi-layer graph (m connections per node)
2. **Layer Assignment**: Assign vectors to layers probabilistically
3. **Greedy Search**: Start from entry point, traverse graph greedily
4. **Connection Management**: Maintain connections to nearest neighbors

**Time Complexity**: O(log n) for search with ef_search parameter
**Space Complexity**: O(n × m) for graph connections

### Hybrid Search Algorithms

#### Vector Similarity + Metadata Filtering
**Algorithm**: Two-stage filtering and search

1. **Metadata Filtering**: Apply metadata filter conditions first
2. **Vector Search**: Perform vector similarity search on filtered set
3. **Result Combination**: Combine filtered metadata with search results

**Time Complexity**: O(m × f) for filtering + O(n × d) for search where m = metadata count, f = filter conditions, n = filtered vectors
**Space Complexity**: O(n) for filtered vectors

```python
# Hybrid search
filter = MetadataFilter().eq("category", "science")
results = hybrid_search(query_vector, vectors, metadata, vector_ids, filter=filter, k=10)
```

#### Reciprocal Rank Fusion (RRF)
**Algorithm**: Result fusion using reciprocal ranks

1. **Rank Collection**: Collect ranks of each result from all result lists
2. **Score Calculation**: score = sum(1 / (k + rank)) for each result across all lists
3. **Result Ranking**: Sort results by RRF score in descending order
4. **Top-k Selection**: Select top k results after fusion

**Time Complexity**: O(r × l) where r = results per list, l = number of lists
**Space Complexity**: O(r × l) for result storage

```python
# RRF fusion
ranker = SearchRanker(strategy="reciprocal_rank_fusion")
fused = ranker.rank([results1, results2], k=60)
```

#### Weighted Average Fusion
**Algorithm**: Weighted sum of scores

1. **Score Collection**: Collect scores for each result from all lists
2. **Weighted Sum**: weighted_score = sum(score × weight) for each result
3. **Result Ranking**: Sort results by weighted score
4. **Top-k Selection**: Select top k results

**Time Complexity**: O(r × l) where r = results per list, l = number of lists
**Space Complexity**: O(r × l) for result storage

### Metadata Management Algorithms

#### Metadata Indexing
**Algorithm**: Inverted index per field

1. **Field Indexing**: Create inverted index for each metadata field
2. **Value Mapping**: Map field values to sets of vector IDs
3. **List Handling**: Index each item in list values
4. **Fast Lookup**: O(1) lookup per field-value pair

**Time Complexity**: O(m × f) where m = metadata count, f = fields per metadata
**Space Complexity**: O(m × f) for indexes

```python
# Metadata indexing
index = MetadataIndex()
index.index_metadata("vec_1", {"category": "science", "year": 2023})
matching_ids = index.query({"category": "science"})
```

#### Metadata Filtering
**Algorithm**: Condition-based filtering

1. **Condition Evaluation**: Evaluate each filter condition (eq, ne, gt, gte, lt, lte, in, contains)
2. **AND/OR Combination**: Combine conditions using AND (intersection) or OR (union)
3. **Result Set Construction**: Build set of matching vector IDs

**Time Complexity**: O(m × c) where m = metadata count, c = conditions
**Space Complexity**: O(m) for result set

```python
# Metadata filtering
filter = MetadataFilter().eq("category", "science").gt("year", 2020)
matching = [meta for meta in metadata if filter.matches(meta)]
```

### Namespace Management Algorithms

#### Namespace Isolation
**Algorithm**: Vector-to-namespace mapping

1. **Mapping Storage**: Dictionary mapping vector_id -> namespace
2. **Namespace Organization**: Group vectors by namespace
3. **Access Control**: Store permissions per namespace-entity pair
4. **Isolation Enforcement**: Filter operations by namespace

**Time Complexity**: O(1) for vector-to-namespace lookup
**Space Complexity**: O(n) where n = vectors

```python
# Namespace isolation
manager.add_vector_to_namespace("vec_1", "user1")
vectors = manager.get_namespace_vectors("user1")
```

### Methods

#### VectorStore Methods

- `store_vectors(vectors, metadata, **options)`: Store vectors with metadata
- `search_vectors(query_vector, k, **options)`: Search for similar vectors
- `update_vectors(vector_ids, new_vectors, **options)`: Update existing vectors
- `delete_vectors(vector_ids, **options)`: Delete vectors
- `get_vector(vector_id)`: Get vector by ID
- `get_metadata(vector_id)`: Get metadata by vector ID

#### VectorIndexer Methods

- `create_index(vectors, ids, **options)`: Create vector index
- `update_index(index, new_vectors, **options)`: Update existing index
- `optimize_index(index, **options)`: Optimize index for performance

#### VectorRetriever Methods

- `search_similar(query_vector, vectors, ids, k, **options)`: Search for similar vectors
- `search_by_metadata(metadata_filters, vectors, metadata, **options)`: Search by metadata
- `search_hybrid(query_vector, metadata_filters, vectors, metadata, **options)`: Hybrid search

#### HybridSearch Methods

- `search(query_vector, vectors, metadata, vector_ids, k, metadata_filter, **options)`: Perform hybrid search
- `multi_source_search(query_vector, sources, k, **options)`: Search across multiple sources
- `filter_by_metadata(results, metadata_filter)`: Filter results by metadata

#### MetadataStore Methods

- `store_metadata(vector_id, metadata)`: Store metadata for vector
- `get_metadata(vector_id)`: Get metadata for vector
- `query_metadata(conditions, operator)`: Query vectors by metadata conditions
- `update_metadata(vector_id, metadata)`: Update metadata
- `delete_metadata(vector_id)`: Delete metadata

#### NamespaceManager Methods

- `create_namespace(name, description, metadata, **options)`: Create namespace
- `delete_namespace(name)`: Delete namespace
- `add_vector_to_namespace(vector_id, namespace_name)`: Add vector to namespace
- `remove_vector_from_namespace(vector_id, namespace_name)`: Remove vector from namespace
- `get_namespace_vectors(namespace_name)`: Get all vectors in namespace
- `get_namespace(namespace_name)`: Get namespace object

#### Convenience Functions

- `store_vectors(vectors, metadata, method, **options)`: Store vectors wrapper
- `search_vectors(query_vector, vectors, vector_ids, k, method, **options)`: Search vectors wrapper
- `update_vectors(vector_ids, new_vectors, method, **options)`: Update vectors wrapper
- `delete_vectors(vector_ids, method, **options)`: Delete vectors wrapper
- `create_index(vectors, ids, method, **options)`: Create index wrapper
- `hybrid_search(query_vector, vectors, metadata, vector_ids, k, metadata_filter, method, **options)`: Hybrid search wrapper
- `filter_metadata(metadata, filter_conditions, operator, method, **options)`: Filter metadata wrapper
- `manage_namespace(namespace_name, operation, **options)`: Namespace management wrapper

## Configuration

### Environment Variables

```bash
# Vector store configuration
export VECTOR_STORE_DEFAULT_BACKEND=faiss
export VECTOR_STORE_DIMENSION=768
export VECTOR_STORE_BATCH_SIZE=1000
export VECTOR_STORE_INDEX_TYPE=flat
export VECTOR_STORE_METRIC=cosine
export VECTOR_STORE_ENABLE_HYBRID_SEARCH=true
export VECTOR_STORE_NAMESPACE=default

# FAISS configuration
export VECTOR_STORE_FAISS_INDEX_TYPE=flat

# Weaviate configuration
export VECTOR_STORE_WEAVIATE_URL=http://localhost:8080

# Qdrant configuration
export VECTOR_STORE_QDRANT_URL=http://localhost:6333

# Milvus configuration
export VECTOR_STORE_MILVUS_HOST=localhost
export VECTOR_STORE_MILVUS_PORT=19530
```

### Programmatic Configuration

```python
from semantica.vector_store.config import vector_store_config

# Get configuration
backend = vector_store_config.get("default_backend", default="faiss")
dimension = vector_store_config.get("dimension", default=768)

# Set configuration
vector_store_config.set("default_backend", "faiss")
vector_store_config.set("dimension", 768)

# Update with dictionary
vector_store_config.update({
    "default_backend": "faiss",
    "dimension": 768,
    "batch_size": 1000
})
```

### Configuration File (YAML)

```yaml
# config.yaml
vector_store:
  default_backend: faiss
  dimension: 768
  batch_size: 1000
  index_type: flat
  metric: cosine
  enable_hybrid_search: true
  default_namespace: default
  faiss_index_type: flat
  weaviate_url: http://localhost:8080
  qdrant_url: http://localhost:6333
  milvus_host: localhost
  milvus_port: 19530
```

## Advanced Examples

### Complete Vector Store Pipeline

```python
from semantica.vector_store import (
    VectorStore,
    HybridSearch,
    MetadataFilter,
    store_vectors,
    search_vectors,
    hybrid_search
)
import numpy as np

# 1. Create vector store
store = VectorStore(backend="faiss", dimension=768)

# 2. Generate and store vectors
vectors = [np.random.rand(768) for _ in range(10000)]
metadata = [
    {"category": "science" if i % 2 == 0 else "technology", "year": 2020 + (i % 4)}
    for i in range(10000)
]
vector_ids = store_vectors(vectors, metadata=metadata)

# 3. Create metadata filter
filter = MetadataFilter().eq("category", "science").gt("year", 2021)

# 4. Perform hybrid search
query_vector = np.random.rand(768)
results = hybrid_search(
    query_vector, vectors, metadata, vector_ids,
    filter=filter, k=10
)

print(f"Stored {len(vector_ids)} vectors")
print(f"Found {len(results)} hybrid search results")
```

### Multi-Backend Vector Store

```python
from semantica.vector_store import FAISSStore, WeaviateStore
import numpy as np

# Local FAISS store
faiss_store = FAISSStore(dimension=768)
faiss_index = faiss_store.create_index(index_type="flat", metric="L2")
faiss_vectors = np.random.rand(1000, 768).astype('float32')
faiss_store.add_vectors(faiss_index, faiss_vectors, ids=[f"faiss_{i}" for i in range(1000)])

# Self-hosted Weaviate store
weaviate_store = WeaviateStore(url="http://localhost:8080")
weaviate_store.connect()
weaviate_index = weaviate_store.create_index("my-index", dimension=768)
weaviate_vectors = [np.random.rand(768).tolist() for _ in range(1000)]
weaviate_store.upsert_vectors(weaviate_vectors, [f"weaviate_{i}" for i in range(1000)])

# Search both
query_vector = np.random.rand(768)
faiss_results = faiss_store.search(faiss_index, query_vector, k=10)
weaviate_results = weaviate_store.query_vectors(query_vector, top_k=10)
```

### Hybrid Search with Custom Ranking

```python
from semantica.vector_store import HybridSearch, SearchRanker, MetadataFilter
import numpy as np

# Create hybrid search with custom ranker
ranker = SearchRanker(strategy="weighted_average")
search = HybridSearch(ranking_strategy="weighted_average")

# Prepare data
query_vector = np.random.rand(768)
vectors = [np.random.rand(768) for _ in range(1000)]
metadata = [{"category": "science", "year": 2023} for _ in range(1000)]
vector_ids = [f"vec_{i}" for i in range(1000)]

# Create filter
filter = MetadataFilter().eq("category", "science")

# Perform search
results = search.search(
    query_vector, vectors, metadata, vector_ids,
    filter=filter, k=10
)

# Custom ranking with weights
results1 = search.search(query_vector, vectors[:500], metadata[:500], vector_ids[:500], k=10)
results2 = search.search(query_vector, vectors[500:], metadata[500:], vector_ids[500:], k=10)

fused = ranker.rank([results1, results2], weights=[0.7, 0.3])
print(f"Fused {len(fused)} results with weighted average")
```

### Namespace-Based Multi-Tenant System

```python
from semantica.vector_store import NamespaceManager, VectorStore
import numpy as np

# Create namespace manager
namespace_manager = NamespaceManager()

# Create namespaces for tenants
namespace_manager.create_namespace("tenant1", description="Tenant 1")
namespace_manager.create_namespace("tenant2", description="Tenant 2")

# Create vector store
store = VectorStore(backend="faiss", dimension=768)

# Store vectors for each tenant
for tenant in ["tenant1", "tenant2"]:
    vectors = [np.random.rand(768) for _ in range(1000)]
    vector_ids = store.store_vectors(vectors)
    
    # Add to namespace
    for vec_id in vector_ids:
        namespace_manager.add_vector_to_namespace(vec_id, tenant)

# Tenant-specific search
tenant1_vectors = namespace_manager.get_namespace_vectors("tenant1")
query_vector = np.random.rand(768)
# Search only tenant1 vectors
results = store.search_vectors(query_vector, k=10)
tenant1_results = [r for r in results if r['id'] in tenant1_vectors]
```

### Custom Method Registration

```python
from semantica.vector_store.registry import method_registry
from semantica.vector_store import store_vectors

# Register custom store method
def custom_store_vectors(vectors, metadata=None, **options):
    # Custom logic (e.g., normalization)
    import numpy as np
    normalized_vectors = [v / np.linalg.norm(v) for v in vectors]
    # Call default implementation
    from semantica.vector_store.methods import _get_store
    store = _get_store()
    return store.store_vectors(normalized_vectors, metadata=metadata, **options)

method_registry.register("store", "normalized", custom_store_vectors)

# Use custom method
vector_ids = store_vectors(vectors, metadata=metadata, method="normalized")
```

## Best Practices

1. **Vector Storage**:
   - Normalize vectors for cosine similarity
   - Use batch operations for large datasets
   - Associate metadata at storage time
   - Validate vector dimensions before storage

2. **Similarity Search**:
   - Choose appropriate metric (cosine for normalized, L2 for distance)
   - Use approximate indices (IVF, HNSW) for large datasets
   - Set appropriate k value (not too large)
   - Cache query vectors when possible

3. **Index Management**:
   - Train indices on representative data
   - Choose index type based on dataset size and accuracy requirements
   - Optimize index parameters (nlist for IVF, m for HNSW)
   - Persist indices for reuse

4. **Hybrid Search**:
   - Apply metadata filters before vector search for efficiency
   - Use RRF for multi-source fusion
   - Tune ranking parameters (k for RRF, weights for weighted average)
   - Combine complementary filters

5. **Metadata Management**:
   - Index frequently queried fields
   - Use schema validation for data quality
   - Keep metadata consistent with vectors
   - Use appropriate operators (AND/OR)

6. **Namespace Management**:
   - Use namespaces for multi-tenant isolation
   - Set appropriate access controls
   - Monitor namespace statistics
   - Clean up unused namespaces

7. **Performance**:
   - Use batch operations when possible
   - Choose appropriate index type for use case
   - Normalize vectors for cosine similarity
   - Cache frequently accessed vectors

8. **Error Handling**:
   - Validate vector dimensions
   - Handle missing metadata gracefully
   - Check namespace existence before operations
   - Handle adapter connection failures

9. **Configuration**:
   - Use environment variables for deployment
   - Use config files for development
   - Set appropriate defaults
   - Document configuration options

10. **Testing**:
    - Test with sample data first
    - Validate search results
    - Test metadata filtering
    - Test namespace isolation
    - Monitor performance metrics

