{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
                "\n",
                "# Vector Store - Comprehensive Guide\n",
                "\n",
                "## Overview\n",
                "\n",
                "This notebook provides a **comprehensive walkthrough** of Semantica's vector_store module, demonstrating vector storage, similarity search, hybrid search, and multi-backend support for semantic retrieval.\n",
                "\n",
                "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
                "\n",
                "### Learning Objectives\n",
                "\n",
                "By the end of this notebook, you will be able to:\n",
                "\n",
                "- Store and manage vectors with metadata\n",
                "- Perform similarity search with different metrics\n",
                "- Use hybrid search combining vectors and metadata\n",
                "- Work with multiple vector store backends (FAISS, Pinecone, etc.)\n",
                "- Create and manage vector indices\n",
                "- Filter and rank search results\n",
                "- Implement namespace isolation for multi-tenancy\n",
                "\n",
                "### What You'll Learn\n",
                "\n",
                "| Component | Purpose | When to Use |\n",
                "|-----------|---------|-------------|\n",
                "| `VectorStore` | Main vector storage | All vector operations |\n",
                "| `VectorIndexer` | Index creation | Performance optimization |\n",
                "| `VectorRetriever` | Similarity search | Finding similar vectors |\n",
                "| `HybridSearch` | Combined search | Vector + metadata filtering |\n",
                "| `MetadataFilter` | Metadata filtering | Filtering by attributes |\n",
                "| `MetadataStore` | Metadata management | Storing vector metadata |\n",
                "| `NamespaceManager` | Multi-tenancy | Isolating vector collections |\n",
                "\n",
                "---\n",
                "\n",
                "## Installation\n",
                "\n",
                "Install Semantica from PyPI:\n",
                "\n",
                "```bash\n",
                "pip install semantica\n",
                "# Or with all optional dependencies:\n",
                "pip install semantica[all]\n",
                "```\n",
                "\n",
                "---"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 1: Basic Vector Storage\n",
                "\n",
                "Let's start with the `VectorStore` for basic vector storage and retrieval.\n",
                "\n",
                "### What is VectorStore?\n",
                "\n",
                "`VectorStore` is the main interface for vector operations:\n",
                "- **Storage**: Store vectors with metadata\n",
                "- **Search**: Find similar vectors\n",
                "- **CRUD**: Create, Read, Update, Delete operations\n",
                "- **Multi-backend**: Support for FAISS, Pinecone, Weaviate, Qdrant, Milvus"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import VectorStore\n",
                "from semantica.embeddings import TextEmbedder\n",
                "import numpy as np\n",
                "\n",
                "# 1. Initialize Embedder (Select Provider & Model)\n",
                "# You can choose 'sentence_transformers' or 'fastembed'\n",
                "embedder = TextEmbedder(method=\"sentence_transformers\", model_name=\"all-MiniLM-L6-v2\")\n",
                "dimension = embedder.get_embedding_dimension()\n",
                "\n",
                "# 2. Create vector store\n",
                "store = VectorStore(backend=\"faiss\", dimension=dimension)\n",
                "\n",
                "# 3. Generate Real Embeddings\n",
                "texts = [f\"Document {i}\" for i in range(100)]\n",
                "vectors = embedder.embed_batch(texts)\n",
                "\n",
                "metadata = [\n",
                "    {\"text\": txt, \"category\": \"science\" if i % 2 == 0 else \"technology\", \"year\": 2020 + (i % 4)}\n",
                "    for i, txt in enumerate(texts)\n",
                "]\n",
                "\n",
                "# 4. Store vectors\n",
                "vector_ids = store.store_vectors(vectors, metadata=metadata)\n",
                "\n",
                "print(f\"Stored {len(vector_ids)} vectors\")\n",
                "print(f\"First 3 IDs: {vector_ids[:3]}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 2: Similarity Search\n",
                "\n",
                "Search for similar vectors using different similarity metrics.\n",
                "\n",
                "### Similarity Metrics\n",
                "\n",
                "- **Cosine Similarity**: Best for semantic similarity\n",
                "- **L2 Distance**: Euclidean distance\n",
                "- **Dot Product**: Fast, requires normalized vectors"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create query vector\n",
                "query_vector = np.random.rand(768)\n",
                "\n",
                "# Search for similar vectors\n",
                "results = store.search_vectors(query_vector, k=10)\n",
                "\n",
                "print(f\"Found {len(results)} similar vectors\")\n",
                "print(\"\\nTop 5 results:\")\n",
                "for i, result in enumerate(results[:5], 1):\n",
                "    print(f\"{i}. ID: {result['id']}, Score: {result['score']:.3f}\")\n",
                "    print(f\"   Metadata: {result.get('metadata', {})}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 3: Vector Indexing\n",
                "\n",
                "Create indices for faster search on large datasets.\n",
                "\n",
                "### Index Types (FAISS)\n",
                "\n",
                "- **Flat**: Exact search (brute force)\n",
                "- **IVF**: Inverted file index (approximate)\n",
                "- **HNSW**: Hierarchical graph (best balance)\n",
                "- **PQ**: Product quantization (compressed)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import VectorIndexer, FAISSAdapter\n",
                "\n",
                "# Create indexer\n",
                "indexer = VectorIndexer(backend=\"faiss\", dimension=768)\n",
                "\n",
                "# Create HNSW index for fast approximate search\n",
                "adapter = FAISSAdapter(dimension=768)\n",
                "index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
                "\n",
                "# Add vectors to index\n",
                "vectors_array = np.array(vectors).astype('float32')\n",
                "adapter.add_vectors(index, vectors_array, ids=vector_ids)\n",
                "\n",
                "# Search using index\n",
                "query_array = query_vector.astype('float32')\n",
                "distances, indices = adapter.search(index, query_array, k=10)\n",
                "\n",
                "print(f\"Index search found {len(indices)} results\")\n",
                "print(f\"Distances: {distances[:5]}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 4: Hybrid Search\n",
                "\n",
                "Combine vector similarity with metadata filtering.\n",
                "\n",
                "### Hybrid Search Benefits\n",
                "\n",
                "- Filter by metadata before vector search\n",
                "- Combine multiple search criteria\n",
                "- More precise results"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import HybridSearch, MetadataFilter\n",
                "\n",
                "# Create hybrid search\n",
                "hybrid_search = HybridSearch()\n",
                "\n",
                "# Create metadata filter\n",
                "filter = MetadataFilter() \\\n",
                "    .eq(\"category\", \"science\") \\\n",
                "    .gt(\"year\", 2021)\n",
                "\n",
                "# Perform hybrid search\n",
                "hybrid_results = hybrid_search.search(\n",
                "    query_vector,\n",
                "    vectors,\n",
                "    metadata,\n",
                "    vector_ids,\n",
                "    filter=filter,\n",
                "    k=10\n",
                ")\n",
                "\n",
                "print(f\"Hybrid search found {len(hybrid_results)} results\")\n",
                "print(\"\\nFiltered results (science, year > 2021):\")\n",
                "for i, result in enumerate(hybrid_results[:5], 1):\n",
                "    meta = result.get('metadata', {})\n",
                "    print(f\"{i}. Category: {meta.get('category')}, Year: {meta.get('year')}, Score: {result['score']:.3f}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 5: Metadata Management\n",
                "\n",
                "Store and query metadata separately from vectors.\n",
                "\n",
                "### Metadata Operations\n",
                "\n",
                "- Store metadata for vectors\n",
                "- Query by metadata conditions\n",
                "- Update metadata\n",
                "- Schema validation"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import MetadataStore, MetadataSchema\n",
                "\n",
                "# Create metadata store\n",
                "meta_store = MetadataStore()\n",
                "\n",
                "# Store metadata\n",
                "for i, vec_id in enumerate(vector_ids[:10]):\n",
                "    meta_store.store_metadata(vec_id, metadata[i])\n",
                "\n",
                "# Query metadata\n",
                "matching_ids = meta_store.query_metadata(\n",
                "    {\"category\": \"science\"},\n",
                "    operator=\"AND\"\n",
                ")\n",
                "\n",
                "print(f\"Found {len(matching_ids)} vectors with category='science'\")\n",
                "\n",
                "# Define schema for validation\n",
                "schema = MetadataSchema({\n",
                "    \"text\": {\"type\": str, \"required\": True},\n",
                "    \"category\": {\"type\": str, \"required\": True},\n",
                "    \"year\": {\"type\": int, \"required\": True}\n",
                "})\n",
                "\n",
                "# Validate metadata\n",
                "is_valid = schema.validate(metadata[0])\n",
                "print(f\"\\nMetadata validation: {is_valid}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 6: Result Ranking and Fusion\n",
                "\n",
                "Combine and rank results from multiple searches.\n",
                "\n",
                "### Ranking Strategies\n",
                "\n",
                "- **Reciprocal Rank Fusion (RRF)**: Combine ranked lists\n",
                "- **Weighted Average**: Weight scores from different sources"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import SearchRanker\n",
                "\n",
                "# Create ranker with RRF strategy\n",
                "ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
                "\n",
                "# Simulate multiple search results\n",
                "results1 = [\n",
                "    {\"id\": \"vec_1\", \"score\": 0.9},\n",
                "    {\"id\": \"vec_2\", \"score\": 0.8},\n",
                "    {\"id\": \"vec_3\", \"score\": 0.7}\n",
                "]\n",
                "\n",
                "results2 = [\n",
                "    {\"id\": \"vec_2\", \"score\": 0.85},\n",
                "    {\"id\": \"vec_4\", \"score\": 0.75},\n",
                "    {\"id\": \"vec_1\", \"score\": 0.7}\n",
                "]\n",
                "\n",
                "# Fuse results using RRF\n",
                "fused_results = ranker.rank([results1, results2], k=60)\n",
                "\n",
                "print(\"Fused results using RRF:\")\n",
                "for i, result in enumerate(fused_results, 1):\n",
                "    print(f\"{i}. ID: {result['id']}, Fused Score: {result['score']:.3f}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 7: Namespace Management\n",
                "\n",
                "Isolate vectors for multi-tenant applications.\n",
                "\n",
                "### Namespace Features\n",
                "\n",
                "- Tenant isolation\n",
                "- Access control\n",
                "- Per-namespace operations"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import NamespaceManager\n",
                "\n",
                "# Create namespace manager\n",
                "ns_manager = NamespaceManager()\n",
                "\n",
                "# Create namespaces for different tenants\n",
                "ns1 = ns_manager.create_namespace(\"tenant1\", \"Tenant 1 vectors\")\n",
                "ns2 = ns_manager.create_namespace(\"tenant2\", \"Tenant 2 vectors\")\n",
                "\n",
                "# Add vectors to namespaces\n",
                "for i in range(5):\n",
                "    ns_manager.add_vector_to_namespace(f\"t1_vec_{i}\", \"tenant1\")\n",
                "    ns_manager.add_vector_to_namespace(f\"t2_vec_{i}\", \"tenant2\")\n",
                "\n",
                "# Get namespace vectors\n",
                "tenant1_vectors = ns_manager.get_namespace_vectors(\"tenant1\")\n",
                "tenant2_vectors = ns_manager.get_namespace_vectors(\"tenant2\")\n",
                "\n",
                "print(f\"Tenant 1: {len(tenant1_vectors)} vectors\")\n",
                "print(f\"Tenant 2: {len(tenant2_vectors)} vectors\")\n",
                "\n",
                "# Set access control\n",
                "ns1.set_access_control(\"user1\", [\"read\", \"write\"])\n",
                "ns1.set_access_control(\"user2\", [\"read\"])\n",
                "\n",
                "print(f\"\\nUser1 can write: {ns1.has_permission('user1', 'write')}\")\n",
                "print(f\"User2 can write: {ns1.has_permission('user2', 'write')}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 8: Convenience Functions\n",
                "\n",
                "Use convenience functions for quick operations.\n",
                "\n",
                "### Available Functions\n",
                "\n",
                "- `store_vectors()`: Store vectors\n",
                "- `search_vectors()`: Search vectors\n",
                "- `hybrid_search()`: Hybrid search\n",
                "- `update_vectors()`: Update vectors\n",
                "- `delete_vectors()`: Delete vectors"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import (\n",
                "    store_vectors,\n",
                "    search_vectors,\n",
                "    hybrid_search as hybrid_search_func,\n",
                "    update_vectors,\n",
                "    delete_vectors\n",
                ")\n",
                "\n",
                "# Store vectors using convenience function\n",
                "new_vectors = [np.random.rand(768) for _ in range(10)]\n",
                "new_metadata = [{\"text\": f\"New doc {i}\"} for i in range(10)]\n",
                "new_ids = store_vectors(new_vectors, metadata=new_metadata, method=\"default\")\n",
                "\n",
                "print(f\"Stored {len(new_ids)} new vectors\")\n",
                "\n",
                "# Search using convenience function\n",
                "search_results = search_vectors(\n",
                "    query_vector,\n",
                "    new_vectors,\n",
                "    new_ids,\n",
                "    k=5,\n",
                "    method=\"default\"\n",
                ")\n",
                "\n",
                "print(f\"Search found {len(search_results)} results\")\n",
                "\n",
                "# Update vectors\n",
                "updated_vectors = [np.random.rand(768) for _ in range(2)]\n",
                "success = update_vectors(new_ids[:2], updated_vectors, method=\"default\")\n",
                "print(f\"\\nUpdated vectors: {success}\")\n",
                "\n",
                "# Delete vectors\n",
                "success = delete_vectors(new_ids[-2:], method=\"default\")\n",
                "print(f\"Deleted vectors: {success}\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 9: Multi-Backend Support\n",
                "\n",
                "Work with different vector store backends.\n",
                "\n",
                "### Supported Backends\n",
                "\n",
                "| Backend | Type | Best For |\n",
                "|---------|------|----------|\n",
                "| FAISS | Local | Development, small datasets |\n",
                "| Pinecone | Cloud | Production, managed service |\n",
                "| Weaviate | Self-hosted | Schema-aware storage |\n",
                "| Qdrant | Self-hosted | High performance |\n",
                "| Milvus | Cloud/Self-hosted | Large scale |"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "from semantica.vector_store import FAISSAdapter, VectorManager\n",
                "\n",
                "# FAISS (local)\n",
                "faiss_adapter = FAISSAdapter(dimension=768)\n",
                "faiss_index = faiss_adapter.create_index(index_type=\"flat\", metric=\"L2\")\n",
                "print(\"Created FAISS index\")\n",
                "\n",
                "# Vector Manager for multi-store management\n",
                "manager = VectorManager()\n",
                "faiss_store = manager.create_store(\"faiss\", {\"dimension\": 768})\n",
                "print(f\"\\nCreated store via manager\")\n",
                "\n",
                "# List all stores\n",
                "stores = manager.list_stores()\n",
                "print(f\"Active stores: {stores}\")\n",
                "\n",
                "# Note: For cloud backends (Pinecone, Weaviate, etc.),\n",
                "# you would need API keys and endpoints\n",
                "# Example:\n",
                "# from semantica.vector_store import PineconeAdapter\n",
                "# pinecone = PineconeAdapter(api_key=\"your-key\", environment=\"us-west1-gcp\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Step 10: Best Practices\n",
                "\n",
                "### Performance Tips\n",
                "\n",
                "1. **Normalize Vectors**: Always normalize for cosine similarity\n",
                "2. **Use HNSW**: Best balance for speed/accuracy\n",
                "3. **Batch Operations**: Process in batches (100-1000)\n",
                "4. **Filter First**: Apply metadata filters before vector search\n",
                "\n",
                "### Backend Selection\n",
                "\n",
                "- **Development**: FAISS (local, fast)\n",
                "- **Production**: Pinecone (managed, scalable)\n",
                "- **Self-hosted**: Qdrant or Milvus (control, performance)\n",
                "- **Schema-aware**: Weaviate (rich metadata)\n",
                "\n",
                "### Index Configuration\n",
                "\n",
                "- **Small datasets (<10K)**: Flat index\n",
                "- **Medium datasets (10K-1M)**: HNSW\n",
                "- **Large datasets (>1M)**: IVF + PQ"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Summary\n",
                "\n",
                "### What You've Learned\n",
                "\n",
                "In this notebook, you've learned how to:\n",
                "\n",
                "- Store and search vectors with VectorStore\n",
                "- Create indices for performance optimization\n",
                "- Use hybrid search with metadata filtering\n",
                "- Manage metadata separately from vectors\n",
                "- Rank and fuse search results\n",
                "- Implement namespace isolation\n",
                "- Use convenience functions for quick operations\n",
                "- Work with multiple backend adapters\n",
                "- Apply best practices for production use\n",
                "\n",
                "### Key Takeaways\n",
                "\n",
                "1. **Multi-Backend**: Choose the right backend for your needs\n",
                "2. **Hybrid Search**: Combine vectors with metadata for precision\n",
                "3. **Indexing**: Use appropriate index types for performance\n",
                "4. **Metadata**: Separate metadata management for flexibility\n",
                "5. **Namespaces**: Isolate vectors for multi-tenancy\n",
                "\n",
                "### Next Steps\n",
                "\n",
                "**Further Reading**:\n",
                "- [Vector Store API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
                "- [Advanced Vector Store Notebook](../advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
                "- [Embedding Generation](12_Embedding_Generation.ipynb)\n",
                "\n",
                "---\n",
                "\n",
                "**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.10.0"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}
