{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
    "\n",
    "# Vector Store - Comprehensive Guide\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook provides a **comprehensive walkthrough** of Semantica's vector_store module, demonstrating vector storage, similarity search, hybrid search, and multi-backend support for semantic retrieval.\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "By the end of this notebook, you will be able to:\n",
    "\n",
    "- Store and manage vectors with metadata\n",
    "- Perform similarity search with different metrics\n",
    "- Use hybrid search combining vectors and metadata\n",
    "- Work with multiple vector store backends (FAISS, Weaviate, etc.)\n",
    "- Create and manage vector indices\n",
    "- Filter and rank search results\n",
    "- Implement namespace isolation for multi-tenancy\n",
    "\n",
    "### What You'll Learn\n",
    "\n",
    "| Component | Purpose | When to Use |\n",
    "|-----------|---------|-------------|\n",
    "| `VectorStore` | Main vector storage | All vector operations |\n",
    "| `VectorIndexer` | Index creation | Performance optimization |\n",
    "| `VectorRetriever` | Similarity search | Finding similar vectors |\n",
    "| `HybridSearch` | Combined search | Vector + metadata filtering |\n",
    "| `MetadataFilter` | Metadata filtering | Filtering by attributes |\n",
    "| `MetadataStore` | Metadata management | Storing vector metadata |\n",
    "| `NamespaceManager` | Multi-tenancy | Isolating vector collections |\n",
    "\n",
    "---\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q semantica\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Basic Vector Storage\n",
    "\n",
    "Let's start with the `VectorStore` for basic vector storage and retrieval.\n",
    "\n",
    "### What is VectorStore?\n",
    "\n",
    "`VectorStore` is the main interface for vector operations:\n",
    "- **Storage**: Store vectors with metadata\n",
    "- **Search**: Find similar vectors\n",
    "- **CRUD**: Create, Read, Update, Delete operations\n",
    "- **Multi-backend**: Support for FAISS, Weaviate, Qdrant, Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "from semantica.embeddings import TextEmbedder\n",
    "import numpy as np\n",
    "\n",
    "# 1. Initialize Embedder (Select Provider & Model)\n",
    "# You can choose 'sentence_transformers' or 'fastembed'\n",
    "embedder = TextEmbedder(method=\"sentence_transformers\", model_name=\"all-MiniLM-L6-v2\")\n",
    "dimension = embedder.get_embedding_dimension()\n",
    "\n",
    "# 2. Create vector store\n",
    "store = VectorStore(backend=\"faiss\", dimension=dimension)\n",
    "\n",
    "# 3. Generate Real Embeddings\n",
    "texts = [f\"Document {i}\" for i in range(100)]\n",
    "vectors = embedder.embed_batch(texts)\n",
    "\n",
    "metadata = [\n",
    "    {\"text\": txt, \"category\": \"science\" if i % 2 == 0 else \"technology\", \"year\": 2020 + (i % 4)}\n",
    "    for i, txt in enumerate(texts)\n",
    "]\n",
    "# 4. Store vectors\n",
    "vector_ids = store.store_vectors(vectors, metadata=metadata)\n",
    "\n",
    "print(f\"Stored {len(vector_ids)} vectors\")\n",
    "print(f\"First 3 IDs: {vector_ids[:3]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Similarity Search\n",
    "\n",
    "Search for similar vectors using different similarity metrics.\n",
    "\n",
    "### Similarity Metrics\n",
    "\n",
    "- **Cosine Similarity**: Best for semantic similarity\n",
    "- **L2 Distance**: Euclidean distance\n",
    "- **Dot Product**: Fast, requires normalized vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorIndexer, FAISSStore \n",
    "import numpy as np \n",
    "# We use the first vector from the dataset as a sample query\n",
    "if 'query_vector' not in locals():\n",
    "    if 'vectors' in locals() and len(vectors) > 0:\n",
    "        query_vector = vectors[0]\n",
    "    else:\n",
    "        # Fallback if vectors are also missing (safety check)\n",
    "        query_vector = np.random.rand(dimension).astype('float32')\n",
    "\n",
    "# Create indexer \n",
    "indexer = VectorIndexer(backend=\"faiss\", dimension=dimension) \n",
    "\n",
    "# Create HNSW index for fast approximate search \n",
    "adapter = FAISSStore(dimension=dimension) \n",
    "index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16) \n",
    "\n",
    "# Add vectors to index \n",
    "vectors_array = np.array(vectors).astype('float32') \n",
    "# FIX: Removed 'index' argument. The adapter uses its internal self.index \n",
    "adapter.add_vectors(vectors_array, ids=vector_ids) \n",
    "\n",
    "# Search using index \n",
    "query_array = np.array(query_vector).astype('float32') \n",
    "# FIX: Call search on the 'index' object directly, not the adapter \n",
    "distances, indices = index.search(query_array.reshape(1, -1), k=10) \n",
    "\n",
    "print(f\"Index search found {len(indices[0])} results\") \n",
    "print(f\"Distances: {distances[0][:5]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Vector Indexing\n",
    "\n",
    "Create indices for faster search on large datasets.\n",
    "\n",
    "### Index Types (FAISS)\n",
    "\n",
    "- **Flat**: Exact search (brute force)\n",
    "- **IVF**: Inverted file index (approximate)\n",
    "- **HNSW**: Hierarchical graph (best balance)\n",
    "- **PQ**: Product quantization (compressed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorIndexer, FAISSStore\n",
    "\n",
    "# Get dimension from vectors to ensure consistency\n",
    "dimension = len(vectors[0]) if len(vectors) > 0 else 384\n",
    "\n",
    "# Create indexer\n",
    "indexer = VectorIndexer(backend=\"faiss\", dimension=dimension)\n",
    "\n",
    "# Create HNSW index for fast approximate search\n",
    "adapter = FAISSStore(dimension=dimension)\n",
    "index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
    "\n",
    "# Add vectors to index\n",
    "vectors_array = np.array(vectors).astype('float32')\n",
    "adapter.add_vectors(vectors_array, ids=vector_ids)\n",
    "\n",
    "# Search using index\n",
    "query_array = query_vector.astype('float32')\n",
    "distances, indices = index.search(query_array.reshape(1, -1), k=10)\n",
    "\n",
    "print(f\"Index search found {len(indices[0])} results\")\n",
    "print(f\"Distances: {distances[0][:5]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Hybrid Search\n",
    "\n",
    "Combine vector similarity with metadata filtering.\n",
    "\n",
    "### Hybrid Search Benefits\n",
    "\n",
    "- Filter by metadata before vector search\n",
    "- Combine multiple search criteria\n",
    "- More precise results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import HybridSearch, MetadataFilter \n",
    "import numpy as np\n",
    "\n",
    "# Create hybrid search \n",
    "hybrid_search = HybridSearch() \n",
    "\n",
    "# Create metadata filter \n",
    "filter = MetadataFilter() \\\n",
    "    .eq(\"category\", \"science\") \\\n",
    "    .gt(\"year\", 2021) \n",
    "\n",
    "# Perform hybrid search \n",
    "# Ensure vectors and metadata are available\n",
    "if 'vectors' not in locals() or 'metadata' not in locals() or 'vector_ids' not in locals():\n",
    "    print(\"Warning: vectors, metadata, or vector_ids are missing. Please run previous cells.\")\n",
    "else:\n",
    "    hybrid_results = hybrid_search.search( \n",
    "        query_vector, \n",
    "        vectors, \n",
    "        metadata, \n",
    "        vector_ids, \n",
    "        filter=filter, \n",
    "        k=10 \n",
    "    ) \n",
    "    \n",
    "    print(f\"Hybrid search found {len(hybrid_results)} results\") \n",
    "    print(\"\\nFiltered results (science, year > 2021):\") \n",
    "    for i, result in enumerate(hybrid_results[:5], 1): \n",
    "        meta = result.get('metadata', {}) \n",
    "        print(f\"{i}. Category: {meta.get('category')}, Year: {meta.get('year')}, Score: {result['score']:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Metadata Management\n",
    "\n",
    "Store and query metadata separately from vectors.\n",
    "\n",
    "### Metadata Operations\n",
    "\n",
    "- Store metadata for vectors\n",
    "- Query by metadata conditions\n",
    "- Update metadata\n",
    "- Schema validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import MetadataStore, MetadataSchema\n",
    "\n",
    "# Create metadata store\n",
    "meta_store = MetadataStore()\n",
    "\n",
    "# Store metadata\n",
    "for i, vec_id in enumerate(vector_ids[:10]):\n",
    "    meta_store.store_metadata(vec_id, metadata[i])\n",
    "\n",
    "# Query metadata\n",
    "matching_ids = meta_store.query_metadata(\n",
    "    {\"category\": \"science\"},\n",
    "    operator=\"AND\"\n",
    ")\n",
    "\n",
    "print(f\"Found {len(matching_ids)} vectors with category='science'\")\n",
    "\n",
    "# Define schema for validation\n",
    "schema = MetadataSchema({\n",
    "    \"text\": {\"type\": str, \"required\": True},\n",
    "    \"category\": {\"type\": str, \"required\": True},\n",
    "    \"year\": {\"type\": int, \"required\": True}\n",
    "})\n",
    "\n",
    "# Validate metadata\n",
    "is_valid = schema.validate(metadata[0])\n",
    "print(f\"\\nMetadata validation: {is_valid}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Result Ranking and Fusion\n",
    "\n",
    "Combine and rank results from multiple searches.\n",
    "\n",
    "### Ranking Strategies\n",
    "\n",
    "- **Reciprocal Rank Fusion (RRF)**: Combine ranked lists\n",
    "- **Weighted Average**: Weight scores from different sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import SearchRanker\n",
    "\n",
    "# Create ranker with RRF strategy\n",
    "ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
    "\n",
    "# Simulate multiple search results\n",
    "results1 = [\n",
    "    {\"id\": \"vec_1\", \"score\": 0.9},\n",
    "    {\"id\": \"vec_2\", \"score\": 0.8},\n",
    "    {\"id\": \"vec_3\", \"score\": 0.7}\n",
    "]\n",
    "\n",
    "results2 = [\n",
    "    {\"id\": \"vec_2\", \"score\": 0.85},\n",
    "    {\"id\": \"vec_4\", \"score\": 0.75},\n",
    "    {\"id\": \"vec_1\", \"score\": 0.7}\n",
    "]\n",
    "\n",
    "# Fuse results using RRF\n",
    "fused_results = ranker.rank([results1, results2], k=60)\n",
    "\n",
    "print(\"Fused results using RRF:\")\n",
    "for i, result in enumerate(fused_results, 1):\n",
    "    print(f\"{i}. ID: {result['id']}, Fused Score: {result['score']:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Namespace Management\n",
    "\n",
    "Isolate vectors for multi-tenant applications.\n",
    "\n",
    "### Namespace Features\n",
    "\n",
    "- Tenant isolation\n",
    "- Access control\n",
    "- Per-namespace operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import NamespaceManager\n",
    "\n",
    "# Create namespace manager\n",
    "ns_manager = NamespaceManager()\n",
    "\n",
    "# Create namespaces for different tenants\n",
    "ns1 = ns_manager.create_namespace(\"tenant1\", \"Tenant 1 vectors\")\n",
    "ns2 = ns_manager.create_namespace(\"tenant2\", \"Tenant 2 vectors\")\n",
    "\n",
    "# Add vectors to namespaces\n",
    "for i in range(5):\n",
    "    ns_manager.add_vector_to_namespace(f\"t1_vec_{i}\", \"tenant1\")\n",
    "    ns_manager.add_vector_to_namespace(f\"t2_vec_{i}\", \"tenant2\")\n",
    "\n",
    "# Get namespace vectors\n",
    "tenant1_vectors = ns_manager.get_namespace_vectors(\"tenant1\")\n",
    "tenant2_vectors = ns_manager.get_namespace_vectors(\"tenant2\")\n",
    "\n",
    "print(f\"Tenant 1: {len(tenant1_vectors)} vectors\")\n",
    "print(f\"Tenant 2: {len(tenant2_vectors)} vectors\")\n",
    "\n",
    "# Set access control\n",
    "ns1.set_access_control(\"user1\", [\"read\", \"write\"])\n",
    "ns1.set_access_control(\"user2\", [\"read\"])\n",
    "\n",
    "print(f\"\\nUser1 can write: {ns1.has_permission('user1', 'write')}\")\n",
    "print(f\"User2 can write: {ns1.has_permission('user2', 'write')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Multi-Backend Support\n",
    "\n",
    "Work with different vector store backends.\n",
    "\n",
    "### Supported Backends\n",
    "\n",
    "| Backend | Type | Best For |\n",
    "|---------|------|----------|\n",
    "| FAISS | Local | Development, small datasets |\n",
    "| Weaviate | Self-hosted | Schema-aware storage |\n",
    "| Qdrant | Self-hosted | High performance |\n",
    "| Milvus | Cloud/Self-hosted | Large scale |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Best Practices\n",
    "\n",
    "### Performance Tips\n",
    "\n",
    "1. **Normalize Vectors**: Always normalize for cosine similarity\n",
    "2. **Use HNSW**: Best balance for speed/accuracy\n",
    "3. **Batch Operations**: Process in batches (100-1000)\n",
    "4. **Filter First**: Apply metadata filters before vector search\n",
    "\n",
    "### Backend Selection\n",
    "\n",
    "- **Development**: FAISS (local, fast)\n",
    "- **Production**: Weaviate/Qdrant (scalable, self-hosted)\n",
    "- **Self-hosted**: Qdrant or Milvus (control, performance)\n",
    "- **Schema-aware**: Weaviate (rich metadata)\n",
    "\n",
    "### Index Configuration\n",
    "\n",
    "- **Small datasets (<10K)**: Flat index\n",
    "- **Medium datasets (10K-1M)**: HNSW\n",
    "- **Large datasets (>1M)**: IVF + PQ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "### What You've Learned\n",
    "\n",
    "In this notebook, you've learned how to:\n",
    "\n",
    "- Store and search vectors with VectorStore\n",
    "- Create indices for performance optimization\n",
    "- Use hybrid search with metadata filtering\n",
    "- Manage metadata separately from vectors\n",
    "- Rank and fuse search results\n",
    "- Implement namespace isolation\n",
    "- Use convenience functions for quick operations\n",
    "- Work with multiple backend adapters\n",
    "- Apply best practices for production use\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "1. **Multi-Backend**: Choose the right backend for your needs\n",
    "2. **Hybrid Search**: Combine vectors with metadata for precision\n",
    "3. **Indexing**: Use appropriate index types for performance\n",
    "4. **Metadata**: Separate metadata management for flexibility\n",
    "5. **Namespaces**: Isolate vectors for multi-tenancy\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "**Further Reading**:\n",
    "- [Vector Store API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
    "- [Advanced Vector Store Notebook](../advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
    "- [Embedding Generation](12_Embedding_Generation.ipynb)\n",
    "\n",
    "---\n",
    "\n",
    "**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
