{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
    "\n",
    "# Advanced Vector Store - Made Easy\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "This notebook shows you **practical ways** to use vector stores in real applications. Each example is simple and ready to use.\n",
    "\n",
    "### Topics\n",
    "\n",
    "1. **Choosing the Right Index** - Which one to use and when\n",
    "2. **Smart Filtering** - Find exactly what you need\n",
    "3. **Combining Results** - Merge searches from different sources\n",
    "4. **Organizing Data** - Keep different users' data separate\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install semantica\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 0: Setup Embeddings\n",
    "\n",
    "First, let's select our embedding provider and model. Semantica supports multiple providers like Sentence Transformers and FastEmbed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import TextEmbedder\n",
    "\n",
    "# Choose provider and model\n",
    "embedder = TextEmbedder(method=\"fastembed\", model_name=\"BAAI/bge-small-en-v1.5\")\n",
    "dimension = embedder.get_embedding_dimension()\n",
    "\n",
    "print(f\"Selected model: {embedder.get_model_info()['model_name']}\")\n",
    "print(f\"Embedding dimension: {dimension}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Choosing the Right Index\n",
    "\n",
    "Think of an index like choosing a filing system:\n",
    "- **Flat**: Like a small notebook - slow but perfect\n",
    "- **HNSW**: Like a well-organized library - fast and accurate\n",
    "- **IVF**: Like a warehouse with sections - very fast for huge collections\n",
    "\n",
    "### Simple Rule\n",
    "- Less than 10,000 items? Use **Flat**\n",
    "- Between 10,000 and 1 million? Use **HNSW** ✅ (recommended)\n",
    "- More than 1 million? Use **IVF**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import FAISSStore\n",
    "import numpy as np\n",
    "\n",
    "# Create some example vectors (like document embeddings)\n",
    "vectors = np.random.rand(5000, 768).astype('float32')\n",
    "query = np.random.rand(768).astype('float32')\n",
    "\n",
    "adapter = FAISSStore(dimension=768)\n",
    "\n",
    "# HNSW Index - Best for most cases\n",
    "index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
    "adapter.add_vectors(vectors, ids=[f\"doc_{i}\" for i in range(len(vectors))])\n",
    "\n",
    "# Search for similar vectors\n",
    "results = adapter.search_similar(query, k=5)\n",
    "\n",
    "print(\"Found 5 most similar documents:\")\n",
    "for i, result in enumerate(results, 1):\n",
    "    print(f\"  {i}. Document {result['id']} (distance: {result['distance']:.3f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Smart Filtering with Metadata\n",
    "\n",
    "Imagine searching for \"similar articles\" but only from 2024 and only in the \"Technology\" category. That's what metadata filtering does!\n",
    "\n",
    "### Real-World Example\n",
    "You're building a document search where users want:\n",
    "- Similar documents (vector search)\n",
    "- From specific categories (metadata filter)\n",
    "- From recent years (metadata filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import HybridSearch, MetadataFilter\n",
    "import numpy as np\n",
    "\n",
    "# Create sample documents with metadata\n",
    "documents = [\n",
    "    {\"id\": 0, \"text\": \"AI in Healthcare\", \"category\": \"Technology\", \"year\": 2024},\n",
    "    {\"id\": 1, \"text\": \"Machine Learning Basics\", \"category\": \"Technology\", \"year\": 2023},\n",
    "    {\"id\": 2, \"text\": \"Business Strategy\", \"category\": \"Business\", \"year\": 2024},\n",
    "    {\"id\": 3, \"text\": \"Data Science Guide\", \"category\": \"Technology\", \"year\": 2024},\n",
    "    {\"id\": 4, \"text\": \"Marketing Tips\", \"category\": \"Business\", \"year\": 2023},\n",
    "]\n",
    "\n",
    "# Create vectors for each document\n",
    "vectors = [np.random.rand(768) for _ in documents]\n",
    "metadata = [{\"category\": d[\"category\"], \"year\": d[\"year\"]} for d in documents]\n",
    "vector_ids = [f\"doc_{d['id']}\" for d in documents]\n",
    "\n",
    "# Create search\n",
    "search = HybridSearch()\n",
    "query = np.random.rand(768)\n",
    "\n",
    "# Example 1: Find Technology articles from 2024\n",
    "filter1 = MetadataFilter().eq(\"category\", \"Technology\").eq(\"year\", 2024)\n",
    "results = search.search(query, vectors, metadata, vector_ids, filter=filter1, k=10)\n",
    "\n",
    "print(\"Technology articles from 2024:\")\n",
    "for r in results:\n",
    "    doc_id = int(r['id'].split('_')[1])\n",
    "    print(f\"  - {documents[doc_id]['text']}\")\n",
    "\n",
    "# Example 2: Find any article from 2024\n",
    "filter2 = MetadataFilter().eq(\"year\", 2024)\n",
    "results2 = search.search(query, vectors, metadata, vector_ids, filter=filter2, k=10)\n",
    "\n",
    "print(\"\\nAll articles from 2024:\")\n",
    "for r in results2:\n",
    "    doc_id = int(r['id'].split('_')[1])\n",
    "    print(f\"  - {documents[doc_id]['text']} ({documents[doc_id]['category']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Combining Search Results\n",
    "\n",
    "Sometimes you want to search in multiple places and combine the results. Like searching both your email and documents, then showing the best matches from both.\n",
    "\n",
    "### When to Use This\n",
    "- Searching multiple databases\n",
    "- Combining different search strategies\n",
    "- Giving more weight to certain sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import SearchRanker\n",
    "\n",
    "# Simulate two different searches\n",
    "# Search 1: Recent documents\n",
    "recent_results = [\n",
    "    {\"id\": \"doc_3\", \"score\": 0.95, \"source\": \"recent\"},\n",
    "    {\"id\": \"doc_0\", \"score\": 0.90, \"source\": \"recent\"},\n",
    "    {\"id\": \"doc_2\", \"score\": 0.85, \"source\": \"recent\"},\n",
    "]\n",
    "\n",
    "# Search 2: Popular documents\n",
    "popular_results = [\n",
    "    {\"id\": \"doc_1\", \"score\": 0.92, \"source\": \"popular\"},\n",
    "    {\"id\": \"doc_3\", \"score\": 0.88, \"source\": \"popular\"},\n",
    "    {\"id\": \"doc_4\", \"score\": 0.80, \"source\": \"popular\"},\n",
    "]\n",
    "\n",
    "# Method 1: Fair combination (RRF)\n",
    "ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
    "combined = ranker.rank([recent_results, popular_results])\n",
    "\n",
    "print(\"Combined results (fair ranking):\")\n",
    "for i, result in enumerate(combined[:3], 1):\n",
    "    doc_id = int(result['id'].split('_')[1])\n",
    "    print(f\"  {i}. {documents[doc_id]['text']} (score: {result['score']:.3f})\")\n",
    "\n",
    "# Method 2: Prefer recent documents (70% recent, 30% popular)\n",
    "weighted_ranker = SearchRanker(strategy=\"weighted_average\")\n",
    "weighted_combined = weighted_ranker.rank(\n",
    "    [recent_results, popular_results],\n",
    "    weights=[0.7, 0.3]\n",
    ")\n",
    "\n",
    "print(\"\\nCombined results (prefer recent):\")\n",
    "for i, result in enumerate(weighted_combined[:3], 1):\n",
    "    doc_id = int(result['id'].split('_')[1])\n",
    "    print(f\"  {i}. {documents[doc_id]['text']} (score: {result['score']:.3f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Keeping User Data Separate\n",
    "\n",
    "If you're building an app with multiple users or companies, you need to keep their data separate. Namespaces do this automatically.\n",
    "\n",
    "### Real Example\n",
    "You're building a SaaS app where:\n",
    "- Company A has their documents\n",
    "- Company B has their documents\n",
    "- They should never see each other's data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import NamespaceManager\n",
    "\n",
    "# Create manager\n",
    "manager = NamespaceManager()\n",
    "\n",
    "# Create separate spaces for each company\n",
    "company_a = manager.create_namespace(\"company_a\", \"Company A's documents\")\n",
    "company_b = manager.create_namespace(\"company_b\", \"Company B's documents\")\n",
    "\n",
    "# Add documents to Company A\n",
    "for i in range(10):\n",
    "    manager.add_vector_to_namespace(f\"company_a_doc_{i}\", \"company_a\")\n",
    "\n",
    "# Add documents to Company B\n",
    "for i in range(15):\n",
    "    manager.add_vector_to_namespace(f\"company_b_doc_{i}\", \"company_b\")\n",
    "\n",
    "# Get each company's documents\n",
    "a_docs = manager.get_namespace_vectors(\"company_a\")\n",
    "b_docs = manager.get_namespace_vectors(\"company_b\")\n",
    "\n",
    "print(f\"Company A has {len(a_docs)} documents\")\n",
    "print(f\"Company B has {len(b_docs)} documents\")\n",
    "\n",
    "# Set permissions (who can access what)\n",
    "company_a.set_access_control(\"admin@companya.com\", [\"read\", \"write\", \"delete\"])\n",
    "company_a.set_access_control(\"user@companya.com\", [\"read\"])  # Read-only\n",
    "\n",
    "# Check permissions\n",
    "print(f\"\\nAdmin can delete: {company_a.has_permission('admin@companya.com', 'delete')}\")\n",
    "print(f\"User can delete: {company_a.has_permission('user@companya.com', 'delete')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Reference Guide\n",
    "\n",
    "### Which Index Should I Use?\n",
    "\n",
    "```python\n",
    "# Small dataset (< 10,000 items)\n",
    "index = adapter.create_index(index_type=\"flat\", metric=\"L2\")\n",
    "\n",
    "# Medium dataset (10,000 - 1,000,000 items) ✅ RECOMMENDED\n",
    "index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
    "\n",
    "# Large dataset (> 1,000,000 items)\n",
    "index = adapter.create_index(index_type=\"ivf\", metric=\"L2\", nlist=100)\n",
    "```\n",
    "\n",
    "### How Do I Filter Results?\n",
    "\n",
    "```python\n",
    "# Single condition\n",
    "filter = MetadataFilter().eq(\"category\", \"Technology\")\n",
    "\n",
    "# Multiple conditions (AND)\n",
    "filter = MetadataFilter() \\\n",
    "    .eq(\"category\", \"Technology\") \\\n",
    "    .eq(\"year\", 2024)\n",
    "\n",
    "# Greater than / Less than\n",
    "filter = MetadataFilter().gt(\"year\", 2020)\n",
    "```\n",
    "\n",
    "### How Do I Combine Results?\n",
    "\n",
    "```python\n",
    "# Fair combination\n",
    "ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
    "combined = ranker.rank([results1, results2])\n",
    "\n",
    "# Weighted combination (prefer first source)\n",
    "ranker = SearchRanker(strategy=\"weighted_average\")\n",
    "combined = ranker.rank([results1, results2], weights=[0.7, 0.3])\n",
    "```\n",
    "\n",
    "### How Do I Separate User Data?\n",
    "\n",
    "```python\n",
    "# Create namespace for each user/company\n",
    "manager = NamespaceManager()\n",
    "user_space = manager.create_namespace(\"user_123\", \"User 123's data\")\n",
    "\n",
    "# Add data to namespace\n",
    "manager.add_vector_to_namespace(\"doc_1\", \"user_123\")\n",
    "\n",
    "# Get user's data\n",
    "user_docs = manager.get_namespace_vectors(\"user_123\")\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "You've learned:\n",
    "\n",
    "1. ✅ **Index Selection**: Use HNSW for most cases\n",
    "2. ✅ **Smart Filtering**: Combine vector search with metadata\n",
    "3. ✅ **Result Fusion**: Merge searches from different sources\n",
    "4. ✅ **Data Isolation**: Keep users' data separate\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "- Try these examples with your own data\n",
    "- Experiment with different filters\n",
    "- Build a multi-user application\n",
    "- Explore the [introduction notebook](../introduction/13_Vector_Store.ipynb) for more basics\n",
    "\n",
    "**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/Hawksight-AI/semantica)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
