docs(vector_store): finalize documentation with simplified notebooks

- Enhanced docs/reference/vector_store.md (~575 lines)
  - All 32 classes documented
  - All 10 convenience functions
  - Complete adapter documentation

- Updated cookbook/introduction/13_Vector_Store.ipynb
  - 10-step comprehensive guide

- Created cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb
  - 4 focused parts (removed error handling per user request)
  - Part 1: Index selection (Flat, HNSW, IVF)
  - Part 2: Smart filtering with metadata
  - Part 3: Result fusion (RRF, weighted)
  - Part 4: Multi-tenant data isolation
  - Beginner-friendly with clear examples
  - Quick reference guide included

All vector_store documentation complete and production-ready.
This commit is contained in:
KaifAhmad1
2025-12-08 19:55:23 +05:30
parent 6c555b49a0
commit 06145fd4b9
3 changed files with 1474 additions and 151 deletions
@@ -0,0 +1,346 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"\n",
"# Advanced Vector Store - Made Easy\n",
"\n",
"## What You'll Learn\n",
"\n",
"This notebook shows you **practical ways** to use vector stores in real applications. Each example is simple and ready to use.\n",
"\n",
"### Topics\n",
"\n",
"1. **Choosing the Right Index** - Which one to use and when\n",
"2. **Smart Filtering** - Find exactly what you need\n",
"3. **Combining Results** - Merge searches from different sources\n",
"4. **Organizing Data** - Keep different users' data separate\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 1: Choosing the Right Index\n",
"\n",
"Think of an index like choosing a filing system:\n",
"- **Flat**: Like a small notebook - slow but perfect\n",
"- **HNSW**: Like a well-organized library - fast and accurate\n",
"- **IVF**: Like a warehouse with sections - very fast for huge collections\n",
"\n",
"### Simple Rule\n",
"- Less than 10,000 items? Use **Flat**\n",
"- Between 10,000 and 1 million? Use **HNSW** ✅ (recommended)\n",
"- More than 1 million? Use **IVF**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import FAISSAdapter\n",
"import numpy as np\n",
"\n",
"# Create some example vectors (like document embeddings)\n",
"vectors = np.random.rand(5000, 768).astype('float32')\n",
"query = np.random.rand(768).astype('float32')\n",
"\n",
"adapter = FAISSAdapter(dimension=768)\n",
"\n",
"# HNSW Index - Best for most cases\n",
"index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
"adapter.add_vectors(index, vectors, ids=[f\"doc_{i}\" for i in range(len(vectors))])\n",
"\n",
"# Search for similar vectors\n",
"distances, indices = adapter.search(index, query, k=5)\n",
"\n",
"print(\"Found 5 most similar documents:\")\n",
"for i, (dist, idx) in enumerate(zip(distances, indices), 1):\n",
" print(f\" {i}. Document {idx} (distance: {dist:.3f})\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 2: Smart Filtering with Metadata\n",
"\n",
"Imagine searching for \"similar articles\" but only from 2024 and only in the \"Technology\" category. That's what metadata filtering does!\n",
"\n",
"### Real-World Example\n",
"You're building a document search where users want:\n",
"- Similar documents (vector search)\n",
"- From specific categories (metadata filter)\n",
"- From recent years (metadata filter)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import HybridSearch, MetadataFilter\n",
"import numpy as np\n",
"\n",
"# Create sample documents with metadata\n",
"documents = [\n",
" {\"id\": 0, \"text\": \"AI in Healthcare\", \"category\": \"Technology\", \"year\": 2024},\n",
" {\"id\": 1, \"text\": \"Machine Learning Basics\", \"category\": \"Technology\", \"year\": 2023},\n",
" {\"id\": 2, \"text\": \"Business Strategy\", \"category\": \"Business\", \"year\": 2024},\n",
" {\"id\": 3, \"text\": \"Data Science Guide\", \"category\": \"Technology\", \"year\": 2024},\n",
" {\"id\": 4, \"text\": \"Marketing Tips\", \"category\": \"Business\", \"year\": 2023},\n",
"]\n",
"\n",
"# Create vectors for each document\n",
"vectors = [np.random.rand(768) for _ in documents]\n",
"metadata = [{\"category\": d[\"category\"], \"year\": d[\"year\"]} for d in documents]\n",
"vector_ids = [f\"doc_{d['id']}\" for d in documents]\n",
"\n",
"# Create search\n",
"search = HybridSearch()\n",
"query = np.random.rand(768)\n",
"\n",
"# Example 1: Find Technology articles from 2024\n",
"filter1 = MetadataFilter().eq(\"category\", \"Technology\").eq(\"year\", 2024)\n",
"results = search.search(query, vectors, metadata, vector_ids, filter=filter1, k=10)\n",
"\n",
"print(\"Technology articles from 2024:\")\n",
"for r in results:\n",
" doc_id = int(r['id'].split('_')[1])\n",
" print(f\" - {documents[doc_id]['text']}\")\n",
"\n",
"# Example 2: Find any article from 2024\n",
"filter2 = MetadataFilter().eq(\"year\", 2024)\n",
"results2 = search.search(query, vectors, metadata, vector_ids, filter=filter2, k=10)\n",
"\n",
"print(\"\\nAll articles from 2024:\")\n",
"for r in results2:\n",
" doc_id = int(r['id'].split('_')[1])\n",
" print(f\" - {documents[doc_id]['text']} ({documents[doc_id]['category']})\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 3: Combining Search Results\n",
"\n",
"Sometimes you want to search in multiple places and combine the results. Like searching both your email and documents, then showing the best matches from both.\n",
"\n",
"### When to Use This\n",
"- Searching multiple databases\n",
"- Combining different search strategies\n",
"- Giving more weight to certain sources"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import SearchRanker\n",
"\n",
"# Simulate two different searches\n",
"# Search 1: Recent documents\n",
"recent_results = [\n",
" {\"id\": \"doc_3\", \"score\": 0.95, \"source\": \"recent\"},\n",
" {\"id\": \"doc_0\", \"score\": 0.90, \"source\": \"recent\"},\n",
" {\"id\": \"doc_2\", \"score\": 0.85, \"source\": \"recent\"},\n",
"]\n",
"\n",
"# Search 2: Popular documents\n",
"popular_results = [\n",
" {\"id\": \"doc_1\", \"score\": 0.92, \"source\": \"popular\"},\n",
" {\"id\": \"doc_3\", \"score\": 0.88, \"source\": \"popular\"},\n",
" {\"id\": \"doc_4\", \"score\": 0.80, \"source\": \"popular\"},\n",
"]\n",
"\n",
"# Method 1: Fair combination (RRF)\n",
"ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
"combined = ranker.rank([recent_results, popular_results])\n",
"\n",
"print(\"Combined results (fair ranking):\")\n",
"for i, result in enumerate(combined[:3], 1):\n",
" doc_id = int(result['id'].split('_')[1])\n",
" print(f\" {i}. {documents[doc_id]['text']} (score: {result['score']:.3f})\")\n",
"\n",
"# Method 2: Prefer recent documents (70% recent, 30% popular)\n",
"weighted_ranker = SearchRanker(strategy=\"weighted_average\")\n",
"weighted_combined = weighted_ranker.rank(\n",
" [recent_results, popular_results],\n",
" weights=[0.7, 0.3]\n",
")\n",
"\n",
"print(\"\\nCombined results (prefer recent):\")\n",
"for i, result in enumerate(weighted_combined[:3], 1):\n",
" doc_id = int(result['id'].split('_')[1])\n",
" print(f\" {i}. {documents[doc_id]['text']} (score: {result['score']:.3f})\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 4: Keeping User Data Separate\n",
"\n",
"If you're building an app with multiple users or companies, you need to keep their data separate. Namespaces do this automatically.\n",
"\n",
"### Real Example\n",
"You're building a SaaS app where:\n",
"- Company A has their documents\n",
"- Company B has their documents\n",
"- They should never see each other's data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import NamespaceManager\n",
"\n",
"# Create manager\n",
"manager = NamespaceManager()\n",
"\n",
"# Create separate spaces for each company\n",
"company_a = manager.create_namespace(\"company_a\", \"Company A's documents\")\n",
"company_b = manager.create_namespace(\"company_b\", \"Company B's documents\")\n",
"\n",
"# Add documents to Company A\n",
"for i in range(10):\n",
" manager.add_vector_to_namespace(f\"company_a_doc_{i}\", \"company_a\")\n",
"\n",
"# Add documents to Company B\n",
"for i in range(15):\n",
" manager.add_vector_to_namespace(f\"company_b_doc_{i}\", \"company_b\")\n",
"\n",
"# Get each company's documents\n",
"a_docs = manager.get_namespace_vectors(\"company_a\")\n",
"b_docs = manager.get_namespace_vectors(\"company_b\")\n",
"\n",
"print(f\"Company A has {len(a_docs)} documents\")\n",
"print(f\"Company B has {len(b_docs)} documents\")\n",
"\n",
"# Set permissions (who can access what)\n",
"company_a.set_access_control(\"admin@companya.com\", [\"read\", \"write\", \"delete\"])\n",
"company_a.set_access_control(\"user@companya.com\", [\"read\"]) # Read-only\n",
"\n",
"# Check permissions\n",
"print(f\"\\nAdmin can delete: {company_a.has_permission('admin@companya.com', 'delete')}\")\n",
"print(f\"User can delete: {company_a.has_permission('user@companya.com', 'delete')}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Quick Reference Guide\n",
"\n",
"### Which Index Should I Use?\n",
"\n",
"```python\n",
"# Small dataset (< 10,000 items)\n",
"index = adapter.create_index(index_type=\"flat\", metric=\"L2\")\n",
"\n",
"# Medium dataset (10,000 - 1,000,000 items) ✅ RECOMMENDED\n",
"index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
"\n",
"# Large dataset (> 1,000,000 items)\n",
"index = adapter.create_index(index_type=\"ivf\", metric=\"L2\", nlist=100)\n",
"```\n",
"\n",
"### How Do I Filter Results?\n",
"\n",
"```python\n",
"# Single condition\n",
"filter = MetadataFilter().eq(\"category\", \"Technology\")\n",
"\n",
"# Multiple conditions (AND)\n",
"filter = MetadataFilter() \\\n",
" .eq(\"category\", \"Technology\") \\\n",
" .eq(\"year\", 2024)\n",
"\n",
"# Greater than / Less than\n",
"filter = MetadataFilter().gt(\"year\", 2020)\n",
"```\n",
"\n",
"### How Do I Combine Results?\n",
"\n",
"```python\n",
"# Fair combination\n",
"ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
"combined = ranker.rank([results1, results2])\n",
"\n",
"# Weighted combination (prefer first source)\n",
"ranker = SearchRanker(strategy=\"weighted_average\")\n",
"combined = ranker.rank([results1, results2], weights=[0.7, 0.3])\n",
"```\n",
"\n",
"### How Do I Separate User Data?\n",
"\n",
"```python\n",
"# Create namespace for each user/company\n",
"manager = NamespaceManager()\n",
"user_space = manager.create_namespace(\"user_123\", \"User 123's data\")\n",
"\n",
"# Add data to namespace\n",
"manager.add_vector_to_namespace(\"doc_1\", \"user_123\")\n",
"\n",
"# Get user's data\n",
"user_docs = manager.get_namespace_vectors(\"user_123\")\n",
"```\n",
"\n",
"---\n",
"\n",
"## Summary\n",
"\n",
"You've learned:\n",
"\n",
"1. ✅ **Index Selection**: Use HNSW for most cases\n",
"2. ✅ **Smart Filtering**: Combine vector search with metadata\n",
"3. ✅ **Result Fusion**: Merge searches from different sources\n",
"4. ✅ **Data Isolation**: Keep users' data separate\n",
"\n",
"### Next Steps\n",
"\n",
"- Try these examples with your own data\n",
"- Experiment with different filters\n",
"- Build a multi-user application\n",
"- Explore the [introduction notebook](../introduction/13_Vector_Store.ipynb) for more basics\n",
"\n",
"**Need Help?** Check our [documentation](https://semantica.readthedocs.io) or ask on [GitHub](https://github.com/Hawksight-AI/semantica)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
+566 -146
View File
@@ -1,147 +1,567 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Vector_Store.ipynb)\n",
"\n",
"# Vector Store\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates how to store and search vectors using Semantica's vector store modules. You'll learn to use `VectorStore` and `HybridSearch` for vector storage and retrieval.\n",
"\n",
"\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
"\n",
"### Learning Objectives\n",
"\n",
"- Use `VectorStore` to store vectors\n",
"- Search vectors using similarity\n",
"- Use `HybridSearch` for hybrid search\n",
"- Manage vector metadata\n",
"\n",
"## Installation\n",
"\n",
"Install Semantica from PyPI:\n",
"\n",
"```bash\n",
"pip install semantica\n",
"# Or with all optional dependencies:\n",
"pip install semantica[all]\n",
"```\n",
"\n",
"---\n",
"\n",
"## Step 1: Store Vectors\n",
"\n",
"Store vectors in the vector store.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import VectorStore\n",
"from semantica.embeddings import EmbeddingGenerator\n",
"import numpy as np\n",
"\n",
"vector_store = VectorStore()\n",
"generator = EmbeddingGenerator()\n",
"\n",
"texts = [\"Apple Inc.\", \"Microsoft Corporation\", \"Amazon Web Services\"]\n",
"embeddings = generator.generate_embeddings(texts, data_type=\"text\")\n",
"\n",
"metadata = [\n",
" {\"id\": \"1\", \"type\": \"company\"},\n",
" {\"id\": \"2\", \"type\": \"company\"},\n",
" {\"id\": \"3\", \"type\": \"service\"}\n",
"]\n",
"\n",
"vector_ids = vector_store.store_vectors(embeddings, metadata)\n",
"\n",
"print(f\"Stored {len(vector_ids)} vectors\")\n",
"print(f\"Vector IDs: {vector_ids[:3]}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Search Vectors\n",
"\n",
"Search for similar vectors.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"query_text = \"technology company\"\n",
"query_embedding = generator.generate_embeddings(query_text, data_type=\"text\")\n",
"\n",
"results = vector_store.search_vectors(query_embedding, k=3)\n",
"\n",
"print(f\"Found {len(results)} similar vectors\")\n",
"for result in results[:3]:\n",
" print(f\" ID: {result.get('id')}, Score: {result.get('score', 0):.3f}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Hybrid Search\n",
"\n",
"Use HybridSearch for combined vector and metadata search.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import HybridSearch\n",
"\n",
"hybrid_search = HybridSearch()\n",
"\n",
"hybrid_results = hybrid_search.search(\n",
" query_vector=query_embedding,\n",
" vectors=embeddings,\n",
" metadata=metadata,\n",
" vector_ids=vector_ids,\n",
" k=3\n",
")\n",
"\n",
"print(f\"Hybrid search found {len(hybrid_results)} results\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"You've learned how to use vector stores:\n",
"\n",
"- **VectorStore**: Store and search vectors\n",
"- **HybridSearch**: Hybrid vector and metadata search\n",
"\n",
"Next: Learn how to generate ontologies in the Ontology notebook.\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
"\n",
"# Vector Store - Comprehensive Guide\n",
"\n",
"## Overview\n",
"\n",
"This notebook provides a **comprehensive walkthrough** of Semantica's vector_store module, demonstrating vector storage, similarity search, hybrid search, and multi-backend support for semantic retrieval.\n",
"\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
"\n",
"### Learning Objectives\n",
"\n",
"By the end of this notebook, you will be able to:\n",
"\n",
"- Store and manage vectors with metadata\n",
"- Perform similarity search with different metrics\n",
"- Use hybrid search combining vectors and metadata\n",
"- Work with multiple vector store backends (FAISS, Pinecone, etc.)\n",
"- Create and manage vector indices\n",
"- Filter and rank search results\n",
"- Implement namespace isolation for multi-tenancy\n",
"\n",
"### What You'll Learn\n",
"\n",
"| Component | Purpose | When to Use |\n",
"|-----------|---------|-------------|\n",
"| `VectorStore` | Main vector storage | All vector operations |\n",
"| `VectorIndexer` | Index creation | Performance optimization |\n",
"| `VectorRetriever` | Similarity search | Finding similar vectors |\n",
"| `HybridSearch` | Combined search | Vector + metadata filtering |\n",
"| `MetadataFilter` | Metadata filtering | Filtering by attributes |\n",
"| `MetadataStore` | Metadata management | Storing vector metadata |\n",
"| `NamespaceManager` | Multi-tenancy | Isolating vector collections |\n",
"\n",
"---\n",
"\n",
"## Installation\n",
"\n",
"Install Semantica from PyPI:\n",
"\n",
"```bash\n",
"pip install semantica\n",
"# Or with all optional dependencies:\n",
"pip install semantica[all]\n",
"```\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Basic Vector Storage\n",
"\n",
"Let's start with the `VectorStore` for basic vector storage and retrieval.\n",
"\n",
"### What is VectorStore?\n",
"\n",
"`VectorStore` is the main interface for vector operations:\n",
"- **Storage**: Store vectors with metadata\n",
"- **Search**: Find similar vectors\n",
"- **CRUD**: Create, Read, Update, Delete operations\n",
"- **Multi-backend**: Support for FAISS, Pinecone, Weaviate, Qdrant, Milvus"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import VectorStore\n",
"import numpy as np\n",
"\n",
"# Create vector store (defaults to FAISS)\n",
"store = VectorStore(backend=\"faiss\", dimension=768)\n",
"\n",
"# Generate sample vectors\n",
"vectors = [np.random.rand(768) for _ in range(100)]\n",
"metadata = [\n",
" {\"text\": f\"Document {i}\", \"category\": \"science\" if i % 2 == 0 else \"technology\", \"year\": 2020 + (i % 4)}\n",
" for i in range(100)\n",
"]\n",
"\n",
"# Store vectors\n",
"vector_ids = store.store_vectors(vectors, metadata=metadata)\n",
"\n",
"print(f\"Stored {len(vector_ids)} vectors\")\n",
"print(f\"First 3 IDs: {vector_ids[:3]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Similarity Search\n",
"\n",
"Search for similar vectors using different similarity metrics.\n",
"\n",
"### Similarity Metrics\n",
"\n",
"- **Cosine Similarity**: Best for semantic similarity\n",
"- **L2 Distance**: Euclidean distance\n",
"- **Dot Product**: Fast, requires normalized vectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create query vector\n",
"query_vector = np.random.rand(768)\n",
"\n",
"# Search for similar vectors\n",
"results = store.search_vectors(query_vector, k=10)\n",
"\n",
"print(f\"Found {len(results)} similar vectors\")\n",
"print(\"\\nTop 5 results:\")\n",
"for i, result in enumerate(results[:5], 1):\n",
" print(f\"{i}. ID: {result['id']}, Score: {result['score']:.3f}\")\n",
" print(f\" Metadata: {result.get('metadata', {})}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Vector Indexing\n",
"\n",
"Create indices for faster search on large datasets.\n",
"\n",
"### Index Types (FAISS)\n",
"\n",
"- **Flat**: Exact search (brute force)\n",
"- **IVF**: Inverted file index (approximate)\n",
"- **HNSW**: Hierarchical graph (best balance)\n",
"- **PQ**: Product quantization (compressed)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import VectorIndexer, FAISSAdapter\n",
"\n",
"# Create indexer\n",
"indexer = VectorIndexer(backend=\"faiss\", dimension=768)\n",
"\n",
"# Create HNSW index for fast approximate search\n",
"adapter = FAISSAdapter(dimension=768)\n",
"index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
"\n",
"# Add vectors to index\n",
"vectors_array = np.array(vectors).astype('float32')\n",
"adapter.add_vectors(index, vectors_array, ids=vector_ids)\n",
"\n",
"# Search using index\n",
"query_array = query_vector.astype('float32')\n",
"distances, indices = adapter.search(index, query_array, k=10)\n",
"\n",
"print(f\"Index search found {len(indices)} results\")\n",
"print(f\"Distances: {distances[:5]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Hybrid Search\n",
"\n",
"Combine vector similarity with metadata filtering.\n",
"\n",
"### Hybrid Search Benefits\n",
"\n",
"- Filter by metadata before vector search\n",
"- Combine multiple search criteria\n",
"- More precise results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import HybridSearch, MetadataFilter\n",
"\n",
"# Create hybrid search\n",
"hybrid_search = HybridSearch()\n",
"\n",
"# Create metadata filter\n",
"filter = MetadataFilter() \\\n",
" .eq(\"category\", \"science\") \\\n",
" .gt(\"year\", 2021)\n",
"\n",
"# Perform hybrid search\n",
"hybrid_results = hybrid_search.search(\n",
" query_vector,\n",
" vectors,\n",
" metadata,\n",
" vector_ids,\n",
" filter=filter,\n",
" k=10\n",
")\n",
"\n",
"print(f\"Hybrid search found {len(hybrid_results)} results\")\n",
"print(\"\\nFiltered results (science, year > 2021):\")\n",
"for i, result in enumerate(hybrid_results[:5], 1):\n",
" meta = result.get('metadata', {})\n",
" print(f\"{i}. Category: {meta.get('category')}, Year: {meta.get('year')}, Score: {result['score']:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Metadata Management\n",
"\n",
"Store and query metadata separately from vectors.\n",
"\n",
"### Metadata Operations\n",
"\n",
"- Store metadata for vectors\n",
"- Query by metadata conditions\n",
"- Update metadata\n",
"- Schema validation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import MetadataStore, MetadataSchema\n",
"\n",
"# Create metadata store\n",
"meta_store = MetadataStore()\n",
"\n",
"# Store metadata\n",
"for i, vec_id in enumerate(vector_ids[:10]):\n",
" meta_store.store_metadata(vec_id, metadata[i])\n",
"\n",
"# Query metadata\n",
"matching_ids = meta_store.query_metadata(\n",
" {\"category\": \"science\"},\n",
" operator=\"AND\"\n",
")\n",
"\n",
"print(f\"Found {len(matching_ids)} vectors with category='science'\")\n",
"\n",
"# Define schema for validation\n",
"schema = MetadataSchema({\n",
" \"text\": {\"type\": str, \"required\": True},\n",
" \"category\": {\"type\": str, \"required\": True},\n",
" \"year\": {\"type\": int, \"required\": True}\n",
"})\n",
"\n",
"# Validate metadata\n",
"is_valid = schema.validate(metadata[0])\n",
"print(f\"\\nMetadata validation: {is_valid}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Result Ranking and Fusion\n",
"\n",
"Combine and rank results from multiple searches.\n",
"\n",
"### Ranking Strategies\n",
"\n",
"- **Reciprocal Rank Fusion (RRF)**: Combine ranked lists\n",
"- **Weighted Average**: Weight scores from different sources"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import SearchRanker\n",
"\n",
"# Create ranker with RRF strategy\n",
"ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
"\n",
"# Simulate multiple search results\n",
"results1 = [\n",
" {\"id\": \"vec_1\", \"score\": 0.9},\n",
" {\"id\": \"vec_2\", \"score\": 0.8},\n",
" {\"id\": \"vec_3\", \"score\": 0.7}\n",
"]\n",
"\n",
"results2 = [\n",
" {\"id\": \"vec_2\", \"score\": 0.85},\n",
" {\"id\": \"vec_4\", \"score\": 0.75},\n",
" {\"id\": \"vec_1\", \"score\": 0.7}\n",
"]\n",
"\n",
"# Fuse results using RRF\n",
"fused_results = ranker.rank([results1, results2], k=60)\n",
"\n",
"print(\"Fused results using RRF:\")\n",
"for i, result in enumerate(fused_results, 1):\n",
" print(f\"{i}. ID: {result['id']}, Fused Score: {result['score']:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 7: Namespace Management\n",
"\n",
"Isolate vectors for multi-tenant applications.\n",
"\n",
"### Namespace Features\n",
"\n",
"- Tenant isolation\n",
"- Access control\n",
"- Per-namespace operations"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import NamespaceManager\n",
"\n",
"# Create namespace manager\n",
"ns_manager = NamespaceManager()\n",
"\n",
"# Create namespaces for different tenants\n",
"ns1 = ns_manager.create_namespace(\"tenant1\", \"Tenant 1 vectors\")\n",
"ns2 = ns_manager.create_namespace(\"tenant2\", \"Tenant 2 vectors\")\n",
"\n",
"# Add vectors to namespaces\n",
"for i in range(5):\n",
" ns_manager.add_vector_to_namespace(f\"t1_vec_{i}\", \"tenant1\")\n",
" ns_manager.add_vector_to_namespace(f\"t2_vec_{i}\", \"tenant2\")\n",
"\n",
"# Get namespace vectors\n",
"tenant1_vectors = ns_manager.get_namespace_vectors(\"tenant1\")\n",
"tenant2_vectors = ns_manager.get_namespace_vectors(\"tenant2\")\n",
"\n",
"print(f\"Tenant 1: {len(tenant1_vectors)} vectors\")\n",
"print(f\"Tenant 2: {len(tenant2_vectors)} vectors\")\n",
"\n",
"# Set access control\n",
"ns1.set_access_control(\"user1\", [\"read\", \"write\"])\n",
"ns1.set_access_control(\"user2\", [\"read\"])\n",
"\n",
"print(f\"\\nUser1 can write: {ns1.has_permission('user1', 'write')}\")\n",
"print(f\"User2 can write: {ns1.has_permission('user2', 'write')}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 8: Convenience Functions\n",
"\n",
"Use convenience functions for quick operations.\n",
"\n",
"### Available Functions\n",
"\n",
"- `store_vectors()`: Store vectors\n",
"- `search_vectors()`: Search vectors\n",
"- `hybrid_search()`: Hybrid search\n",
"- `update_vectors()`: Update vectors\n",
"- `delete_vectors()`: Delete vectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import (\n",
" store_vectors,\n",
" search_vectors,\n",
" hybrid_search as hybrid_search_func,\n",
" update_vectors,\n",
" delete_vectors\n",
")\n",
"\n",
"# Store vectors using convenience function\n",
"new_vectors = [np.random.rand(768) for _ in range(10)]\n",
"new_metadata = [{\"text\": f\"New doc {i}\"} for i in range(10)]\n",
"new_ids = store_vectors(new_vectors, metadata=new_metadata, method=\"default\")\n",
"\n",
"print(f\"Stored {len(new_ids)} new vectors\")\n",
"\n",
"# Search using convenience function\n",
"search_results = search_vectors(\n",
" query_vector,\n",
" new_vectors,\n",
" new_ids,\n",
" k=5,\n",
" method=\"default\"\n",
")\n",
"\n",
"print(f\"Search found {len(search_results)} results\")\n",
"\n",
"# Update vectors\n",
"updated_vectors = [np.random.rand(768) for _ in range(2)]\n",
"success = update_vectors(new_ids[:2], updated_vectors, method=\"default\")\n",
"print(f\"\\nUpdated vectors: {success}\")\n",
"\n",
"# Delete vectors\n",
"success = delete_vectors(new_ids[-2:], method=\"default\")\n",
"print(f\"Deleted vectors: {success}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 9: Multi-Backend Support\n",
"\n",
"Work with different vector store backends.\n",
"\n",
"### Supported Backends\n",
"\n",
"| Backend | Type | Best For |\n",
"|---------|------|----------|\n",
"| FAISS | Local | Development, small datasets |\n",
"| Pinecone | Cloud | Production, managed service |\n",
"| Weaviate | Self-hosted | Schema-aware storage |\n",
"| Qdrant | Self-hosted | High performance |\n",
"| Milvus | Cloud/Self-hosted | Large scale |"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.vector_store import FAISSAdapter, VectorManager\n",
"\n",
"# FAISS (local)\n",
"faiss_adapter = FAISSAdapter(dimension=768)\n",
"faiss_index = faiss_adapter.create_index(index_type=\"flat\", metric=\"L2\")\n",
"print(\"Created FAISS index\")\n",
"\n",
"# Vector Manager for multi-store management\n",
"manager = VectorManager()\n",
"faiss_store = manager.create_store(\"faiss\", {\"dimension\": 768})\n",
"print(f\"\\nCreated store via manager\")\n",
"\n",
"# List all stores\n",
"stores = manager.list_stores()\n",
"print(f\"Active stores: {stores}\")\n",
"\n",
"# Note: For cloud backends (Pinecone, Weaviate, etc.),\n",
"# you would need API keys and endpoints\n",
"# Example:\n",
"# from semantica.vector_store import PineconeAdapter\n",
"# pinecone = PineconeAdapter(api_key=\"your-key\", environment=\"us-west1-gcp\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 10: Best Practices\n",
"\n",
"### Performance Tips\n",
"\n",
"1. **Normalize Vectors**: Always normalize for cosine similarity\n",
"2. **Use HNSW**: Best balance for speed/accuracy\n",
"3. **Batch Operations**: Process in batches (100-1000)\n",
"4. **Filter First**: Apply metadata filters before vector search\n",
"\n",
"### Backend Selection\n",
"\n",
"- **Development**: FAISS (local, fast)\n",
"- **Production**: Pinecone (managed, scalable)\n",
"- **Self-hosted**: Qdrant or Milvus (control, performance)\n",
"- **Schema-aware**: Weaviate (rich metadata)\n",
"\n",
"### Index Configuration\n",
"\n",
"- **Small datasets (<10K)**: Flat index\n",
"- **Medium datasets (10K-1M)**: HNSW\n",
"- **Large datasets (>1M)**: IVF + PQ"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"### What You've Learned\n",
"\n",
"In this notebook, you've learned how to:\n",
"\n",
"- Store and search vectors with VectorStore\n",
"- Create indices for performance optimization\n",
"- Use hybrid search with metadata filtering\n",
"- Manage metadata separately from vectors\n",
"- Rank and fuse search results\n",
"- Implement namespace isolation\n",
"- Use convenience functions for quick operations\n",
"- Work with multiple backend adapters\n",
"- Apply best practices for production use\n",
"\n",
"### Key Takeaways\n",
"\n",
"1. **Multi-Backend**: Choose the right backend for your needs\n",
"2. **Hybrid Search**: Combine vectors with metadata for precision\n",
"3. **Indexing**: Use appropriate index types for performance\n",
"4. **Metadata**: Separate metadata management for flexibility\n",
"5. **Namespaces**: Isolate vectors for multi-tenancy\n",
"\n",
"### Next Steps\n",
"\n",
"**Further Reading**:\n",
"- [Vector Store API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
"- [Advanced Vector Store Notebook](../advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
"- [Embedding Generation](12_Embedding_Generation.ipynb)\n",
"\n",
"---\n",
"\n",
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}