mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
KaifAhmad1
7a6f1d0417
Add a Cite Us section to the README with BibTeX citation info, and align it with docs/citation.md (author/organization: Semantica, 2026). Update LICENSE and docs/project-license.md copyright holder to Semantica, and replace the stale Hawksight-AI GitHub org slug with semantica-agi across READMEs, plugin manifests, cookbook notebooks, and GitHub templates.
521 lines
17 KiB
Plaintext
521 lines
17 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)\n",
|
|
"\n",
|
|
"# Vector Store - Comprehensive Guide\n",
|
|
"\n",
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook provides a **comprehensive walkthrough** of Semantica's vector_store module, demonstrating vector storage, similarity search, hybrid search, and multi-backend support for semantic retrieval.\n",
|
|
"\n",
|
|
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
|
|
"\n",
|
|
"### Learning Objectives\n",
|
|
"\n",
|
|
"By the end of this notebook, you will be able to:\n",
|
|
"\n",
|
|
"- Store and manage vectors with metadata\n",
|
|
"- Perform similarity search with different metrics\n",
|
|
"- Use hybrid search combining vectors and metadata\n",
|
|
"- Work with multiple vector store backends (FAISS, Weaviate, etc.)\n",
|
|
"- Create and manage vector indices\n",
|
|
"- Filter and rank search results\n",
|
|
"- Implement namespace isolation for multi-tenancy\n",
|
|
"\n",
|
|
"### What You'll Learn\n",
|
|
"\n",
|
|
"| Component | Purpose | When to Use |\n",
|
|
"|-----------|---------|-------------|\n",
|
|
"| `VectorStore` | Main vector storage | All vector operations |\n",
|
|
"| `VectorIndexer` | Index creation | Performance optimization |\n",
|
|
"| `VectorRetriever` | Similarity search | Finding similar vectors |\n",
|
|
"| `HybridSearch` | Combined search | Vector + metadata filtering |\n",
|
|
"| `MetadataFilter` | Metadata filtering | Filtering by attributes |\n",
|
|
"| `MetadataStore` | Metadata management | Storing vector metadata |\n",
|
|
"| `NamespaceManager` | Multi-tenancy | Isolating vector collections |\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"## Installation\n",
|
|
"\n",
|
|
"Install Semantica from PyPI:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"pip install semantica\n",
|
|
"# Or with all optional dependencies:\n",
|
|
"pip install semantica[all]\n",
|
|
"```\n",
|
|
"\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install -q semantica\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 1: Basic Vector Storage\n",
|
|
"\n",
|
|
"Let's start with the `VectorStore` for basic vector storage and retrieval.\n",
|
|
"\n",
|
|
"### What is VectorStore?\n",
|
|
"\n",
|
|
"`VectorStore` is the main interface for vector operations:\n",
|
|
"- **Storage**: Store vectors with metadata\n",
|
|
"- **Search**: Find similar vectors\n",
|
|
"- **CRUD**: Create, Read, Update, Delete operations\n",
|
|
"- **Multi-backend**: Support for FAISS, Weaviate, Qdrant, Milvus"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import VectorStore\n",
|
|
"from semantica.embeddings import TextEmbedder\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"# 1. Initialize Embedder (Select Provider & Model)\n",
|
|
"# You can choose 'sentence_transformers' or 'fastembed'\n",
|
|
"embedder = TextEmbedder(method=\"sentence_transformers\", model_name=\"all-MiniLM-L6-v2\")\n",
|
|
"dimension = embedder.get_embedding_dimension()\n",
|
|
"\n",
|
|
"# 2. Create vector store\n",
|
|
"store = VectorStore(backend=\"faiss\", dimension=dimension)\n",
|
|
"\n",
|
|
"# 3. Generate Real Embeddings\n",
|
|
"texts = [f\"Document {i}\" for i in range(100)]\n",
|
|
"vectors = embedder.embed_batch(texts)\n",
|
|
"\n",
|
|
"metadata = [\n",
|
|
" {\"text\": txt, \"category\": \"science\" if i % 2 == 0 else \"technology\", \"year\": 2020 + (i % 4)}\n",
|
|
" for i, txt in enumerate(texts)\n",
|
|
"]\n",
|
|
"# 4. Store vectors\n",
|
|
"vector_ids = store.store_vectors(vectors, metadata=metadata)\n",
|
|
"\n",
|
|
"print(f\"Stored {len(vector_ids)} vectors\")\n",
|
|
"print(f\"First 3 IDs: {vector_ids[:3]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 2: Similarity Search\n",
|
|
"\n",
|
|
"Search for similar vectors using different similarity metrics.\n",
|
|
"\n",
|
|
"### Similarity Metrics\n",
|
|
"\n",
|
|
"- **Cosine Similarity**: Best for semantic similarity\n",
|
|
"- **L2 Distance**: Euclidean distance\n",
|
|
"- **Dot Product**: Fast, requires normalized vectors"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import VectorIndexer, FAISSStore \n",
|
|
"import numpy as np \n",
|
|
"# We use the first vector from the dataset as a sample query\n",
|
|
"if 'query_vector' not in locals():\n",
|
|
" if 'vectors' in locals() and len(vectors) > 0:\n",
|
|
" query_vector = vectors[0]\n",
|
|
" else:\n",
|
|
" # Fallback if vectors are also missing (safety check)\n",
|
|
" query_vector = np.random.rand(dimension).astype('float32')\n",
|
|
"\n",
|
|
"# Create indexer \n",
|
|
"indexer = VectorIndexer(backend=\"faiss\", dimension=dimension) \n",
|
|
"\n",
|
|
"# Create HNSW index for fast approximate search \n",
|
|
"adapter = FAISSStore(dimension=dimension) \n",
|
|
"index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16) \n",
|
|
"\n",
|
|
"# Add vectors to index \n",
|
|
"vectors_array = np.array(vectors).astype('float32') \n",
|
|
"# FIX: Removed 'index' argument. The adapter uses its internal self.index \n",
|
|
"adapter.add_vectors(vectors_array, ids=vector_ids) \n",
|
|
"\n",
|
|
"# Search using index \n",
|
|
"query_array = np.array(query_vector).astype('float32') \n",
|
|
"# FIX: Call search on the 'index' object directly, not the adapter \n",
|
|
"distances, indices = index.search(query_array.reshape(1, -1), k=10) \n",
|
|
"\n",
|
|
"print(f\"Index search found {len(indices[0])} results\") \n",
|
|
"print(f\"Distances: {distances[0][:5]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 3: Vector Indexing\n",
|
|
"\n",
|
|
"Create indices for faster search on large datasets.\n",
|
|
"\n",
|
|
"### Index Types (FAISS)\n",
|
|
"\n",
|
|
"- **Flat**: Exact search (brute force)\n",
|
|
"- **IVF**: Inverted file index (approximate)\n",
|
|
"- **HNSW**: Hierarchical graph (best balance)\n",
|
|
"- **PQ**: Product quantization (compressed)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import VectorIndexer, FAISSStore\n",
|
|
"\n",
|
|
"# Get dimension from vectors to ensure consistency\n",
|
|
"dimension = len(vectors[0]) if len(vectors) > 0 else 384\n",
|
|
"\n",
|
|
"# Create indexer\n",
|
|
"indexer = VectorIndexer(backend=\"faiss\", dimension=dimension)\n",
|
|
"\n",
|
|
"# Create HNSW index for fast approximate search\n",
|
|
"adapter = FAISSStore(dimension=dimension)\n",
|
|
"index = adapter.create_index(index_type=\"hnsw\", metric=\"L2\", m=16)\n",
|
|
"\n",
|
|
"# Add vectors to index\n",
|
|
"vectors_array = np.array(vectors).astype('float32')\n",
|
|
"adapter.add_vectors(vectors_array, ids=vector_ids)\n",
|
|
"\n",
|
|
"# Search using index\n",
|
|
"query_array = query_vector.astype('float32')\n",
|
|
"distances, indices = index.search(query_array.reshape(1, -1), k=10)\n",
|
|
"\n",
|
|
"print(f\"Index search found {len(indices[0])} results\")\n",
|
|
"print(f\"Distances: {distances[0][:5]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 4: Hybrid Search\n",
|
|
"\n",
|
|
"Combine vector similarity with metadata filtering.\n",
|
|
"\n",
|
|
"### Hybrid Search Benefits\n",
|
|
"\n",
|
|
"- Filter by metadata before vector search\n",
|
|
"- Combine multiple search criteria\n",
|
|
"- More precise results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import HybridSearch, MetadataFilter \n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"# Create hybrid search \n",
|
|
"hybrid_search = HybridSearch() \n",
|
|
"\n",
|
|
"# Create metadata filter \n",
|
|
"filter = MetadataFilter() \\\n",
|
|
" .eq(\"category\", \"science\") \\\n",
|
|
" .gt(\"year\", 2021) \n",
|
|
"\n",
|
|
"# Perform hybrid search \n",
|
|
"# Ensure vectors and metadata are available\n",
|
|
"if 'vectors' not in locals() or 'metadata' not in locals() or 'vector_ids' not in locals():\n",
|
|
" print(\"Warning: vectors, metadata, or vector_ids are missing. Please run previous cells.\")\n",
|
|
"else:\n",
|
|
" hybrid_results = hybrid_search.search( \n",
|
|
" query_vector, \n",
|
|
" vectors, \n",
|
|
" metadata, \n",
|
|
" vector_ids, \n",
|
|
" filter=filter, \n",
|
|
" k=10 \n",
|
|
" ) \n",
|
|
" \n",
|
|
" print(f\"Hybrid search found {len(hybrid_results)} results\") \n",
|
|
" print(\"\\nFiltered results (science, year > 2021):\") \n",
|
|
" for i, result in enumerate(hybrid_results[:5], 1): \n",
|
|
" meta = result.get('metadata', {}) \n",
|
|
" print(f\"{i}. Category: {meta.get('category')}, Year: {meta.get('year')}, Score: {result['score']:.3f}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 5: Metadata Management\n",
|
|
"\n",
|
|
"Store and query metadata separately from vectors.\n",
|
|
"\n",
|
|
"### Metadata Operations\n",
|
|
"\n",
|
|
"- Store metadata for vectors\n",
|
|
"- Query by metadata conditions\n",
|
|
"- Update metadata\n",
|
|
"- Schema validation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import MetadataStore, MetadataSchema\n",
|
|
"\n",
|
|
"# Create metadata store\n",
|
|
"meta_store = MetadataStore()\n",
|
|
"\n",
|
|
"# Store metadata\n",
|
|
"for i, vec_id in enumerate(vector_ids[:10]):\n",
|
|
" meta_store.store_metadata(vec_id, metadata[i])\n",
|
|
"\n",
|
|
"# Query metadata\n",
|
|
"matching_ids = meta_store.query_metadata(\n",
|
|
" {\"category\": \"science\"},\n",
|
|
" operator=\"AND\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Found {len(matching_ids)} vectors with category='science'\")\n",
|
|
"\n",
|
|
"# Define schema for validation\n",
|
|
"schema = MetadataSchema({\n",
|
|
" \"text\": {\"type\": str, \"required\": True},\n",
|
|
" \"category\": {\"type\": str, \"required\": True},\n",
|
|
" \"year\": {\"type\": int, \"required\": True}\n",
|
|
"})\n",
|
|
"\n",
|
|
"# Validate metadata\n",
|
|
"is_valid = schema.validate(metadata[0])\n",
|
|
"print(f\"\\nMetadata validation: {is_valid}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 6: Result Ranking and Fusion\n",
|
|
"\n",
|
|
"Combine and rank results from multiple searches.\n",
|
|
"\n",
|
|
"### Ranking Strategies\n",
|
|
"\n",
|
|
"- **Reciprocal Rank Fusion (RRF)**: Combine ranked lists\n",
|
|
"- **Weighted Average**: Weight scores from different sources"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import SearchRanker\n",
|
|
"\n",
|
|
"# Create ranker with RRF strategy\n",
|
|
"ranker = SearchRanker(strategy=\"reciprocal_rank_fusion\")\n",
|
|
"\n",
|
|
"# Simulate multiple search results\n",
|
|
"results1 = [\n",
|
|
" {\"id\": \"vec_1\", \"score\": 0.9},\n",
|
|
" {\"id\": \"vec_2\", \"score\": 0.8},\n",
|
|
" {\"id\": \"vec_3\", \"score\": 0.7}\n",
|
|
"]\n",
|
|
"\n",
|
|
"results2 = [\n",
|
|
" {\"id\": \"vec_2\", \"score\": 0.85},\n",
|
|
" {\"id\": \"vec_4\", \"score\": 0.75},\n",
|
|
" {\"id\": \"vec_1\", \"score\": 0.7}\n",
|
|
"]\n",
|
|
"\n",
|
|
"# Fuse results using RRF\n",
|
|
"fused_results = ranker.rank([results1, results2], k=60)\n",
|
|
"\n",
|
|
"print(\"Fused results using RRF:\")\n",
|
|
"for i, result in enumerate(fused_results, 1):\n",
|
|
" print(f\"{i}. ID: {result['id']}, Fused Score: {result['score']:.3f}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 7: Namespace Management\n",
|
|
"\n",
|
|
"Isolate vectors for multi-tenant applications.\n",
|
|
"\n",
|
|
"### Namespace Features\n",
|
|
"\n",
|
|
"- Tenant isolation\n",
|
|
"- Access control\n",
|
|
"- Per-namespace operations"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.vector_store import NamespaceManager\n",
|
|
"\n",
|
|
"# Create namespace manager\n",
|
|
"ns_manager = NamespaceManager()\n",
|
|
"\n",
|
|
"# Create namespaces for different tenants\n",
|
|
"ns1 = ns_manager.create_namespace(\"tenant1\", \"Tenant 1 vectors\")\n",
|
|
"ns2 = ns_manager.create_namespace(\"tenant2\", \"Tenant 2 vectors\")\n",
|
|
"\n",
|
|
"# Add vectors to namespaces\n",
|
|
"for i in range(5):\n",
|
|
" ns_manager.add_vector_to_namespace(f\"t1_vec_{i}\", \"tenant1\")\n",
|
|
" ns_manager.add_vector_to_namespace(f\"t2_vec_{i}\", \"tenant2\")\n",
|
|
"\n",
|
|
"# Get namespace vectors\n",
|
|
"tenant1_vectors = ns_manager.get_namespace_vectors(\"tenant1\")\n",
|
|
"tenant2_vectors = ns_manager.get_namespace_vectors(\"tenant2\")\n",
|
|
"\n",
|
|
"print(f\"Tenant 1: {len(tenant1_vectors)} vectors\")\n",
|
|
"print(f\"Tenant 2: {len(tenant2_vectors)} vectors\")\n",
|
|
"\n",
|
|
"# Set access control\n",
|
|
"ns1.set_access_control(\"user1\", [\"read\", \"write\"])\n",
|
|
"ns1.set_access_control(\"user2\", [\"read\"])\n",
|
|
"\n",
|
|
"print(f\"\\nUser1 can write: {ns1.has_permission('user1', 'write')}\")\n",
|
|
"print(f\"User2 can write: {ns1.has_permission('user2', 'write')}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 8: Multi-Backend Support\n",
|
|
"\n",
|
|
"Work with different vector store backends.\n",
|
|
"\n",
|
|
"### Supported Backends\n",
|
|
"\n",
|
|
"| Backend | Type | Best For |\n",
|
|
"|---------|------|----------|\n",
|
|
"| FAISS | Local | Development, small datasets |\n",
|
|
"| Weaviate | Self-hosted | Schema-aware storage |\n",
|
|
"| Qdrant | Self-hosted | High performance |\n",
|
|
"| Milvus | Cloud/Self-hosted | Large scale |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Step 9: Best Practices\n",
|
|
"\n",
|
|
"### Performance Tips\n",
|
|
"\n",
|
|
"1. **Normalize Vectors**: Always normalize for cosine similarity\n",
|
|
"2. **Use HNSW**: Best balance for speed/accuracy\n",
|
|
"3. **Batch Operations**: Process in batches (100-1000)\n",
|
|
"4. **Filter First**: Apply metadata filters before vector search\n",
|
|
"\n",
|
|
"### Backend Selection\n",
|
|
"\n",
|
|
"- **Development**: FAISS (local, fast)\n",
|
|
"- **Production**: Weaviate/Qdrant (scalable, self-hosted)\n",
|
|
"- **Self-hosted**: Qdrant or Milvus (control, performance)\n",
|
|
"- **Schema-aware**: Weaviate (rich metadata)\n",
|
|
"\n",
|
|
"### Index Configuration\n",
|
|
"\n",
|
|
"- **Small datasets (<10K)**: Flat index\n",
|
|
"- **Medium datasets (10K-1M)**: HNSW\n",
|
|
"- **Large datasets (>1M)**: IVF + PQ"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"### What You've Learned\n",
|
|
"\n",
|
|
"In this notebook, you've learned how to:\n",
|
|
"\n",
|
|
"- Store and search vectors with VectorStore\n",
|
|
"- Create indices for performance optimization\n",
|
|
"- Use hybrid search with metadata filtering\n",
|
|
"- Manage metadata separately from vectors\n",
|
|
"- Rank and fuse search results\n",
|
|
"- Implement namespace isolation\n",
|
|
"- Use convenience functions for quick operations\n",
|
|
"- Work with multiple backend adapters\n",
|
|
"- Apply best practices for production use\n",
|
|
"\n",
|
|
"### Key Takeaways\n",
|
|
"\n",
|
|
"1. **Multi-Backend**: Choose the right backend for your needs\n",
|
|
"2. **Hybrid Search**: Combine vectors with metadata for precision\n",
|
|
"3. **Indexing**: Use appropriate index types for performance\n",
|
|
"4. **Metadata**: Separate metadata management for flexibility\n",
|
|
"5. **Namespaces**: Isolate vectors for multi-tenancy\n",
|
|
"\n",
|
|
"### Next Steps\n",
|
|
"\n",
|
|
"**Further Reading**:\n",
|
|
"- [Vector Store API Reference](https://semantica.readthedocs.io/reference/vector_store/)\n",
|
|
"- [Advanced Vector Store Notebook](../advanced/Advanced_Vector_Store_and_Search.ipynb)\n",
|
|
"- [Embedding Generation](12_Embedding_Generation.ipynb)\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|