Refactor: Remove Pinecone and enhance vector store backend support

- Removed all Pinecone references, adapters, and documentation to align with open-source, self-hosted focus.
- Removed PineconeAdapter and related dependencies.
- Updated VectorStore to enforce supported backends (FAISS, Weaviate, Qdrant, Milvus, InMemory).
- Updated cookbooks (e.g., 13_Vector_Store.ipynb) to use Weaviate/FAISS examples instead of Pinecone.
- Updated core documentation (modules.md, rchitecture.md, etc.) to reflect backend changes.
- Added new tests (	est_pinecone_removal.py, 	est_vector_store_deepdive.py) to verify removal and validate remaining backends.
- Verified all vector store tests pass.
This commit is contained in:
KaifAhmad1
2025-12-12 20:19:17 +05:30
parent 6856580a7a
commit f3dd7a05bd
30 changed files with 1113 additions and 1318 deletions
+1 -1
View File
@@ -57,7 +57,7 @@ The **Context Module** provides agents with a persistent, searchable, and struct
The high-level facade that unifies all context operations. It routes data to the appropriate subsystems (Memory, Graph, Vector Store) and manages the lifecycle of context.
#### **Constructor Parameters**
* `vector_store` (Required): The backing vector database instance (e.g., FAISS, Pinecone).
* `vector_store` (Required): The backing vector database instance (e.g., FAISS, Weaviate).
* `knowledge_graph` (Optional): The graph store instance for structured knowledge.
* `token_limit` (Default: `2000`): The maximum number of tokens allowed in short-term memory before pruning occurs.
* `short_term_limit` (Default: `10`): The maximum number of distinct memory items in short-term memory.
+3 -3
View File
@@ -34,7 +34,7 @@ The **Embeddings Module** provides a unified interface for generating vector rep
---
Automatic formatting and validation for FAISS, Pinecone, Qdrant, and Weaviate.
Automatic formatting and validation for FAISS, Qdrant, and Weaviate.
</div>
@@ -122,13 +122,13 @@ print(f"Dimension: {embedder.get_embedding_dimension()}")
---
### VectorEmbeddingManager (The Bridge)
A utility class that prepares raw embeddings for insertion into specific vector databases. It handles formatting differences between backends like FAISS and Pinecone.
A utility class that prepares raw embeddings for insertion into specific vector databases. It handles formatting differences between backends like FAISS and Weaviate.
#### **Core Methods**
| Method | Description |
|--------|-------------|
| `prepare_for_vector_db(embeddings, backend, ...)` | Formats data for the target DB. |
| `prepare_for_vector_db(embeddings, metadata, backend)` | Formats data for the target DB. |
| `validate_dimensions(embeddings, expected_dim)` | Ensures vectors match the index configuration. |
| `batch_prepare(embeddings_list)` | Prepares a batch of embeddings for storage. |
+8 -46
View File
@@ -1,6 +1,6 @@
# Vector Store
> **Unified vector database interface supporting FAISS, Pinecone, Weaviate, Qdrant, and Milvus with Hybrid Search.**
> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
---
@@ -12,7 +12,7 @@
---
Seamlessly switch between FAISS (Local), Pinecone, Weaviate, Qdrant, and Milvus
Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
- :material-magnify-plus:{ .lg .middle } **Hybrid Search**
@@ -230,7 +230,6 @@ results = searcher.search(
Backend-specific implementations:
- `FAISSAdapter`: Local, in-memory/disk.
- `PineconeAdapter`: Managed cloud service.
- `WeaviateAdapter`: Schema-aware vector DB.
- `QdrantAdapter`: Rust-based high-performance DB.
- `MilvusAdapter`: Scalable cloud-native DB.
@@ -265,41 +264,6 @@ query = np.random.rand(768).astype('float32')
distances, indices = adapter.search(index, query, k=10)
```
#### PineconeAdapter
Managed cloud vector database.
**Helper Classes:**
- `PineconeIndex`: Index management
- `PineconeQuery`: Query operations
- `PineconeMetadata`: Metadata handling
**Example:**
```python
from semantica.vector_store import PineconeAdapter
adapter = PineconeAdapter(api_key="your-key", environment="us-west1-gcp")
adapter.connect()
# Create index
index = adapter.create_index("my-index", dimension=768, metric="cosine")
# Upsert with metadata
adapter.upsert_vectors(
vectors=[[0.1, 0.2, ...], ...],
ids=["vec_1", "vec_2"],
metadata=[{"category": "news"}, ...]
)
# Query with filter
results = adapter.query_vectors(
query_vector=[0.1, 0.2, ...],
top_k=10,
filter={"category": {"$eq": "news"}}
)
```
#### WeaviateAdapter
Schema-aware vector database with GraphQL.
@@ -716,25 +680,23 @@ print(f"Available methods: {methods}")
### Environment Variables
```bash
export VECTOR_STORE_BACKEND=pinecone
export PINECONE_API_KEY=sk-...
export PINECONE_ENV=us-west1-gcp
export VECTOR_STORE_BACKEND=weaviate
export WEAVIATE_URL=http://localhost:8080
```
### YAML Configuration
```yaml
vector_store:
backend: faiss # or pinecone, weaviate, etc.
backend: faiss # or weaviate, qdrant, milvus
dimension: 1536
metric: cosine
faiss:
index_type: HNSW
pinecone:
environment: us-west1-gcp
index_name: my-index
weaviate:
url: http://localhost:8080
```
---
@@ -777,7 +739,7 @@ print(f"Context: {context}")
**Solution**: Ensure your embedding model dimension (e.g., 1536 for OpenAI) matches the VectorStore dimension.
**Issue**: FAISS index not saved.
**Solution**: Call `store.save("index.faiss")` explicitly for local FAISS indices, or use a persistent backend like Pinecone/Qdrant.
**Solution**: Call `store.save("index.faiss")` explicitly for local FAISS indices, or use a persistent backend like Weaviate/Qdrant.
---