mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-15 04:00:33 +00:00
Refactor: Remove Pinecone and enhance vector store backend support
- Removed all Pinecone references, adapters, and documentation to align with open-source, self-hosted focus. - Removed PineconeAdapter and related dependencies. - Updated VectorStore to enforce supported backends (FAISS, Weaviate, Qdrant, Milvus, InMemory). - Updated cookbooks (e.g., 13_Vector_Store.ipynb) to use Weaviate/FAISS examples instead of Pinecone. - Updated core documentation (modules.md, rchitecture.md, etc.) to reflect backend changes. - Added new tests ( est_pinecone_removal.py, est_vector_store_deepdive.py) to verify removal and validate remaining backends. - Verified all vector store tests pass.
This commit is contained in:
@@ -57,7 +57,7 @@ The **Context Module** provides agents with a persistent, searchable, and struct
|
||||
The high-level facade that unifies all context operations. It routes data to the appropriate subsystems (Memory, Graph, Vector Store) and manages the lifecycle of context.
|
||||
|
||||
#### **Constructor Parameters**
|
||||
* `vector_store` (Required): The backing vector database instance (e.g., FAISS, Pinecone).
|
||||
* `vector_store` (Required): The backing vector database instance (e.g., FAISS, Weaviate).
|
||||
* `knowledge_graph` (Optional): The graph store instance for structured knowledge.
|
||||
* `token_limit` (Default: `2000`): The maximum number of tokens allowed in short-term memory before pruning occurs.
|
||||
* `short_term_limit` (Default: `10`): The maximum number of distinct memory items in short-term memory.
|
||||
|
||||
@@ -34,7 +34,7 @@ The **Embeddings Module** provides a unified interface for generating vector rep
|
||||
|
||||
---
|
||||
|
||||
Automatic formatting and validation for FAISS, Pinecone, Qdrant, and Weaviate.
|
||||
Automatic formatting and validation for FAISS, Qdrant, and Weaviate.
|
||||
|
||||
</div>
|
||||
|
||||
@@ -122,13 +122,13 @@ print(f"Dimension: {embedder.get_embedding_dimension()}")
|
||||
---
|
||||
|
||||
### VectorEmbeddingManager (The Bridge)
|
||||
A utility class that prepares raw embeddings for insertion into specific vector databases. It handles formatting differences between backends like FAISS and Pinecone.
|
||||
A utility class that prepares raw embeddings for insertion into specific vector databases. It handles formatting differences between backends like FAISS and Weaviate.
|
||||
|
||||
#### **Core Methods**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `prepare_for_vector_db(embeddings, backend, ...)` | Formats data for the target DB. |
|
||||
| `prepare_for_vector_db(embeddings, metadata, backend)` | Formats data for the target DB. |
|
||||
| `validate_dimensions(embeddings, expected_dim)` | Ensures vectors match the index configuration. |
|
||||
| `batch_prepare(embeddings_list)` | Prepares a batch of embeddings for storage. |
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Vector Store
|
||||
|
||||
> **Unified vector database interface supporting FAISS, Pinecone, Weaviate, Qdrant, and Milvus with Hybrid Search.**
|
||||
> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
|
||||
|
||||
---
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
|
||||
---
|
||||
|
||||
Seamlessly switch between FAISS (Local), Pinecone, Weaviate, Qdrant, and Milvus
|
||||
Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
|
||||
|
||||
- :material-magnify-plus:{ .lg .middle } **Hybrid Search**
|
||||
|
||||
@@ -230,7 +230,6 @@ results = searcher.search(
|
||||
|
||||
Backend-specific implementations:
|
||||
- `FAISSAdapter`: Local, in-memory/disk.
|
||||
- `PineconeAdapter`: Managed cloud service.
|
||||
- `WeaviateAdapter`: Schema-aware vector DB.
|
||||
- `QdrantAdapter`: Rust-based high-performance DB.
|
||||
- `MilvusAdapter`: Scalable cloud-native DB.
|
||||
@@ -265,41 +264,6 @@ query = np.random.rand(768).astype('float32')
|
||||
distances, indices = adapter.search(index, query, k=10)
|
||||
```
|
||||
|
||||
#### PineconeAdapter
|
||||
|
||||
Managed cloud vector database.
|
||||
|
||||
**Helper Classes:**
|
||||
- `PineconeIndex`: Index management
|
||||
- `PineconeQuery`: Query operations
|
||||
- `PineconeMetadata`: Metadata handling
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.vector_store import PineconeAdapter
|
||||
|
||||
adapter = PineconeAdapter(api_key="your-key", environment="us-west1-gcp")
|
||||
adapter.connect()
|
||||
|
||||
# Create index
|
||||
index = adapter.create_index("my-index", dimension=768, metric="cosine")
|
||||
|
||||
# Upsert with metadata
|
||||
adapter.upsert_vectors(
|
||||
vectors=[[0.1, 0.2, ...], ...],
|
||||
ids=["vec_1", "vec_2"],
|
||||
metadata=[{"category": "news"}, ...]
|
||||
)
|
||||
|
||||
# Query with filter
|
||||
results = adapter.query_vectors(
|
||||
query_vector=[0.1, 0.2, ...],
|
||||
top_k=10,
|
||||
filter={"category": {"$eq": "news"}}
|
||||
)
|
||||
```
|
||||
|
||||
#### WeaviateAdapter
|
||||
|
||||
Schema-aware vector database with GraphQL.
|
||||
@@ -716,25 +680,23 @@ print(f"Available methods: {methods}")
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
export VECTOR_STORE_BACKEND=pinecone
|
||||
export PINECONE_API_KEY=sk-...
|
||||
export PINECONE_ENV=us-west1-gcp
|
||||
export VECTOR_STORE_BACKEND=weaviate
|
||||
export WEAVIATE_URL=http://localhost:8080
|
||||
```
|
||||
|
||||
### YAML Configuration
|
||||
|
||||
```yaml
|
||||
vector_store:
|
||||
backend: faiss # or pinecone, weaviate, etc.
|
||||
backend: faiss # or weaviate, qdrant, milvus
|
||||
dimension: 1536
|
||||
metric: cosine
|
||||
|
||||
faiss:
|
||||
index_type: HNSW
|
||||
|
||||
pinecone:
|
||||
environment: us-west1-gcp
|
||||
index_name: my-index
|
||||
weaviate:
|
||||
url: http://localhost:8080
|
||||
```
|
||||
|
||||
---
|
||||
@@ -777,7 +739,7 @@ print(f"Context: {context}")
|
||||
**Solution**: Ensure your embedding model dimension (e.g., 1536 for OpenAI) matches the VectorStore dimension.
|
||||
|
||||
**Issue**: FAISS index not saved.
|
||||
**Solution**: Call `store.save("index.faiss")` explicitly for local FAISS indices, or use a persistent backend like Pinecone/Qdrant.
|
||||
**Solution**: Call `store.save("index.faiss")` explicitly for local FAISS indices, or use a persistent backend like Weaviate/Qdrant.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user