mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
Compare commits
23
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b3f3ac413c | ||
|
|
ea8a250186 | ||
|
|
1d64d58741 | ||
|
|
3a091872ee | ||
|
|
979653e498 | ||
|
|
1a95b0d35f | ||
|
|
589dd8c61e | ||
|
|
95ea8de455 | ||
|
|
327792c830 | ||
|
|
017a36591d | ||
|
|
e9ec904d87 | ||
|
|
b6f7542600 | ||
|
|
e8c93def07 | ||
|
|
74cb3c6ac2 | ||
|
|
0197062dfc | ||
|
|
274114ae67 | ||
|
|
4ec94b6a5d | ||
|
|
eb1886bee3 | ||
|
|
cb91321360 | ||
|
|
d514e6b4cf | ||
|
|
bc875450fa | ||
|
|
400a70986d | ||
|
|
15b32f49be |
@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added / Changed
|
||||
|
||||
- **Enhanced Change Management Module**:
|
||||
- New `semantica.change_management` module with persistent version storage and audit trails
|
||||
- **Core Classes**: `TemporalVersionManager` (KG versioning), `OntologyVersionManager` (ontology versioning), `ChangeLogEntry` (metadata)
|
||||
- **Storage**: SQLite (persistent) and in-memory backends with thread-safe operations
|
||||
- **Features**: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
|
||||
- **Compliance**: HIPAA, SOX, FDA 21 CFR Part 11 support with immutable audit trails
|
||||
- **Testing**: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
|
||||
- **Performance**: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
|
||||
- **Migration**: Backward compatible, simplified class names, zero external dependencies
|
||||
|
||||
- CSV Ingestion Enhancements (PR #244 by @saloni0318)
|
||||
- Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
|
||||
- Tolerant decoding and malformed-row handling (`on_bad_lines='warn'`)
|
||||
- Optional chunked reading for large files; metadata tracks detected values
|
||||
- Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation
|
||||
|
||||
- Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)
|
||||
- Added focused test coverage for TextNormalizer behavior across inputs
|
||||
|
||||
- Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)
|
||||
- Introduced integration test marker and reduced noisy warnings in ingest tests
|
||||
|
||||
- Tests (ingest): Add unit tests for file, web, and feed ingestors (PR #239 by @Mohammed2372)
|
||||
- Broadened ingest test coverage across multiple source types
|
||||
|
||||
## [0.2.5] - 2026-01-27
|
||||
|
||||
### Added
|
||||
|
||||
@@ -8,7 +8,6 @@
|
||||
[](https://www.python.org/)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://pypi.org/project/semantica/)
|
||||
[](https://pypi.org/project/semantica/)
|
||||
[](https://pepy.tech/project/semantica)
|
||||
[](https://github.com/Hawksight-AI/semantica/actions)
|
||||
[](https://discord.gg/RgaGTj9J)
|
||||
@@ -75,6 +74,7 @@ print(f"Built KG with {len(kg.get('entities', []))} entities")
|
||||
| **Provenance-Aware** | Source-level provenance from documents to responses |
|
||||
| **Validated** | Built-in conflict detection, deduplication, QA |
|
||||
| **Governed** | Rule-based validation and semantic consistency |
|
||||
| **Version Control** | Enterprise-grade change management with HIPAA/SOX/FDA compliance |
|
||||
|
||||
### Perfect For High-Stakes Use Cases
|
||||
|
||||
@@ -183,6 +183,7 @@ The **semantic gap** is the fundamental disconnect between what AI systems can p
|
||||
- ✅ **Quality Assurance** — Conflict detection, validation
|
||||
- 📊 **Provenance Tracking** — Source, time, confidence metadata
|
||||
- 🧠 **Reasoning Traces** — Explainable inference paths
|
||||
- 🔐 **Change Management** — Version control with audit trails, checksums, HIPAA/SOX/FDA compliance
|
||||
|
||||
### 3️⃣ Output Layer — Auditable Knowledge Assets
|
||||
- 📊 **Knowledge Graphs** — Queryable, temporal, explainable
|
||||
@@ -447,6 +448,43 @@ print(f"Classes: {len(custom_ontology.classes)}")
|
||||
|
||||
[**Cookbook: Ontology**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/14_Ontology.ipynb)
|
||||
|
||||
### Change Management & Version Control
|
||||
|
||||
> **Enterprise-Grade Versioning** • Persistent Storage • Audit Trails • HIPAA/SOX/FDA Compliance • SHA-256 Checksums
|
||||
|
||||
```python
|
||||
from semantica.change_management import TemporalVersionManager, OntologyVersionManager
|
||||
|
||||
# Knowledge Graph versioning with audit trails
|
||||
kg_manager = TemporalVersionManager(storage_path="kg_versions.db")
|
||||
|
||||
# Create versioned snapshot
|
||||
snapshot = kg_manager.create_snapshot(
|
||||
knowledge_graph,
|
||||
version_label="v1.0",
|
||||
author="user@company.com",
|
||||
description="Initial patient record"
|
||||
)
|
||||
|
||||
# Compare versions with detailed diffs
|
||||
diff = kg_manager.compare_versions("v1.0", "v2.0")
|
||||
print(f"Entities added: {diff['summary']['entities_added']}")
|
||||
print(f"Entities modified: {diff['summary']['entities_modified']}")
|
||||
|
||||
# Verify data integrity
|
||||
is_valid = kg_manager.verify_checksum(snapshot)
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- 🔐 **Persistent Storage** — SQLite and in-memory backends
|
||||
- 📊 **Detailed Diffs** — Entity-level and relationship-level change tracking
|
||||
- ✅ **Data Integrity** — SHA-256 checksums with tamper detection
|
||||
- 🏥 **Compliance Ready** — HIPAA, SOX, FDA 21 CFR Part 11 support
|
||||
- ⚡ **High Performance** — 17.6ms for 10k entities, 510+ ops/sec concurrent
|
||||
- 🧪 **Fully Tested** — 104 tests covering real-world scenarios
|
||||
|
||||
[**Documentation: Change Management**](docs/reference/change_management.md) • [**Usage Guide**](semantica/change_management/change_management_usage.md)
|
||||
|
||||
### Context Engineering & Memory Systems
|
||||
|
||||
> **Persistent Memory** • **Context Graph** • **Context Retriever** • **Hybrid Retrieval (Vector + Graph)** • **Production Graph Store (Neo4j)** • **Entity Linking** • **Multi-Hop Reasoning**
|
||||
|
||||
@@ -0,0 +1,938 @@
|
||||
# Change Management API Reference
|
||||
|
||||
Comprehensive API documentation for the Enhanced Change Management module in Semantica.
|
||||
|
||||
## Overview
|
||||
|
||||
The `semantica.change_management` module provides enterprise-grade version control, audit trails, and compliance tracking for knowledge graphs and ontologies. It includes persistent storage backends, detailed change tracking, data integrity verification, and standardized metadata structures.
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
semantica.change_management/
|
||||
├── change_log.py # Standardized metadata structures
|
||||
├── version_storage.py # Storage abstraction and implementations
|
||||
├── managers.py # Enhanced version managers
|
||||
├── ontology_version_manager.py # Ontology version management
|
||||
└── change_management_usage.md # Usage guide
|
||||
```
|
||||
|
||||
## Quick Import
|
||||
|
||||
```python
|
||||
from semantica.change_management import (
|
||||
# Metadata
|
||||
ChangeLogEntry,
|
||||
|
||||
# Storage
|
||||
VersionStorage,
|
||||
InMemoryVersionStorage,
|
||||
SQLiteVersionStorage,
|
||||
|
||||
# Utilities
|
||||
compute_checksum,
|
||||
verify_checksum,
|
||||
|
||||
# Version Managers
|
||||
BaseVersionManager,
|
||||
TemporalVersionManager,
|
||||
OntologyVersionManager,
|
||||
VersionManager,
|
||||
OntologyVersion
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Classes
|
||||
|
||||
### ChangeLogEntry
|
||||
|
||||
Standardized metadata structure for version changes with validation.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ChangeLogEntry:
|
||||
"""
|
||||
Standardized change log entry with validation.
|
||||
|
||||
Attributes:
|
||||
timestamp: ISO 8601 formatted timestamp
|
||||
author: Email address of the change author
|
||||
description: Change description (max 500 characters)
|
||||
change_id: Optional ID linking to external systems
|
||||
"""
|
||||
timestamp: str
|
||||
author: str
|
||||
description: str
|
||||
change_id: Optional[str] = None
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
##### `__post_init__()`
|
||||
|
||||
Validates all fields after initialization.
|
||||
|
||||
**Raises:**
|
||||
- `ValidationError`: If any field validation fails
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
entry = ChangeLogEntry(
|
||||
timestamp="2024-01-30T12:00:00Z",
|
||||
author="user@example.com",
|
||||
description="Updated entity relationships",
|
||||
change_id="TICKET-123"
|
||||
)
|
||||
```
|
||||
|
||||
##### `create_now(author, description, change_id=None)` (classmethod)
|
||||
|
||||
Creates a change log entry with the current timestamp.
|
||||
|
||||
**Parameters:**
|
||||
- `author` (str): Email address of the change author
|
||||
- `description` (str): Change description (max 500 characters)
|
||||
- `change_id` (str, optional): ID linking to external systems
|
||||
|
||||
**Returns:**
|
||||
- `ChangeLogEntry`: New instance with current timestamp
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
entry = ChangeLogEntry.create_now(
|
||||
author="developer@company.com",
|
||||
description="Fixed entity resolution bug",
|
||||
change_id="JIRA-1234"
|
||||
)
|
||||
```
|
||||
|
||||
#### Validation Rules
|
||||
|
||||
- **Timestamp**: Must be valid ISO 8601 format with 'T' separator
|
||||
- **Author**: Must be valid email format (RFC 5322)
|
||||
- **Description**: Maximum 500 characters
|
||||
- **Change ID**: Optional, no validation
|
||||
|
||||
---
|
||||
|
||||
### VersionStorage
|
||||
|
||||
Abstract base class for storage implementations.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
class VersionStorage(ABC):
|
||||
"""
|
||||
Abstract base class for version storage backends.
|
||||
|
||||
Provides interface for saving, retrieving, and managing version snapshots.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Abstract Methods
|
||||
|
||||
##### `save(snapshot)`
|
||||
|
||||
Save a version snapshot.
|
||||
|
||||
**Parameters:**
|
||||
- `snapshot` (Dict[str, Any]): Version snapshot dictionary with metadata
|
||||
|
||||
**Raises:**
|
||||
- `ValidationError`: If snapshot data is invalid
|
||||
- `ProcessingError`: If save operation fails
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
snapshot = {
|
||||
"label": "v1.0",
|
||||
"timestamp": "2024-01-30T12:00:00Z",
|
||||
"author": "user@example.com",
|
||||
"description": "Initial version",
|
||||
"data": {...}
|
||||
}
|
||||
storage.save(snapshot)
|
||||
```
|
||||
|
||||
##### `get(label)`
|
||||
|
||||
Retrieve a version snapshot by label.
|
||||
|
||||
**Parameters:**
|
||||
- `label` (str): Version label to retrieve
|
||||
|
||||
**Returns:**
|
||||
- `Optional[Dict[str, Any]]`: Snapshot dictionary or None if not found
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
snapshot = storage.get("v1.0")
|
||||
if snapshot:
|
||||
print(f"Retrieved: {snapshot['label']}")
|
||||
```
|
||||
|
||||
##### `list_all()`
|
||||
|
||||
List all version snapshots.
|
||||
|
||||
**Returns:**
|
||||
- `List[Dict[str, Any]]`: List of snapshot metadata dictionaries
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
versions = storage.list_all()
|
||||
for v in versions:
|
||||
print(f"{v['label']}: {v['description']}")
|
||||
```
|
||||
|
||||
##### `exists(label)`
|
||||
|
||||
Check if a version exists.
|
||||
|
||||
**Parameters:**
|
||||
- `label` (str): Version label to check
|
||||
|
||||
**Returns:**
|
||||
- `bool`: True if version exists, False otherwise
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
if storage.exists("v1.0"):
|
||||
print("Version exists")
|
||||
```
|
||||
|
||||
##### `delete(label)`
|
||||
|
||||
Delete a version snapshot.
|
||||
|
||||
**Parameters:**
|
||||
- `label` (str): Version label to delete
|
||||
|
||||
**Returns:**
|
||||
- `bool`: True if deleted, False if not found
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
if storage.delete("v1.0"):
|
||||
print("Version deleted")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### InMemoryVersionStorage
|
||||
|
||||
In-memory version storage implementation.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
class InMemoryVersionStorage(VersionStorage):
|
||||
"""
|
||||
In-memory version storage implementation.
|
||||
|
||||
Fast, volatile storage for development and testing.
|
||||
Data is lost when the process ends.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Constructor
|
||||
|
||||
```python
|
||||
def __init__(self):
|
||||
"""Initialize in-memory storage."""
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
storage = InMemoryVersionStorage()
|
||||
```
|
||||
|
||||
#### Performance Characteristics
|
||||
|
||||
- **Save**: 0.37-16ms (10-1000 entities)
|
||||
- **Get**: 0.20-16ms (10-1000 entities)
|
||||
- **List**: <0.03ms
|
||||
- **Thread-safe**: Yes (uses RLock)
|
||||
|
||||
#### Use Cases
|
||||
|
||||
- Development and testing
|
||||
- Temporary version tracking
|
||||
- High-performance scenarios where persistence is not required
|
||||
|
||||
---
|
||||
|
||||
### SQLiteVersionStorage
|
||||
|
||||
SQLite-based persistent version storage implementation.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
class SQLiteVersionStorage(VersionStorage):
|
||||
"""
|
||||
SQLite-based persistent version storage implementation.
|
||||
|
||||
Provides persistence across process restarts with ACID guarantees.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Constructor
|
||||
|
||||
```python
|
||||
def __init__(self, storage_path: str):
|
||||
"""
|
||||
Initialize SQLite storage.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file
|
||||
"""
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `storage_path` (str): Path to SQLite database file (created if doesn't exist)
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
storage = SQLiteVersionStorage("versions.db")
|
||||
```
|
||||
|
||||
#### Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE versions (
|
||||
label TEXT PRIMARY KEY,
|
||||
timestamp TEXT NOT NULL,
|
||||
author TEXT NOT NULL,
|
||||
description TEXT,
|
||||
checksum TEXT,
|
||||
snapshot_data TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL
|
||||
)
|
||||
```
|
||||
|
||||
#### Performance Characteristics
|
||||
|
||||
- **Save**: 7-25ms (10-1000 entities)
|
||||
- **Get**: 2-8ms (10-1000 entities)
|
||||
- **List**: 0.6-13ms
|
||||
- **Thread-safe**: Yes (uses RLock)
|
||||
- **ACID**: Full transaction support
|
||||
|
||||
#### Use Cases
|
||||
|
||||
- Production deployments
|
||||
- Long-term version storage
|
||||
- Compliance and audit requirements
|
||||
- Multi-process environments
|
||||
|
||||
---
|
||||
|
||||
### BaseVersionManager
|
||||
|
||||
Abstract base class for version managers.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
class BaseVersionManager(ABC):
|
||||
"""
|
||||
Abstract base class for version managers.
|
||||
|
||||
Provides common functionality for version management across
|
||||
different data types (knowledge graphs, ontologies, etc.).
|
||||
"""
|
||||
```
|
||||
|
||||
#### Constructor
|
||||
|
||||
```python
|
||||
def __init__(self, storage_path: Optional[str] = None):
|
||||
"""
|
||||
Initialize base version manager.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file for persistent storage.
|
||||
If None, uses in-memory storage.
|
||||
"""
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `storage_path` (str, optional): Path to SQLite database file
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# In-memory storage
|
||||
manager = BaseVersionManager()
|
||||
|
||||
# Persistent storage
|
||||
manager = BaseVersionManager(storage_path="versions.db")
|
||||
```
|
||||
|
||||
#### Abstract Methods
|
||||
|
||||
##### `create_snapshot(data, version_label, author, description, **options)`
|
||||
|
||||
Create a versioned snapshot of the data.
|
||||
|
||||
**Parameters:**
|
||||
- `data` (Any): Data to snapshot
|
||||
- `version_label` (str): Version label
|
||||
- `author` (str): Email address of the author
|
||||
- `description` (str): Change description
|
||||
- `**options`: Additional options
|
||||
|
||||
**Returns:**
|
||||
- `Dict[str, Any]`: Snapshot with metadata and checksum
|
||||
|
||||
##### `compare_versions(version1, version2, **options)`
|
||||
|
||||
Compare two versions and return detailed differences.
|
||||
|
||||
**Parameters:**
|
||||
- `version1` (Any): First version (label or snapshot)
|
||||
- `version2` (Any): Second version (label or snapshot)
|
||||
- `**options`: Comparison options
|
||||
|
||||
**Returns:**
|
||||
- `Dict[str, Any]`: Detailed differences
|
||||
|
||||
#### Concrete Methods
|
||||
|
||||
##### `list_versions()`
|
||||
|
||||
List all version snapshots.
|
||||
|
||||
**Returns:**
|
||||
- `List[Dict[str, Any]]`: List of version metadata
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
versions = manager.list_versions()
|
||||
for v in versions:
|
||||
print(f"{v['label']}: {v['description']}")
|
||||
```
|
||||
|
||||
##### `get_version(label)`
|
||||
|
||||
Retrieve specific version by label.
|
||||
|
||||
**Parameters:**
|
||||
- `label` (str): Version label
|
||||
|
||||
**Returns:**
|
||||
- `Optional[Dict[str, Any]]`: Version snapshot or None
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
version = manager.get_version("v1.0")
|
||||
```
|
||||
|
||||
##### `verify_checksum(snapshot)`
|
||||
|
||||
Verify data integrity using checksum.
|
||||
|
||||
**Parameters:**
|
||||
- `snapshot` (Dict[str, Any]): Snapshot to verify
|
||||
|
||||
**Returns:**
|
||||
- `bool`: True if checksum is valid
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
is_valid = manager.verify_checksum(snapshot)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### TemporalVersionManager
|
||||
|
||||
Enhanced temporal version management engine for knowledge graphs.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
class TemporalVersionManager(BaseVersionManager):
|
||||
"""
|
||||
Enhanced temporal version management engine for knowledge graphs.
|
||||
|
||||
Features:
|
||||
- Persistent snapshot storage (SQLite or in-memory)
|
||||
- Detailed change tracking with entity-level diffs
|
||||
- SHA-256 checksums for data integrity
|
||||
- Standardized metadata with author attribution
|
||||
- Version comparison with backward compatibility
|
||||
- Input validation and security features
|
||||
"""
|
||||
```
|
||||
|
||||
#### Constructor
|
||||
|
||||
```python
|
||||
def __init__(self, storage_path: Optional[str] = None, **config):
|
||||
"""
|
||||
Initialize enhanced temporal version manager.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file for persistent storage.
|
||||
If None, uses in-memory storage
|
||||
**config: Additional configuration options
|
||||
"""
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `storage_path` (str, optional): Path to SQLite database file
|
||||
- `**config`: Additional configuration options
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# In-memory storage
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Persistent storage
|
||||
manager = TemporalVersionManager(storage_path="kg_versions.db")
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
##### `create_snapshot(graph, version_label, author, description, **options)`
|
||||
|
||||
Create and store snapshot with checksum and metadata.
|
||||
|
||||
**Parameters:**
|
||||
- `graph` (Dict[str, Any]): Knowledge graph dict with "entities" and "relationships"
|
||||
- `version_label` (str): Version string (e.g., "v1.0")
|
||||
- `author` (str): Email address of the change author
|
||||
- `description` (str): Change description (max 500 chars)
|
||||
- `**options`: Additional options
|
||||
|
||||
**Returns:**
|
||||
- `Dict[str, Any]`: Snapshot with metadata and checksum
|
||||
|
||||
**Raises:**
|
||||
- `ValidationError`: If input validation fails
|
||||
- `ProcessingError`: If snapshot creation fails
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
graph = {
|
||||
"entities": [
|
||||
{"id": "e1", "name": "Entity 1", "type": "Person"},
|
||||
{"id": "e2", "name": "Entity 2", "type": "Organization"}
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "e1", "target": "e2", "type": "works_for"}
|
||||
]
|
||||
}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
graph,
|
||||
version_label="v1.0",
|
||||
author="user@example.com",
|
||||
description="Initial knowledge graph"
|
||||
)
|
||||
|
||||
print(f"Created: {snapshot['label']}")
|
||||
print(f"Checksum: {snapshot['checksum']}")
|
||||
```
|
||||
|
||||
##### `compare_versions(version1, version2, **options)`
|
||||
|
||||
Compare two versions with detailed entity and relationship diffs.
|
||||
|
||||
**Parameters:**
|
||||
- `version1` (Union[str, Dict]): First version (label or snapshot dict)
|
||||
- `version2` (Union[str, Dict]): Second version (label or snapshot dict)
|
||||
- `**options`: Comparison options
|
||||
|
||||
**Returns:**
|
||||
- `Dict[str, Any]`: Detailed differences including:
|
||||
- `summary`: Aggregate statistics
|
||||
- `entity_changes`: Entity-level changes
|
||||
- `relationship_changes`: Relationship-level changes
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
diff = manager.compare_versions("v1.0", "v2.0")
|
||||
|
||||
print(f"Entities added: {diff['summary']['entities_added']}")
|
||||
print(f"Entities modified: {diff['summary']['entities_modified']}")
|
||||
print(f"Relationships added: {diff['summary']['relationships_added']}")
|
||||
|
||||
# Detailed entity changes
|
||||
for entity_id, changes in diff['entity_changes'].items():
|
||||
print(f"Entity {entity_id}: {changes['status']}")
|
||||
if changes['status'] == 'modified':
|
||||
print(f" Before: {changes['before']}")
|
||||
print(f" After: {changes['after']}")
|
||||
```
|
||||
|
||||
#### Performance
|
||||
|
||||
- **Snapshot Creation**: 1.40-54ms (50-2000 entities)
|
||||
- **Version Retrieval**: 0.65-26ms (50-2000 entities)
|
||||
- **Version Comparison**: 3.46-33ms (100-1000 entities)
|
||||
- **Concurrent Throughput**: 500+ operations/second
|
||||
|
||||
---
|
||||
|
||||
### OntologyVersionManager
|
||||
|
||||
Enhanced version management for ontologies.
|
||||
|
||||
#### Class Definition
|
||||
|
||||
```python
|
||||
class OntologyVersionManager(BaseVersionManager):
|
||||
"""
|
||||
Enhanced version management for ontologies.
|
||||
|
||||
Features:
|
||||
- Persistent ontology snapshot storage
|
||||
- Structural comparison (classes, properties, axioms)
|
||||
- SHA-256 checksums for data integrity
|
||||
- Standardized metadata with author attribution
|
||||
"""
|
||||
```
|
||||
|
||||
#### Constructor
|
||||
|
||||
```python
|
||||
def __init__(self, storage_path: Optional[str] = None, **config):
|
||||
"""
|
||||
Initialize enhanced version manager for ontologies.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file for persistent storage.
|
||||
If None, uses in-memory storage
|
||||
**config: Additional configuration options
|
||||
"""
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
manager = OntologyVersionManager(storage_path="ontology_versions.db")
|
||||
```
|
||||
|
||||
#### Methods
|
||||
|
||||
##### `create_snapshot(ontology, version_label, author, description, **options)`
|
||||
|
||||
Create ontology snapshot with metadata.
|
||||
|
||||
**Parameters:**
|
||||
- `ontology` (Dict[str, Any]): Ontology dict with structure information
|
||||
- `version_label` (str): Version label
|
||||
- `author` (str): Email address of the author
|
||||
- `description` (str): Change description
|
||||
- `**options`: Additional options
|
||||
|
||||
**Returns:**
|
||||
- `Dict[str, Any]`: Ontology snapshot with metadata
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
ontology = {
|
||||
"uri": "https://example.com/ontology",
|
||||
"version_info": {"version": "1.0", "date": "2024-01-30"},
|
||||
"structure": {
|
||||
"classes": ["Person", "Organization", "Location"],
|
||||
"properties": ["name", "address", "email"],
|
||||
"individuals": ["JohnDoe", "ACME_Corp"],
|
||||
"axioms": ["Person hasAddress exactly 1 Location"]
|
||||
}
|
||||
}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
ontology,
|
||||
version_label="ont_v1.0",
|
||||
author="architect@example.com",
|
||||
description="Initial ontology design"
|
||||
)
|
||||
```
|
||||
|
||||
##### `compare_versions(version1, version2, **options)`
|
||||
|
||||
Compare ontology versions with structural analysis.
|
||||
|
||||
**Parameters:**
|
||||
- `version1` (Union[str, Dict]): First version
|
||||
- `version2` (Union[str, Dict]): Second version
|
||||
- `**options`: Comparison options
|
||||
|
||||
**Returns:**
|
||||
- `Dict[str, Any]`: Structural differences including:
|
||||
- `classes_added`, `classes_removed`
|
||||
- `properties_added`, `properties_removed`
|
||||
- `individuals_added`, `individuals_removed`
|
||||
- `axioms_added`, `axioms_removed`, `axioms_modified`
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
diff = manager.compare_versions("ont_v1.0", "ont_v2.0")
|
||||
|
||||
print(f"Classes added: {diff['classes_added']}")
|
||||
print(f"Properties added: {diff['properties_added']}")
|
||||
print(f"Axioms modified: {diff['axioms_modified']}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Utility Functions
|
||||
|
||||
### compute_checksum
|
||||
|
||||
Compute SHA-256 checksum for data integrity.
|
||||
|
||||
#### Function Signature
|
||||
|
||||
```python
|
||||
def compute_checksum(data: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Compute SHA-256 checksum for data.
|
||||
|
||||
Args:
|
||||
data: Dictionary to compute checksum for
|
||||
|
||||
Returns:
|
||||
SHA-256 checksum as hexadecimal string
|
||||
"""
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `data` (Dict[str, Any]): Dictionary to compute checksum for
|
||||
|
||||
**Returns:**
|
||||
- `str`: SHA-256 checksum as hexadecimal string
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from semantica.change_management import compute_checksum
|
||||
|
||||
data = {"entities": [...], "relationships": [...]}
|
||||
checksum = compute_checksum(data)
|
||||
print(f"Checksum: {checksum}")
|
||||
```
|
||||
|
||||
**Performance:** 1.29-110ms (100-10,000 entities)
|
||||
|
||||
---
|
||||
|
||||
### verify_checksum
|
||||
|
||||
Verify data integrity using stored checksum.
|
||||
|
||||
#### Function Signature
|
||||
|
||||
```python
|
||||
def verify_checksum(snapshot: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Verify data integrity using checksum.
|
||||
|
||||
Args:
|
||||
snapshot: Snapshot dictionary with 'checksum' field
|
||||
|
||||
Returns:
|
||||
True if checksum is valid, False otherwise
|
||||
"""
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `snapshot` (Dict[str, Any]): Snapshot dictionary with 'checksum' field
|
||||
|
||||
**Returns:**
|
||||
- `bool`: True if checksum is valid, False otherwise
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from semantica.change_management import verify_checksum
|
||||
|
||||
snapshot = manager.get_version("v1.0")
|
||||
is_valid = verify_checksum(snapshot)
|
||||
|
||||
if not is_valid:
|
||||
print("WARNING: Data integrity compromised!")
|
||||
```
|
||||
|
||||
**Performance:** 0.82-96ms (100-10,000 entities)
|
||||
|
||||
---
|
||||
|
||||
## Legacy Classes
|
||||
|
||||
### VersionManager
|
||||
|
||||
Original ontology version manager (moved from `semantica.ontology`).
|
||||
|
||||
#### Import
|
||||
|
||||
```python
|
||||
from semantica.change_management import VersionManager, OntologyVersion
|
||||
```
|
||||
|
||||
**Note:** This class is maintained for backward compatibility. New projects should use `OntologyVersionManager`.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### ValidationError
|
||||
|
||||
Raised when input validation fails.
|
||||
|
||||
**Common Causes:**
|
||||
- Invalid email format
|
||||
- Description exceeds 500 characters
|
||||
- Invalid ISO 8601 timestamp
|
||||
- Missing required fields
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
|
||||
try:
|
||||
entry = ChangeLogEntry(
|
||||
timestamp="invalid",
|
||||
author="not-an-email",
|
||||
description="x" * 501
|
||||
)
|
||||
except ValidationError as e:
|
||||
print(f"Validation failed: {e}")
|
||||
```
|
||||
|
||||
### ProcessingError
|
||||
|
||||
Raised when operations fail.
|
||||
|
||||
**Common Causes:**
|
||||
- Database connection issues
|
||||
- File system errors
|
||||
- Concurrent modification conflicts
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from semantica.utils.exceptions import ProcessingError
|
||||
|
||||
try:
|
||||
storage.save(snapshot)
|
||||
except ProcessingError as e:
|
||||
print(f"Save failed: {e}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Benchmarks
|
||||
|
||||
Based on comprehensive performance testing:
|
||||
|
||||
| Component | Small (100) | Medium (500) | Large (2000) |
|
||||
|-----------|-------------|--------------|--------------|
|
||||
| Snapshot Creation | 2.33ms | 10.70ms | 54.23ms |
|
||||
| Version Retrieval | 1.88ms | 7.33ms | 26.04ms |
|
||||
| Version Comparison | 3.46ms | 17.39ms | 32.83ms |
|
||||
| Checksum Compute | 1.29ms | 5.48ms | 22.15ms |
|
||||
| SQLite Save | 8.69ms | 13.37ms | 25.33ms |
|
||||
| InMemory Save | 1.18ms | 10.60ms | 14.11ms |
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Use appropriate storage backend:**
|
||||
- Development: `InMemoryVersionStorage`
|
||||
- Production: `SQLiteVersionStorage`
|
||||
|
||||
2. **Batch operations when possible:**
|
||||
```python
|
||||
for data in batch:
|
||||
manager.create_snapshot(data, ...)
|
||||
```
|
||||
|
||||
3. **Implement retention policies:**
|
||||
```python
|
||||
# Delete old versions periodically
|
||||
for version in old_versions:
|
||||
storage.delete(version['label'])
|
||||
```
|
||||
|
||||
4. **Use concurrent operations:**
|
||||
- Thread-safe: 500+ operations/second
|
||||
- No performance degradation under load
|
||||
|
||||
---
|
||||
|
||||
## Compliance Features
|
||||
|
||||
### HIPAA Compliance
|
||||
|
||||
- Complete audit trails with author attribution
|
||||
- Timestamp tracking for all changes
|
||||
- Data integrity verification with checksums
|
||||
- Secure storage with access controls
|
||||
|
||||
### SOX Compliance
|
||||
|
||||
- Immutable change records
|
||||
- Detailed change descriptions
|
||||
- External system linking (change IDs)
|
||||
- Comprehensive audit reports
|
||||
|
||||
### FDA 21 CFR Part 11
|
||||
|
||||
- Electronic signatures (author email)
|
||||
- Data integrity verification
|
||||
- Audit trail generation
|
||||
- Tamper detection
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Complete Healthcare Example
|
||||
|
||||
```python
|
||||
from semantica.change_management import TemporalVersionManager
|
||||
|
||||
# Initialize with HIPAA-compliant storage
|
||||
manager = TemporalVersionManager(storage_path="hipaa_records.db")
|
||||
|
||||
# Patient knowledge graph
|
||||
patient_kg = {
|
||||
"entities": [
|
||||
{"id": "patient_001", "type": "Patient", "name": "Jane Smith"},
|
||||
{"id": "diagnosis_001", "type": "Diagnosis", "code": "I10"}
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "patient_001", "target": "diagnosis_001", "type": "has_diagnosis"}
|
||||
]
|
||||
}
|
||||
|
||||
# Create versioned record
|
||||
snapshot = manager.create_snapshot(
|
||||
patient_kg,
|
||||
"patient_001_v1.0",
|
||||
"dr.williams@hospital.com",
|
||||
"Initial diagnosis - Essential hypertension"
|
||||
)
|
||||
|
||||
# Verify integrity
|
||||
assert manager.verify_checksum(snapshot), "Data integrity check failed"
|
||||
|
||||
# Generate audit report
|
||||
for version in manager.list_versions():
|
||||
print(f"{version['timestamp']}: {version['label']} by {version['author']}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- **Usage Guide**: `semantica/change_management/change_management_usage.md`
|
||||
- **Performance Tests**: `tests/change_management/test_performance.py`
|
||||
- **CHANGELOG**: `CHANGELOG.md`
|
||||
- **GitHub**: https://github.com/Hawksight-AI/semantica
|
||||
@@ -0,0 +1,4 @@
|
||||
[pytest]
|
||||
markers =
|
||||
integration: marks tests as integration (deselect with '-m "not integration"')
|
||||
addopts = -ra
|
||||
@@ -0,0 +1,63 @@
|
||||
"""
|
||||
Enhanced Change Management Module for Semantica
|
||||
|
||||
This module provides comprehensive change management capabilities including:
|
||||
- Persistent version storage (SQLite and in-memory)
|
||||
- Detailed change tracking and diff algorithms
|
||||
- Standardized metadata and audit trails
|
||||
- Data integrity verification with checksums
|
||||
- Enhanced version managers for KG and ontologies
|
||||
- Enterprise compliance support (HIPAA, SOX, FDA)
|
||||
|
||||
Public API:
|
||||
ChangeLogEntry: Standardized metadata for version changes
|
||||
VersionStorage: Abstract storage interface
|
||||
InMemoryVersionStorage: Fast in-memory storage backend
|
||||
SQLiteVersionStorage: Persistent SQLite storage backend
|
||||
compute_checksum: SHA-256 checksum computation
|
||||
verify_checksum: Data integrity verification
|
||||
EnhancedTemporalVersionManager: Advanced KG version management
|
||||
EnhancedVersionManager: Advanced ontology version management
|
||||
"""
|
||||
|
||||
from .change_log import ChangeLogEntry
|
||||
from .version_storage import (
|
||||
VersionStorage,
|
||||
InMemoryVersionStorage,
|
||||
SQLiteVersionStorage,
|
||||
compute_checksum,
|
||||
verify_checksum
|
||||
)
|
||||
from .managers import (
|
||||
BaseVersionManager,
|
||||
TemporalVersionManager,
|
||||
OntologyVersionManager
|
||||
)
|
||||
from .ontology_version_manager import VersionManager, OntologyVersion
|
||||
|
||||
__all__ = [
|
||||
# Change metadata
|
||||
"ChangeLogEntry",
|
||||
|
||||
# Storage backends
|
||||
"VersionStorage",
|
||||
"InMemoryVersionStorage",
|
||||
"SQLiteVersionStorage",
|
||||
|
||||
# Integrity utilities
|
||||
"compute_checksum",
|
||||
"verify_checksum",
|
||||
|
||||
# Version managers
|
||||
"BaseVersionManager",
|
||||
"TemporalVersionManager",
|
||||
"OntologyVersionManager",
|
||||
|
||||
# Ontology version management
|
||||
"VersionManager",
|
||||
"OntologyVersion"
|
||||
]
|
||||
|
||||
__version__ = "1.0.0"
|
||||
__author__ = "Semantica Team"
|
||||
__description__ = "Enhanced Change Management for Semantica"
|
||||
@@ -0,0 +1,108 @@
|
||||
"""
|
||||
Change Log Module
|
||||
|
||||
This module provides standardized metadata structures for version changes
|
||||
across both ontology and knowledge graph versioning systems.
|
||||
|
||||
Key Features:
|
||||
- Standardized ChangeLogEntry dataclass
|
||||
- Email validation for authors
|
||||
- Timestamp handling in ISO 8601 format
|
||||
- Optional change linking and tracking
|
||||
|
||||
Main Classes:
|
||||
- ChangeLogEntry: Standard metadata for version changes
|
||||
|
||||
Example Usage:
|
||||
>>> from semantica.common.change_log import ChangeLogEntry
|
||||
>>> entry = ChangeLogEntry(
|
||||
... timestamp="2024-01-15T10:30:00Z",
|
||||
... author="alice@company.com",
|
||||
... description="Added Customer entity"
|
||||
... )
|
||||
|
||||
Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from ..utils.exceptions import ValidationError
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChangeLogEntry:
|
||||
"""
|
||||
Standard metadata for version changes.
|
||||
|
||||
This dataclass provides a consistent structure for tracking changes
|
||||
across both ontology and knowledge graph versioning systems.
|
||||
|
||||
Attributes:
|
||||
timestamp: ISO 8601 timestamp of the change
|
||||
author: Email address of the change author
|
||||
description: Description of the change (max 500 characters)
|
||||
change_id: Optional unique identifier for the change
|
||||
related_changes: Optional list of related change IDs
|
||||
"""
|
||||
|
||||
timestamp: str
|
||||
author: str
|
||||
description: str
|
||||
change_id: Optional[str] = None
|
||||
related_changes: List[str] = field(default_factory=list)
|
||||
|
||||
def __post_init__(self):
|
||||
"""Validate fields after initialization."""
|
||||
self._validate_timestamp()
|
||||
self._validate_author()
|
||||
self._validate_description()
|
||||
|
||||
def _validate_timestamp(self):
|
||||
"""Validate timestamp is in ISO 8601 format."""
|
||||
try:
|
||||
# More strict validation for ISO 8601 format
|
||||
if 'T' not in self.timestamp:
|
||||
raise ValueError("Missing 'T' separator")
|
||||
datetime.fromisoformat(self.timestamp.replace('Z', '+00:00'))
|
||||
except ValueError:
|
||||
raise ValidationError(f"Invalid timestamp format: {self.timestamp}. Expected ISO 8601 format.")
|
||||
|
||||
def _validate_author(self):
|
||||
"""Validate author is a valid email address."""
|
||||
email_pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
if not re.match(email_pattern, self.author):
|
||||
raise ValidationError(f"Invalid email format: {self.author}")
|
||||
|
||||
def _validate_description(self):
|
||||
"""Validate description length."""
|
||||
if len(self.description) > 500:
|
||||
raise ValidationError(f"Description too long: {len(self.description)} characters (max 500)")
|
||||
if not self.description.strip():
|
||||
raise ValidationError("Description cannot be empty")
|
||||
|
||||
@classmethod
|
||||
def create_now(cls, author: str, description: str, change_id: Optional[str] = None,
|
||||
related_changes: Optional[List[str]] = None) -> 'ChangeLogEntry':
|
||||
"""
|
||||
Create a ChangeLogEntry with current timestamp.
|
||||
|
||||
Args:
|
||||
author: Email address of the change author
|
||||
description: Description of the change
|
||||
change_id: Optional unique identifier for the change
|
||||
related_changes: Optional list of related change IDs
|
||||
|
||||
Returns:
|
||||
ChangeLogEntry with current timestamp
|
||||
"""
|
||||
return cls(
|
||||
timestamp=datetime.now().isoformat(),
|
||||
author=author,
|
||||
description=description,
|
||||
change_id=change_id,
|
||||
related_changes=related_changes or []
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,497 @@
|
||||
"""
|
||||
Enhanced Version Managers Module
|
||||
|
||||
This module provides enhanced version management capabilities for both knowledge graphs
|
||||
and ontologies, with comprehensive change tracking, persistent storage, and audit trails.
|
||||
|
||||
Key Features:
|
||||
- Enhanced TemporalVersionManager for knowledge graphs
|
||||
- Enhanced VersionManager for ontologies
|
||||
- Detailed diff algorithms for entities and relationships
|
||||
- Structural comparison for ontology elements
|
||||
- Integration with storage backends and metadata
|
||||
|
||||
Main Classes:
|
||||
- EnhancedTemporalVersionManager: Advanced KG version management
|
||||
- EnhancedVersionManager: Advanced ontology version management
|
||||
|
||||
Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .change_log import ChangeLogEntry
|
||||
from .version_storage import VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage, compute_checksum, verify_checksum
|
||||
from ..utils.exceptions import ValidationError, ProcessingError
|
||||
from ..utils.logging import get_logger
|
||||
|
||||
|
||||
class BaseVersionManager(ABC):
|
||||
"""
|
||||
Abstract base class for enhanced version managers.
|
||||
|
||||
Provides common functionality for version management across different data types.
|
||||
"""
|
||||
|
||||
def __init__(self, storage_path: Optional[str] = None):
|
||||
"""
|
||||
Initialize base version manager.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database for persistent storage.
|
||||
If None, uses in-memory storage.
|
||||
"""
|
||||
self.logger = get_logger(self.__class__.__name__.lower())
|
||||
|
||||
# Initialize storage backend
|
||||
if storage_path:
|
||||
self.storage = SQLiteVersionStorage(storage_path)
|
||||
self.logger.info(f"Initialized with SQLite storage: {storage_path}")
|
||||
else:
|
||||
self.storage = InMemoryVersionStorage()
|
||||
self.logger.info("Initialized with in-memory storage")
|
||||
|
||||
@abstractmethod
|
||||
def create_snapshot(self, data: Any, version_label: str, author: str, description: str, **options) -> Dict[str, Any]:
|
||||
"""Create a versioned snapshot of the data."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def compare_versions(self, version1: Any, version2: Any, **options) -> Dict[str, Any]:
|
||||
"""Compare two versions and return detailed differences."""
|
||||
pass
|
||||
|
||||
def list_versions(self) -> List[Dict[str, Any]]:
|
||||
"""List all version snapshots."""
|
||||
return self.storage.list_all()
|
||||
|
||||
def get_version(self, label: str) -> Optional[Dict[str, Any]]:
|
||||
"""Retrieve specific version by label."""
|
||||
return self.storage.get(label)
|
||||
|
||||
def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
|
||||
"""Verify the integrity of a snapshot using its checksum."""
|
||||
return verify_checksum(snapshot)
|
||||
|
||||
|
||||
class TemporalVersionManager(BaseVersionManager):
|
||||
"""
|
||||
Temporal version management engine for knowledge graphs.
|
||||
|
||||
Provides comprehensive version/snapshot management capabilities including
|
||||
persistent storage, detailed change tracking, and audit trails.
|
||||
|
||||
Features:
|
||||
- Persistent snapshot storage (SQLite or in-memory)
|
||||
- Detailed change tracking with entity-level diffs
|
||||
- SHA-256 checksums for data integrity
|
||||
- Standardized metadata with author attribution
|
||||
- Version comparison with backward compatibility
|
||||
- Input validation and security features
|
||||
"""
|
||||
|
||||
def __init__(self, storage_path: Optional[str] = None, **config):
|
||||
"""
|
||||
Initialize enhanced temporal version manager.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file for persistent storage.
|
||||
If None, uses in-memory storage
|
||||
**config: Additional configuration options
|
||||
"""
|
||||
super().__init__(storage_path)
|
||||
self.config = config
|
||||
|
||||
def create_snapshot(
|
||||
self,
|
||||
graph: Dict[str, Any],
|
||||
version_label: str,
|
||||
author: str,
|
||||
description: str,
|
||||
**options
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create and store snapshot with checksum and metadata.
|
||||
|
||||
Args:
|
||||
graph: Knowledge graph dict with "entities" and "relationships"
|
||||
version_label: Version string (e.g., "v1.0")
|
||||
author: Email address of the change author
|
||||
description: Change description (max 500 chars)
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
dict: Snapshot with metadata and checksum
|
||||
|
||||
Raises:
|
||||
ValidationError: If input validation fails
|
||||
ProcessingError: If storage operation fails
|
||||
"""
|
||||
# Validate inputs
|
||||
change_entry = ChangeLogEntry(
|
||||
timestamp=datetime.now().isoformat(),
|
||||
author=author,
|
||||
description=description
|
||||
)
|
||||
|
||||
# Create snapshot
|
||||
snapshot = {
|
||||
"label": version_label,
|
||||
"timestamp": change_entry.timestamp,
|
||||
"author": change_entry.author,
|
||||
"description": change_entry.description,
|
||||
"entities": graph.get("entities", []).copy(),
|
||||
"relationships": graph.get("relationships", []).copy(),
|
||||
"metadata": options.get("metadata", {})
|
||||
}
|
||||
|
||||
# Compute and add checksum
|
||||
snapshot["checksum"] = compute_checksum(snapshot)
|
||||
|
||||
# Store snapshot
|
||||
self.storage.save(snapshot)
|
||||
|
||||
self.logger.info(f"Created snapshot '{version_label}' by {author}")
|
||||
return snapshot
|
||||
|
||||
def compare_versions(
|
||||
self,
|
||||
v1_label_or_dict,
|
||||
v2_label_or_dict,
|
||||
comparison_metrics: Optional[List[str]] = None,
|
||||
**options,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Compare two graph versions with detailed entity-level differences.
|
||||
|
||||
Args:
|
||||
v1_label_or_dict: First version (label string or snapshot dict)
|
||||
v2_label_or_dict: Second version (label string or snapshot dict)
|
||||
comparison_metrics: List of metrics to calculate (optional, unused)
|
||||
**options: Additional comparison options (unused)
|
||||
|
||||
Returns:
|
||||
dict: Detailed version comparison results
|
||||
"""
|
||||
# Handle both label strings and snapshot dictionaries
|
||||
if isinstance(v1_label_or_dict, str):
|
||||
version1 = self.storage.get(v1_label_or_dict)
|
||||
if not version1:
|
||||
raise ValidationError(f"Version not found: {v1_label_or_dict}")
|
||||
else:
|
||||
version1 = v1_label_or_dict
|
||||
|
||||
if isinstance(v2_label_or_dict, str):
|
||||
version2 = self.storage.get(v2_label_or_dict)
|
||||
if not version2:
|
||||
raise ValidationError(f"Version not found: {v2_label_or_dict}")
|
||||
else:
|
||||
version2 = v2_label_or_dict
|
||||
|
||||
# Compute detailed diff
|
||||
detailed_diff = self._compute_detailed_diff(version1, version2)
|
||||
|
||||
# Maintain backward compatibility with summary
|
||||
summary = {
|
||||
"entities_added": len(detailed_diff["entities_added"]),
|
||||
"entities_removed": len(detailed_diff["entities_removed"]),
|
||||
"entities_modified": len(detailed_diff["entities_modified"]),
|
||||
"relationships_added": len(detailed_diff["relationships_added"]),
|
||||
"relationships_removed": len(detailed_diff["relationships_removed"]),
|
||||
"relationships_modified": len(detailed_diff["relationships_modified"])
|
||||
}
|
||||
|
||||
return {
|
||||
"version1": version1.get("label", "unknown"),
|
||||
"version2": version2.get("label", "unknown"),
|
||||
"summary": summary,
|
||||
**detailed_diff
|
||||
}
|
||||
|
||||
def _compute_detailed_diff(self, version1: Dict[str, Any], version2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Compute detailed entity and relationship differences between versions.
|
||||
|
||||
Args:
|
||||
version1: First version snapshot
|
||||
version2: Second version snapshot
|
||||
|
||||
Returns:
|
||||
Dict with detailed diff information
|
||||
"""
|
||||
entities1 = {e.get("id", str(i)): e for i, e in enumerate(version1.get("entities", []))}
|
||||
entities2 = {e.get("id", str(i)): e for i, e in enumerate(version2.get("entities", []))}
|
||||
|
||||
relationships1 = {self._relationship_key(r): r for r in version1.get("relationships", [])}
|
||||
relationships2 = {self._relationship_key(r): r for r in version2.get("relationships", [])}
|
||||
|
||||
# Entity differences
|
||||
entity_ids1 = set(entities1.keys())
|
||||
entity_ids2 = set(entities2.keys())
|
||||
|
||||
entities_added = [entities2[eid] for eid in entity_ids2 - entity_ids1]
|
||||
entities_removed = [entities1[eid] for eid in entity_ids1 - entity_ids2]
|
||||
|
||||
entities_modified = []
|
||||
for eid in entity_ids1 & entity_ids2:
|
||||
if entities1[eid] != entities2[eid]:
|
||||
changes = self._compute_entity_changes(entities1[eid], entities2[eid])
|
||||
entities_modified.append({
|
||||
"id": eid,
|
||||
"before": entities1[eid],
|
||||
"after": entities2[eid],
|
||||
"changes": changes
|
||||
})
|
||||
|
||||
# Relationship differences
|
||||
rel_keys1 = set(relationships1.keys())
|
||||
rel_keys2 = set(relationships2.keys())
|
||||
|
||||
relationships_added = [relationships2[key] for key in rel_keys2 - rel_keys1]
|
||||
relationships_removed = [relationships1[key] for key in rel_keys1 - rel_keys2]
|
||||
|
||||
relationships_modified = []
|
||||
for key in rel_keys1 & rel_keys2:
|
||||
if relationships1[key] != relationships2[key]:
|
||||
changes = self._compute_relationship_changes(relationships1[key], relationships2[key])
|
||||
relationships_modified.append({
|
||||
"key": key,
|
||||
"before": relationships1[key],
|
||||
"after": relationships2[key],
|
||||
"changes": changes
|
||||
})
|
||||
|
||||
return {
|
||||
"entities_added": entities_added,
|
||||
"entities_removed": entities_removed,
|
||||
"entities_modified": entities_modified,
|
||||
"relationships_added": relationships_added,
|
||||
"relationships_removed": relationships_removed,
|
||||
"relationships_modified": relationships_modified
|
||||
}
|
||||
|
||||
def _relationship_key(self, relationship: Dict[str, Any]) -> str:
|
||||
"""Generate a unique key for a relationship."""
|
||||
source = relationship.get("source", "")
|
||||
target = relationship.get("target", "")
|
||||
rel_type = relationship.get("type", relationship.get("relationship", ""))
|
||||
return f"{source}|{rel_type}|{target}"
|
||||
|
||||
def _compute_entity_changes(self, entity1: Dict[str, Any], entity2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Compute changes between two entity versions."""
|
||||
changes = {}
|
||||
all_keys = set(entity1.keys()) | set(entity2.keys())
|
||||
|
||||
for key in all_keys:
|
||||
val1 = entity1.get(key)
|
||||
val2 = entity2.get(key)
|
||||
|
||||
if val1 != val2:
|
||||
changes[key] = {"from": val1, "to": val2}
|
||||
|
||||
return changes
|
||||
|
||||
def _compute_relationship_changes(self, rel1: Dict[str, Any], rel2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Compute changes between two relationship versions."""
|
||||
changes = {}
|
||||
all_keys = set(rel1.keys()) | set(rel2.keys())
|
||||
|
||||
for key in all_keys:
|
||||
val1 = rel1.get(key)
|
||||
val2 = rel2.get(key)
|
||||
|
||||
if val1 != val2:
|
||||
changes[key] = {"from": val1, "to": val2}
|
||||
|
||||
return changes
|
||||
|
||||
|
||||
class OntologyVersionManager(BaseVersionManager):
|
||||
"""
|
||||
Version management for ontologies with structural comparison.
|
||||
|
||||
Provides comprehensive version management for ontologies including
|
||||
detailed structural analysis and change tracking.
|
||||
|
||||
Features:
|
||||
- Structural comparison of ontology elements
|
||||
- Detailed diff for classes, properties, individuals, axioms
|
||||
- Persistent storage with metadata
|
||||
- Change tracking and audit trails
|
||||
"""
|
||||
|
||||
def __init__(self, storage_path: Optional[str] = None, **config):
|
||||
"""
|
||||
Initialize enhanced version manager.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file for persistent storage.
|
||||
If None, uses in-memory storage
|
||||
**config: Additional configuration options
|
||||
"""
|
||||
super().__init__(storage_path)
|
||||
self.config = config
|
||||
self.versions = {} # In-memory version tracking for compatibility
|
||||
|
||||
def create_snapshot(
|
||||
self,
|
||||
ontology_data: Dict[str, Any],
|
||||
version_label: str,
|
||||
author: str,
|
||||
description: str,
|
||||
**options
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create ontology version snapshot.
|
||||
|
||||
Args:
|
||||
ontology_data: Ontology data dictionary
|
||||
version_label: Version string (e.g., "v1.0")
|
||||
author: Email address of the change author
|
||||
description: Change description
|
||||
**options: Additional options including metadata
|
||||
|
||||
Returns:
|
||||
dict: Ontology version snapshot
|
||||
"""
|
||||
# Validate inputs
|
||||
change_entry = ChangeLogEntry(
|
||||
timestamp=datetime.now().isoformat(),
|
||||
author=author,
|
||||
description=description
|
||||
)
|
||||
|
||||
# Create snapshot
|
||||
snapshot = {
|
||||
"label": version_label,
|
||||
"timestamp": change_entry.timestamp,
|
||||
"author": change_entry.author,
|
||||
"description": change_entry.description,
|
||||
"ontology_iri": ontology_data.get("uri", ""),
|
||||
"version_info": ontology_data.get("version_info", {}),
|
||||
"structure": ontology_data.get("structure", {}),
|
||||
"metadata": options.get("metadata", {})
|
||||
}
|
||||
|
||||
# Compute and add checksum
|
||||
snapshot["checksum"] = compute_checksum(snapshot)
|
||||
|
||||
# Store snapshot
|
||||
self.storage.save(snapshot)
|
||||
|
||||
# Also store in memory for compatibility
|
||||
self.versions[version_label] = snapshot
|
||||
|
||||
self.logger.info(f"Created ontology snapshot '{version_label}' by {author}")
|
||||
return snapshot
|
||||
|
||||
def compare_versions(self, version1: str, version2: str, **options) -> Dict[str, Any]:
|
||||
"""
|
||||
Compare two ontology versions with detailed structural analysis.
|
||||
|
||||
Args:
|
||||
version1: First version label
|
||||
version2: Second version label
|
||||
**options: Additional comparison options
|
||||
|
||||
Returns:
|
||||
Detailed comparison results including structural differences
|
||||
"""
|
||||
# Get versions from storage
|
||||
v1_snapshot = self.storage.get(version1)
|
||||
v2_snapshot = self.storage.get(version2)
|
||||
|
||||
if not v1_snapshot:
|
||||
raise ValidationError(f"Version not found: {version1}")
|
||||
if not v2_snapshot:
|
||||
raise ValidationError(f"Version not found: {version2}")
|
||||
|
||||
# Basic metadata comparison
|
||||
metadata_changes = {}
|
||||
if v1_snapshot.get("ontology_iri") != v2_snapshot.get("ontology_iri"):
|
||||
metadata_changes["ontology_iri"] = {
|
||||
"from": v1_snapshot.get("ontology_iri"),
|
||||
"to": v2_snapshot.get("ontology_iri")
|
||||
}
|
||||
if v1_snapshot.get("version_info") != v2_snapshot.get("version_info"):
|
||||
metadata_changes["version_info"] = {
|
||||
"from": v1_snapshot.get("version_info"),
|
||||
"to": v2_snapshot.get("version_info")
|
||||
}
|
||||
|
||||
# Structural comparison
|
||||
structural_diff = self._compare_ontology_structures(v1_snapshot, v2_snapshot)
|
||||
|
||||
return {
|
||||
"version1": version1,
|
||||
"version2": version2,
|
||||
"metadata_changes": metadata_changes,
|
||||
**structural_diff
|
||||
}
|
||||
|
||||
def _compare_ontology_structures(self, v1_snapshot: Dict[str, Any], v2_snapshot: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Compare structural elements between two ontology versions.
|
||||
|
||||
Args:
|
||||
v1_snapshot: First ontology version snapshot
|
||||
v2_snapshot: Second ontology version snapshot
|
||||
|
||||
Returns:
|
||||
Dictionary with structural differences
|
||||
"""
|
||||
# Extract structural information
|
||||
v1_structure = v1_snapshot.get("structure", {})
|
||||
v2_structure = v2_snapshot.get("structure", {})
|
||||
|
||||
# Compare classes
|
||||
v1_classes = set(v1_structure.get("classes", []))
|
||||
v2_classes = set(v2_structure.get("classes", []))
|
||||
|
||||
classes_added = list(v2_classes - v1_classes)
|
||||
classes_removed = list(v1_classes - v2_classes)
|
||||
|
||||
# Compare properties
|
||||
v1_properties = set(v1_structure.get("properties", []))
|
||||
v2_properties = set(v2_structure.get("properties", []))
|
||||
|
||||
properties_added = list(v2_properties - v1_properties)
|
||||
properties_removed = list(v1_properties - v2_properties)
|
||||
|
||||
# Compare individuals
|
||||
v1_individuals = set(v1_structure.get("individuals", []))
|
||||
v2_individuals = set(v2_structure.get("individuals", []))
|
||||
|
||||
individuals_added = list(v2_individuals - v1_individuals)
|
||||
individuals_removed = list(v1_individuals - v2_individuals)
|
||||
|
||||
# Compare axioms/rules
|
||||
v1_axioms = set(v1_structure.get("axioms", []))
|
||||
v2_axioms = set(v2_structure.get("axioms", []))
|
||||
|
||||
axioms_added = list(v2_axioms - v1_axioms)
|
||||
axioms_removed = list(v1_axioms - v2_axioms)
|
||||
|
||||
return {
|
||||
"classes_added": classes_added,
|
||||
"classes_removed": classes_removed,
|
||||
"properties_added": properties_added,
|
||||
"properties_removed": properties_removed,
|
||||
"individuals_added": individuals_added,
|
||||
"individuals_removed": individuals_removed,
|
||||
"axioms_added": axioms_added,
|
||||
"axioms_removed": axioms_removed,
|
||||
"summary": {
|
||||
"classes_added": len(classes_added),
|
||||
"classes_removed": len(classes_removed),
|
||||
"properties_added": len(properties_added),
|
||||
"properties_removed": len(properties_removed),
|
||||
"individuals_added": len(individuals_added),
|
||||
"individuals_removed": len(individuals_removed),
|
||||
"axioms_added": len(axioms_added),
|
||||
"axioms_removed": len(axioms_removed)
|
||||
}
|
||||
}
|
||||
+76
-10
@@ -42,7 +42,7 @@ from typing import Any, Dict, List, Optional
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .namespace_manager import NamespaceManager
|
||||
from ..ontology.namespace_manager import NamespaceManager
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -198,14 +198,14 @@ class VersionManager:
|
||||
|
||||
def compare_versions(self, version1: str, version2: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Compare two ontology versions.
|
||||
Compare two ontology versions with detailed structural analysis.
|
||||
|
||||
Args:
|
||||
version1: First version
|
||||
version2: Second version
|
||||
|
||||
Returns:
|
||||
Comparison results
|
||||
Detailed comparison results including structural differences
|
||||
"""
|
||||
if version1 not in self.versions:
|
||||
raise ValidationError(f"Version not found: {version1}")
|
||||
@@ -215,19 +215,85 @@ class VersionManager:
|
||||
v1 = self.versions[version1]
|
||||
v2 = self.versions[version2]
|
||||
|
||||
# Basic comparison
|
||||
changes = []
|
||||
# Basic metadata comparison
|
||||
metadata_changes = {}
|
||||
if v1.ontology_iri != v2.ontology_iri:
|
||||
changes.append("Ontology IRI changed")
|
||||
metadata_changes["ontology_iri"] = {"from": v1.ontology_iri, "to": v2.ontology_iri}
|
||||
if v1.version_info != v2.version_info:
|
||||
changes.append("Version info changed")
|
||||
metadata_changes["version_info"] = {"from": v1.version_info, "to": v2.version_info}
|
||||
|
||||
# Structural comparison (if ontology data is available in metadata)
|
||||
structural_diff = self._compare_ontology_structures(v1, v2)
|
||||
|
||||
return {
|
||||
"version1": version1,
|
||||
"version2": version2,
|
||||
"changes": changes,
|
||||
"v1_iri": v1.ontology_iri,
|
||||
"v2_iri": v2.ontology_iri,
|
||||
"metadata_changes": metadata_changes,
|
||||
**structural_diff
|
||||
}
|
||||
|
||||
def _compare_ontology_structures(self, v1: OntologyVersion, v2: OntologyVersion) -> Dict[str, Any]:
|
||||
"""
|
||||
Compare structural elements between two ontology versions.
|
||||
|
||||
Args:
|
||||
v1: First ontology version
|
||||
v2: Second ontology version
|
||||
|
||||
Returns:
|
||||
Dictionary with structural differences
|
||||
"""
|
||||
# Extract structural information from metadata if available
|
||||
v1_structure = v1.metadata.get("structure", {})
|
||||
v2_structure = v2.metadata.get("structure", {})
|
||||
|
||||
# Compare classes
|
||||
v1_classes = set(v1_structure.get("classes", []))
|
||||
v2_classes = set(v2_structure.get("classes", []))
|
||||
|
||||
classes_added = list(v2_classes - v1_classes)
|
||||
classes_removed = list(v1_classes - v2_classes)
|
||||
|
||||
# Compare properties
|
||||
v1_properties = set(v1_structure.get("properties", []))
|
||||
v2_properties = set(v2_structure.get("properties", []))
|
||||
|
||||
properties_added = list(v2_properties - v1_properties)
|
||||
properties_removed = list(v1_properties - v2_properties)
|
||||
|
||||
# Compare individuals
|
||||
v1_individuals = set(v1_structure.get("individuals", []))
|
||||
v2_individuals = set(v2_structure.get("individuals", []))
|
||||
|
||||
individuals_added = list(v2_individuals - v1_individuals)
|
||||
individuals_removed = list(v1_individuals - v2_individuals)
|
||||
|
||||
# Compare axioms/rules if available
|
||||
v1_axioms = set(v1_structure.get("axioms", []))
|
||||
v2_axioms = set(v2_structure.get("axioms", []))
|
||||
|
||||
axioms_added = list(v2_axioms - v1_axioms)
|
||||
axioms_removed = list(v1_axioms - v2_axioms)
|
||||
|
||||
return {
|
||||
"classes_added": classes_added,
|
||||
"classes_removed": classes_removed,
|
||||
"properties_added": properties_added,
|
||||
"properties_removed": properties_removed,
|
||||
"individuals_added": individuals_added,
|
||||
"individuals_removed": individuals_removed,
|
||||
"axioms_added": axioms_added,
|
||||
"axioms_removed": axioms_removed,
|
||||
"summary": {
|
||||
"classes_added": len(classes_added),
|
||||
"classes_removed": len(classes_removed),
|
||||
"properties_added": len(properties_added),
|
||||
"properties_removed": len(properties_removed),
|
||||
"individuals_added": len(individuals_added),
|
||||
"individuals_removed": len(individuals_removed),
|
||||
"axioms_added": len(axioms_added),
|
||||
"axioms_removed": len(axioms_removed)
|
||||
}
|
||||
}
|
||||
|
||||
def get_version(self, version: str) -> Optional[OntologyVersion]:
|
||||
@@ -0,0 +1,391 @@
|
||||
"""
|
||||
Version Storage Module
|
||||
|
||||
This module provides abstract storage interfaces and concrete implementations
|
||||
for persistent version management in Semantica.
|
||||
|
||||
Key Features:
|
||||
- Abstract VersionStorage interface
|
||||
- In-memory storage implementation
|
||||
- SQLite-based persistent storage implementation
|
||||
- Checksum computation and validation
|
||||
- Thread-safe operations
|
||||
|
||||
Main Classes:
|
||||
- VersionStorage: Abstract base class for storage backends
|
||||
- InMemoryVersionStorage: Dictionary-based in-memory storage
|
||||
- SQLiteVersionStorage: SQLite-based persistent storage
|
||||
|
||||
Example Usage:
|
||||
>>> from semantica.common.version_storage import SQLiteVersionStorage
|
||||
>>> storage = SQLiteVersionStorage("versions.db")
|
||||
>>> storage.save(snapshot)
|
||||
>>> versions = storage.list_all()
|
||||
|
||||
Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
import threading
|
||||
from abc import ABC, abstractmethod
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
|
||||
|
||||
class VersionStorage(ABC):
|
||||
"""
|
||||
Abstract base class for version storage backends.
|
||||
|
||||
This interface defines the contract that all storage implementations
|
||||
must follow for version management operations.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def save(self, snapshot: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Save a version snapshot.
|
||||
|
||||
Args:
|
||||
snapshot: Version snapshot dictionary with metadata
|
||||
|
||||
Raises:
|
||||
ValidationError: If snapshot data is invalid
|
||||
ProcessingError: If save operation fails
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get(self, label: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Retrieve a version snapshot by label.
|
||||
|
||||
Args:
|
||||
label: Version label to retrieve
|
||||
|
||||
Returns:
|
||||
Snapshot dictionary or None if not found
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def list_all(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
List all version snapshots.
|
||||
|
||||
Returns:
|
||||
List of snapshot metadata dictionaries
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def exists(self, label: str) -> bool:
|
||||
"""
|
||||
Check if a version exists.
|
||||
|
||||
Args:
|
||||
label: Version label to check
|
||||
|
||||
Returns:
|
||||
True if version exists, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete(self, label: str) -> bool:
|
||||
"""
|
||||
Delete a version snapshot.
|
||||
|
||||
Args:
|
||||
label: Version label to delete
|
||||
|
||||
Returns:
|
||||
True if deleted, False if not found
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class InMemoryVersionStorage(VersionStorage):
|
||||
"""
|
||||
In-memory version storage implementation.
|
||||
|
||||
This implementation stores all version data in memory using a dictionary.
|
||||
Data is lost when the process ends.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize in-memory storage."""
|
||||
self._storage: Dict[str, Dict[str, Any]] = {}
|
||||
self._lock = threading.RLock()
|
||||
self.logger = get_logger("in_memory_storage")
|
||||
|
||||
def save(self, snapshot: Dict[str, Any]) -> None:
|
||||
"""Save snapshot to memory."""
|
||||
label = snapshot.get("label")
|
||||
if not label:
|
||||
raise ValidationError("Snapshot must have a 'label' field")
|
||||
|
||||
with self._lock:
|
||||
if label in self._storage:
|
||||
raise ValidationError(f"Version '{label}' already exists")
|
||||
|
||||
# Deep copy to prevent external modifications
|
||||
self._storage[label] = json.loads(json.dumps(snapshot))
|
||||
self.logger.debug(f"Saved version '{label}' to memory")
|
||||
|
||||
def get(self, label: str) -> Optional[Dict[str, Any]]:
|
||||
"""Retrieve snapshot from memory."""
|
||||
with self._lock:
|
||||
snapshot = self._storage.get(label)
|
||||
if snapshot:
|
||||
# Return deep copy to prevent external modifications
|
||||
return json.loads(json.dumps(snapshot))
|
||||
return None
|
||||
|
||||
def list_all(self) -> List[Dict[str, Any]]:
|
||||
"""List all snapshots in memory."""
|
||||
with self._lock:
|
||||
# Return metadata only (without full graph data)
|
||||
metadata_list = []
|
||||
for label, snapshot in self._storage.items():
|
||||
metadata = {
|
||||
"label": snapshot.get("label"),
|
||||
"timestamp": snapshot.get("timestamp"),
|
||||
"author": snapshot.get("author"),
|
||||
"description": snapshot.get("description"),
|
||||
"checksum": snapshot.get("checksum"),
|
||||
"entity_count": len(snapshot.get("entities", [])),
|
||||
"relationship_count": len(snapshot.get("relationships", []))
|
||||
}
|
||||
metadata_list.append(metadata)
|
||||
return metadata_list
|
||||
|
||||
def exists(self, label: str) -> bool:
|
||||
"""Check if version exists in memory."""
|
||||
with self._lock:
|
||||
return label in self._storage
|
||||
|
||||
def delete(self, label: str) -> bool:
|
||||
"""Delete version from memory."""
|
||||
with self._lock:
|
||||
if label in self._storage:
|
||||
del self._storage[label]
|
||||
self.logger.debug(f"Deleted version '{label}' from memory")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class SQLiteVersionStorage(VersionStorage):
|
||||
"""
|
||||
SQLite-based persistent version storage implementation.
|
||||
|
||||
This implementation stores version data in a SQLite database file,
|
||||
providing persistence across process restarts.
|
||||
"""
|
||||
|
||||
def __init__(self, storage_path: str):
|
||||
"""
|
||||
Initialize SQLite storage.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file
|
||||
"""
|
||||
self.storage_path = Path(storage_path)
|
||||
self._lock = threading.RLock()
|
||||
self.logger = get_logger("sqlite_storage")
|
||||
|
||||
# Create directory if it doesn't exist
|
||||
self.storage_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Initialize database
|
||||
self._init_database()
|
||||
|
||||
def _init_database(self) -> None:
|
||||
"""Initialize SQLite database schema."""
|
||||
with self._lock:
|
||||
conn = sqlite3.connect(str(self.storage_path))
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS versions (
|
||||
label TEXT PRIMARY KEY,
|
||||
timestamp TEXT NOT NULL,
|
||||
author TEXT NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
checksum TEXT NOT NULL,
|
||||
snapshot_data TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL
|
||||
)
|
||||
""")
|
||||
conn.commit()
|
||||
self.logger.debug(f"Initialized SQLite database at {self.storage_path}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def save(self, snapshot: Dict[str, Any]) -> None:
|
||||
"""Save snapshot to SQLite database."""
|
||||
label = snapshot.get("label")
|
||||
if not label:
|
||||
raise ValidationError("Snapshot must have a 'label' field")
|
||||
|
||||
with self._lock:
|
||||
conn = sqlite3.connect(str(self.storage_path))
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if version already exists
|
||||
cursor.execute("SELECT label FROM versions WHERE label = ?", (label,))
|
||||
if cursor.fetchone():
|
||||
raise ValidationError(f"Version '{label}' already exists")
|
||||
|
||||
# Insert new version
|
||||
cursor.execute("""
|
||||
INSERT INTO versions
|
||||
(label, timestamp, author, description, checksum, snapshot_data, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
label,
|
||||
snapshot.get("timestamp", ""),
|
||||
snapshot.get("author", ""),
|
||||
snapshot.get("description", ""),
|
||||
snapshot.get("checksum", ""),
|
||||
json.dumps(snapshot),
|
||||
datetime.now().isoformat()
|
||||
))
|
||||
|
||||
conn.commit()
|
||||
self.logger.debug(f"Saved version '{label}' to SQLite database")
|
||||
|
||||
except sqlite3.Error as e:
|
||||
raise ProcessingError(f"Failed to save version to database: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def get(self, label: str) -> Optional[Dict[str, Any]]:
|
||||
"""Retrieve snapshot from SQLite database."""
|
||||
with self._lock:
|
||||
conn = sqlite3.connect(str(self.storage_path))
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT snapshot_data FROM versions WHERE label = ?
|
||||
""", (label,))
|
||||
|
||||
row = cursor.fetchone()
|
||||
if not row:
|
||||
return None
|
||||
|
||||
return json.loads(row[0])
|
||||
|
||||
except sqlite3.Error as e:
|
||||
raise ProcessingError(f"Failed to retrieve version from database: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def list_all(self) -> List[Dict[str, Any]]:
|
||||
"""List all snapshots in SQLite database."""
|
||||
with self._lock:
|
||||
conn = sqlite3.connect(str(self.storage_path))
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
SELECT snapshot_data FROM versions ORDER BY timestamp DESC
|
||||
""")
|
||||
|
||||
metadata_list = []
|
||||
for row in cursor.fetchall():
|
||||
snapshot = json.loads(row[0])
|
||||
|
||||
metadata = {
|
||||
"label": snapshot.get("label"),
|
||||
"timestamp": snapshot.get("timestamp"),
|
||||
"author": snapshot.get("author"),
|
||||
"description": snapshot.get("description"),
|
||||
"checksum": snapshot.get("checksum"),
|
||||
"entity_count": len(snapshot.get("entities", [])),
|
||||
"relationship_count": len(snapshot.get("relationships", []))
|
||||
}
|
||||
metadata_list.append(metadata)
|
||||
|
||||
return metadata_list
|
||||
|
||||
except sqlite3.Error as e:
|
||||
raise ProcessingError(f"Failed to list versions from database: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def exists(self, label: str) -> bool:
|
||||
"""Check if version exists in SQLite database."""
|
||||
with self._lock:
|
||||
conn = sqlite3.connect(str(self.storage_path))
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT 1 FROM versions WHERE label = ?", (label,))
|
||||
return cursor.fetchone() is not None
|
||||
except sqlite3.Error as e:
|
||||
raise ProcessingError(f"Failed to check version existence: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def delete(self, label: str) -> bool:
|
||||
"""Delete version from SQLite database."""
|
||||
with self._lock:
|
||||
conn = sqlite3.connect(str(self.storage_path))
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM versions WHERE label = ?", (label,))
|
||||
deleted = cursor.rowcount > 0
|
||||
conn.commit()
|
||||
|
||||
if deleted:
|
||||
self.logger.debug(f"Deleted version '{label}' from SQLite database")
|
||||
|
||||
return deleted
|
||||
|
||||
except sqlite3.Error as e:
|
||||
raise ProcessingError(f"Failed to delete version from database: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def compute_checksum(data: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Compute SHA-256 checksum for version data.
|
||||
|
||||
Args:
|
||||
data: Dictionary containing version data
|
||||
|
||||
Returns:
|
||||
SHA-256 checksum as hexadecimal string
|
||||
"""
|
||||
# Create a deterministic JSON representation
|
||||
json_str = json.dumps(data, sort_keys=True, separators=(',', ':'))
|
||||
return hashlib.sha256(json_str.encode('utf-8')).hexdigest()
|
||||
|
||||
|
||||
def verify_checksum(snapshot: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Verify the integrity of a snapshot using its checksum.
|
||||
|
||||
Args:
|
||||
snapshot: Snapshot dictionary with checksum field
|
||||
|
||||
Returns:
|
||||
True if checksum is valid, False otherwise
|
||||
"""
|
||||
stored_checksum = snapshot.get("checksum")
|
||||
if not stored_checksum:
|
||||
return False
|
||||
|
||||
# Create copy without checksum for verification
|
||||
data_copy = snapshot.copy()
|
||||
data_copy.pop("checksum", None)
|
||||
|
||||
computed_checksum = compute_checksum(data_copy)
|
||||
return stored_checksum == computed_checksum
|
||||
@@ -359,9 +359,14 @@ class FeedParser:
|
||||
|
||||
return parser.parse(date_string)
|
||||
except (ImportError, OSError):
|
||||
# If dateutil isn't available, fall through to raising ValueError
|
||||
pass
|
||||
except Exception as e:
|
||||
# If dateutil fails to parse, raise ValueError to signal invalid input
|
||||
raise ValueError(f"Invalid date format: {date_string}") from e
|
||||
|
||||
return None
|
||||
# No known formats matched and dateutil is unavailable; raise ValueError
|
||||
raise ValueError(f"Invalid date format: {date_string}")
|
||||
|
||||
def validate_feed(self, feed_data: FeedData) -> bool:
|
||||
"""
|
||||
|
||||
@@ -25,6 +25,8 @@ Example Usage:
|
||||
"""
|
||||
|
||||
import json
|
||||
import csv
|
||||
import chardet
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
@@ -193,23 +195,9 @@ class PandasIngestor:
|
||||
def from_csv(
|
||||
self,
|
||||
file_path: Union[str, Path],
|
||||
chunksize: Optional[int] = None,
|
||||
**pandas_options,
|
||||
) -> PandasData:
|
||||
"""
|
||||
Ingest data from CSV file.
|
||||
|
||||
This method reads a CSV file using pandas and ingests it as a DataFrame.
|
||||
|
||||
Args:
|
||||
file_path: Path to CSV file
|
||||
**pandas_options: Additional options passed to pd.read_csv()
|
||||
|
||||
Returns:
|
||||
PandasData: Ingested data object
|
||||
|
||||
Raises:
|
||||
ProcessingError: If CSV reading fails
|
||||
"""
|
||||
file_path = Path(file_path)
|
||||
|
||||
if not file_path.exists():
|
||||
@@ -223,15 +211,104 @@ class PandasIngestor:
|
||||
)
|
||||
|
||||
try:
|
||||
# Read CSV with pandas
|
||||
dataframe = pd.read_csv(file_path, **pandas_options)
|
||||
|
||||
self.progress_tracker.update_tracking(
|
||||
tracking_id, message="CSV read successfully, processing DataFrame..."
|
||||
# ---------- Encoding Detection ----------
|
||||
with open(file_path, "rb") as f:
|
||||
raw = f.read(100_000)
|
||||
encoding_info = chardet.detect(raw)
|
||||
encoding = encoding_info.get("encoding") or "utf-8"
|
||||
|
||||
|
||||
# ---------- Delimiter & Header Detection ----------
|
||||
with open(file_path, "r", encoding=encoding, errors="replace") as f:
|
||||
sample = f.read(10000)
|
||||
sniffer = csv.Sniffer()
|
||||
|
||||
try:
|
||||
dialect = sniffer.sniff(sample, delimiters=[",", ";", "\t", "|"])
|
||||
delimiter = dialect.delimiter
|
||||
quotechar = dialect.quotechar
|
||||
except Exception:
|
||||
delimiter = ","
|
||||
quotechar = '"'
|
||||
|
||||
# Header handling: default to True (treat first row as header)
|
||||
# unless user explicitly overrides via pandas_options['header'].
|
||||
has_header = True
|
||||
header_opt = pandas_options.get("header", None)
|
||||
if header_opt is None:
|
||||
has_header = True
|
||||
elif header_opt == 0 or header_opt == "infer":
|
||||
has_header = True
|
||||
else:
|
||||
# Any explicit non-header setting (e.g., None or int>0) implies no header
|
||||
try:
|
||||
has_header = False if header_opt is None or int(header_opt) != 0 else True
|
||||
except Exception:
|
||||
has_header = False
|
||||
|
||||
|
||||
skipped_rows = 0
|
||||
dataframes = []
|
||||
|
||||
# ---------- CSV Reading (Chunked if needed) ----------
|
||||
# Preserve explicit header setting (including None) if user provided it.
|
||||
has_explicit_header = "header" in pandas_options
|
||||
explicit_header = pandas_options.pop("header", None) if has_explicit_header else None
|
||||
header_arg = explicit_header if has_explicit_header else (0 if has_header else None)
|
||||
|
||||
reader = pd.read_csv(
|
||||
file_path,
|
||||
sep=delimiter,
|
||||
encoding=encoding,
|
||||
encoding_errors="replace",
|
||||
quoting=csv.QUOTE_MINIMAL,
|
||||
header=header_arg,
|
||||
quotechar=quotechar,
|
||||
escapechar="\\",
|
||||
engine="python",
|
||||
on_bad_lines="warn",
|
||||
chunksize=chunksize,
|
||||
**pandas_options,
|
||||
)
|
||||
|
||||
# Ingest the DataFrame
|
||||
return self.ingest_dataframe(dataframe, **pandas_options)
|
||||
if chunksize:
|
||||
for chunk in reader:
|
||||
dataframes.append(chunk)
|
||||
else:
|
||||
dataframes.append(reader)
|
||||
|
||||
dataframe = pd.concat(dataframes, ignore_index=True)
|
||||
|
||||
self.progress_tracker.update_tracking(
|
||||
tracking_id,
|
||||
message="CSV parsed successfully, ingesting DataFrame...",
|
||||
)
|
||||
|
||||
# ---------- Ingest ----------
|
||||
pandas_data = self.ingest_dataframe(dataframe)
|
||||
|
||||
# ---------- Metadata ----------
|
||||
pandas_data.metadata.update(
|
||||
{
|
||||
"source": "csv",
|
||||
"file": str(file_path),
|
||||
"detected_encoding": encoding,
|
||||
"encoding_confidence": encoding_info.get("confidence"),
|
||||
"detected_delimiter": delimiter,
|
||||
"header_detected": has_header,
|
||||
"chunksize": chunksize,
|
||||
"malformed_rows_skipped": skipped_rows,
|
||||
}
|
||||
)
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message=f"CSV ingestion completed: {pandas_data.row_count} rows",
|
||||
)
|
||||
|
||||
return pandas_data
|
||||
|
||||
except Exception as e:
|
||||
self.progress_tracker.stop_tracking(
|
||||
|
||||
+298
-33
@@ -631,39 +631,47 @@ class TemporalPatternDetector:
|
||||
|
||||
class TemporalVersionManager:
|
||||
"""
|
||||
Temporal version management engine.
|
||||
Enhanced temporal version management engine with persistent storage.
|
||||
|
||||
This class provides version/snapshot management capabilities for knowledge
|
||||
graphs, enabling creation of temporal versions, version comparison, and
|
||||
version history tracking.
|
||||
This class provides comprehensive version/snapshot management capabilities for knowledge
|
||||
graphs, including persistent storage, detailed change tracking, and audit trails.
|
||||
|
||||
Features:
|
||||
- Version snapshot creation
|
||||
- Version comparison
|
||||
- Version history tracking
|
||||
- Automatic snapshotting (planned)
|
||||
- Version rollback (planned)
|
||||
- Persistent snapshot storage (SQLite or in-memory)
|
||||
- Detailed change tracking with entity-level diffs
|
||||
- SHA-256 checksums for data integrity
|
||||
- Standardized metadata with author attribution
|
||||
- Version comparison with backward compatibility
|
||||
- Input validation and security features
|
||||
|
||||
Example Usage:
|
||||
>>> # In-memory storage
|
||||
>>> manager = TemporalVersionManager()
|
||||
>>> version = manager.create_version(graph, version_label="v1.0")
|
||||
>>> comparison = manager.compare_versions(version1, version2)
|
||||
>>> # Persistent storage
|
||||
>>> manager = TemporalVersionManager(storage_path="versions.db")
|
||||
>>> snapshot = manager.create_snapshot(graph, "v1.0",
|
||||
... author="alice@company.com", description="Initial release")
|
||||
>>> versions = manager.list_versions()
|
||||
>>> diff = manager.compare_versions("v1.0", "v1.1")
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_path: Optional[str] = None,
|
||||
snapshot_interval: Optional[int] = None,
|
||||
auto_snapshot: bool = False,
|
||||
version_strategy: str = "timestamp",
|
||||
**config,
|
||||
):
|
||||
"""
|
||||
Initialize temporal version manager.
|
||||
Initialize enhanced temporal version manager.
|
||||
|
||||
Sets up the version manager with snapshot configuration and versioning
|
||||
strategy.
|
||||
Sets up the version manager with storage backend, snapshot configuration,
|
||||
and versioning strategy.
|
||||
|
||||
Args:
|
||||
storage_path: Path to SQLite database file for persistent storage.
|
||||
If None, uses in-memory storage (default: None)
|
||||
snapshot_interval: Interval for automatic snapshots in seconds
|
||||
(optional, auto_snapshot must be True)
|
||||
auto_snapshot: Enable automatic snapshots (default: False)
|
||||
@@ -671,11 +679,23 @@ class TemporalVersionManager:
|
||||
- "timestamp": Use timestamps for version labels (default)
|
||||
- "incremental": Use incremental version numbers (planned)
|
||||
- "semantic": Use semantic versioning (planned)
|
||||
**config: Additional configuration options (unused)
|
||||
**config: Additional configuration options
|
||||
"""
|
||||
from semantica.change_management import ChangeLogEntry, VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage
|
||||
from ..utils.logging import get_logger
|
||||
|
||||
self.snapshot_interval = snapshot_interval
|
||||
self.auto_snapshot = auto_snapshot
|
||||
self.version_strategy = version_strategy
|
||||
self.logger = get_logger("temporal_version_manager")
|
||||
|
||||
# Initialize storage backend
|
||||
if storage_path:
|
||||
self.storage = SQLiteVersionStorage(storage_path)
|
||||
self.logger.info(f"Initialized with SQLite storage: {storage_path}")
|
||||
else:
|
||||
self.storage = InMemoryVersionStorage()
|
||||
self.logger.info("Initialized with in-memory storage")
|
||||
|
||||
def create_version(
|
||||
self,
|
||||
@@ -722,37 +742,282 @@ class TemporalVersionManager:
|
||||
|
||||
def compare_versions(
|
||||
self,
|
||||
version1: Dict[str, Any],
|
||||
version2: Dict[str, Any],
|
||||
v1_label_or_dict,
|
||||
v2_label_or_dict,
|
||||
comparison_metrics: Optional[List[str]] = None,
|
||||
**options,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Compare two graph versions.
|
||||
Compare two graph versions with detailed entity-level differences.
|
||||
|
||||
This method compares two version snapshots and calculates differences
|
||||
in entities and relationships.
|
||||
This method compares two version snapshots and calculates detailed differences
|
||||
in entities and relationships, maintaining backward compatibility.
|
||||
|
||||
Args:
|
||||
version1: First version snapshot dictionary
|
||||
version2: Second version snapshot dictionary
|
||||
v1_label_or_dict: First version (label string or snapshot dict)
|
||||
v2_label_or_dict: Second version (label string or snapshot dict)
|
||||
comparison_metrics: List of metrics to calculate (optional, unused)
|
||||
**options: Additional comparison options (unused)
|
||||
|
||||
Returns:
|
||||
dict: Version comparison results containing:
|
||||
- version1: Label of first version
|
||||
- version2: Label of second version
|
||||
- entities_added: Change in entity count (version2 - version1)
|
||||
- relationships_added: Change in relationship count (version2 - version1)
|
||||
dict: Detailed version comparison results containing:
|
||||
- summary: Backward-compatible summary counts
|
||||
- entities_added: List of added entities
|
||||
- entities_removed: List of removed entities
|
||||
- entities_modified: List of modified entities with changes
|
||||
- relationships_added: List of added relationships
|
||||
- relationships_removed: List of removed relationships
|
||||
- relationships_modified: List of modified relationships
|
||||
"""
|
||||
comparison = {
|
||||
from ..utils.exceptions import ValidationError
|
||||
|
||||
# Handle both label strings and snapshot dictionaries
|
||||
if isinstance(v1_label_or_dict, str):
|
||||
version1 = self.storage.get(v1_label_or_dict)
|
||||
if not version1:
|
||||
raise ValidationError(f"Version not found: {v1_label_or_dict}")
|
||||
else:
|
||||
version1 = v1_label_or_dict
|
||||
|
||||
if isinstance(v2_label_or_dict, str):
|
||||
version2 = self.storage.get(v2_label_or_dict)
|
||||
if not version2:
|
||||
raise ValidationError(f"Version not found: {v2_label_or_dict}")
|
||||
else:
|
||||
version2 = v2_label_or_dict
|
||||
|
||||
# Compute detailed diff
|
||||
detailed_diff = self._compute_detailed_diff(version1, version2)
|
||||
|
||||
# Maintain backward compatibility with summary
|
||||
summary = {
|
||||
"entities_added": len(detailed_diff["entities_added"]),
|
||||
"entities_removed": len(detailed_diff["entities_removed"]),
|
||||
"entities_modified": len(detailed_diff["entities_modified"]),
|
||||
"relationships_added": len(detailed_diff["relationships_added"]),
|
||||
"relationships_removed": len(detailed_diff["relationships_removed"]),
|
||||
"relationships_modified": len(detailed_diff["relationships_modified"])
|
||||
}
|
||||
|
||||
return {
|
||||
"version1": version1.get("label", "unknown"),
|
||||
"version2": version2.get("label", "unknown"),
|
||||
"entities_added": len(version2.get("entities", []))
|
||||
- len(version1.get("entities", [])),
|
||||
"relationships_added": len(version2.get("relationships", []))
|
||||
- len(version1.get("relationships", [])),
|
||||
"summary": summary,
|
||||
**detailed_diff
|
||||
}
|
||||
|
||||
return comparison
|
||||
def create_snapshot(
|
||||
self,
|
||||
graph: Dict[str, Any],
|
||||
version_label: str,
|
||||
author: str,
|
||||
description: str,
|
||||
**options
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create and store snapshot with checksum and metadata.
|
||||
|
||||
Args:
|
||||
graph: Knowledge graph dict with "entities" and "relationships"
|
||||
version_label: Version string (e.g., "v1.0")
|
||||
author: Email address of the change author
|
||||
description: Change description (max 500 chars)
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
dict: Snapshot with metadata and checksum
|
||||
|
||||
Raises:
|
||||
ValidationError: If input validation fails
|
||||
ProcessingError: If storage operation fails
|
||||
"""
|
||||
from ..change_management import ChangeLogEntry, compute_checksum
|
||||
from datetime import datetime
|
||||
|
||||
# Validate inputs
|
||||
change_entry = ChangeLogEntry(
|
||||
timestamp=datetime.now().isoformat(),
|
||||
author=author,
|
||||
description=description
|
||||
)
|
||||
|
||||
# Create snapshot
|
||||
snapshot = {
|
||||
"label": version_label,
|
||||
"timestamp": change_entry.timestamp,
|
||||
"author": change_entry.author,
|
||||
"description": change_entry.description,
|
||||
"entities": graph.get("entities", []).copy(),
|
||||
"relationships": graph.get("relationships", []).copy(),
|
||||
"metadata": options.get("metadata", {})
|
||||
}
|
||||
|
||||
# Compute and add checksum
|
||||
snapshot["checksum"] = compute_checksum(snapshot)
|
||||
|
||||
# Store snapshot
|
||||
self.storage.save(snapshot)
|
||||
|
||||
self.logger.info(f"Created snapshot '{version_label}' by {author}")
|
||||
return snapshot
|
||||
|
||||
def list_versions(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
List all version snapshots.
|
||||
|
||||
Returns:
|
||||
List of version metadata dictionaries
|
||||
"""
|
||||
return self.storage.list_all()
|
||||
|
||||
def get_version(self, label: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Retrieve specific version by label.
|
||||
|
||||
Args:
|
||||
label: Version label to retrieve
|
||||
|
||||
Returns:
|
||||
Snapshot dictionary or None if not found
|
||||
"""
|
||||
return self.storage.get(label)
|
||||
|
||||
def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Verify the integrity of a snapshot using its checksum.
|
||||
|
||||
Args:
|
||||
snapshot: Snapshot dictionary with checksum field
|
||||
|
||||
Returns:
|
||||
True if checksum is valid, False otherwise
|
||||
"""
|
||||
from ..change_management import verify_checksum
|
||||
return verify_checksum(snapshot)
|
||||
|
||||
def _compute_detailed_diff(self, version1: Dict[str, Any], version2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Compute detailed entity and relationship differences between versions.
|
||||
|
||||
Args:
|
||||
version1: First version snapshot
|
||||
version2: Second version snapshot
|
||||
|
||||
Returns:
|
||||
Dict with detailed diff information
|
||||
"""
|
||||
entities1 = {e.get("id", str(i)): e for i, e in enumerate(version1.get("entities", []))}
|
||||
entities2 = {e.get("id", str(i)): e for i, e in enumerate(version2.get("entities", []))}
|
||||
|
||||
relationships1 = {self._relationship_key(r): r for r in version1.get("relationships", [])}
|
||||
relationships2 = {self._relationship_key(r): r for r in version2.get("relationships", [])}
|
||||
|
||||
# Entity differences
|
||||
entity_ids1 = set(entities1.keys())
|
||||
entity_ids2 = set(entities2.keys())
|
||||
|
||||
entities_added = [entities2[eid] for eid in entity_ids2 - entity_ids1]
|
||||
entities_removed = [entities1[eid] for eid in entity_ids1 - entity_ids2]
|
||||
|
||||
entities_modified = []
|
||||
for eid in entity_ids1 & entity_ids2:
|
||||
if entities1[eid] != entities2[eid]:
|
||||
changes = self._compute_entity_changes(entities1[eid], entities2[eid])
|
||||
entities_modified.append({
|
||||
"id": eid,
|
||||
"before": entities1[eid],
|
||||
"after": entities2[eid],
|
||||
"changes": changes
|
||||
})
|
||||
|
||||
# Relationship differences
|
||||
rel_keys1 = set(relationships1.keys())
|
||||
rel_keys2 = set(relationships2.keys())
|
||||
|
||||
relationships_added = [relationships2[key] for key in rel_keys2 - rel_keys1]
|
||||
relationships_removed = [relationships1[key] for key in rel_keys1 - rel_keys2]
|
||||
|
||||
relationships_modified = []
|
||||
for key in rel_keys1 & rel_keys2:
|
||||
if relationships1[key] != relationships2[key]:
|
||||
changes = self._compute_relationship_changes(relationships1[key], relationships2[key])
|
||||
relationships_modified.append({
|
||||
"key": key,
|
||||
"before": relationships1[key],
|
||||
"after": relationships2[key],
|
||||
"changes": changes
|
||||
})
|
||||
|
||||
return {
|
||||
"entities_added": entities_added,
|
||||
"entities_removed": entities_removed,
|
||||
"entities_modified": entities_modified,
|
||||
"relationships_added": relationships_added,
|
||||
"relationships_removed": relationships_removed,
|
||||
"relationships_modified": relationships_modified
|
||||
}
|
||||
|
||||
def _relationship_key(self, relationship: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Generate a unique key for a relationship.
|
||||
|
||||
Args:
|
||||
relationship: Relationship dictionary
|
||||
|
||||
Returns:
|
||||
Unique string key for the relationship
|
||||
"""
|
||||
source = relationship.get("source", "")
|
||||
target = relationship.get("target", "")
|
||||
rel_type = relationship.get("type", relationship.get("relationship", ""))
|
||||
return f"{source}|{rel_type}|{target}"
|
||||
|
||||
def _compute_entity_changes(self, entity1: Dict[str, Any], entity2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Compute changes between two entity versions.
|
||||
|
||||
Args:
|
||||
entity1: Original entity
|
||||
entity2: Modified entity
|
||||
|
||||
Returns:
|
||||
Dictionary of changes
|
||||
"""
|
||||
changes = {}
|
||||
|
||||
# Check all keys from both entities
|
||||
all_keys = set(entity1.keys()) | set(entity2.keys())
|
||||
|
||||
for key in all_keys:
|
||||
val1 = entity1.get(key)
|
||||
val2 = entity2.get(key)
|
||||
|
||||
if val1 != val2:
|
||||
changes[key] = {"from": val1, "to": val2}
|
||||
|
||||
return changes
|
||||
|
||||
def _compute_relationship_changes(self, rel1: Dict[str, Any], rel2: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Compute changes between two relationship versions.
|
||||
|
||||
Args:
|
||||
rel1: Original relationship
|
||||
rel2: Modified relationship
|
||||
|
||||
Returns:
|
||||
Dictionary of changes
|
||||
"""
|
||||
changes = {}
|
||||
|
||||
# Check all keys from both relationships
|
||||
all_keys = set(rel1.keys()) | set(rel2.keys())
|
||||
|
||||
for key in all_keys:
|
||||
val1 = rel1.get(key)
|
||||
val2 = rel2.get(key)
|
||||
|
||||
if val1 != val2:
|
||||
changes[key] = {"from": val1, "to": val2}
|
||||
|
||||
return changes
|
||||
|
||||
@@ -562,15 +562,19 @@ class SpecialCharacterProcessor:
|
||||
Returns:
|
||||
str: Text with normalized punctuation marks
|
||||
"""
|
||||
# Normalize quotes
|
||||
text = re.sub(r'["""]', '"', text)
|
||||
text = re.sub(r"[''']", "'", text)
|
||||
# Replace common smart punctuation with ASCII equivalents
|
||||
replacements = {
|
||||
"\u2018": "'", # Left single quotation mark
|
||||
"\u2019": "'", # Right single quotation mark
|
||||
"\u201C": '"', # Left double quotation mark
|
||||
"\u201D": '"', # Right double quotation mark
|
||||
"\u2013": "-", # En dash
|
||||
"\u2014": "--", # Em dash
|
||||
"\u2026": "...", # Ellipsis
|
||||
}
|
||||
|
||||
# Normalize dashes
|
||||
text = re.sub(r"[–—]", "-", text)
|
||||
|
||||
# Normalize ellipsis
|
||||
text = text.replace("…", "...")
|
||||
for old, new in replacements.items():
|
||||
text = text.replace(old, new)
|
||||
|
||||
return text
|
||||
|
||||
|
||||
@@ -156,7 +156,8 @@ from .property_generator import PropertyGenerator
|
||||
from .registry import MethodRegistry, method_registry
|
||||
from .requirements_spec import RequirementsSpec, RequirementsSpecManager
|
||||
from .reuse_manager import ReuseDecision, ReuseManager
|
||||
from .version_manager import OntologyVersion, VersionManager
|
||||
# VersionManager and OntologyVersion moved to change_management module
|
||||
# Import them directly from there: from semantica.change_management import VersionManager, OntologyVersion
|
||||
from semantica.ingest import OntologyData, OntologyIngestor
|
||||
from .methods import ingest_ontology
|
||||
|
||||
@@ -184,8 +185,7 @@ __all__ = [
|
||||
# Management
|
||||
"ReuseManager",
|
||||
"ReuseDecision",
|
||||
"VersionManager",
|
||||
"OntologyVersion",
|
||||
# VersionManager and OntologyVersion moved to change_management module
|
||||
"NamespaceManager",
|
||||
"NamingConventions",
|
||||
"ModuleManager",
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
"""
|
||||
Tests for the ChangeLogEntry module.
|
||||
|
||||
This module tests the standardized metadata structures for version changes.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime
|
||||
|
||||
from semantica.change_management import ChangeLogEntry
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
|
||||
|
||||
class TestChangeLogEntry:
|
||||
"""Test cases for ChangeLogEntry dataclass."""
|
||||
|
||||
def test_valid_change_log_entry(self):
|
||||
"""Test creating a valid change log entry."""
|
||||
entry = ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author="alice@company.com",
|
||||
description="Added Customer entity"
|
||||
)
|
||||
|
||||
assert entry.timestamp == "2024-01-15T10:30:00Z"
|
||||
assert entry.author == "alice@company.com"
|
||||
assert entry.description == "Added Customer entity"
|
||||
assert entry.change_id is None
|
||||
assert entry.related_changes == []
|
||||
|
||||
def test_change_log_entry_with_optional_fields(self):
|
||||
"""Test creating a change log entry with optional fields."""
|
||||
entry = ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author="bob@company.com",
|
||||
description="Modified Product entity",
|
||||
change_id="CHG-001",
|
||||
related_changes=["CHG-000"]
|
||||
)
|
||||
|
||||
assert entry.change_id == "CHG-001"
|
||||
assert entry.related_changes == ["CHG-000"]
|
||||
|
||||
def test_invalid_timestamp_format(self):
|
||||
"""Test that invalid timestamp format raises ValidationError."""
|
||||
with pytest.raises(ValidationError, match="Invalid timestamp format"):
|
||||
ChangeLogEntry(
|
||||
timestamp="2024-01-15 10:30:00", # Wrong format
|
||||
author="alice@company.com",
|
||||
description="Test change"
|
||||
)
|
||||
|
||||
def test_invalid_email_format(self):
|
||||
"""Test that invalid email format raises ValidationError."""
|
||||
with pytest.raises(ValidationError, match="Invalid email format"):
|
||||
ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author="invalid-email", # Invalid email
|
||||
description="Test change"
|
||||
)
|
||||
|
||||
def test_empty_description(self):
|
||||
"""Test that empty description raises ValidationError."""
|
||||
with pytest.raises(ValidationError, match="Description cannot be empty"):
|
||||
ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author="alice@company.com",
|
||||
description=" " # Empty/whitespace only
|
||||
)
|
||||
|
||||
def test_description_too_long(self):
|
||||
"""Test that description over 500 chars raises ValidationError."""
|
||||
long_description = "x" * 501
|
||||
with pytest.raises(ValidationError, match="Description too long"):
|
||||
ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author="alice@company.com",
|
||||
description=long_description
|
||||
)
|
||||
|
||||
def test_description_exactly_500_chars(self):
|
||||
"""Test that description of exactly 500 chars is valid."""
|
||||
description_500 = "x" * 500
|
||||
entry = ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author="alice@company.com",
|
||||
description=description_500
|
||||
)
|
||||
assert len(entry.description) == 500
|
||||
|
||||
def test_create_now_class_method(self):
|
||||
"""Test the create_now class method."""
|
||||
entry = ChangeLogEntry.create_now(
|
||||
author="charlie@company.com",
|
||||
description="Test change with current timestamp"
|
||||
)
|
||||
|
||||
# Verify timestamp is recent (within last minute)
|
||||
entry_time = datetime.fromisoformat(entry.timestamp)
|
||||
now = datetime.now()
|
||||
time_diff = abs((now - entry_time).total_seconds())
|
||||
assert time_diff < 60 # Within 1 minute
|
||||
|
||||
assert entry.author == "charlie@company.com"
|
||||
assert entry.description == "Test change with current timestamp"
|
||||
|
||||
def test_create_now_with_optional_fields(self):
|
||||
"""Test create_now with optional fields."""
|
||||
entry = ChangeLogEntry.create_now(
|
||||
author="dave@company.com",
|
||||
description="Test change",
|
||||
change_id="CHG-002",
|
||||
related_changes=["CHG-001", "CHG-000"]
|
||||
)
|
||||
|
||||
assert entry.change_id == "CHG-002"
|
||||
assert entry.related_changes == ["CHG-001", "CHG-000"]
|
||||
|
||||
def test_various_valid_email_formats(self):
|
||||
"""Test various valid email formats."""
|
||||
valid_emails = [
|
||||
"user@domain.com",
|
||||
"user.name@domain.co.uk",
|
||||
"user+tag@domain.org",
|
||||
"user123@domain123.net",
|
||||
"user_name@sub.domain.com"
|
||||
]
|
||||
|
||||
for email in valid_emails:
|
||||
entry = ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author=email,
|
||||
description="Test change"
|
||||
)
|
||||
assert entry.author == email
|
||||
|
||||
def test_various_invalid_email_formats(self):
|
||||
"""Test various invalid email formats."""
|
||||
invalid_emails = [
|
||||
"plainaddress",
|
||||
"@missingdomain.com",
|
||||
"missing@.com",
|
||||
"missing@domain",
|
||||
"spaces @domain.com",
|
||||
"double@@domain.com"
|
||||
]
|
||||
|
||||
for email in invalid_emails:
|
||||
with pytest.raises(ValidationError, match="Invalid email format"):
|
||||
ChangeLogEntry(
|
||||
timestamp="2024-01-15T10:30:00Z",
|
||||
author=email,
|
||||
description="Test change"
|
||||
)
|
||||
|
||||
def test_various_valid_timestamp_formats(self):
|
||||
"""Test various valid ISO 8601 timestamp formats."""
|
||||
valid_timestamps = [
|
||||
"2024-01-15T10:30:00Z",
|
||||
"2024-01-15T10:30:00+00:00",
|
||||
"2024-01-15T10:30:00.123Z",
|
||||
"2024-01-15T10:30:00.123456Z",
|
||||
"2024-01-15T10:30:00+05:30",
|
||||
"2024-01-15T10:30:00-08:00"
|
||||
]
|
||||
|
||||
for timestamp in valid_timestamps:
|
||||
entry = ChangeLogEntry(
|
||||
timestamp=timestamp,
|
||||
author="test@company.com",
|
||||
description="Test change"
|
||||
)
|
||||
assert entry.timestamp == timestamp
|
||||
@@ -0,0 +1,960 @@
|
||||
"""
|
||||
Comprehensive Integration Tests for Real-World Scenarios
|
||||
|
||||
This module contains integration tests covering real-world use cases including:
|
||||
- Healthcare compliance (HIPAA)
|
||||
- Financial compliance (SOX)
|
||||
- Pharmaceutical compliance (FDA 21 CFR Part 11)
|
||||
- Large-scale production scenarios
|
||||
- Data corruption and recovery
|
||||
- Migration and upgrade paths
|
||||
- Multi-user concurrent scenarios
|
||||
- Long-running production workflows
|
||||
|
||||
Author: Semantica Contributors
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
import json
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timedelta
|
||||
import pytest
|
||||
|
||||
from semantica.change_management import (
|
||||
TemporalVersionManager,
|
||||
OntologyVersionManager,
|
||||
ChangeLogEntry,
|
||||
InMemoryVersionStorage,
|
||||
SQLiteVersionStorage,
|
||||
compute_checksum,
|
||||
verify_checksum
|
||||
)
|
||||
from semantica.utils.exceptions import ValidationError, ProcessingError
|
||||
|
||||
|
||||
class TestHealthcareCompliance:
|
||||
"""Test healthcare compliance scenarios (HIPAA § 164.312(b))"""
|
||||
|
||||
def test_patient_record_audit_trail(self):
|
||||
"""Test complete audit trail for patient records"""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
|
||||
# Initial patient record
|
||||
patient_record = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "patient_001",
|
||||
"type": "Patient",
|
||||
"name": "John Doe",
|
||||
"dob": "1980-05-15",
|
||||
"mrn": "MR-2024-001",
|
||||
"ssn_last4": "1234"
|
||||
},
|
||||
{
|
||||
"id": "diagnosis_001",
|
||||
"type": "Diagnosis",
|
||||
"code": "I10",
|
||||
"description": "Essential hypertension",
|
||||
"date": "2024-01-15"
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source": "patient_001",
|
||||
"target": "diagnosis_001",
|
||||
"type": "has_diagnosis",
|
||||
"date": "2024-01-15"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Create initial version
|
||||
v1 = manager.create_snapshot(
|
||||
patient_record,
|
||||
"patient_001_v1.0",
|
||||
"dr.smith@hospital.com",
|
||||
"Initial patient record with hypertension diagnosis"
|
||||
)
|
||||
|
||||
assert v1 is not None
|
||||
assert manager.verify_checksum(v1)
|
||||
|
||||
# Add medication
|
||||
patient_record["entities"].append({
|
||||
"id": "medication_001",
|
||||
"type": "Medication",
|
||||
"name": "Lisinopril",
|
||||
"dosage": "10mg",
|
||||
"frequency": "once daily",
|
||||
"prescribed_date": "2024-01-15"
|
||||
})
|
||||
patient_record["relationships"].append({
|
||||
"source": "patient_001",
|
||||
"target": "medication_001",
|
||||
"type": "prescribed",
|
||||
"date": "2024-01-15"
|
||||
})
|
||||
|
||||
v2 = manager.create_snapshot(
|
||||
patient_record,
|
||||
"patient_001_v1.1",
|
||||
"dr.smith@hospital.com",
|
||||
"Added Lisinopril 10mg prescription"
|
||||
)
|
||||
|
||||
# Verify audit trail
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 2
|
||||
|
||||
# Verify all changes are tracked
|
||||
diff = manager.compare_versions("patient_001_v1.0", "patient_001_v1.1")
|
||||
assert diff["summary"]["entities_added"] == 1
|
||||
assert diff["summary"]["relationships_added"] == 1
|
||||
|
||||
# Verify data integrity for compliance
|
||||
for version in versions:
|
||||
retrieved = manager.get_version(version["label"])
|
||||
assert manager.verify_checksum(retrieved), f"Integrity check failed for {version['label']}"
|
||||
|
||||
# Verify author attribution
|
||||
assert all(v["author"] == "dr.smith@hospital.com" for v in versions)
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_hipaa_access_logging(self):
|
||||
"""Test that all access is logged for HIPAA compliance"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create patient record
|
||||
patient_data = {
|
||||
"entities": [{"id": "patient_123", "name": "Jane Smith", "ssn": "***-**-5678"}],
|
||||
"relationships": []
|
||||
}
|
||||
|
||||
# Multiple healthcare providers accessing and modifying
|
||||
providers = [
|
||||
("dr.jones@hospital.com", "Initial examination"),
|
||||
("nurse.williams@hospital.com", "Vital signs recorded"),
|
||||
("dr.chen@hospital.com", "Lab results added"),
|
||||
("pharmacist@hospital.com", "Medication dispensed")
|
||||
]
|
||||
|
||||
for i, (provider, description) in enumerate(providers):
|
||||
snapshot = manager.create_snapshot(
|
||||
patient_data,
|
||||
f"patient_123_v{i+1}",
|
||||
provider,
|
||||
description
|
||||
)
|
||||
assert snapshot["author"] == provider
|
||||
assert snapshot["description"] == description
|
||||
|
||||
# Verify complete access log
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 4
|
||||
|
||||
# Verify each access is properly attributed
|
||||
for i, version in enumerate(versions):
|
||||
assert version["author"] == providers[i][0]
|
||||
assert version["description"] == providers[i][1]
|
||||
|
||||
def test_phi_data_integrity(self):
|
||||
"""Test Protected Health Information (PHI) data integrity"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# PHI data
|
||||
phi_data = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "patient_456",
|
||||
"name": "Robert Johnson",
|
||||
"dob": "1975-03-20",
|
||||
"ssn": "***-**-9012",
|
||||
"address": "123 Main St, City, State",
|
||||
"phone": "555-0123",
|
||||
"email": "robert.j@email.com"
|
||||
}
|
||||
],
|
||||
"relationships": []
|
||||
}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
phi_data,
|
||||
"phi_v1",
|
||||
"admin@hospital.com",
|
||||
"PHI data snapshot"
|
||||
)
|
||||
|
||||
# Verify integrity
|
||||
assert manager.verify_checksum(snapshot)
|
||||
|
||||
# Simulate tampering
|
||||
snapshot["entities"][0]["ssn"] = "123-45-6789" # Unauthorized modification
|
||||
|
||||
# Should detect tampering
|
||||
assert not manager.verify_checksum(snapshot)
|
||||
|
||||
|
||||
class TestFinancialCompliance:
|
||||
"""Test financial compliance scenarios (SOX § 404)"""
|
||||
|
||||
def test_financial_transaction_audit(self):
|
||||
"""Test audit trail for financial transactions"""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
|
||||
# Financial transaction graph
|
||||
transaction_graph = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "txn_001",
|
||||
"type": "Transaction",
|
||||
"amount": 10000.00,
|
||||
"currency": "USD",
|
||||
"date": "2024-01-15",
|
||||
"status": "pending"
|
||||
},
|
||||
{
|
||||
"id": "account_001",
|
||||
"type": "Account",
|
||||
"number": "****1234",
|
||||
"balance": 50000.00
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source": "txn_001",
|
||||
"target": "account_001",
|
||||
"type": "debits"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Create transaction record
|
||||
v1 = manager.create_snapshot(
|
||||
transaction_graph,
|
||||
"txn_001_initial",
|
||||
"system@bank.com",
|
||||
"Transaction initiated"
|
||||
)
|
||||
|
||||
# Approval workflow
|
||||
transaction_graph["entities"][0]["status"] = "approved"
|
||||
transaction_graph["entities"][0]["approved_by"] = "manager@bank.com"
|
||||
transaction_graph["entities"][0]["approved_date"] = "2024-01-15T10:30:00Z"
|
||||
|
||||
v2 = manager.create_snapshot(
|
||||
transaction_graph,
|
||||
"txn_001_approved",
|
||||
"manager@bank.com",
|
||||
"Transaction approved by manager"
|
||||
)
|
||||
|
||||
# Completion
|
||||
transaction_graph["entities"][0]["status"] = "completed"
|
||||
transaction_graph["entities"][1]["balance"] = 40000.00
|
||||
|
||||
v3 = manager.create_snapshot(
|
||||
transaction_graph,
|
||||
"txn_001_completed",
|
||||
"system@bank.com",
|
||||
"Transaction completed and balance updated"
|
||||
)
|
||||
|
||||
# Verify complete audit trail
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 3
|
||||
|
||||
# Verify immutability - cannot overwrite
|
||||
with pytest.raises((ValidationError, ProcessingError)):
|
||||
manager.create_snapshot(
|
||||
transaction_graph,
|
||||
"txn_001_initial", # Duplicate label
|
||||
"hacker@evil.com",
|
||||
"Attempting to modify history"
|
||||
)
|
||||
|
||||
# Verify all versions have integrity
|
||||
for version in versions:
|
||||
retrieved = manager.get_version(version["label"])
|
||||
assert manager.verify_checksum(retrieved)
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_sox_change_control(self):
|
||||
"""Test SOX-compliant change control process"""
|
||||
manager = OntologyVersionManager()
|
||||
|
||||
# Financial ontology
|
||||
financial_ontology = {
|
||||
"uri": "https://bank.com/ontology/financial",
|
||||
"version_info": {"version": "1.0", "date": "2024-01-30"},
|
||||
"structure": {
|
||||
"classes": ["Account", "Transaction", "Customer"],
|
||||
"properties": ["accountNumber", "balance", "transactionAmount"],
|
||||
"individuals": ["CheckingAccount", "SavingsAccount"],
|
||||
"axioms": [
|
||||
"Account belongsTo exactly 1 Customer",
|
||||
"Transaction involves exactly 1 Account"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
# Initial version
|
||||
v1 = manager.create_snapshot(
|
||||
financial_ontology,
|
||||
"financial_ont_v1.0",
|
||||
"architect@bank.com",
|
||||
"Initial financial ontology"
|
||||
)
|
||||
|
||||
# Add compliance requirements
|
||||
financial_ontology["structure"]["classes"].extend(["ComplianceCheck", "AuditLog"])
|
||||
financial_ontology["structure"]["properties"].extend(["complianceStatus", "auditTimestamp"])
|
||||
financial_ontology["structure"]["axioms"].append(
|
||||
"Transaction requiresCompliance exactly 1 ComplianceCheck"
|
||||
)
|
||||
|
||||
v2 = manager.create_snapshot(
|
||||
financial_ontology,
|
||||
"financial_ont_v2.0",
|
||||
"compliance@bank.com",
|
||||
"Added SOX compliance requirements"
|
||||
)
|
||||
|
||||
# Verify structural changes are tracked
|
||||
diff = manager.compare_versions("financial_ont_v1.0", "financial_ont_v2.0")
|
||||
assert "ComplianceCheck" in diff["classes_added"]
|
||||
assert "AuditLog" in diff["classes_added"]
|
||||
assert len(diff["axioms_added"]) == 1
|
||||
|
||||
|
||||
class TestPharmaceuticalCompliance:
|
||||
"""Test pharmaceutical compliance scenarios (FDA 21 CFR Part 11)"""
|
||||
|
||||
def test_clinical_trial_data_integrity(self):
|
||||
"""Test FDA 21 CFR Part 11 compliant clinical trial data"""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
|
||||
# Clinical trial data
|
||||
trial_data = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "trial_001",
|
||||
"type": "ClinicalTrial",
|
||||
"name": "Phase III Efficacy Study",
|
||||
"drug": "Compound-X",
|
||||
"protocol": "PROTO-2024-001",
|
||||
"status": "active"
|
||||
},
|
||||
{
|
||||
"id": "cohort_001",
|
||||
"type": "PatientCohort",
|
||||
"size": 500,
|
||||
"demographics": "Adults 18-65",
|
||||
"enrollment_date": "2024-01-01"
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source": "trial_001",
|
||||
"target": "cohort_001",
|
||||
"type": "includes_cohort"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Baseline data with electronic signature
|
||||
v1 = manager.create_snapshot(
|
||||
trial_data,
|
||||
"trial_001_baseline",
|
||||
"principal.investigator@pharma.com",
|
||||
"Baseline clinical trial data - FDA 21 CFR Part 11 compliant"
|
||||
)
|
||||
|
||||
# Verify electronic signature (author email)
|
||||
assert v1["author"] == "principal.investigator@pharma.com"
|
||||
|
||||
# Verify data integrity
|
||||
assert manager.verify_checksum(v1)
|
||||
|
||||
# Add interim results
|
||||
trial_data["entities"].append({
|
||||
"id": "results_001",
|
||||
"type": "InterimResults",
|
||||
"date": "2024-02-15",
|
||||
"efficacy_rate": 0.75,
|
||||
"adverse_events": 12
|
||||
})
|
||||
|
||||
v2 = manager.create_snapshot(
|
||||
trial_data,
|
||||
"trial_001_interim",
|
||||
"data.manager@pharma.com",
|
||||
"Interim results - 6 week analysis"
|
||||
)
|
||||
|
||||
# Verify audit trail
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 2
|
||||
|
||||
# Verify data integrity for all versions (FDA requirement)
|
||||
for version in versions:
|
||||
retrieved = manager.get_version(version["label"])
|
||||
assert manager.verify_checksum(retrieved), \
|
||||
f"FDA 21 CFR Part 11 integrity check failed for {version['label']}"
|
||||
|
||||
# Generate audit report
|
||||
audit_report = []
|
||||
for version in versions:
|
||||
audit_report.append({
|
||||
"version": version["label"],
|
||||
"timestamp": version["timestamp"],
|
||||
"author": version["author"],
|
||||
"description": version["description"],
|
||||
"checksum": version["checksum"],
|
||||
"integrity_verified": manager.verify_checksum(
|
||||
manager.get_version(version["label"])
|
||||
)
|
||||
})
|
||||
|
||||
# All entries should have verified integrity
|
||||
assert all(entry["integrity_verified"] for entry in audit_report)
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_electronic_signature_validation(self):
|
||||
"""Test electronic signature validation for FDA compliance"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Valid electronic signature (email)
|
||||
data = {"entities": [], "relationships": []}
|
||||
snapshot = manager.create_snapshot(
|
||||
data,
|
||||
"v1",
|
||||
"qualified.person@pharma.com",
|
||||
"Signed by qualified person"
|
||||
)
|
||||
|
||||
assert snapshot["author"] == "qualified.person@pharma.com"
|
||||
|
||||
# Invalid signature should be rejected
|
||||
with pytest.raises(ValidationError, match="email"):
|
||||
manager.create_snapshot(
|
||||
data,
|
||||
"v2",
|
||||
"not-a-valid-signature",
|
||||
"Invalid signature"
|
||||
)
|
||||
|
||||
|
||||
class TestLargeScaleProduction:
|
||||
"""Test large-scale production scenarios"""
|
||||
|
||||
def test_high_volume_snapshots(self):
|
||||
"""Test handling high volume of snapshots"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create 100 versions rapidly
|
||||
base_graph = {"entities": [], "relationships": []}
|
||||
|
||||
start_time = time.perf_counter()
|
||||
for i in range(100):
|
||||
base_graph["entities"].append({"id": f"entity_{i}", "value": i})
|
||||
manager.create_snapshot(
|
||||
base_graph.copy(),
|
||||
f"v{i}",
|
||||
"system@company.com",
|
||||
f"Version {i}"
|
||||
)
|
||||
duration = time.perf_counter() - start_time
|
||||
|
||||
# Should handle 100 versions efficiently
|
||||
assert duration < 5.0, f"High volume test took {duration}s, should be <5s"
|
||||
|
||||
# Verify all versions are retrievable
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 100
|
||||
|
||||
# Spot check some versions
|
||||
for i in [0, 25, 50, 75, 99]:
|
||||
version = manager.get_version(f"v{i}")
|
||||
assert version is not None
|
||||
assert len(version["entities"]) == i + 1
|
||||
|
||||
def test_large_graph_performance(self):
|
||||
"""Test performance with very large graphs"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create graph with 5000 entities and 10000 relationships
|
||||
large_graph = {
|
||||
"entities": [
|
||||
{"id": f"entity_{i}", "type": "Node", "value": i}
|
||||
for i in range(5000)
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source": f"entity_{i}",
|
||||
"target": f"entity_{i+1}",
|
||||
"type": "connects"
|
||||
}
|
||||
for i in range(4999)
|
||||
] + [
|
||||
{
|
||||
"source": f"entity_{i}",
|
||||
"target": f"entity_{i+2}",
|
||||
"type": "skips"
|
||||
}
|
||||
for i in range(4998)
|
||||
]
|
||||
}
|
||||
|
||||
# Should handle large graph efficiently
|
||||
start = time.perf_counter()
|
||||
snapshot = manager.create_snapshot(
|
||||
large_graph,
|
||||
"large_v1",
|
||||
"system@company.com",
|
||||
"Large graph snapshot"
|
||||
)
|
||||
duration = time.perf_counter() - start
|
||||
|
||||
assert duration < 1.0, f"Large graph snapshot took {duration}s"
|
||||
assert len(snapshot["entities"]) == 5000
|
||||
assert len(snapshot["relationships"]) == 9997
|
||||
|
||||
def test_concurrent_write_load(self):
|
||||
"""Test concurrent write operations under load"""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
|
||||
def create_version(thread_id, count):
|
||||
"""Create versions from a thread"""
|
||||
results = []
|
||||
for i in range(count):
|
||||
graph = {
|
||||
"entities": [{"id": f"t{thread_id}_e{i}", "value": i}],
|
||||
"relationships": []
|
||||
}
|
||||
try:
|
||||
snapshot = manager.create_snapshot(
|
||||
graph,
|
||||
f"thread_{thread_id}_v{i}",
|
||||
f"user{thread_id}@company.com",
|
||||
f"Thread {thread_id} version {i}"
|
||||
)
|
||||
results.append(True)
|
||||
except Exception as e:
|
||||
results.append(False)
|
||||
return results
|
||||
|
||||
# Run 10 threads, each creating 20 versions
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [
|
||||
executor.submit(create_version, thread_id, 20)
|
||||
for thread_id in range(10)
|
||||
]
|
||||
|
||||
all_results = []
|
||||
for future in as_completed(futures):
|
||||
all_results.extend(future.result())
|
||||
|
||||
# All operations should succeed
|
||||
assert all(all_results), f"Some concurrent operations failed"
|
||||
|
||||
# Verify all 200 versions were created
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 200
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
|
||||
class TestDataCorruptionRecovery:
|
||||
"""Test data corruption detection and recovery scenarios"""
|
||||
|
||||
def test_detect_corrupted_data(self):
|
||||
"""Test detection of corrupted data"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
graph = {
|
||||
"entities": [{"id": "e1", "value": "original"}],
|
||||
"relationships": []
|
||||
}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
graph,
|
||||
"v1",
|
||||
"user@company.com",
|
||||
"Original data"
|
||||
)
|
||||
|
||||
# Verify original is valid
|
||||
assert manager.verify_checksum(snapshot)
|
||||
|
||||
# Simulate corruption
|
||||
snapshot["entities"][0]["value"] = "corrupted"
|
||||
|
||||
# Should detect corruption
|
||||
assert not manager.verify_checksum(snapshot)
|
||||
|
||||
def test_checksum_mismatch_detection(self):
|
||||
"""Test detection of checksum mismatches"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
graph = {"entities": [{"id": "e1"}], "relationships": []}
|
||||
snapshot = manager.create_snapshot(
|
||||
graph,
|
||||
"v1",
|
||||
"user@company.com",
|
||||
"Test"
|
||||
)
|
||||
|
||||
# Tamper with checksum
|
||||
original_checksum = snapshot["checksum"]
|
||||
snapshot["checksum"] = "0" * 64 # Invalid checksum
|
||||
|
||||
assert not manager.verify_checksum(snapshot)
|
||||
|
||||
# Restore correct checksum
|
||||
snapshot["checksum"] = original_checksum
|
||||
assert manager.verify_checksum(snapshot)
|
||||
|
||||
def test_recovery_from_valid_snapshot(self):
|
||||
"""Test recovery by reverting to valid snapshot"""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
|
||||
# Create valid snapshots
|
||||
graph = {"entities": [{"id": "e1", "status": "good"}], "relationships": []}
|
||||
|
||||
v1 = manager.create_snapshot(graph, "v1", "user@company.com", "Good v1")
|
||||
v2 = manager.create_snapshot(graph, "v2", "user@company.com", "Good v2")
|
||||
|
||||
# Simulate corruption in v2
|
||||
v2["entities"][0]["status"] = "corrupted"
|
||||
|
||||
# Detect corruption
|
||||
assert not manager.verify_checksum(v2)
|
||||
|
||||
# Recover by retrieving v1
|
||||
recovered = manager.get_version("v1")
|
||||
assert manager.verify_checksum(recovered)
|
||||
assert recovered["entities"][0]["status"] == "good"
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
|
||||
class TestMigrationUpgrade:
|
||||
"""Test migration and upgrade path scenarios"""
|
||||
|
||||
def test_storage_backend_migration(self):
|
||||
"""Test migration from in-memory to SQLite storage"""
|
||||
# Start with in-memory
|
||||
memory_manager = TemporalVersionManager()
|
||||
|
||||
graph = {
|
||||
"entities": [{"id": "e1", "name": "Entity 1"}],
|
||||
"relationships": []
|
||||
}
|
||||
|
||||
# Create versions in memory
|
||||
for i in range(5):
|
||||
memory_manager.create_snapshot(
|
||||
graph,
|
||||
f"v{i}",
|
||||
"user@company.com",
|
||||
f"Version {i}"
|
||||
)
|
||||
|
||||
memory_versions = memory_manager.list_versions()
|
||||
assert len(memory_versions) == 5
|
||||
|
||||
# Migrate to SQLite
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
sqlite_manager = TemporalVersionManager(storage_path=db_path)
|
||||
|
||||
# Manually migrate data
|
||||
for version_info in memory_versions:
|
||||
version_data = memory_manager.get_version(version_info["label"])
|
||||
sqlite_manager.storage.save(version_data)
|
||||
|
||||
# Verify migration
|
||||
sqlite_versions = sqlite_manager.list_versions()
|
||||
assert len(sqlite_versions) == 5
|
||||
|
||||
# Verify data integrity after migration
|
||||
for version_info in sqlite_versions:
|
||||
version = sqlite_manager.get_version(version_info["label"])
|
||||
assert sqlite_manager.verify_checksum(version)
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_backward_compatibility(self):
|
||||
"""Test backward compatibility with legacy API"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Old-style usage (should still work)
|
||||
graph = {"entities": [], "relationships": []}
|
||||
|
||||
# New API
|
||||
snapshot = manager.create_snapshot(
|
||||
graph,
|
||||
"v1",
|
||||
"user@company.com",
|
||||
"New API"
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert "checksum" in snapshot
|
||||
assert "author" in snapshot
|
||||
|
||||
|
||||
class TestEdgeCasesStress:
|
||||
"""Test edge cases and stress scenarios"""
|
||||
|
||||
def test_empty_graph_operations(self):
|
||||
"""Test operations on empty graphs"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
empty_graph = {"entities": [], "relationships": []}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
empty_graph,
|
||||
"empty_v1",
|
||||
"user@company.com",
|
||||
"Empty graph"
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert len(snapshot["entities"]) == 0
|
||||
assert len(snapshot["relationships"]) == 0
|
||||
assert manager.verify_checksum(snapshot)
|
||||
|
||||
def test_unicode_and_special_characters(self):
|
||||
"""Test handling of Unicode and special characters"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
unicode_graph = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "e1",
|
||||
"name": "Test with 中文字符",
|
||||
"emoji": "🎉🚀💻",
|
||||
"special": "Special chars: @#$%^&*()",
|
||||
"quotes": 'Single "double" quotes'
|
||||
}
|
||||
],
|
||||
"relationships": []
|
||||
}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
unicode_graph,
|
||||
"unicode_v1",
|
||||
"user@company.com",
|
||||
"Unicode test with émojis 🎉"
|
||||
)
|
||||
|
||||
# Verify data is preserved
|
||||
retrieved = manager.get_version("unicode_v1")
|
||||
assert retrieved["entities"][0]["name"] == "Test with 中文字符"
|
||||
assert retrieved["entities"][0]["emoji"] == "🎉🚀💻"
|
||||
assert manager.verify_checksum(retrieved)
|
||||
|
||||
def test_deeply_nested_structures(self):
|
||||
"""Test handling of deeply nested data structures"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
nested_graph = {
|
||||
"entities": [
|
||||
{
|
||||
"id": "e1",
|
||||
"level1": {
|
||||
"level2": {
|
||||
"level3": {
|
||||
"level4": {
|
||||
"level5": {
|
||||
"value": "deep nested value"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"relationships": []
|
||||
}
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
nested_graph,
|
||||
"nested_v1",
|
||||
"user@company.com",
|
||||
"Deeply nested structure"
|
||||
)
|
||||
|
||||
retrieved = manager.get_version("nested_v1")
|
||||
assert retrieved["entities"][0]["level1"]["level2"]["level3"]["level4"]["level5"]["value"] == "deep nested value"
|
||||
assert manager.verify_checksum(retrieved)
|
||||
|
||||
def test_very_long_descriptions(self):
|
||||
"""Test handling of maximum description length"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
graph = {"entities": [], "relationships": []}
|
||||
|
||||
# Maximum allowed length (500 chars)
|
||||
max_description = "x" * 500
|
||||
snapshot = manager.create_snapshot(
|
||||
graph,
|
||||
"v1",
|
||||
"user@company.com",
|
||||
max_description
|
||||
)
|
||||
assert snapshot["description"] == max_description
|
||||
|
||||
# Exceeding maximum should fail
|
||||
too_long = "x" * 501
|
||||
with pytest.raises(ValidationError, match="too long"):
|
||||
manager.create_snapshot(
|
||||
graph,
|
||||
"v2",
|
||||
"user@company.com",
|
||||
too_long
|
||||
)
|
||||
|
||||
def test_rapid_version_creation(self):
|
||||
"""Test rapid creation of many versions"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
graph = {"entities": [], "relationships": []}
|
||||
|
||||
# Create 50 versions as fast as possible
|
||||
start = time.perf_counter()
|
||||
for i in range(50):
|
||||
manager.create_snapshot(
|
||||
graph,
|
||||
f"rapid_v{i}",
|
||||
"user@company.com",
|
||||
f"Rapid version {i}"
|
||||
)
|
||||
duration = time.perf_counter() - start
|
||||
|
||||
# Should complete quickly
|
||||
assert duration < 2.0, f"Rapid creation took {duration}s"
|
||||
|
||||
# All versions should be present
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 50
|
||||
|
||||
|
||||
class TestLongRunningWorkflows:
|
||||
"""Test long-running production workflows"""
|
||||
|
||||
def test_daily_snapshot_workflow(self):
|
||||
"""Simulate daily snapshot workflow over extended period"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Simulate 30 days of daily snapshots
|
||||
base_date = datetime(2024, 1, 1)
|
||||
graph = {"entities": [], "relationships": []}
|
||||
|
||||
for day in range(30):
|
||||
current_date = base_date + timedelta(days=day)
|
||||
|
||||
# Add daily data
|
||||
graph["entities"].append({
|
||||
"id": f"daily_entity_{day}",
|
||||
"date": current_date.isoformat(),
|
||||
"value": day
|
||||
})
|
||||
|
||||
manager.create_snapshot(
|
||||
graph.copy(),
|
||||
f"daily_{current_date.strftime('%Y%m%d')}",
|
||||
"system@company.com",
|
||||
f"Daily snapshot for {current_date.date()}"
|
||||
)
|
||||
|
||||
# Verify all 30 days are recorded
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 30
|
||||
|
||||
# Verify data accumulation
|
||||
final_version = manager.get_version(f"daily_20240130")
|
||||
assert len(final_version["entities"]) == 30
|
||||
|
||||
def test_version_retention_policy(self):
|
||||
"""Test implementation of version retention policy"""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
graph = {"entities": [], "relationships": []}
|
||||
|
||||
# Create versions with different ages
|
||||
old_date = datetime(2023, 1, 1)
|
||||
recent_date = datetime(2024, 1, 1)
|
||||
|
||||
# Old versions
|
||||
for i in range(5):
|
||||
manager.create_snapshot(
|
||||
graph,
|
||||
f"old_v{i}",
|
||||
"user@company.com",
|
||||
f"Old version {i}"
|
||||
)
|
||||
|
||||
# Recent versions
|
||||
for i in range(5):
|
||||
manager.create_snapshot(
|
||||
graph,
|
||||
f"recent_v{i}",
|
||||
"user@company.com",
|
||||
f"Recent version {i}"
|
||||
)
|
||||
|
||||
# Simulate retention policy (keep only recent)
|
||||
all_versions = manager.list_versions()
|
||||
assert len(all_versions) == 10
|
||||
|
||||
# In production, would implement cleanup based on timestamp
|
||||
# For now, verify all versions are accessible
|
||||
for version in all_versions:
|
||||
retrieved = manager.get_version(version["label"])
|
||||
assert retrieved is not None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v", "--tb=short"])
|
||||
@@ -0,0 +1,332 @@
|
||||
"""
|
||||
Tests for Enhanced Version Managers
|
||||
|
||||
This module tests the enhanced version management capabilities for both
|
||||
knowledge graphs and ontologies with comprehensive change tracking.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from semantica.change_management import (
|
||||
TemporalVersionManager,
|
||||
OntologyVersionManager,
|
||||
ChangeLogEntry
|
||||
)
|
||||
from semantica.utils.exceptions import ValidationError, ProcessingError
|
||||
|
||||
|
||||
class TestTemporalVersionManager:
|
||||
"""Test cases for TemporalVersionManager."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.sample_graph = {
|
||||
"entities": [
|
||||
{"id": "entity1", "name": "Entity 1", "type": "Person"},
|
||||
{"id": "entity2", "name": "Entity 2", "type": "Organization"}
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "entity1", "target": "entity2", "type": "works_for"}
|
||||
]
|
||||
}
|
||||
|
||||
def test_in_memory_initialization(self):
|
||||
"""Test initialization with in-memory storage."""
|
||||
manager = TemporalVersionManager()
|
||||
assert manager.storage is not None
|
||||
assert manager.logger is not None
|
||||
|
||||
def test_sqlite_initialization(self):
|
||||
"""Test initialization with SQLite storage."""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
assert manager.storage is not None
|
||||
assert os.path.exists(db_path)
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_create_snapshot_basic(self):
|
||||
"""Test basic snapshot creation."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
self.sample_graph,
|
||||
"test_v1.0",
|
||||
"test@example.com",
|
||||
"Test snapshot creation"
|
||||
)
|
||||
|
||||
assert snapshot["label"] == "test_v1.0"
|
||||
assert snapshot["author"] == "test@example.com"
|
||||
assert snapshot["description"] == "Test snapshot creation"
|
||||
assert "checksum" in snapshot
|
||||
assert "timestamp" in snapshot
|
||||
assert len(snapshot["entities"]) == 2
|
||||
assert len(snapshot["relationships"]) == 1
|
||||
|
||||
def test_create_snapshot_with_invalid_author(self):
|
||||
"""Test snapshot creation with invalid author email."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
with pytest.raises(ValidationError, match="Invalid email format"):
|
||||
manager.create_snapshot(
|
||||
self.sample_graph,
|
||||
"test_v1.0",
|
||||
"invalid-email",
|
||||
"Test snapshot"
|
||||
)
|
||||
|
||||
def test_create_snapshot_with_long_description(self):
|
||||
"""Test snapshot creation with description too long."""
|
||||
manager = TemporalVersionManager()
|
||||
long_description = "x" * 501 # Exceeds 500 character limit
|
||||
|
||||
with pytest.raises(ValidationError, match="Description too long"):
|
||||
manager.create_snapshot(
|
||||
self.sample_graph,
|
||||
"test_v1.0",
|
||||
"test@example.com",
|
||||
long_description
|
||||
)
|
||||
|
||||
def test_list_versions(self):
|
||||
"""Test listing versions."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Initially empty
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 0
|
||||
|
||||
# Create snapshot
|
||||
manager.create_snapshot(
|
||||
self.sample_graph,
|
||||
"test_v1.0",
|
||||
"test@example.com",
|
||||
"Test snapshot"
|
||||
)
|
||||
|
||||
# Should have one version
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 1
|
||||
assert versions[0]["label"] == "test_v1.0"
|
||||
|
||||
def test_get_version(self):
|
||||
"""Test retrieving specific version."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create snapshot
|
||||
original_snapshot = manager.create_snapshot(
|
||||
self.sample_graph,
|
||||
"test_v1.0",
|
||||
"test@example.com",
|
||||
"Test snapshot"
|
||||
)
|
||||
|
||||
# Retrieve version
|
||||
retrieved = manager.get_version("test_v1.0")
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "test_v1.0"
|
||||
assert retrieved["checksum"] == original_snapshot["checksum"]
|
||||
|
||||
# Non-existent version
|
||||
assert manager.get_version("nonexistent") is None
|
||||
|
||||
def test_verify_checksum(self):
|
||||
"""Test checksum verification."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
self.sample_graph,
|
||||
"test_v1.0",
|
||||
"test@example.com",
|
||||
"Test snapshot"
|
||||
)
|
||||
|
||||
# Valid checksum
|
||||
assert manager.verify_checksum(snapshot) is True
|
||||
|
||||
# Invalid checksum
|
||||
snapshot["checksum"] = "invalid_checksum"
|
||||
assert manager.verify_checksum(snapshot) is False
|
||||
|
||||
def test_compare_versions_detailed(self):
|
||||
"""Test detailed version comparison."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create first version
|
||||
graph_v1 = {
|
||||
"entities": [
|
||||
{"id": "entity1", "name": "Entity 1", "type": "Person"},
|
||||
{"id": "entity2", "name": "Entity 2", "type": "Organization"}
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "entity1", "target": "entity2", "type": "works_for"}
|
||||
]
|
||||
}
|
||||
|
||||
manager.create_snapshot(graph_v1, "v1.0", "test@example.com", "Version 1")
|
||||
|
||||
# Create second version with changes
|
||||
graph_v2 = {
|
||||
"entities": [
|
||||
{"id": "entity1", "name": "Entity 1 Updated", "type": "Person"}, # Modified
|
||||
{"id": "entity3", "name": "Entity 3", "type": "Project"} # Added
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "entity1", "target": "entity3", "type": "manages"} # Added
|
||||
]
|
||||
}
|
||||
|
||||
manager.create_snapshot(graph_v2, "v2.0", "test@example.com", "Version 2")
|
||||
|
||||
# Compare versions
|
||||
diff = manager.compare_versions("v1.0", "v2.0")
|
||||
|
||||
assert diff["version1"] == "v1.0"
|
||||
assert diff["version2"] == "v2.0"
|
||||
assert diff["summary"]["entities_added"] == 1
|
||||
assert diff["summary"]["entities_removed"] == 1
|
||||
assert diff["summary"]["entities_modified"] == 1
|
||||
assert diff["summary"]["relationships_added"] == 1
|
||||
assert diff["summary"]["relationships_removed"] == 1
|
||||
|
||||
# Check detailed changes
|
||||
assert len(diff["entities_added"]) == 1
|
||||
assert diff["entities_added"][0]["id"] == "entity3"
|
||||
|
||||
assert len(diff["entities_modified"]) == 1
|
||||
assert diff["entities_modified"][0]["id"] == "entity1"
|
||||
assert diff["entities_modified"][0]["changes"]["name"]["from"] == "Entity 1"
|
||||
assert diff["entities_modified"][0]["changes"]["name"]["to"] == "Entity 1 Updated"
|
||||
|
||||
|
||||
class TestOntologyVersionManager:
|
||||
"""Test cases for OntologyVersionManager."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.sample_ontology = {
|
||||
"uri": "https://example.com/ontology",
|
||||
"version_info": {"version": "1.0", "date": "2024-01-15"},
|
||||
"structure": {
|
||||
"classes": ["Person", "Organization"],
|
||||
"properties": ["name", "email"],
|
||||
"individuals": ["john_doe", "acme_corp"],
|
||||
"axioms": ["Person hasName exactly 1 string"]
|
||||
}
|
||||
}
|
||||
|
||||
def test_initialization(self):
|
||||
"""Test initialization."""
|
||||
manager = OntologyVersionManager()
|
||||
assert manager.storage is not None
|
||||
assert manager.logger is not None
|
||||
assert manager.versions == {}
|
||||
|
||||
def test_create_snapshot(self):
|
||||
"""Test ontology snapshot creation."""
|
||||
manager = OntologyVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
self.sample_ontology,
|
||||
"ont_v1.0",
|
||||
"test@example.com",
|
||||
"Initial ontology version"
|
||||
)
|
||||
|
||||
assert snapshot["label"] == "ont_v1.0"
|
||||
assert snapshot["author"] == "test@example.com"
|
||||
assert snapshot["ontology_iri"] == "https://example.com/ontology"
|
||||
assert "checksum" in snapshot
|
||||
assert "timestamp" in snapshot
|
||||
assert snapshot["structure"]["classes"] == ["Person", "Organization"]
|
||||
|
||||
def test_compare_versions_structural(self):
|
||||
"""Test structural comparison between ontology versions."""
|
||||
manager = OntologyVersionManager()
|
||||
|
||||
# Create first version
|
||||
ontology_v1 = {
|
||||
"uri": "https://example.com/ontology",
|
||||
"structure": {
|
||||
"classes": ["Person", "Organization"],
|
||||
"properties": ["name", "email"],
|
||||
"individuals": ["john_doe"],
|
||||
"axioms": ["Person hasName exactly 1 string"]
|
||||
}
|
||||
}
|
||||
|
||||
manager.create_snapshot(ontology_v1, "v1.0", "test@example.com", "Version 1")
|
||||
|
||||
# Create second version with structural changes
|
||||
ontology_v2 = {
|
||||
"uri": "https://example.com/ontology",
|
||||
"structure": {
|
||||
"classes": ["Person", "Organization", "Project"], # Added Project
|
||||
"properties": ["name", "email", "description"], # Added description
|
||||
"individuals": ["john_doe", "acme_corp"], # Added acme_corp
|
||||
"axioms": [
|
||||
"Person hasName exactly 1 string",
|
||||
"Project hasDescription some string" # Added axiom
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
manager.create_snapshot(ontology_v2, "v2.0", "test@example.com", "Version 2")
|
||||
|
||||
# Compare versions
|
||||
diff = manager.compare_versions("v1.0", "v2.0")
|
||||
|
||||
assert diff["version1"] == "v1.0"
|
||||
assert diff["version2"] == "v2.0"
|
||||
|
||||
# Check structural changes
|
||||
assert "Project" in diff["classes_added"]
|
||||
assert "description" in diff["properties_added"]
|
||||
assert "acme_corp" in diff["individuals_added"]
|
||||
assert "Project hasDescription some string" in diff["axioms_added"]
|
||||
|
||||
# Check summary counts
|
||||
assert diff["summary"]["classes_added"] == 1
|
||||
assert diff["summary"]["properties_added"] == 1
|
||||
assert diff["summary"]["individuals_added"] == 1
|
||||
assert diff["summary"]["axioms_added"] == 1
|
||||
|
||||
def test_compare_versions_nonexistent(self):
|
||||
"""Test comparison with nonexistent version."""
|
||||
manager = OntologyVersionManager()
|
||||
|
||||
with pytest.raises(ValidationError, match="Version not found"):
|
||||
manager.compare_versions("nonexistent1", "nonexistent2")
|
||||
|
||||
def test_persistence_across_instances(self):
|
||||
"""Test that data persists across manager instances."""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
# Create snapshot with first instance
|
||||
manager1 = OntologyVersionManager(storage_path=db_path)
|
||||
manager1.create_snapshot(
|
||||
self.sample_ontology,
|
||||
"persistent_v1.0",
|
||||
"test@example.com",
|
||||
"Persistent test"
|
||||
)
|
||||
|
||||
# Retrieve with second instance
|
||||
manager2 = OntologyVersionManager(storage_path=db_path)
|
||||
retrieved = manager2.get_version("persistent_v1.0")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "persistent_v1.0"
|
||||
assert retrieved["ontology_iri"] == "https://example.com/ontology"
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
@@ -0,0 +1,619 @@
|
||||
"""
|
||||
Performance and Latency Tests for Enhanced Change Management Module
|
||||
|
||||
This module provides comprehensive performance testing for all change management
|
||||
components including storage backends, version managers, and diff algorithms.
|
||||
"""
|
||||
|
||||
import time
|
||||
import tempfile
|
||||
import os
|
||||
import threading
|
||||
import psutil
|
||||
import statistics
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from typing import List, Dict, Any, Tuple
|
||||
|
||||
import pytest
|
||||
from semantica.change_management import (
|
||||
TemporalVersionManager,
|
||||
OntologyVersionManager,
|
||||
InMemoryVersionStorage,
|
||||
SQLiteVersionStorage,
|
||||
ChangeLogEntry,
|
||||
compute_checksum,
|
||||
verify_checksum
|
||||
)
|
||||
|
||||
|
||||
class PerformanceTestSuite:
|
||||
"""Comprehensive performance test suite for change management module."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize performance test suite."""
|
||||
self.results = {}
|
||||
self.process = psutil.Process()
|
||||
|
||||
def measure_time(self, func, *args, **kwargs) -> Tuple[Any, float]:
|
||||
"""Measure execution time of a function."""
|
||||
start_time = time.perf_counter()
|
||||
result = func(*args, **kwargs)
|
||||
end_time = time.perf_counter()
|
||||
return result, end_time - start_time
|
||||
|
||||
def measure_memory(self, func, *args, **kwargs) -> Tuple[Any, float]:
|
||||
"""Measure memory usage of a function."""
|
||||
initial_memory = self.process.memory_info().rss / 1024 / 1024 # MB
|
||||
result = func(*args, **kwargs)
|
||||
final_memory = self.process.memory_info().rss / 1024 / 1024 # MB
|
||||
return result, final_memory - initial_memory
|
||||
|
||||
def generate_test_graph(self, num_entities: int, num_relationships: int) -> Dict[str, Any]:
|
||||
"""Generate test knowledge graph with specified size."""
|
||||
entities = []
|
||||
for i in range(num_entities):
|
||||
entities.append({
|
||||
"id": f"entity_{i}",
|
||||
"name": f"Entity {i}",
|
||||
"type": f"Type_{i % 10}",
|
||||
"description": f"Description for entity {i}" * 5, # Make it longer
|
||||
"properties": {
|
||||
"category": f"Category_{i % 5}",
|
||||
"score": i * 0.1,
|
||||
"active": i % 2 == 0
|
||||
}
|
||||
})
|
||||
|
||||
relationships = []
|
||||
for i in range(num_relationships):
|
||||
source_idx = i % num_entities
|
||||
target_idx = (i + 1) % num_entities
|
||||
relationships.append({
|
||||
"source": f"entity_{source_idx}",
|
||||
"target": f"entity_{target_idx}",
|
||||
"type": f"relation_type_{i % 5}",
|
||||
"weight": i * 0.01,
|
||||
"properties": {
|
||||
"strength": i % 10,
|
||||
"confidence": 0.8 + (i % 20) * 0.01
|
||||
}
|
||||
})
|
||||
|
||||
return {
|
||||
"entities": entities,
|
||||
"relationships": relationships
|
||||
}
|
||||
|
||||
def generate_test_ontology(self, num_classes: int, num_properties: int) -> Dict[str, Any]:
|
||||
"""Generate test ontology with specified size."""
|
||||
classes = [f"Class_{i}" for i in range(num_classes)]
|
||||
properties = [f"property_{i}" for i in range(num_properties)]
|
||||
individuals = [f"individual_{i}" for i in range(num_classes // 2)]
|
||||
axioms = [f"Class_{i} hasProperty property_{i % num_properties}" for i in range(num_classes)]
|
||||
|
||||
return {
|
||||
"uri": "https://test.com/ontology",
|
||||
"version_info": {"version": "1.0", "date": "2024-01-30"},
|
||||
"structure": {
|
||||
"classes": classes,
|
||||
"properties": properties,
|
||||
"individuals": individuals,
|
||||
"axioms": axioms
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class TestStoragePerformance:
|
||||
"""Test performance of storage backends."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.perf_suite = PerformanceTestSuite()
|
||||
self.test_sizes = [10, 50, 100, 500, 1000]
|
||||
|
||||
def test_inmemory_storage_performance(self):
|
||||
"""Test InMemoryVersionStorage performance across different data sizes."""
|
||||
print("\n=== InMemoryVersionStorage Performance ===")
|
||||
|
||||
storage = InMemoryVersionStorage()
|
||||
results = {}
|
||||
|
||||
for size in self.test_sizes:
|
||||
graph = self.perf_suite.generate_test_graph(size, size * 2)
|
||||
snapshot = {
|
||||
"label": f"test_v{size}",
|
||||
"timestamp": "2024-01-30T12:00:00Z",
|
||||
"author": "test@example.com",
|
||||
"description": f"Test snapshot with {size} entities",
|
||||
"entities": graph["entities"],
|
||||
"relationships": graph["relationships"],
|
||||
"checksum": compute_checksum(graph)
|
||||
}
|
||||
|
||||
# Test save performance
|
||||
_, save_time = self.perf_suite.measure_time(storage.save, snapshot)
|
||||
|
||||
# Test get performance
|
||||
_, get_time = self.perf_suite.measure_time(storage.get, f"test_v{size}")
|
||||
|
||||
# Test list performance
|
||||
_, list_time = self.perf_suite.measure_time(storage.list_all)
|
||||
|
||||
results[size] = {
|
||||
"save_time": save_time,
|
||||
"get_time": get_time,
|
||||
"list_time": list_time
|
||||
}
|
||||
|
||||
print(f"Size {size:4d}: Save={save_time*1000:6.2f}ms, Get={get_time*1000:6.2f}ms, List={list_time*1000:6.2f}ms")
|
||||
|
||||
# Verify performance requirements
|
||||
assert results[1000]["save_time"] < 0.1, "Large snapshot save should be under 100ms"
|
||||
assert results[1000]["get_time"] < 0.05, "Large snapshot retrieval should be under 50ms"
|
||||
|
||||
self.perf_suite.results["inmemory_storage"] = results
|
||||
|
||||
def test_sqlite_storage_performance(self):
|
||||
"""Test SQLiteVersionStorage performance across different data sizes."""
|
||||
print("\n=== SQLiteVersionStorage Performance ===")
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
storage = SQLiteVersionStorage(db_path)
|
||||
results = {}
|
||||
|
||||
for size in self.test_sizes:
|
||||
graph = self.perf_suite.generate_test_graph(size, size * 2)
|
||||
snapshot = {
|
||||
"label": f"test_v{size}",
|
||||
"timestamp": "2024-01-30T12:00:00Z",
|
||||
"author": "test@example.com",
|
||||
"description": f"Test snapshot with {size} entities",
|
||||
"entities": graph["entities"],
|
||||
"relationships": graph["relationships"],
|
||||
"checksum": compute_checksum(graph)
|
||||
}
|
||||
|
||||
# Test save performance
|
||||
_, save_time = self.perf_suite.measure_time(storage.save, snapshot)
|
||||
|
||||
# Test get performance
|
||||
_, get_time = self.perf_suite.measure_time(storage.get, f"test_v{size}")
|
||||
|
||||
# Test list performance
|
||||
_, list_time = self.perf_suite.measure_time(storage.list_all)
|
||||
|
||||
results[size] = {
|
||||
"save_time": save_time,
|
||||
"get_time": get_time,
|
||||
"list_time": list_time
|
||||
}
|
||||
|
||||
print(f"Size {size:4d}: Save={save_time*1000:6.2f}ms, Get={get_time*1000:6.2f}ms, List={list_time*1000:6.2f}ms")
|
||||
|
||||
# Verify performance requirements
|
||||
assert results[1000]["save_time"] < 0.5, "Large snapshot save should be under 500ms"
|
||||
assert results[1000]["get_time"] < 0.1, "Large snapshot retrieval should be under 100ms"
|
||||
|
||||
self.perf_suite.results["sqlite_storage"] = results
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_storage_comparison(self):
|
||||
"""Compare performance between InMemory and SQLite storage."""
|
||||
print("\n=== Storage Backend Comparison ===")
|
||||
|
||||
# Test with medium-sized dataset
|
||||
test_size = 500
|
||||
graph = self.perf_suite.generate_test_graph(test_size, test_size * 2)
|
||||
snapshot = {
|
||||
"label": f"comparison_test",
|
||||
"timestamp": "2024-01-30T12:00:00Z",
|
||||
"author": "test@example.com",
|
||||
"description": f"Comparison test with {test_size} entities",
|
||||
"entities": graph["entities"],
|
||||
"relationships": graph["relationships"],
|
||||
"checksum": compute_checksum(graph)
|
||||
}
|
||||
|
||||
# InMemory performance
|
||||
inmemory_storage = InMemoryVersionStorage()
|
||||
_, inmemory_save = self.perf_suite.measure_time(inmemory_storage.save, snapshot)
|
||||
_, inmemory_get = self.perf_suite.measure_time(inmemory_storage.get, "comparison_test")
|
||||
|
||||
# SQLite performance
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
sqlite_storage = SQLiteVersionStorage(db_path)
|
||||
_, sqlite_save = self.perf_suite.measure_time(sqlite_storage.save, snapshot)
|
||||
_, sqlite_get = self.perf_suite.measure_time(sqlite_storage.get, "comparison_test")
|
||||
|
||||
print(f"InMemory: Save={inmemory_save*1000:6.2f}ms, Get={inmemory_get*1000:6.2f}ms")
|
||||
print(f"SQLite: Save={sqlite_save*1000:6.2f}ms, Get={sqlite_get*1000:6.2f}ms")
|
||||
print(f"SQLite overhead: Save={sqlite_save/inmemory_save:.1f}x, Get={sqlite_get/inmemory_get:.1f}x")
|
||||
|
||||
# SQLite should be reasonably close to InMemory for typical use cases
|
||||
assert sqlite_save < inmemory_save * 10, "SQLite save shouldn't be more than 10x slower"
|
||||
assert sqlite_get < inmemory_get * 5, "SQLite get shouldn't be more than 5x slower"
|
||||
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
|
||||
class TestVersionManagerPerformance:
|
||||
"""Test performance of enhanced version managers."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.perf_suite = PerformanceTestSuite()
|
||||
self.test_sizes = [50, 100, 500, 1000, 2000]
|
||||
|
||||
def test_temporal_version_manager_performance(self):
|
||||
"""Test TemporalVersionManager performance."""
|
||||
print("\n=== TemporalVersionManager Performance ===")
|
||||
|
||||
manager = TemporalVersionManager()
|
||||
results = {}
|
||||
|
||||
for size in self.test_sizes:
|
||||
graph = self.perf_suite.generate_test_graph(size, size * 2)
|
||||
|
||||
# Test snapshot creation performance
|
||||
_, create_time = self.perf_suite.measure_time(
|
||||
manager.create_snapshot,
|
||||
graph,
|
||||
f"perf_test_v{size}",
|
||||
"test@example.com",
|
||||
f"Performance test with {size} entities"
|
||||
)
|
||||
|
||||
# Test version listing performance
|
||||
_, list_time = self.perf_suite.measure_time(manager.list_versions)
|
||||
|
||||
# Test version retrieval performance
|
||||
_, get_time = self.perf_suite.measure_time(manager.get_version, f"perf_test_v{size}")
|
||||
|
||||
results[size] = {
|
||||
"create_time": create_time,
|
||||
"list_time": list_time,
|
||||
"get_time": get_time
|
||||
}
|
||||
|
||||
print(f"Size {size:4d}: Create={create_time*1000:6.2f}ms, List={list_time*1000:6.2f}ms, Get={get_time*1000:6.2f}ms")
|
||||
|
||||
# Verify performance requirements (as specified in original requirements)
|
||||
assert results[2000]["create_time"] < 0.5, "Large snapshot creation should be under 500ms"
|
||||
|
||||
self.perf_suite.results["temporal_manager"] = results
|
||||
|
||||
def test_version_comparison_performance(self):
|
||||
"""Test version comparison performance with different graph sizes."""
|
||||
print("\n=== Version Comparison Performance ===")
|
||||
|
||||
manager = TemporalVersionManager()
|
||||
results = {}
|
||||
|
||||
for size in [100, 500, 1000]:
|
||||
# Create two similar graphs with some differences
|
||||
graph1 = self.perf_suite.generate_test_graph(size, size * 2)
|
||||
graph2 = self.perf_suite.generate_test_graph(size + 10, size * 2 + 20) # Slightly different
|
||||
|
||||
# Create snapshots
|
||||
manager.create_snapshot(graph1, f"v1_{size}", "test@example.com", "Version 1")
|
||||
manager.create_snapshot(graph2, f"v2_{size}", "test@example.com", "Version 2")
|
||||
|
||||
# Test comparison performance
|
||||
_, compare_time = self.perf_suite.measure_time(
|
||||
manager.compare_versions, f"v1_{size}", f"v2_{size}"
|
||||
)
|
||||
|
||||
results[size] = {"compare_time": compare_time}
|
||||
print(f"Size {size:4d}: Compare={compare_time*1000:6.2f}ms")
|
||||
|
||||
# Verify comparison performance
|
||||
assert results[1000]["compare_time"] < 1.0, "Large graph comparison should be under 1 second"
|
||||
|
||||
self.perf_suite.results["version_comparison"] = results
|
||||
|
||||
def test_ontology_version_manager_performance(self):
|
||||
"""Test OntologyVersionManager performance with ontologies."""
|
||||
print("\n=== OntologyVersionManager Performance ===")
|
||||
|
||||
manager = OntologyVersionManager()
|
||||
results = {}
|
||||
|
||||
ontology_sizes = [50, 100, 500, 1000]
|
||||
|
||||
for size in ontology_sizes:
|
||||
ontology = self.perf_suite.generate_test_ontology(size, size // 2)
|
||||
|
||||
# Test ontology snapshot creation
|
||||
_, create_time = self.perf_suite.measure_time(
|
||||
manager.create_snapshot,
|
||||
ontology,
|
||||
f"ont_v{size}",
|
||||
"test@example.com",
|
||||
f"Ontology with {size} classes"
|
||||
)
|
||||
|
||||
results[size] = {"create_time": create_time}
|
||||
print(f"Classes {size:4d}: Create={create_time*1000:6.2f}ms")
|
||||
|
||||
# Test structural comparison
|
||||
ont1 = self.perf_suite.generate_test_ontology(500, 250)
|
||||
ont2 = self.perf_suite.generate_test_ontology(520, 260) # Slightly different
|
||||
|
||||
manager.create_snapshot(ont1, "ont_comp_1", "test@example.com", "Ontology 1")
|
||||
manager.create_snapshot(ont2, "ont_comp_2", "test@example.com", "Ontology 2")
|
||||
|
||||
_, compare_time = self.perf_suite.measure_time(
|
||||
manager.compare_versions, "ont_comp_1", "ont_comp_2"
|
||||
)
|
||||
|
||||
print(f"Ontology comparison: {compare_time*1000:6.2f}ms")
|
||||
|
||||
self.perf_suite.results["ontology_manager"] = results
|
||||
|
||||
|
||||
class TestChecksumPerformance:
|
||||
"""Test checksum computation and verification performance."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.perf_suite = PerformanceTestSuite()
|
||||
|
||||
def test_checksum_performance(self):
|
||||
"""Test checksum computation performance across different data sizes."""
|
||||
print("\n=== Checksum Performance ===")
|
||||
|
||||
results = {}
|
||||
|
||||
for size in [100, 500, 1000, 5000, 10000]:
|
||||
graph = self.perf_suite.generate_test_graph(size, size * 2)
|
||||
|
||||
# Test checksum computation
|
||||
_, compute_time = self.perf_suite.measure_time(compute_checksum, graph)
|
||||
|
||||
# Test checksum verification
|
||||
checksum = compute_checksum(graph)
|
||||
graph_with_checksum = graph.copy()
|
||||
graph_with_checksum["checksum"] = checksum
|
||||
|
||||
_, verify_time = self.perf_suite.measure_time(verify_checksum, graph_with_checksum)
|
||||
|
||||
results[size] = {
|
||||
"compute_time": compute_time,
|
||||
"verify_time": verify_time
|
||||
}
|
||||
|
||||
print(f"Size {size:5d}: Compute={compute_time*1000:6.2f}ms, Verify={verify_time*1000:6.2f}ms")
|
||||
|
||||
# Verify checksum performance requirements
|
||||
assert results[10000]["compute_time"] < 0.5, "Large checksum computation should be under 500ms"
|
||||
assert results[10000]["verify_time"] < 0.5, "Large checksum verification should be under 500ms"
|
||||
|
||||
self.perf_suite.results["checksum"] = results
|
||||
|
||||
|
||||
class TestConcurrencyPerformance:
|
||||
"""Test concurrent operations and thread safety."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.perf_suite = PerformanceTestSuite()
|
||||
|
||||
def test_concurrent_storage_operations(self):
|
||||
"""Test concurrent storage operations."""
|
||||
print("\n=== Concurrent Storage Operations ===")
|
||||
|
||||
storage = InMemoryVersionStorage()
|
||||
num_threads = 10
|
||||
operations_per_thread = 50
|
||||
|
||||
def worker_function(thread_id: int):
|
||||
"""Worker function for concurrent testing."""
|
||||
times = []
|
||||
for i in range(operations_per_thread):
|
||||
graph = self.perf_suite.generate_test_graph(50, 100)
|
||||
snapshot = {
|
||||
"label": f"thread_{thread_id}_snapshot_{i}",
|
||||
"timestamp": "2024-01-30T12:00:00Z",
|
||||
"author": f"thread_{thread_id}@example.com",
|
||||
"description": f"Concurrent test snapshot {i}",
|
||||
"entities": graph["entities"],
|
||||
"relationships": graph["relationships"],
|
||||
"checksum": compute_checksum(graph)
|
||||
}
|
||||
|
||||
start_time = time.perf_counter()
|
||||
storage.save(snapshot)
|
||||
end_time = time.perf_counter()
|
||||
times.append(end_time - start_time)
|
||||
|
||||
return times
|
||||
|
||||
# Run concurrent operations
|
||||
start_time = time.perf_counter()
|
||||
|
||||
with ThreadPoolExecutor(max_workers=num_threads) as executor:
|
||||
futures = [executor.submit(worker_function, i) for i in range(num_threads)]
|
||||
all_times = []
|
||||
|
||||
for future in as_completed(futures):
|
||||
thread_times = future.result()
|
||||
all_times.extend(thread_times)
|
||||
|
||||
end_time = time.perf_counter()
|
||||
total_time = end_time - start_time
|
||||
|
||||
# Calculate statistics
|
||||
avg_operation_time = statistics.mean(all_times)
|
||||
max_operation_time = max(all_times)
|
||||
total_operations = num_threads * operations_per_thread
|
||||
|
||||
print(f"Total operations: {total_operations}")
|
||||
print(f"Total time: {total_time:.2f}s")
|
||||
print(f"Operations per second: {total_operations/total_time:.1f}")
|
||||
print(f"Average operation time: {avg_operation_time*1000:.2f}ms")
|
||||
print(f"Max operation time: {max_operation_time*1000:.2f}ms")
|
||||
|
||||
# Verify concurrent performance
|
||||
assert avg_operation_time < 0.1, "Average concurrent operation should be under 100ms"
|
||||
assert total_operations/total_time > 50, "Should handle at least 50 operations per second"
|
||||
|
||||
def test_concurrent_version_manager_operations(self):
|
||||
"""Test concurrent version manager operations."""
|
||||
print("\n=== Concurrent Version Manager Operations ===")
|
||||
|
||||
manager = TemporalVersionManager()
|
||||
num_threads = 5
|
||||
snapshots_per_thread = 20
|
||||
|
||||
def create_snapshots(thread_id: int):
|
||||
"""Create snapshots concurrently."""
|
||||
times = []
|
||||
for i in range(snapshots_per_thread):
|
||||
graph = self.perf_suite.generate_test_graph(100, 200)
|
||||
|
||||
start_time = time.perf_counter()
|
||||
manager.create_snapshot(
|
||||
graph,
|
||||
f"concurrent_t{thread_id}_s{i}",
|
||||
f"thread{thread_id}@example.com",
|
||||
f"Concurrent snapshot {i}"
|
||||
)
|
||||
end_time = time.perf_counter()
|
||||
times.append(end_time - start_time)
|
||||
|
||||
return times
|
||||
|
||||
# Run concurrent snapshot creation
|
||||
start_time = time.perf_counter()
|
||||
|
||||
with ThreadPoolExecutor(max_workers=num_threads) as executor:
|
||||
futures = [executor.submit(create_snapshots, i) for i in range(num_threads)]
|
||||
all_times = []
|
||||
|
||||
for future in as_completed(futures):
|
||||
thread_times = future.result()
|
||||
all_times.extend(thread_times)
|
||||
|
||||
end_time = time.perf_counter()
|
||||
total_time = end_time - start_time
|
||||
|
||||
# Verify all snapshots were created
|
||||
versions = manager.list_versions()
|
||||
expected_count = num_threads * snapshots_per_thread
|
||||
|
||||
print(f"Created {len(versions)} snapshots in {total_time:.2f}s")
|
||||
print(f"Average creation time: {statistics.mean(all_times)*1000:.2f}ms")
|
||||
|
||||
assert len(versions) == expected_count, f"Expected {expected_count} snapshots, got {len(versions)}"
|
||||
|
||||
|
||||
class TestMemoryUsage:
|
||||
"""Test memory usage and resource consumption."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.perf_suite = PerformanceTestSuite()
|
||||
|
||||
def test_memory_usage_scaling(self):
|
||||
"""Test memory usage scaling with data size."""
|
||||
print("\n=== Memory Usage Scaling ===")
|
||||
|
||||
manager = TemporalVersionManager()
|
||||
initial_memory = self.perf_suite.process.memory_info().rss / 1024 / 1024 # MB
|
||||
|
||||
memory_usage = {}
|
||||
|
||||
for size in [100, 500, 1000, 2000]:
|
||||
graph = self.perf_suite.generate_test_graph(size, size * 2)
|
||||
|
||||
# Create snapshot and measure memory
|
||||
manager.create_snapshot(
|
||||
graph,
|
||||
f"memory_test_{size}",
|
||||
"test@example.com",
|
||||
f"Memory test with {size} entities"
|
||||
)
|
||||
|
||||
current_memory = self.perf_suite.process.memory_info().rss / 1024 / 1024 # MB
|
||||
memory_used = current_memory - initial_memory
|
||||
memory_usage[size] = memory_used
|
||||
|
||||
print(f"Size {size:4d}: Memory used: {memory_used:.1f}MB")
|
||||
|
||||
# Verify memory usage is reasonable
|
||||
memory_per_entity = memory_usage[2000] / 2000
|
||||
print(f"Memory per entity: {memory_per_entity*1024:.2f}KB")
|
||||
|
||||
# Should use less than 1MB per 1000 entities for reasonable efficiency
|
||||
assert memory_usage[1000] < 50, "Memory usage should be reasonable for large datasets"
|
||||
|
||||
|
||||
def run_comprehensive_performance_tests():
|
||||
"""Run all performance tests and generate summary report."""
|
||||
print("=" * 80)
|
||||
print("COMPREHENSIVE CHANGE MANAGEMENT PERFORMANCE TEST SUITE")
|
||||
print("=" * 80)
|
||||
|
||||
# Initialize test classes
|
||||
storage_tests = TestStoragePerformance()
|
||||
storage_tests.setup_method()
|
||||
|
||||
manager_tests = TestVersionManagerPerformance()
|
||||
manager_tests.setup_method()
|
||||
|
||||
checksum_tests = TestChecksumPerformance()
|
||||
checksum_tests.setup_method()
|
||||
|
||||
concurrency_tests = TestConcurrencyPerformance()
|
||||
concurrency_tests.setup_method()
|
||||
|
||||
memory_tests = TestMemoryUsage()
|
||||
memory_tests.setup_method()
|
||||
|
||||
# Run all tests
|
||||
try:
|
||||
# Storage performance tests
|
||||
storage_tests.test_inmemory_storage_performance()
|
||||
storage_tests.test_sqlite_storage_performance()
|
||||
storage_tests.test_storage_comparison()
|
||||
|
||||
# Version manager performance tests
|
||||
manager_tests.test_temporal_version_manager_performance()
|
||||
manager_tests.test_version_comparison_performance()
|
||||
manager_tests.test_ontology_version_manager_performance()
|
||||
|
||||
# Checksum performance tests
|
||||
checksum_tests.test_checksum_performance()
|
||||
|
||||
# Concurrency tests
|
||||
concurrency_tests.test_concurrent_storage_operations()
|
||||
concurrency_tests.test_concurrent_version_manager_operations()
|
||||
|
||||
# Memory usage tests
|
||||
memory_tests.test_memory_usage_scaling()
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("ALL PERFORMANCE TESTS COMPLETED SUCCESSFULLY!")
|
||||
print("=" * 80)
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nPERFORMANCE TEST FAILED: {e}")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = run_comprehensive_performance_tests()
|
||||
exit(0 if success else 1)
|
||||
@@ -0,0 +1,373 @@
|
||||
"""
|
||||
Tests for enhanced TemporalVersionManager with persistent storage.
|
||||
|
||||
This module tests the comprehensive version management capabilities for knowledge
|
||||
graphs, including persistent storage, detailed change tracking, and audit trails.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from semantica.kg.temporal_query import TemporalVersionManager
|
||||
from semantica.change_management import ChangeLogEntry, InMemoryVersionStorage, SQLiteVersionStorage
|
||||
from semantica.utils.exceptions import ValidationError, ProcessingError
|
||||
|
||||
|
||||
class TestTemporalVersionManager:
|
||||
"""Test cases for enhanced TemporalVersionManager."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.sample_graph = {
|
||||
"entities": [
|
||||
{"id": "1", "name": "Entity1", "type": "Person"},
|
||||
{"id": "2", "name": "Entity2", "type": "Organization"}
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "1", "target": "2", "type": "works_for"}
|
||||
]
|
||||
}
|
||||
|
||||
self.modified_graph = {
|
||||
"entities": [
|
||||
{"id": "1", "name": "Entity1 Modified", "type": "Person"},
|
||||
{"id": "2", "name": "Entity2", "type": "Organization"},
|
||||
{"id": "3", "name": "Entity3", "type": "Product"}
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "1", "target": "2", "type": "works_for"},
|
||||
{"source": "2", "target": "3", "type": "produces"}
|
||||
]
|
||||
}
|
||||
|
||||
def test_in_memory_initialization(self):
|
||||
"""Test initialization with in-memory storage."""
|
||||
manager = TemporalVersionManager()
|
||||
assert manager.storage is not None
|
||||
assert manager.version_strategy == "timestamp"
|
||||
|
||||
def test_sqlite_initialization(self):
|
||||
"""Test initialization with SQLite storage."""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
manager = TemporalVersionManager(storage_path=db_path)
|
||||
assert manager.storage is not None
|
||||
assert os.path.exists(db_path)
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_create_snapshot_basic(self):
|
||||
"""Test creating a basic snapshot."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
graph=self.sample_graph,
|
||||
version_label="v1.0",
|
||||
author="alice@company.com",
|
||||
description="Initial version"
|
||||
)
|
||||
|
||||
assert snapshot["label"] == "v1.0"
|
||||
assert snapshot["author"] == "alice@company.com"
|
||||
assert snapshot["description"] == "Initial version"
|
||||
assert "checksum" in snapshot
|
||||
assert len(snapshot["entities"]) == 2
|
||||
assert len(snapshot["relationships"]) == 1
|
||||
|
||||
def test_create_snapshot_with_invalid_author(self):
|
||||
"""Test that invalid author email raises ValidationError."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
with pytest.raises(ValidationError, match="Invalid email format"):
|
||||
manager.create_snapshot(
|
||||
graph=self.sample_graph,
|
||||
version_label="v1.0",
|
||||
author="invalid-email",
|
||||
description="Test version"
|
||||
)
|
||||
|
||||
def test_create_snapshot_with_long_description(self):
|
||||
"""Test that description over 500 chars raises ValidationError."""
|
||||
manager = TemporalVersionManager()
|
||||
long_description = "x" * 501
|
||||
|
||||
with pytest.raises(ValidationError, match="Description too long"):
|
||||
manager.create_snapshot(
|
||||
graph=self.sample_graph,
|
||||
version_label="v1.0",
|
||||
author="alice@company.com",
|
||||
description=long_description
|
||||
)
|
||||
|
||||
def test_list_versions_empty(self):
|
||||
"""Test listing versions from empty storage."""
|
||||
manager = TemporalVersionManager()
|
||||
versions = manager.list_versions()
|
||||
assert versions == []
|
||||
|
||||
def test_list_versions_with_data(self):
|
||||
"""Test listing versions with data."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create multiple snapshots
|
||||
manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
manager.create_snapshot(
|
||||
self.modified_graph, "v2.0", "bob@company.com", "Version 2"
|
||||
)
|
||||
|
||||
versions = manager.list_versions()
|
||||
assert len(versions) == 2
|
||||
|
||||
labels = [v["label"] for v in versions]
|
||||
assert "v1.0" in labels
|
||||
assert "v2.0" in labels
|
||||
|
||||
def test_get_version_existing(self):
|
||||
"""Test retrieving an existing version."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create snapshot
|
||||
manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
|
||||
# Retrieve it
|
||||
retrieved = manager.get_version("v1.0")
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "v1.0"
|
||||
assert retrieved["author"] == "alice@company.com"
|
||||
|
||||
def test_get_version_nonexistent(self):
|
||||
"""Test retrieving a nonexistent version."""
|
||||
manager = TemporalVersionManager()
|
||||
retrieved = manager.get_version("nonexistent")
|
||||
assert retrieved is None
|
||||
|
||||
def test_verify_checksum_valid(self):
|
||||
"""Test verifying a valid checksum."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
|
||||
assert manager.verify_checksum(snapshot) is True
|
||||
|
||||
def test_verify_checksum_invalid(self):
|
||||
"""Test verifying an invalid checksum."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
|
||||
# Corrupt the checksum
|
||||
snapshot["checksum"] = "invalid_checksum"
|
||||
assert manager.verify_checksum(snapshot) is False
|
||||
|
||||
def test_compare_versions_with_labels(self):
|
||||
"""Test comparing versions using labels."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create two versions
|
||||
manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
manager.create_snapshot(
|
||||
self.modified_graph, "v2.0", "bob@company.com", "Version 2"
|
||||
)
|
||||
|
||||
# Compare them
|
||||
diff = manager.compare_versions("v1.0", "v2.0")
|
||||
|
||||
assert diff["version1"] == "v1.0"
|
||||
assert diff["version2"] == "v2.0"
|
||||
assert "summary" in diff
|
||||
assert "entities_added" in diff
|
||||
assert "entities_removed" in diff
|
||||
assert "entities_modified" in diff
|
||||
|
||||
# Check summary counts
|
||||
summary = diff["summary"]
|
||||
assert summary["entities_added"] == 1 # Entity3 added
|
||||
assert summary["entities_removed"] == 0
|
||||
assert summary["entities_modified"] == 1 # Entity1 modified
|
||||
assert summary["relationships_added"] == 1 # New relationship added
|
||||
|
||||
def test_compare_versions_with_dicts(self):
|
||||
"""Test comparing versions using snapshot dictionaries."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create snapshots but don't store them
|
||||
snapshot1 = {
|
||||
"label": "v1.0",
|
||||
"entities": self.sample_graph["entities"],
|
||||
"relationships": self.sample_graph["relationships"]
|
||||
}
|
||||
|
||||
snapshot2 = {
|
||||
"label": "v2.0",
|
||||
"entities": self.modified_graph["entities"],
|
||||
"relationships": self.modified_graph["relationships"]
|
||||
}
|
||||
|
||||
# Compare directly
|
||||
diff = manager.compare_versions(snapshot1, snapshot2)
|
||||
|
||||
assert diff["version1"] == "v1.0"
|
||||
assert diff["version2"] == "v2.0"
|
||||
assert len(diff["entities_added"]) == 1
|
||||
assert diff["entities_added"][0]["id"] == "3"
|
||||
|
||||
def test_compare_versions_nonexistent_label(self):
|
||||
"""Test comparing with nonexistent version label."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
|
||||
with pytest.raises(ValidationError, match="Version not found: nonexistent"):
|
||||
manager.compare_versions("v1.0", "nonexistent")
|
||||
|
||||
def test_detailed_entity_diff(self):
|
||||
"""Test detailed entity-level differences."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Create versions
|
||||
manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
manager.create_snapshot(
|
||||
self.modified_graph, "v2.0", "bob@company.com", "Version 2"
|
||||
)
|
||||
|
||||
diff = manager.compare_versions("v1.0", "v2.0")
|
||||
|
||||
# Check entities_added
|
||||
assert len(diff["entities_added"]) == 1
|
||||
assert diff["entities_added"][0]["id"] == "3"
|
||||
assert diff["entities_added"][0]["name"] == "Entity3"
|
||||
|
||||
# Check entities_modified
|
||||
assert len(diff["entities_modified"]) == 1
|
||||
modified_entity = diff["entities_modified"][0]
|
||||
assert modified_entity["id"] == "1"
|
||||
assert "changes" in modified_entity
|
||||
assert "name" in modified_entity["changes"]
|
||||
assert modified_entity["changes"]["name"]["from"] == "Entity1"
|
||||
assert modified_entity["changes"]["name"]["to"] == "Entity1 Modified"
|
||||
|
||||
def test_detailed_relationship_diff(self):
|
||||
"""Test detailed relationship-level differences."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
manager.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
manager.create_snapshot(
|
||||
self.modified_graph, "v2.0", "bob@company.com", "Version 2"
|
||||
)
|
||||
|
||||
diff = manager.compare_versions("v1.0", "v2.0")
|
||||
|
||||
# Check relationships_added
|
||||
assert len(diff["relationships_added"]) == 1
|
||||
added_rel = diff["relationships_added"][0]
|
||||
assert added_rel["source"] == "2"
|
||||
assert added_rel["target"] == "3"
|
||||
assert added_rel["type"] == "produces"
|
||||
|
||||
def test_backward_compatibility_create_version(self):
|
||||
"""Test that old create_version method still works."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
# Use old method signature
|
||||
version = manager.create_version(
|
||||
graph=self.sample_graph,
|
||||
version_label="v1.0"
|
||||
)
|
||||
|
||||
assert version["label"] == "v1.0"
|
||||
assert "entities" in version
|
||||
assert "relationships" in version
|
||||
|
||||
def test_persistence_across_instances(self):
|
||||
"""Test that data persists across manager instances."""
|
||||
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
|
||||
db_path = tmp.name
|
||||
|
||||
try:
|
||||
# Create snapshot with first instance
|
||||
manager1 = TemporalVersionManager(storage_path=db_path)
|
||||
manager1.create_snapshot(
|
||||
self.sample_graph, "v1.0", "alice@company.com", "Version 1"
|
||||
)
|
||||
|
||||
# Retrieve with second instance
|
||||
manager2 = TemporalVersionManager(storage_path=db_path)
|
||||
retrieved = manager2.get_version("v1.0")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "v1.0"
|
||||
assert retrieved["author"] == "alice@company.com"
|
||||
finally:
|
||||
if os.path.exists(db_path):
|
||||
os.remove(db_path)
|
||||
|
||||
def test_relationship_key_generation(self):
|
||||
"""Test the relationship key generation method."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
relationship = {
|
||||
"source": "entity1",
|
||||
"target": "entity2",
|
||||
"type": "relates_to"
|
||||
}
|
||||
|
||||
key = manager._relationship_key(relationship)
|
||||
assert key == "entity1|relates_to|entity2"
|
||||
|
||||
def test_relationship_key_with_missing_fields(self):
|
||||
"""Test relationship key generation with missing fields."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
relationship = {"source": "entity1"} # Missing target and type
|
||||
key = manager._relationship_key(relationship)
|
||||
assert key == "entity1||"
|
||||
|
||||
def test_entity_changes_computation(self):
|
||||
"""Test entity changes computation."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
entity1 = {"id": "1", "name": "Original", "type": "Person"}
|
||||
entity2 = {"id": "1", "name": "Modified", "type": "Person", "age": 30}
|
||||
|
||||
changes = manager._compute_entity_changes(entity1, entity2)
|
||||
|
||||
assert "name" in changes
|
||||
assert changes["name"]["from"] == "Original"
|
||||
assert changes["name"]["to"] == "Modified"
|
||||
assert "age" in changes
|
||||
assert changes["age"]["from"] is None
|
||||
assert changes["age"]["to"] == 30
|
||||
|
||||
def test_snapshot_with_metadata(self):
|
||||
"""Test creating snapshot with additional metadata."""
|
||||
manager = TemporalVersionManager()
|
||||
|
||||
snapshot = manager.create_snapshot(
|
||||
graph=self.sample_graph,
|
||||
version_label="v1.0",
|
||||
author="alice@company.com",
|
||||
description="Version with metadata",
|
||||
metadata={"environment": "production", "build": "123"}
|
||||
)
|
||||
|
||||
assert snapshot["metadata"]["environment"] == "production"
|
||||
assert snapshot["metadata"]["build"] == "123"
|
||||
@@ -0,0 +1,321 @@
|
||||
"""
|
||||
Tests for the Version Storage module.
|
||||
|
||||
This module tests the storage abstraction layer and concrete implementations
|
||||
for persistent version management.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
from semantica.change_management import (
|
||||
VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage,
|
||||
compute_checksum, verify_checksum
|
||||
)
|
||||
from semantica.utils.exceptions import ValidationError, ProcessingError
|
||||
|
||||
|
||||
class TestInMemoryVersionStorage:
|
||||
"""Test cases for InMemoryVersionStorage."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.storage = InMemoryVersionStorage()
|
||||
self.sample_snapshot = {
|
||||
"label": "v1.0",
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"author": "alice@company.com",
|
||||
"description": "Initial version",
|
||||
"checksum": "abc123",
|
||||
"entities": [{"id": "1", "name": "Entity1"}],
|
||||
"relationships": [{"source": "1", "target": "2", "type": "relates"}],
|
||||
"metadata": {"version": "1.0"}
|
||||
}
|
||||
|
||||
def test_save_and_get_snapshot(self):
|
||||
"""Test saving and retrieving a snapshot."""
|
||||
self.storage.save(self.sample_snapshot)
|
||||
retrieved = self.storage.get("v1.0")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "v1.0"
|
||||
assert retrieved["author"] == "alice@company.com"
|
||||
assert len(retrieved["entities"]) == 1
|
||||
|
||||
def test_save_snapshot_without_label_raises_error(self):
|
||||
"""Test that saving snapshot without label raises ValidationError."""
|
||||
invalid_snapshot = self.sample_snapshot.copy()
|
||||
del invalid_snapshot["label"]
|
||||
|
||||
with pytest.raises(ValidationError, match="Snapshot must have a 'label' field"):
|
||||
self.storage.save(invalid_snapshot)
|
||||
|
||||
def test_save_duplicate_label_raises_error(self):
|
||||
"""Test that saving duplicate label raises ValidationError."""
|
||||
self.storage.save(self.sample_snapshot)
|
||||
|
||||
with pytest.raises(ValidationError, match="Version 'v1.0' already exists"):
|
||||
self.storage.save(self.sample_snapshot)
|
||||
|
||||
def test_get_nonexistent_snapshot_returns_none(self):
|
||||
"""Test that getting nonexistent snapshot returns None."""
|
||||
result = self.storage.get("nonexistent")
|
||||
assert result is None
|
||||
|
||||
def test_list_all_empty_storage(self):
|
||||
"""Test listing all snapshots from empty storage."""
|
||||
result = self.storage.list_all()
|
||||
assert result == []
|
||||
|
||||
def test_list_all_with_snapshots(self):
|
||||
"""Test listing all snapshots with data."""
|
||||
self.storage.save(self.sample_snapshot)
|
||||
|
||||
snapshot2 = self.sample_snapshot.copy()
|
||||
snapshot2["label"] = "v2.0"
|
||||
self.storage.save(snapshot2)
|
||||
|
||||
result = self.storage.list_all()
|
||||
assert len(result) == 2
|
||||
|
||||
labels = [item["label"] for item in result]
|
||||
assert "v1.0" in labels
|
||||
assert "v2.0" in labels
|
||||
|
||||
# Check metadata structure
|
||||
for item in result:
|
||||
assert "entity_count" in item
|
||||
assert "relationship_count" in item
|
||||
assert item["entity_count"] == 1
|
||||
assert item["relationship_count"] == 1
|
||||
|
||||
def test_exists_method(self):
|
||||
"""Test the exists method."""
|
||||
assert not self.storage.exists("v1.0")
|
||||
|
||||
self.storage.save(self.sample_snapshot)
|
||||
assert self.storage.exists("v1.0")
|
||||
assert not self.storage.exists("v2.0")
|
||||
|
||||
def test_delete_method(self):
|
||||
"""Test the delete method."""
|
||||
# Delete non-existent returns False
|
||||
assert not self.storage.delete("nonexistent")
|
||||
|
||||
# Save and delete existing returns True
|
||||
self.storage.save(self.sample_snapshot)
|
||||
assert self.storage.delete("v1.0")
|
||||
assert not self.storage.exists("v1.0")
|
||||
|
||||
def test_data_isolation(self):
|
||||
"""Test that returned data is isolated from internal storage."""
|
||||
self.storage.save(self.sample_snapshot)
|
||||
retrieved = self.storage.get("v1.0")
|
||||
|
||||
# Modify retrieved data
|
||||
retrieved["entities"].append({"id": "2", "name": "Entity2"})
|
||||
|
||||
# Original should be unchanged
|
||||
retrieved_again = self.storage.get("v1.0")
|
||||
assert len(retrieved_again["entities"]) == 1
|
||||
|
||||
|
||||
class TestSQLiteVersionStorage:
|
||||
"""Test cases for SQLiteVersionStorage."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.temp_dir = tempfile.mkdtemp()
|
||||
self.db_path = os.path.join(self.temp_dir, "test_versions.db")
|
||||
self.storage = SQLiteVersionStorage(self.db_path)
|
||||
|
||||
self.sample_snapshot = {
|
||||
"label": "v1.0",
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"author": "alice@company.com",
|
||||
"description": "Initial version",
|
||||
"checksum": "abc123",
|
||||
"entities": [{"id": "1", "name": "Entity1"}],
|
||||
"relationships": [{"source": "1", "target": "2", "type": "relates"}],
|
||||
"metadata": {"version": "1.0"}
|
||||
}
|
||||
|
||||
def teardown_method(self):
|
||||
"""Clean up test fixtures."""
|
||||
if os.path.exists(self.db_path):
|
||||
os.remove(self.db_path)
|
||||
os.rmdir(self.temp_dir)
|
||||
|
||||
def test_database_initialization(self):
|
||||
"""Test that database is properly initialized."""
|
||||
assert os.path.exists(self.db_path)
|
||||
|
||||
# Verify table exists
|
||||
import sqlite3
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='versions'")
|
||||
result = cursor.fetchone()
|
||||
conn.close()
|
||||
|
||||
assert result is not None
|
||||
|
||||
def test_save_and_get_snapshot(self):
|
||||
"""Test saving and retrieving a snapshot from SQLite."""
|
||||
self.storage.save(self.sample_snapshot)
|
||||
retrieved = self.storage.get("v1.0")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "v1.0"
|
||||
assert retrieved["author"] == "alice@company.com"
|
||||
assert len(retrieved["entities"]) == 1
|
||||
assert retrieved["entities"][0]["name"] == "Entity1"
|
||||
|
||||
def test_save_duplicate_label_raises_error(self):
|
||||
"""Test that saving duplicate label raises ValidationError."""
|
||||
self.storage.save(self.sample_snapshot)
|
||||
|
||||
with pytest.raises(ValidationError, match="Version 'v1.0' already exists"):
|
||||
self.storage.save(self.sample_snapshot)
|
||||
|
||||
def test_persistence_across_instances(self):
|
||||
"""Test that data persists across storage instances."""
|
||||
# Save with first instance
|
||||
self.storage.save(self.sample_snapshot)
|
||||
|
||||
# Create new instance and retrieve
|
||||
new_storage = SQLiteVersionStorage(self.db_path)
|
||||
retrieved = new_storage.get("v1.0")
|
||||
|
||||
assert retrieved is not None
|
||||
assert retrieved["label"] == "v1.0"
|
||||
|
||||
def test_list_all_with_ordering(self):
|
||||
"""Test that list_all returns items ordered by timestamp."""
|
||||
# Save multiple snapshots
|
||||
snapshot1 = self.sample_snapshot.copy()
|
||||
snapshot1["timestamp"] = "2024-01-15T10:30:00Z"
|
||||
|
||||
snapshot2 = self.sample_snapshot.copy()
|
||||
snapshot2["label"] = "v2.0"
|
||||
snapshot2["timestamp"] = "2024-01-15T11:30:00Z"
|
||||
|
||||
self.storage.save(snapshot1)
|
||||
self.storage.save(snapshot2)
|
||||
|
||||
result = self.storage.list_all()
|
||||
assert len(result) == 2
|
||||
|
||||
# Should be ordered by timestamp DESC (newest first)
|
||||
assert result[0]["label"] == "v2.0"
|
||||
assert result[1]["label"] == "v1.0"
|
||||
|
||||
def test_exists_method(self):
|
||||
"""Test the exists method with SQLite."""
|
||||
assert not self.storage.exists("v1.0")
|
||||
|
||||
self.storage.save(self.sample_snapshot)
|
||||
assert self.storage.exists("v1.0")
|
||||
|
||||
def test_delete_method(self):
|
||||
"""Test the delete method with SQLite."""
|
||||
# Delete non-existent returns False
|
||||
assert not self.storage.delete("nonexistent")
|
||||
|
||||
# Save and delete existing returns True
|
||||
self.storage.save(self.sample_snapshot)
|
||||
assert self.storage.delete("v1.0")
|
||||
assert not self.storage.exists("v1.0")
|
||||
|
||||
def test_directory_creation(self):
|
||||
"""Test that storage creates directories if they don't exist."""
|
||||
nested_path = os.path.join(self.temp_dir, "nested", "path", "versions.db")
|
||||
storage = SQLiteVersionStorage(nested_path)
|
||||
|
||||
assert os.path.exists(nested_path)
|
||||
|
||||
# Clean up
|
||||
os.remove(nested_path)
|
||||
os.rmdir(os.path.dirname(nested_path))
|
||||
os.rmdir(os.path.dirname(os.path.dirname(nested_path)))
|
||||
|
||||
|
||||
class TestChecksumUtilities:
|
||||
"""Test cases for checksum computation and verification."""
|
||||
|
||||
def test_compute_checksum_deterministic(self):
|
||||
"""Test that checksum computation is deterministic."""
|
||||
data = {
|
||||
"entities": [{"id": "1", "name": "Entity1"}],
|
||||
"relationships": [{"source": "1", "target": "2"}],
|
||||
"metadata": {"version": "1.0"}
|
||||
}
|
||||
|
||||
checksum1 = compute_checksum(data)
|
||||
checksum2 = compute_checksum(data)
|
||||
|
||||
assert checksum1 == checksum2
|
||||
assert len(checksum1) == 64 # SHA-256 hex length
|
||||
|
||||
def test_compute_checksum_different_data(self):
|
||||
"""Test that different data produces different checksums."""
|
||||
data1 = {"entities": [{"id": "1", "name": "Entity1"}]}
|
||||
data2 = {"entities": [{"id": "1", "name": "Entity2"}]}
|
||||
|
||||
checksum1 = compute_checksum(data1)
|
||||
checksum2 = compute_checksum(data2)
|
||||
|
||||
assert checksum1 != checksum2
|
||||
|
||||
def test_compute_checksum_order_independence(self):
|
||||
"""Test that key order doesn't affect checksum."""
|
||||
data1 = {"b": 2, "a": 1}
|
||||
data2 = {"a": 1, "b": 2}
|
||||
|
||||
checksum1 = compute_checksum(data1)
|
||||
checksum2 = compute_checksum(data2)
|
||||
|
||||
assert checksum1 == checksum2
|
||||
|
||||
def test_verify_checksum_valid(self):
|
||||
"""Test verifying a valid checksum."""
|
||||
data = {"entities": [{"id": "1"}], "metadata": {}}
|
||||
checksum = compute_checksum(data)
|
||||
|
||||
snapshot = data.copy()
|
||||
snapshot["checksum"] = checksum
|
||||
|
||||
assert verify_checksum(snapshot) is True
|
||||
|
||||
def test_verify_checksum_invalid(self):
|
||||
"""Test verifying an invalid checksum."""
|
||||
snapshot = {
|
||||
"entities": [{"id": "1"}],
|
||||
"metadata": {},
|
||||
"checksum": "invalid_checksum"
|
||||
}
|
||||
|
||||
assert verify_checksum(snapshot) is False
|
||||
|
||||
def test_verify_checksum_missing(self):
|
||||
"""Test verifying snapshot without checksum."""
|
||||
snapshot = {
|
||||
"entities": [{"id": "1"}],
|
||||
"metadata": {}
|
||||
}
|
||||
|
||||
assert verify_checksum(snapshot) is False
|
||||
|
||||
def test_verify_checksum_with_modified_data(self):
|
||||
"""Test that verification fails when data is modified."""
|
||||
data = {"entities": [{"id": "1"}], "metadata": {}}
|
||||
checksum = compute_checksum(data)
|
||||
|
||||
# Modify data after computing checksum
|
||||
snapshot = data.copy()
|
||||
snapshot["entities"].append({"id": "2"})
|
||||
snapshot["checksum"] = checksum
|
||||
|
||||
assert verify_checksum(snapshot) is False
|
||||
@@ -0,0 +1,259 @@
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from semantica.ingest.feed_ingestor import (
|
||||
FeedData,
|
||||
FeedIngestor,
|
||||
FeedItem,
|
||||
FeedParser,
|
||||
ProcessingError,
|
||||
)
|
||||
|
||||
|
||||
# --- Fixtures ---
|
||||
@pytest.fixture
|
||||
def complex_atom_xml() -> str:
|
||||
"""Return a complex Atom feed XML string."""
|
||||
|
||||
return """
|
||||
<feed xmlns="http://www.w3.org/2005/Atom">
|
||||
<title>Atom</title>
|
||||
<link href="http://example.com" rel="self"/>
|
||||
<subtitle>Subtitle</subtitle>
|
||||
<updated>2025-01-01T00:00:00Z</updated>
|
||||
<entry>
|
||||
<title>Entry 1</title>
|
||||
<link href="http://example.com/1" rel="alternate"/>
|
||||
<summary>Summary</summary>
|
||||
<content>Full Content</content>
|
||||
<id>uuid:123</id>
|
||||
<published>2025-01-01T00:00:00Z</published>
|
||||
<updated>2025-01-02T00:00:00Z</updated>
|
||||
<category term="tech"/>
|
||||
<category term="news"/>
|
||||
</entry>
|
||||
</feed>
|
||||
"""
|
||||
|
||||
|
||||
# --- Tests ---
|
||||
def test_parse_atom_complex(complex_atom_xml: str) -> None:
|
||||
"""Test parsing a complex Atom feed."""
|
||||
|
||||
parser = FeedParser()
|
||||
data = parser.parse_feed(complex_atom_xml)
|
||||
item = data.items[0]
|
||||
|
||||
assert data.title == "Atom"
|
||||
assert len(data.items) == 1
|
||||
assert item.description == "Summary"
|
||||
assert item.content == "Full Content"
|
||||
assert "tech" in item.categories
|
||||
assert "news" in item.categories
|
||||
assert item.published.year == 2025
|
||||
|
||||
|
||||
def test_parse_rss_dates() -> None:
|
||||
"""Test date parsing logic specific to RSS."""
|
||||
|
||||
xml = """
|
||||
<rss version="2.0">
|
||||
<channel>
|
||||
<title>T</title>
|
||||
<link>http://l.com</link>
|
||||
<item>
|
||||
<title>T</title>
|
||||
<pubDate>Mon, 27 Jan 2025 12:00:00 GMT</pubDate>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
"""
|
||||
parser = FeedParser()
|
||||
data = parser.parse_feed(xml)
|
||||
|
||||
assert data.items[0].published.year == 2025
|
||||
|
||||
|
||||
def test_ingest_feed_errors() -> None:
|
||||
"""Test error handling in ingest_feed."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
|
||||
with pytest.raises(Exception):
|
||||
ingestor.ingest_feed("not_a_url")
|
||||
|
||||
with patch(
|
||||
"requests.get",
|
||||
side_effect=requests.exceptions.RequestException("Fail"),
|
||||
):
|
||||
with pytest.raises(ProcessingError):
|
||||
ingestor.ingest_feed("http://valid.com")
|
||||
|
||||
|
||||
def test_monitor_loop_lifecycle() -> None:
|
||||
"""Test start, run loop once, and stop."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
ingestor.monitor.add_feed("http://test.com")
|
||||
|
||||
def side_effect_sleep(seconds: float) -> None:
|
||||
ingestor.monitor.monitoring = False
|
||||
|
||||
with patch("time.sleep", side_effect=side_effect_sleep):
|
||||
with patch.object(
|
||||
ingestor.monitor,
|
||||
"check_updates",
|
||||
side_effect=Exception("Check Fail"),
|
||||
):
|
||||
ingestor.monitor.monitoring = True
|
||||
ingestor.monitor._monitoring_loop()
|
||||
|
||||
assert ingestor.monitor.monitoring is False
|
||||
|
||||
|
||||
def test_monitor_threading() -> None:
|
||||
"""Test that start_monitoring actually spawns a thread."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
ingestor.monitor.add_feed("http://test.com")
|
||||
|
||||
with patch("threading.Thread") as mock_thread:
|
||||
ingestor.monitor.start_monitoring()
|
||||
mock_thread.return_value.start.assert_called_once()
|
||||
|
||||
# Test double start and stop
|
||||
ingestor.monitor.start_monitoring()
|
||||
ingestor.monitor.stop_monitoring()
|
||||
|
||||
assert ingestor.monitor.monitoring is False
|
||||
|
||||
|
||||
def test_extract_content_helper() -> None:
|
||||
"""Test the extract_content method full fields."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
item = FeedItem(
|
||||
title="T",
|
||||
link="L",
|
||||
description="D",
|
||||
content="C",
|
||||
categories=["cat"],
|
||||
)
|
||||
|
||||
res = ingestor.extract_content(item)
|
||||
|
||||
assert res["content"] == "C"
|
||||
assert res["categories"] == ["cat"]
|
||||
|
||||
|
||||
def test_extract_content_missing_fields() -> None:
|
||||
"""Test extract_content with missing optional fields."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
item = FeedItem(title="T", link="L", description="D")
|
||||
|
||||
res = ingestor.extract_content(item)
|
||||
|
||||
assert res["content"] == "D"
|
||||
assert res["published"] is None
|
||||
assert res["updated"] is None
|
||||
|
||||
|
||||
def test_parse_date_formats() -> None:
|
||||
"""Test various date formats."""
|
||||
|
||||
parser = FeedParser()
|
||||
d1 = parser._parse_date("Mon, 27 Jan 2025 12:00:00 GMT")
|
||||
d2 = parser._parse_date("2025-01-27T12:00:00Z")
|
||||
d3 = parser._parse_date("2025-01-27")
|
||||
|
||||
assert d1.year == 2025
|
||||
assert d2.year == 2025
|
||||
assert d3.year == 2025
|
||||
with pytest.raises(ValueError):
|
||||
parser._parse_date("Not a date")
|
||||
|
||||
|
||||
def test_validate_feed() -> None:
|
||||
"""Test feed validation logic."""
|
||||
|
||||
parser = FeedParser()
|
||||
|
||||
f1 = FeedData(
|
||||
title="T",
|
||||
link="http://e.com",
|
||||
items=[MagicMock(title="t")],
|
||||
)
|
||||
f2 = FeedData(
|
||||
title="",
|
||||
link="http://e.com",
|
||||
items=[MagicMock(title="t")],
|
||||
)
|
||||
f3 = FeedData(title="T", link="http://e.com", items=[])
|
||||
|
||||
assert parser.validate_feed(f1) is True
|
||||
assert parser.validate_feed(f2) is False
|
||||
assert parser.validate_feed(f3) is False
|
||||
|
||||
|
||||
def test_discover_feeds_empty() -> None:
|
||||
"""Test discovery finding nothing."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
html = "<html><body>No feeds here</body></html>"
|
||||
|
||||
with patch("requests.get", return_value=MagicMock(text=html)):
|
||||
feeds = ingestor.discover_feeds("http://site.com")
|
||||
|
||||
assert len(feeds) == 0
|
||||
|
||||
|
||||
def test_discover_feeds_found() -> None:
|
||||
"""Test discovering feeds in HTML content."""
|
||||
ingestor = FeedIngestor()
|
||||
html = """
|
||||
<html>
|
||||
<head>
|
||||
<link rel="alternate" type="application/rss+xml" href="/rss.xml">
|
||||
</head>
|
||||
<body>
|
||||
<a href="/feed">RSS</a>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.text = html
|
||||
mock_response.status_code = 200
|
||||
|
||||
with patch("requests.get", return_value=mock_response):
|
||||
with patch("requests.head") as mock_head:
|
||||
# Mock HEAD request headers for the verification step
|
||||
mock_head.return_value.headers = {
|
||||
"Content-Type": "application/rss+xml",
|
||||
}
|
||||
mock_head.return_value.status_code = 200
|
||||
|
||||
feeds = ingestor.discover_feeds("http://site.com")
|
||||
|
||||
assert "http://site.com/rss.xml" in feeds
|
||||
assert "http://site.com/feed" in feeds
|
||||
|
||||
|
||||
def test_feed_monitor_options() -> None:
|
||||
"""Test adding feeds with specific options."""
|
||||
|
||||
ingestor = FeedIngestor()
|
||||
|
||||
with patch.object(ingestor.monitor, "add_feed") as mock_add:
|
||||
with patch.object(ingestor.monitor, "start_monitoring") as mock_start:
|
||||
ingestor.monitor_feeds(["http://f.com"], interval=60, start=True)
|
||||
|
||||
mock_add.assert_called_with(
|
||||
"http://f.com",
|
||||
interval=60,
|
||||
start=True,
|
||||
)
|
||||
mock_start.assert_called_once()
|
||||
@@ -0,0 +1,326 @@
|
||||
import sys
|
||||
import types
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# --- Mock missing cloud modules ---
|
||||
# We must mock these BEFORE importing the file_ingestor module
|
||||
# and we must populate the attributes that the code tries to import/patch.
|
||||
|
||||
module_names = [
|
||||
"boto3",
|
||||
"google",
|
||||
"google.cloud",
|
||||
"google.cloud.storage",
|
||||
"azure",
|
||||
"azure.storage",
|
||||
"azure.storage.blob",
|
||||
]
|
||||
|
||||
for name in module_names:
|
||||
if name not in sys.modules:
|
||||
mod = types.ModuleType(name)
|
||||
sys.modules[name] = mod
|
||||
|
||||
# Explicitly add the classes that will be patched/used
|
||||
sys.modules["google.cloud.storage"].Client = MagicMock()
|
||||
sys.modules["azure.storage.blob"].BlobServiceClient = MagicMock()
|
||||
sys.modules["boto3"].client = MagicMock()
|
||||
|
||||
from pathlib import Path # noqa: E402
|
||||
from unittest.mock import patch # noqa: E402
|
||||
|
||||
# Now proceed with normal imports
|
||||
import pytest # noqa: E402
|
||||
|
||||
from semantica.ingest.file_ingestor import ( # noqa: E402
|
||||
CloudStorageIngestor,
|
||||
FileIngestor,
|
||||
FileObject,
|
||||
FileTypeDetector,
|
||||
ProcessingError,
|
||||
ValidationError,
|
||||
)
|
||||
|
||||
|
||||
# --- Fixtures ---
|
||||
@pytest.fixture
|
||||
def temp_files(tmp_path: Path) -> Path:
|
||||
"""Create a temporary directory with some dummy files."""
|
||||
txt_file = tmp_path / "test.txt"
|
||||
txt_file.write_text("Hello World", encoding="utf-8") # 11 bytes
|
||||
|
||||
# Binary file (PDF signature)
|
||||
pdf_file = tmp_path / "test.pdf"
|
||||
pdf_file.write_bytes(b"%PDF-1.4 content")
|
||||
|
||||
# Subdirectory
|
||||
sub_dir = tmp_path / "subdir"
|
||||
sub_dir.mkdir()
|
||||
sub_file = sub_dir / "sub.log"
|
||||
sub_file.write_text("Log content")
|
||||
|
||||
# Latin-1 file to test encoding fallback (4 bytes)
|
||||
latin_file = tmp_path / "latin.txt"
|
||||
latin_file.write_bytes(b"Caf\xe9")
|
||||
|
||||
return tmp_path
|
||||
|
||||
|
||||
# --- FileObject Tests ---
|
||||
def test_file_object_text_decoding() -> None:
|
||||
"""Test text property decoding logic."""
|
||||
f1 = FileObject(
|
||||
path="p",
|
||||
name="n",
|
||||
size=1,
|
||||
file_type="txt",
|
||||
content=b"Hello",
|
||||
)
|
||||
f2 = FileObject(
|
||||
path="p",
|
||||
name="n",
|
||||
size=1,
|
||||
file_type="txt",
|
||||
content=b"Caf\xe9",
|
||||
)
|
||||
f3 = FileObject(
|
||||
path="p",
|
||||
name="n",
|
||||
size=1,
|
||||
file_type="txt",
|
||||
content=None,
|
||||
)
|
||||
f4 = FileObject(
|
||||
path="p", name="n", size=1, file_type="txt", content="Already String"
|
||||
)
|
||||
|
||||
assert f1.text == "Hello"
|
||||
assert f2.text == "Café"
|
||||
assert f3.text == ""
|
||||
assert f4.text == "Already String"
|
||||
|
||||
|
||||
# --- FileTypeDetector Tests ---
|
||||
def test_type_detector_extended() -> None:
|
||||
"""Test extended file type detection logic."""
|
||||
|
||||
detector = FileTypeDetector()
|
||||
png_sig = b"\x89\x50\x4e\x47\x0d\x0a\x1a\x0a"
|
||||
|
||||
# We must mock .exists() so detect_type enters the mime detection block
|
||||
with patch("pathlib.Path.exists", return_value=True):
|
||||
with patch("mimetypes.guess_type", return_value=("video/mp4", None)):
|
||||
is_mp4 = detector.detect_type("movie.mp4")
|
||||
|
||||
detected_gz = detector.detect_type("file.tar.gz")
|
||||
detected_png = detector.detect_type("test", content=png_sig)
|
||||
detected_unknown = detector.detect_type("unknown")
|
||||
|
||||
assert detected_gz == "gz"
|
||||
assert is_mp4 == "mp4"
|
||||
assert detected_png == "png"
|
||||
assert detected_unknown == "unknown"
|
||||
|
||||
|
||||
# --- CloudStorageIngestor Tests ---
|
||||
@patch("boto3.client")
|
||||
def test_cloud_storage_s3(mock_boto: MagicMock) -> None:
|
||||
"""Test S3 provider."""
|
||||
|
||||
mock_s3 = mock_boto.return_value
|
||||
mock_s3.get_paginator.return_value.paginate.return_value = [
|
||||
{
|
||||
"Contents": [
|
||||
{
|
||||
"Key": "doc.txt",
|
||||
"Size": 100,
|
||||
"LastModified": "2025",
|
||||
"ETag": "tag",
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
mock_s3.get_object.return_value = {
|
||||
"Body": MagicMock(read=lambda: b"s3_data"),
|
||||
}
|
||||
|
||||
ingestor = CloudStorageIngestor(
|
||||
"s3",
|
||||
access_key_id="x",
|
||||
secret_access_key="y",
|
||||
)
|
||||
objects = ingestor.list_objects("bucket")
|
||||
content = ingestor.download_object("bucket", "doc.txt")
|
||||
|
||||
assert objects[0]["key"] == "doc.txt"
|
||||
assert content == b"s3_data"
|
||||
|
||||
|
||||
@patch("google.cloud.storage.Client")
|
||||
def test_cloud_storage_gcs(mock_gcs_cls: MagicMock) -> None:
|
||||
"""Test Google Cloud Storage provider."""
|
||||
|
||||
mock_client = mock_gcs_cls.return_value
|
||||
mock_blob = MagicMock()
|
||||
mock_blob.name = "gcs.txt"
|
||||
mock_blob.size = 200
|
||||
mock_blob.updated = "2025"
|
||||
mock_blob.etag = "tag"
|
||||
mock_blob.download_as_bytes.return_value = b"gcs_data"
|
||||
|
||||
mock_client.bucket.return_value.list_blobs.return_value = [mock_blob]
|
||||
mock_client.bucket.return_value.blob.return_value = mock_blob
|
||||
|
||||
ingestor = CloudStorageIngestor("gcs")
|
||||
objects = ingestor.list_objects("bucket")
|
||||
content = ingestor.download_object("bucket", "gcs.txt")
|
||||
|
||||
assert objects[0]["key"] == "gcs.txt"
|
||||
assert content == b"gcs_data"
|
||||
|
||||
|
||||
@patch("azure.storage.blob.BlobServiceClient.from_connection_string")
|
||||
def test_cloud_storage_azure(mock_azure_cls: MagicMock) -> None:
|
||||
"""Test Azure Blob Storage provider."""
|
||||
|
||||
mock_client = mock_azure_cls.return_value
|
||||
mock_blob = MagicMock()
|
||||
mock_blob.name = "azure.txt"
|
||||
mock_blob.size = 300
|
||||
mock_blob.last_modified = "2025"
|
||||
mock_blob.etag = "tag"
|
||||
|
||||
mock_container = mock_client.get_container_client.return_value
|
||||
mock_container.list_blobs.return_value = [mock_blob]
|
||||
|
||||
mock_blob_client = mock_container.get_blob_client.return_value
|
||||
mock_download = mock_blob_client.download_blob.return_value
|
||||
mock_download.readall.return_value = b"azure_data"
|
||||
|
||||
ingestor = CloudStorageIngestor("azure", connection_string="conn")
|
||||
objects = ingestor.list_objects("bucket")
|
||||
content = ingestor.download_object("bucket", "azure.txt")
|
||||
|
||||
assert objects[0]["key"] == "azure.txt"
|
||||
assert content == b"azure_data"
|
||||
|
||||
|
||||
def test_cloud_storage_invalid() -> None:
|
||||
"""Test invalid cloud provider raises error."""
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
CloudStorageIngestor("dropbox")
|
||||
|
||||
|
||||
def test_cloud_storage_list_error() -> None:
|
||||
"""Test error handling in list_objects."""
|
||||
|
||||
with patch("boto3.client") as mock_boto:
|
||||
# Raise error on the METHOD call, not the constructor
|
||||
mock_boto.return_value.get_paginator.side_effect = Exception(
|
||||
"Auth Fail",
|
||||
)
|
||||
|
||||
ingestor = CloudStorageIngestor("s3")
|
||||
with pytest.raises(ProcessingError):
|
||||
ingestor.list_objects("bucket")
|
||||
|
||||
|
||||
def test_cloud_storage_download_error() -> None:
|
||||
"""Test error handling in download_object."""
|
||||
|
||||
with patch("boto3.client") as mock_boto:
|
||||
mock_boto.return_value.get_object.side_effect = Exception("Fail")
|
||||
ingestor = CloudStorageIngestor("s3")
|
||||
with pytest.raises(ProcessingError):
|
||||
ingestor.download_object("bucket", "key")
|
||||
|
||||
|
||||
# --- FileIngestor Tests ---
|
||||
def test_ingest_directory_recursive(temp_files: Path) -> None:
|
||||
"""Test recursive directory ingestion."""
|
||||
|
||||
ingestor = FileIngestor()
|
||||
results = ingestor.ingest_directory(temp_files, recursive=True)
|
||||
|
||||
assert len(results) >= 3
|
||||
|
||||
|
||||
def test_ingest_directory_non_recursive(temp_files: Path) -> None:
|
||||
"""Test scanning only top level."""
|
||||
|
||||
ingestor = FileIngestor()
|
||||
results = ingestor.ingest_directory(temp_files, recursive=False)
|
||||
has_sub_log = any("sub.log" in f.name for f in results)
|
||||
|
||||
assert len(results) == 3
|
||||
assert not has_sub_log
|
||||
|
||||
|
||||
def test_ingest_file_callback(temp_files: Path) -> None:
|
||||
"""Test progress callback."""
|
||||
|
||||
ingestor = FileIngestor()
|
||||
mock_cb = MagicMock()
|
||||
ingestor.set_progress_callback(mock_cb)
|
||||
|
||||
ingestor.ingest_directory(temp_files, recursive=False)
|
||||
|
||||
assert mock_cb.called
|
||||
|
||||
|
||||
def test_ingest_file_fail_fast(temp_files: Path) -> None:
|
||||
"""Test directory ingestion failure handling."""
|
||||
|
||||
ingestor = FileIngestor(fail_fast=True)
|
||||
|
||||
with patch.object(ingestor, "ingest_file", side_effect=Exception("Boom")):
|
||||
with pytest.raises(ProcessingError):
|
||||
ingestor.ingest_directory(temp_files)
|
||||
|
||||
|
||||
def test_ingest_alias(temp_files: Path) -> None:
|
||||
"""Test the .ingest() alias method."""
|
||||
|
||||
ingestor = FileIngestor()
|
||||
res_dir = ingestor.ingest(temp_files)
|
||||
res_file = ingestor.ingest(temp_files / "test.txt")
|
||||
|
||||
assert len(res_dir) > 0
|
||||
assert len(res_file) == 1
|
||||
with pytest.raises(ValidationError):
|
||||
ingestor.ingest("ghost_path")
|
||||
|
||||
|
||||
@patch("semantica.ingest.file_ingestor.CloudStorageIngestor")
|
||||
def test_ingest_cloud_loop_errors(mock_cloud_cls: MagicMock) -> None:
|
||||
"""Test cloud ingestion where one file fails."""
|
||||
|
||||
ingestor = FileIngestor(fail_fast=False)
|
||||
mock_inst = mock_cloud_cls.return_value
|
||||
mock_inst.list_objects.return_value = [
|
||||
{"key": "good.txt", "size": 10, "last_modified": "2025", "etag": "1"},
|
||||
{"key": "bad.txt", "size": 10, "last_modified": "2025", "etag": "2"},
|
||||
]
|
||||
mock_inst.download_object.side_effect = [b"good", Exception("Bad dl")]
|
||||
|
||||
results = ingestor.ingest_cloud("s3", "bucket")
|
||||
|
||||
assert len(results) == 1
|
||||
assert results[0].name == "good.txt"
|
||||
|
||||
|
||||
def test_scan_directory_filters(temp_files: Path) -> None:
|
||||
"""Deep dive into filter logic."""
|
||||
|
||||
ingestor = FileIngestor()
|
||||
|
||||
# latin.txt is 4 bytes. 4 <= 5 is True.
|
||||
# So we expect latin.txt to survive.
|
||||
res_max = ingestor.scan_directory(temp_files, max_size=5)
|
||||
|
||||
# All files are small.
|
||||
res_min = ingestor.scan_directory(temp_files, min_size=1)
|
||||
|
||||
assert len(res_max) == 1 # Expect latin.txt
|
||||
assert len(res_min) >= 3
|
||||
@@ -0,0 +1,235 @@
|
||||
import os
|
||||
import tempfile
|
||||
import pandas as pd
|
||||
|
||||
from semantica.ingest.pandas_ingestor import PandasIngestor
|
||||
|
||||
def write_temp_csv(content: str, encoding="utf-8"):
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".csv")
|
||||
tmp.close()
|
||||
with open(tmp.name, "w", encoding=encoding) as f:
|
||||
f.write(content)
|
||||
return tmp.name
|
||||
|
||||
|
||||
# =======================================================
|
||||
# ENCODING (5 tests)
|
||||
# =======================================================
|
||||
|
||||
def test_encoding_latin1():
|
||||
content = "name,city\nJosé,São Paulo\n"
|
||||
path = write_temp_csv(content, encoding="latin-1")
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.dataframe.iloc[0]["name"] == "José"
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_encoding_utf8():
|
||||
content = "user,country\n李雷,China\n"
|
||||
path = write_temp_csv(content, encoding="utf-8")
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.dataframe.iloc[0]["user"] == "李雷"
|
||||
os.remove(path)
|
||||
|
||||
def test_from_csv_detects_tab_delimiter():
|
||||
content = (
|
||||
"user_id\trole\n"
|
||||
"1\tadmin\n"
|
||||
"2\tuser\n"
|
||||
)
|
||||
|
||||
path = write_temp_csv(content)
|
||||
|
||||
ingestor = PandasIngestor()
|
||||
data = ingestor.from_csv(path)
|
||||
|
||||
assert data.row_count == 2
|
||||
assert data.columns == ["user_id", "role"]
|
||||
assert data.dataframe.iloc[0]["role"] == "admin"
|
||||
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_from_csv_handles_quoted_fields_with_commas():
|
||||
content = (
|
||||
"company,revenue\n"
|
||||
'"Acme, Inc.",100\n'
|
||||
'"Widgets, LLC",200\n'
|
||||
)
|
||||
|
||||
path = write_temp_csv(content)
|
||||
|
||||
ingestor = PandasIngestor()
|
||||
data = ingestor.from_csv(path)
|
||||
|
||||
assert data.row_count == 2
|
||||
assert data.columns == ["company", "revenue"]
|
||||
assert data.dataframe.iloc[0]["company"] == "Acme, Inc."
|
||||
assert int(data.dataframe.iloc[1]["revenue"]) == 200
|
||||
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_from_csv_handles_multiline_quoted_fields():
|
||||
# Embed actual newlines within quoted fields
|
||||
content = "id,notes\n1,\"line1\nline2\"\n2,\"alpha\nbeta\"\n"
|
||||
|
||||
path = write_temp_csv(content)
|
||||
|
||||
ingestor = PandasIngestor()
|
||||
data = ingestor.from_csv(path)
|
||||
|
||||
assert data.row_count == 2
|
||||
assert "\n" in data.dataframe.iloc[0]["notes"]
|
||||
assert data.dataframe.iloc[1]["notes"].split("\n")[1] == "beta"
|
||||
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_from_csv_no_header_override():
|
||||
content = (
|
||||
"colA,colB\n"
|
||||
"x,1\n"
|
||||
"y,2\n"
|
||||
)
|
||||
|
||||
path = write_temp_csv(content)
|
||||
|
||||
ingestor = PandasIngestor()
|
||||
data = ingestor.from_csv(path, header=None)
|
||||
|
||||
assert data.row_count == 3
|
||||
assert data.columns == [0, 1]
|
||||
assert list(data.dataframe.iloc[0]) == ["colA", "colB"]
|
||||
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_from_csv_with_chunksize_concatenates():
|
||||
rows = ["a,b", "1,x", "2,y", "3,z", "4,w"]
|
||||
content = "\n".join(rows) + "\n"
|
||||
|
||||
path = write_temp_csv(content)
|
||||
|
||||
ingestor = PandasIngestor()
|
||||
data = ingestor.from_csv(path, chunksize=2)
|
||||
|
||||
assert data.row_count == 4
|
||||
assert data.metadata.get("chunksize") == 2
|
||||
assert list(data.dataframe["a"]) == [1, 2, 3, 4]
|
||||
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_from_csv_preserves_nan_values():
|
||||
content = (
|
||||
"name,score\n"
|
||||
"alice,\n"
|
||||
"bob,10\n"
|
||||
)
|
||||
|
||||
path = write_temp_csv(content)
|
||||
|
||||
ingestor = PandasIngestor()
|
||||
data = ingestor.from_csv(path)
|
||||
|
||||
assert data.row_count == 2
|
||||
assert pd.isna(data.dataframe.iloc[0]["score"]) is True
|
||||
assert int(data.dataframe.iloc[1]["score"]) == 10
|
||||
|
||||
os.remove(path)
|
||||
|
||||
|
||||
# -------------------------------------------------------
|
||||
# Test 2: Delimiter Detection (semicolon separated)
|
||||
# -------------------------------------------------------
|
||||
|
||||
|
||||
def test_encoding_accented_text():
|
||||
content = "company,city\nRenée,Zürich\n"
|
||||
path = write_temp_csv(content, encoding="latin-1")
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.row_count == 1
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_encoding_spanish():
|
||||
content = "org,country\nTelefónica,España\n"
|
||||
path = write_temp_csv(content, encoding="latin-1")
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.row_count == 1
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_encoding_ansi_cp1252():
|
||||
content = "brand,city\nPeugeot,Montréal\n"
|
||||
path = write_temp_csv(content, encoding="cp1252")
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.dataframe.iloc[0]["city"] == "Montréal"
|
||||
os.remove(path)
|
||||
|
||||
|
||||
# =======================================================
|
||||
# DELIMITERS (4 tests)
|
||||
# =======================================================
|
||||
|
||||
def test_delimiter_comma():
|
||||
content = "a,b\n1,2\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert list(data.columns) == ["a", "b"]
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_delimiter_semicolon():
|
||||
content = "a;b\n1;2\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert list(data.columns) == ["a", "b"]
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_delimiter_pipe():
|
||||
content = "a|b\n1|2\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert list(data.columns) == ["a", "b"]
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def test_delimiter_tab():
|
||||
content = "a\tb\n1\t2\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert list(data.columns) == ["a", "b"]
|
||||
os.remove(path)
|
||||
|
||||
|
||||
# =======================================================
|
||||
# BAD ROWS (3 tests)
|
||||
# =======================================================
|
||||
|
||||
def test_bad_row_extra_columns():
|
||||
content = "x,y\n1,2\n1,2,3,4\n5,6\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.row_count == 2
|
||||
os.remove(path)
|
||||
|
||||
def test_bad_row_missing_column():
|
||||
content = "x,y\n1,2\n3\n4,5\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
assert data.row_count == 3
|
||||
assert data.dataframe["y"].isna().sum() == 1
|
||||
|
||||
def test_bad_row_unclosed_quote():
|
||||
content = "x,y\n1,2\n\"3,4\n5,6\n"
|
||||
path = write_temp_csv(content)
|
||||
data = PandasIngestor().from_csv(path)
|
||||
# The malformed quoted line consumes the following line; both are skipped.
|
||||
# Only the first valid row remains.
|
||||
assert data.row_count == 1
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,415 @@
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from semantica.ingest.web_ingestor import (
|
||||
ContentExtractor,
|
||||
ProcessingError,
|
||||
RateLimiter,
|
||||
RobotsChecker,
|
||||
SitemapCrawler,
|
||||
WebContent,
|
||||
WebIngestor,
|
||||
)
|
||||
|
||||
|
||||
# --- Fixtures ---
|
||||
@pytest.fixture
|
||||
def sample_html() -> str:
|
||||
"""Return a simple HTML string."""
|
||||
|
||||
return """<html>
|
||||
<head><title>T</title></head>
|
||||
<body><a href='/1'>1</a></body>
|
||||
</html>"""
|
||||
|
||||
|
||||
# --- Sitemap Tests ---
|
||||
def test_sitemap_index_recursion() -> None:
|
||||
"""Test crawling a sitemap index that points to other sitemaps."""
|
||||
|
||||
crawler = SitemapCrawler()
|
||||
|
||||
index_xml = """
|
||||
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<sitemap><loc>http://ex.com/s1.xml</loc></sitemap>
|
||||
</sitemapindex>
|
||||
"""
|
||||
|
||||
child_xml = """
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url><loc>http://ex.com/page1</loc></url>
|
||||
</urlset>
|
||||
"""
|
||||
|
||||
with patch("requests.get") as mock_get:
|
||||
mock_get.side_effect = [
|
||||
MagicMock(status_code=200, content=index_xml.encode()),
|
||||
MagicMock(status_code=200, content=child_xml.encode()),
|
||||
]
|
||||
urls = crawler.crawl_sitemap_index("http://ex.com/index.xml")
|
||||
|
||||
assert "http://ex.com/page1" in urls
|
||||
|
||||
|
||||
def test_sitemap_fallback_parsing() -> None:
|
||||
"""Test sitemap parsing without namespaces."""
|
||||
|
||||
crawler = SitemapCrawler()
|
||||
xml = "<urlset><url><loc>http://a.com</loc></url></urlset>"
|
||||
|
||||
with patch(
|
||||
"requests.get",
|
||||
return_value=MagicMock(
|
||||
status_code=200,
|
||||
content=xml.encode(),
|
||||
),
|
||||
):
|
||||
urls = crawler.parse_sitemap("http://s.xml")
|
||||
|
||||
assert "http://a.com" in urls
|
||||
|
||||
|
||||
def test_sitemap_invalid_xml() -> None:
|
||||
"""Test parsing invalid XML raises ProcessingError."""
|
||||
|
||||
crawler = SitemapCrawler()
|
||||
|
||||
with patch(
|
||||
"requests.get",
|
||||
return_value=MagicMock(status_code=200, content=b"NOT XML"),
|
||||
):
|
||||
with pytest.raises(ProcessingError):
|
||||
crawler.parse_sitemap("http://s.xml")
|
||||
|
||||
|
||||
# --- Content Extraction Tests ---
|
||||
def test_extract_links_schemes() -> None:
|
||||
"""Ensure we ignore mailto and javascript links."""
|
||||
|
||||
extractor = ContentExtractor()
|
||||
html = """
|
||||
<a href="http://good.com">Good</a>
|
||||
<a href="mailto:me@me.com">Mail</a>
|
||||
<a href="tel:123">Phone</a>
|
||||
<a href="javascript:void(0)">JS</a>
|
||||
"""
|
||||
links = extractor.extract_links(html)
|
||||
|
||||
assert len(links) == 1
|
||||
assert links[0] == "http://good.com"
|
||||
|
||||
|
||||
def test_extract_metadata_empty() -> None:
|
||||
"""Test extraction with missing meta tags."""
|
||||
|
||||
extractor = ContentExtractor()
|
||||
html = "<html><body>No Head</body></html>"
|
||||
meta = extractor.extract_metadata(html, "http://u.com")
|
||||
|
||||
assert meta.get("title") is None or meta.get("title") == ""
|
||||
assert meta.get("description") is None or meta.get("description") == ""
|
||||
|
||||
|
||||
# --- WebIngestor Tests ---
|
||||
@patch("requests.Session.get")
|
||||
def test_ingest_url_happy(mock_get: MagicMock, sample_html: str) -> None:
|
||||
"""Test successful URL ingestion."""
|
||||
|
||||
mock_get.return_value = MagicMock(status_code=200, text=sample_html)
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
res = ingestor.ingest_url("http://site.com")
|
||||
|
||||
assert res.title == "T"
|
||||
|
||||
|
||||
@patch("semantica.ingest.web_ingestor.RobotFileParser")
|
||||
def test_robots_blocking(mock_parser_cls: MagicMock) -> None:
|
||||
"""Test that we actually block if robots says no."""
|
||||
|
||||
mock_inst = mock_parser_cls.return_value
|
||||
mock_inst.can_fetch.return_value = False
|
||||
|
||||
ingestor = WebIngestor(respect_robots=True)
|
||||
|
||||
with pytest.raises(ProcessingError):
|
||||
ingestor.ingest_url("http://site.com/private")
|
||||
|
||||
|
||||
def test_crawl_domain_visited_logic() -> None:
|
||||
"""Test that we don't crawl the same page twice."""
|
||||
|
||||
with patch.object(WebIngestor, "ingest_url") as mock_ingest:
|
||||
p1 = WebContent(url="http://a.com", links=["http://a.com"])
|
||||
mock_ingest.return_value = p1
|
||||
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
results = ingestor.crawl_domain("http://a.com", max_pages=10)
|
||||
|
||||
assert len(results) == 1
|
||||
assert mock_ingest.call_count == 1
|
||||
|
||||
|
||||
def test_rate_limiter() -> None:
|
||||
"""Test that rate limiter actually sleeps."""
|
||||
|
||||
limiter = RateLimiter(delay=0.1)
|
||||
|
||||
with patch("time.sleep") as mock_sleep:
|
||||
# Need 4 values: [init_check, init_set, 2nd_check, 2nd_set]
|
||||
with patch("time.time", side_effect=[100.0, 100.0, 100.05, 100.2]):
|
||||
limiter.wait_if_needed()
|
||||
limiter.wait_if_needed()
|
||||
|
||||
assert mock_sleep.called
|
||||
|
||||
|
||||
def test_rate_limiter_no_delay() -> None:
|
||||
"""Test that 0 delay does not sleep."""
|
||||
|
||||
limiter = RateLimiter(delay=0.0)
|
||||
with patch("time.sleep") as mock_sleep:
|
||||
limiter.wait_if_needed()
|
||||
|
||||
assert not mock_sleep.called
|
||||
|
||||
|
||||
@patch("requests.Session.get")
|
||||
def test_ingest_url_retry(mock_get: MagicMock) -> None:
|
||||
"""Test that it retries on failure."""
|
||||
|
||||
mock_get.side_effect = [
|
||||
requests.exceptions.ConnectionError("Fail 1"),
|
||||
requests.exceptions.ConnectionError("Fail 2"),
|
||||
MagicMock(status_code=200, text="<html></html>"),
|
||||
]
|
||||
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
|
||||
assert ingestor.session.adapters["https://"].max_retries.total == 3
|
||||
|
||||
|
||||
def test_url_filters() -> None:
|
||||
"""Test URL filtering logic."""
|
||||
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
urls = ["https://good.com/a", "https://bad.com/b", "https://good.com/skip"]
|
||||
|
||||
f1 = ingestor._apply_url_filters(urls, {"domains": ["good.com"]})
|
||||
f2 = ingestor._apply_url_filters(urls, {"pattern": r"/a$"})
|
||||
f3 = ingestor._apply_url_filters(urls, {"exclude_pattern": "skip"})
|
||||
|
||||
assert len(f1) == 2
|
||||
assert f2 == ["https://good.com/a"]
|
||||
assert "https://good.com/skip" not in f3
|
||||
|
||||
|
||||
def test_extract_text_cleaning() -> None:
|
||||
"""Test stripping scripts and styles from text."""
|
||||
|
||||
extractor = ContentExtractor()
|
||||
html = """
|
||||
<html>
|
||||
<style>body { color: red; }</style>
|
||||
<script>alert('x');</script>
|
||||
<body>
|
||||
<h1>Real Text</h1>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
text = extractor.extract_text(html)
|
||||
|
||||
assert "Real Text" in text
|
||||
assert "alert" not in text
|
||||
assert "color: red" not in text
|
||||
|
||||
|
||||
def test_robots_checker_cache() -> None:
|
||||
"""Test that robots.txt is cached per domain."""
|
||||
|
||||
with patch("semantica.ingest.web_ingestor.RobotFileParser") as mock_parser:
|
||||
mock_parser.return_value.can_fetch.return_value = True
|
||||
checker = RobotsChecker()
|
||||
|
||||
# First call: Should trigger parser creation
|
||||
checker.can_fetch("http://example.com/a")
|
||||
|
||||
# Second call: Should use cache (no new parser)
|
||||
checker.can_fetch("http://example.com/b")
|
||||
|
||||
# Verify parser was initialized only once
|
||||
assert mock_parser.call_count == 1
|
||||
|
||||
|
||||
def test_web_ingestor_crawl_sitemap_integration() -> None:
|
||||
"""Test the high-level crawl_sitemap method in WebIngestor."""
|
||||
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
|
||||
# 1. Mock the SitemapCrawler to return 2 URLs
|
||||
with patch(
|
||||
"semantica.ingest.web_ingestor.SitemapCrawler.parse_sitemap"
|
||||
) as mock_parse:
|
||||
mock_parse.return_value = ["http://site.com/1", "http://site.com/2"]
|
||||
|
||||
# 2. Mock ingest_url to successfully process those URLs
|
||||
with patch.object(ingestor, "ingest_url") as mock_ingest:
|
||||
mock_ingest.return_value = MagicMock(url="http://site.com/1")
|
||||
|
||||
results = ingestor.crawl_sitemap("http://site.com/sitemap.xml")
|
||||
|
||||
# Should have called ingest_url twice
|
||||
assert len(results) == 2
|
||||
assert mock_ingest.call_count == 2
|
||||
|
||||
|
||||
def test_metadata_priority() -> None:
|
||||
"""Test that OpenGraph tags take precedence over standard meta tags."""
|
||||
|
||||
extractor = ContentExtractor()
|
||||
html = """
|
||||
<html>
|
||||
<head>
|
||||
<meta name="description" content="Standard Description">
|
||||
<meta property="og:description" content="OG Description">
|
||||
<meta name="author" content="Standard Author">
|
||||
</head>
|
||||
</html>
|
||||
"""
|
||||
meta = extractor.extract_metadata(html, "http://site.com")
|
||||
|
||||
assert meta["description"] == "Standard Description"
|
||||
assert meta["og"]["description"] == "OG Description"
|
||||
assert meta["author"] == "Standard Author"
|
||||
|
||||
|
||||
def test_crawl_domain_max_depth() -> None:
|
||||
"""Test that crawling respects max depth/pages."""
|
||||
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
|
||||
# Create a chain of links: P1 -> P2 -> P3
|
||||
p1 = WebContent(url="http://a.com/1", links=["http://a.com/2"])
|
||||
p2 = WebContent(url="http://a.com/2", links=["http://a.com/3"])
|
||||
p3 = WebContent(url="http://a.com/3", links=[])
|
||||
|
||||
with patch.object(ingestor, "ingest_url") as mock_ingest:
|
||||
mock_ingest.side_effect = [p1, p2, p3]
|
||||
|
||||
# Limit to 2 pages
|
||||
results = ingestor.crawl_domain("http://a.com/1", max_pages=2)
|
||||
|
||||
assert len(results) == 2
|
||||
# Should have stopped before P3
|
||||
assert "http://a.com/3" not in [r.url for r in results]
|
||||
|
||||
|
||||
def test_crawl_sitemap_integration() -> None:
|
||||
"""Test the full flow of crawling a sitemap and ingesting its URLs."""
|
||||
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
|
||||
# 1. Mock the sitemap parser to return specific URLs
|
||||
with patch(
|
||||
"semantica.ingest.web_ingestor.SitemapCrawler.parse_sitemap"
|
||||
) as mock_parse:
|
||||
mock_parse.return_value = ["http://site.com/1", "http://site.com/2"]
|
||||
|
||||
# 2. Mock ingest_url to simulate successful extraction for each URL
|
||||
with patch.object(ingestor, "ingest_url") as mock_ingest:
|
||||
# Return dummy content for each call
|
||||
mock_ingest.side_effect = [
|
||||
WebContent(
|
||||
url="http://site.com/1",
|
||||
title="Page 1",
|
||||
text="1",
|
||||
html="",
|
||||
metadata={},
|
||||
links=[],
|
||||
),
|
||||
WebContent(
|
||||
url="http://site.com/2",
|
||||
title="Page 2",
|
||||
text="2",
|
||||
html="",
|
||||
metadata={},
|
||||
links=[],
|
||||
),
|
||||
]
|
||||
|
||||
results = ingestor.crawl_sitemap("http://site.com/sitemap.xml")
|
||||
|
||||
# Verify the loop ran correctly
|
||||
assert len(results) == 2
|
||||
assert results[0].title == "Page 1"
|
||||
assert mock_ingest.call_count == 2
|
||||
|
||||
|
||||
def test_crawl_domain_max_pages() -> None:
|
||||
"""Test that the crawler stops exactly at max_pages."""
|
||||
ingestor = WebIngestor(respect_robots=False)
|
||||
|
||||
# Create a chain: P1 -> P2 -> P3 -> P4
|
||||
p1 = WebContent(
|
||||
url="http://a.com/1",
|
||||
links=["http://a.com/2"],
|
||||
title="",
|
||||
text="",
|
||||
html="",
|
||||
metadata={},
|
||||
)
|
||||
p2 = WebContent(
|
||||
url="http://a.com/2",
|
||||
links=["http://a.com/3"],
|
||||
title="",
|
||||
text="",
|
||||
html="",
|
||||
metadata={},
|
||||
)
|
||||
p3 = WebContent(
|
||||
url="http://a.com/3",
|
||||
links=["http://a.com/4"],
|
||||
title="",
|
||||
text="",
|
||||
html="",
|
||||
metadata={},
|
||||
)
|
||||
|
||||
with patch.object(ingestor, "ingest_url") as mock_ingest:
|
||||
mock_ingest.side_effect = [p1, p2, p3]
|
||||
|
||||
# Set limit to 2 pages
|
||||
results = ingestor.crawl_domain("http://a.com/1", max_pages=2)
|
||||
|
||||
assert len(results) == 2
|
||||
assert results[0].url == "http://a.com/1"
|
||||
assert results[1].url == "http://a.com/2"
|
||||
|
||||
# Verify P3 was never ingested
|
||||
res = [c[0][0] for c in mock_ingest.call_args_list]
|
||||
assert "http://a.com/3" not in res
|
||||
|
||||
|
||||
def test_metadata_opengraph_priority() -> None:
|
||||
"""Test that OpenGraph tags are captured correctly."""
|
||||
extractor = ContentExtractor()
|
||||
# HTML with both standard meta and OG tags
|
||||
html = """
|
||||
<html>
|
||||
<head>
|
||||
<meta name="description" content="Basic Desc">
|
||||
<meta property="og:description" content="OG Desc">
|
||||
<meta property="og:title" content="OG Title">
|
||||
<meta property="og:image" content="http://img.jpg">
|
||||
</head>
|
||||
</html>
|
||||
"""
|
||||
meta = extractor.extract_metadata(html, "http://site.com")
|
||||
|
||||
# Check that OG data is structured correctly in the 'og' dict
|
||||
assert meta["og"]["description"] == "OG Desc"
|
||||
assert meta["og"]["title"] == "OG Title"
|
||||
assert meta["og"]["image"] == "http://img.jpg"
|
||||
# Basic description should still be available
|
||||
assert meta["description"] == "Basic Desc"
|
||||
@@ -0,0 +1,325 @@
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from semantica.normalize.text_normalizer import (
|
||||
SpecialCharacterProcessor,
|
||||
TextNormalizer,
|
||||
UnicodeNormalizer,
|
||||
WhitespaceNormalizer,
|
||||
)
|
||||
|
||||
|
||||
class TestTextNormalizer(unittest.TestCase):
|
||||
"""
|
||||
Test suite for the TextNormalizer class.
|
||||
"""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up mocks"""
|
||||
|
||||
self.logger_patcher = patch("semantica.normalize.text_normalizer.get_logger")
|
||||
self.tracker_patcher = patch(
|
||||
"semantica.normalize.text_normalizer.get_progress_tracker"
|
||||
)
|
||||
self.cleaner_patcher = patch("semantica.normalize.text_normalizer.TextCleaner")
|
||||
|
||||
self.mock_logger = self.logger_patcher.start()
|
||||
self.mock_tracker = self.tracker_patcher.start()
|
||||
self.mock_cleaner_cls = self.cleaner_patcher.start()
|
||||
|
||||
# config mocks
|
||||
self.mock_tracker_instance = MagicMock()
|
||||
self.mock_tracker_instance.enabled = True
|
||||
self.mock_tracker.return_value = self.mock_tracker_instance
|
||||
|
||||
self.mock_cleaner_instance = MagicMock()
|
||||
self.mock_cleaner_cls.return_value = self.mock_cleaner_instance
|
||||
|
||||
# init normalization
|
||||
|
||||
self.normalizer = TextNormalizer()
|
||||
|
||||
def tearDown(self):
|
||||
"""Stop all patches."""
|
||||
self.logger_patcher.stop()
|
||||
self.tracker_patcher.stop()
|
||||
self.cleaner_patcher.stop()
|
||||
|
||||
def test_init(self):
|
||||
"""Test initialization"""
|
||||
self.mock_cleaner_cls.assert_called_once()
|
||||
self.assertTrue(hasattr(self.normalizer, "unicode_normalizer"))
|
||||
self.assertTrue(hasattr(self.normalizer, "whitespace_normalizer"))
|
||||
self.assertTrue(hasattr(self.normalizer, "special_char_processor"))
|
||||
|
||||
self.assertTrue(self.normalizer.progress_tracker.enabled)
|
||||
|
||||
def test_normalize_text_basic(self):
|
||||
"""Test basic text normalization"""
|
||||
text = "Hello World"
|
||||
result = self.normalizer.normalize_text(text)
|
||||
self.assertEqual(result, "Hello World")
|
||||
|
||||
# progress bar insurance
|
||||
|
||||
self.mock_tracker_instance.start_tracking.assert_called()
|
||||
self.mock_tracker_instance.stop_tracking.assert_called_with(
|
||||
self.mock_tracker_instance.start_tracking.return_value, status="completed"
|
||||
)
|
||||
|
||||
def test_normalize_empty_string(self):
|
||||
"""Test 'nothingness'"""
|
||||
self.assertEqual(self.normalizer.normalize_text(""), "")
|
||||
self.assertEqual(self.normalizer.normalize_text(None), "")
|
||||
|
||||
def test_normalize_case_options(self):
|
||||
"""Test case normalization"""
|
||||
text = "HeLLo WoRLd"
|
||||
|
||||
self.assertEqual(
|
||||
self.normalizer.normalize_text(text, case="lower"), "hello world"
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
self.normalizer.normalize_text(text, case="upper"), "HELLO WORLD"
|
||||
)
|
||||
self.assertEqual(
|
||||
self.normalizer.normalize_text(text, case="title"), "Hello World"
|
||||
)
|
||||
|
||||
# preserve test ---- default
|
||||
|
||||
self.assertEqual(
|
||||
self.normalizer.normalize_text(text, case="preserve"), "HeLLo WoRLd"
|
||||
)
|
||||
|
||||
def test_normalize_delegation(self):
|
||||
"""Verify that normalize_text correctly delegates to subcomponents."""
|
||||
|
||||
self.normalizer.unicode_normalizer.normalize_unicode = MagicMock(
|
||||
return_value="U"
|
||||
)
|
||||
self.normalizer.whitespace_normalizer.normalize_whitespace = MagicMock(
|
||||
return_value="W"
|
||||
)
|
||||
self.normalizer.special_char_processor.process_special_chars = MagicMock(
|
||||
return_value="S"
|
||||
)
|
||||
|
||||
result = self.normalizer.normalize_text(
|
||||
"input",
|
||||
unicode_form="NFD",
|
||||
line_break_type="windows",
|
||||
normalize_diacritics=True,
|
||||
)
|
||||
|
||||
self.normalizer.unicode_normalizer.normalize_unicode.assert_called_with(
|
||||
"input", form="NFD"
|
||||
)
|
||||
self.normalizer.whitespace_normalizer.normalize_whitespace.assert_called_with(
|
||||
"U", line_break_type="windows"
|
||||
)
|
||||
self.normalizer.special_char_processor.process_special_chars.assert_called_with(
|
||||
"W", normalize_diacritics=True
|
||||
)
|
||||
|
||||
self.assertEqual(result, "S")
|
||||
|
||||
def test_clean_text(self):
|
||||
"""Test delegation to TextCleaner"""
|
||||
text = "<html>body</html>"
|
||||
self.mock_cleaner_instance.clean.return_value = "body"
|
||||
result = self.normalizer.clean_text(text, remove_html=True)
|
||||
|
||||
self.mock_cleaner_instance.clean.assert_called_with(text, remove_html=True)
|
||||
self.assertEqual(result, "body")
|
||||
|
||||
def test_standardize_format(self):
|
||||
"""Test format standardization option"""
|
||||
text = " one two "
|
||||
|
||||
self.assertEqual(
|
||||
self.normalizer.standardize_format(text, format_type="compact"), "one two"
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
self.normalizer.standardize_format(text, format_type="preserve"),
|
||||
"one two",
|
||||
)
|
||||
|
||||
def test_process_batch(self):
|
||||
"""Test batch processing"""
|
||||
texts = ["TEST 1", "Test 2"]
|
||||
results = self.normalizer.process_batch(texts, case="lower")
|
||||
self.assertEqual(results, ["test 1", "test 2"])
|
||||
|
||||
def test_normalize_overloaded_method(self):
|
||||
"""Test generic normalize method"""
|
||||
self.assertEqual(self.normalizer.normalize("TEST", case="lower"), "test")
|
||||
|
||||
# dict
|
||||
|
||||
docs = [
|
||||
{"id": 1, "content": "DOC 1"},
|
||||
{"id": 2, "content": "DOC 2", "other": "meta"},
|
||||
{"id": 3, "nocontent": "skip"},
|
||||
]
|
||||
|
||||
results = self.normalizer.normalize(docs, case="lower")
|
||||
|
||||
self.assertEqual(results[0]["content"], "doc 1")
|
||||
self.assertEqual(results[1]["content"], "doc 2")
|
||||
self.assertEqual(results[1]["other"], "meta")
|
||||
|
||||
self.assertIn("skip", results[2])
|
||||
|
||||
def test_normalize_error_handling(self):
|
||||
"""Test error handling"""
|
||||
|
||||
self.normalizer.unicode_normalizer.normalize_unicode = MagicMock(
|
||||
side_effect=Exception("Test Error")
|
||||
)
|
||||
|
||||
with self.assertRaises(Exception):
|
||||
self.normalizer.normalize_text("input")
|
||||
self.mock_tracker_instance.stop_tracking.assert_called_with(
|
||||
self.mock_tracker_instance.start_tracking.return_value,
|
||||
status="failed",
|
||||
message="Test Error",
|
||||
)
|
||||
|
||||
|
||||
class TestUnicodeNormalizer(unittest.TestCase):
|
||||
"""Test suite for UniCodeNormalizer class"""
|
||||
|
||||
def setUp(self):
|
||||
self.normalizer = UnicodeNormalizer()
|
||||
|
||||
def test_normalize_unicode_forms(self):
|
||||
"""Test diff unicode normalization forms"""
|
||||
|
||||
text_nfc = "\u00e9"
|
||||
text_nfd = "\u0065\u0301"
|
||||
|
||||
self.assertEqual(self.normalizer.normalize_unicode(text_nfd, "NFC"), text_nfc)
|
||||
self.assertEqual(self.normalizer.normalize_unicode(text_nfc, "NFD"), text_nfd)
|
||||
|
||||
def test_normalize_none(self):
|
||||
"""Test empty input"""
|
||||
|
||||
self.assertEqual(self.normalizer.normalize_unicode(None), "")
|
||||
self.assertEqual(self.normalizer.normalize_unicode(""), "")
|
||||
|
||||
def test_normalize_failure_fallback(self):
|
||||
"""Test that it returns og text if unicode fails"""
|
||||
|
||||
with patch("unicodedata.normalize", side_effect=Exception("Boom")):
|
||||
result = self.normalizer.normalize_unicode("test")
|
||||
self.assertEqual(result, "test")
|
||||
|
||||
def test_handle_encoding(self):
|
||||
"""Test encoding handling"""
|
||||
self.assertEqual(self.normalizer.handle_encoding("test", "utf-8"), "test")
|
||||
|
||||
# bytes in
|
||||
|
||||
byte_data = "test".encode("utf-8")
|
||||
self.assertEqual(self.normalizer.handle_encoding(byte_data, "utf-8"), "test")
|
||||
|
||||
# cross encoding
|
||||
|
||||
latin_bytes = "café".encode("latin-1")
|
||||
result = self.normalizer.handle_encoding(latin_bytes, "latin-1", "utf-8")
|
||||
self.assertEqual(result, "café")
|
||||
|
||||
# broken bites
|
||||
|
||||
bad_bytes = b"\xff"
|
||||
self.assertIsInstance(self.normalizer.handle_encoding(bad_bytes, "utf-8"), str)
|
||||
|
||||
def test_process_special_chars_replacement(self):
|
||||
"""Test unicode character replacement"""
|
||||
input_text = "\u2018single\u2019 \u201Cdouble\u201D \u2013 \u2014 \u2026"
|
||||
expected = "'single' \"double\" - -- ..."
|
||||
self.assertEqual(self.normalizer.process_special_chars(input_text), expected)
|
||||
|
||||
|
||||
class TestWhitespaceNormalizer(unittest.TestCase):
|
||||
"""Test suite for WhitespaceNormalizer class"""
|
||||
|
||||
def setUp(self):
|
||||
self.normalizer = WhitespaceNormalizer()
|
||||
|
||||
def test_normalize_whitespace_basic(self):
|
||||
"""Test basic whitespace cleanup"""
|
||||
text = "Hello World\tTest"
|
||||
|
||||
self.assertEqual(self.normalizer.normalize_whitespace(text), "Hello World Test")
|
||||
|
||||
def test_handle_line_breaks(self):
|
||||
"""Test line break conversion"""
|
||||
text = "Row1\r\nRow2\rRow3\n"
|
||||
|
||||
self.assertEqual(
|
||||
self.normalizer.handle_line_breaks(text, "unix"), "Row1\nRow2\nRow3\n"
|
||||
)
|
||||
|
||||
res_windows = self.normalizer.handle_line_breaks("Row1\nRow2", "windows")
|
||||
self.assertEqual(res_windows, "Row1\r\nRow2")
|
||||
|
||||
def test_process_indentation(self):
|
||||
"""Test indentation conversion"""
|
||||
|
||||
spaces = " Code"
|
||||
self.assertEqual(self.normalizer.process_indentation(spaces, "tabs"), "\tCode")
|
||||
|
||||
tabs = "\tCode"
|
||||
self.assertEqual(
|
||||
self.normalizer.process_indentation(tabs, "spaces"), " Code"
|
||||
)
|
||||
|
||||
|
||||
class TestSpecialCharacterProcessor(unittest.TestCase):
|
||||
"""Test suite for SpecialCharacterProcessor class."""
|
||||
|
||||
def setUp(self):
|
||||
self.processor = SpecialCharacterProcessor()
|
||||
|
||||
def test_normalize_punctuation(self):
|
||||
"""Test punctuation cleanup"""
|
||||
|
||||
text = "“Hello” ‘World’ – …"
|
||||
expected = "\"Hello\" 'World' - ..."
|
||||
|
||||
self.assertEqual(self.processor.normalize_punctuation(text), expected)
|
||||
|
||||
def test_process_diacritics_remove(self):
|
||||
"""Test removing diacritics"""
|
||||
|
||||
text = "Crème Brûlée"
|
||||
expected = "Creme Brulee"
|
||||
result = self.processor.process_diacritics(text, remove_diacritics=True)
|
||||
|
||||
self.assertEqual(result, expected)
|
||||
|
||||
def test_process_diacritics_normalize(self):
|
||||
"""Test normalizing diacritics"""
|
||||
|
||||
text = "e\u0301" # NFD ~~ this wastes memory
|
||||
|
||||
expected = "\u00e9" # should become NFC which is uh precomposed single char
|
||||
result = self.processor.process_diacritics(text, remove_diacritics=False)
|
||||
self.assertEqual(result, expected)
|
||||
|
||||
def test_process_special_chars_integration(self):
|
||||
"""Test the main processing method integration"""
|
||||
text = "“Crème”"
|
||||
|
||||
result = self.processor.process_special_chars(
|
||||
text, normalize_diacritics=True, remove_diacritics=True
|
||||
)
|
||||
self.assertEqual(result, '"Creme"')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -6,7 +6,7 @@ import os
|
||||
|
||||
from semantica.ontology.ontology_evaluator import OntologyEvaluator, EvaluationResult
|
||||
from semantica.ontology.competency_questions import CompetencyQuestionsManager, CompetencyQuestion
|
||||
from semantica.ontology.version_manager import VersionManager, OntologyVersion
|
||||
from semantica.change_management import VersionManager, OntologyVersion
|
||||
from semantica.ontology.associative_class import AssociativeClassBuilder, AssociativeClass
|
||||
|
||||
class TestOntologyAdvanced(unittest.TestCase):
|
||||
@@ -22,8 +22,8 @@ class TestOntologyAdvanced(unittest.TestCase):
|
||||
patch('semantica.ontology.ontology_evaluator.get_progress_tracker', return_value=self.mock_tracker),
|
||||
patch('semantica.ontology.competency_questions.get_logger', return_value=self.mock_logger),
|
||||
patch('semantica.ontology.competency_questions.get_progress_tracker', return_value=self.mock_tracker),
|
||||
patch('semantica.ontology.version_manager.get_logger', return_value=self.mock_logger),
|
||||
patch('semantica.ontology.version_manager.get_progress_tracker', return_value=self.mock_tracker),
|
||||
patch('semantica.change_management.ontology_version_manager.get_logger', return_value=self.mock_logger),
|
||||
patch('semantica.change_management.ontology_version_manager.get_progress_tracker', return_value=self.mock_tracker),
|
||||
patch('semantica.ontology.associative_class.get_logger', return_value=self.mock_logger),
|
||||
patch('semantica.ontology.associative_class.get_progress_tracker', return_value=self.mock_tracker),
|
||||
]
|
||||
@@ -93,7 +93,7 @@ class TestOntologyAdvanced(unittest.TestCase):
|
||||
self.assertEqual(result.completeness_score, 0.9)
|
||||
|
||||
# --- VersionManager Tests ---
|
||||
@patch('semantica.ontology.version_manager.NamespaceManager')
|
||||
@patch('semantica.change_management.ontology_version_manager.NamespaceManager')
|
||||
def test_version_manager_create(self, mock_ns_cls):
|
||||
manager = VersionManager(base_uri="http://example.org/")
|
||||
ontology = {"metadata": {}}
|
||||
|
||||
Reference in New Issue
Block a user