mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
KaifAhmad1
409f3fdd27
- Added grid‑card layouts, icons, and admonitions to all reference pages - Removed all mkdocstrings ::: directives that caused alias‑resolution errors - Fixed export module alias issue (RDFExporter) by cleaning the docs - Updated README‑style sections for better readability - Added a PowerShell helper script (remove_mkdocstrings.ps1) for future clean‑ups
181 lines
4.3 KiB
Markdown
181 lines
4.3 KiB
Markdown
# Utils Module
|
|
|
|
Utility functions and helper classes for common operations including file handling, text processing, and data manipulation.
|
|
|
|
## Overview
|
|
|
|
- **File Operations**: File I/O, path manipulation, compression
|
|
- **Text Utilities**: String manipulation, encoding, hashing
|
|
- **Data Utilities**: JSON/YAML handling, serialization
|
|
- **Logging**: Structured logging with multiple handlers
|
|
- **Validation**: Schema validation, type checking
|
|
|
|
---
|
|
|
|
## Algorithms Used
|
|
|
|
### File Operations
|
|
- **Hashing**: MD5, SHA256 for file integrity checking
|
|
- **Compression**: GZIP, LZMA algorithms for file compression
|
|
- **Chunking**: Fixed-size or sliding window chunking for large files
|
|
|
|
### Text Processing
|
|
- **Tokenization**: Whitespace and punctuation-based tokenization
|
|
- **Similarity**: Jaccard similarity, Cosine similarity
|
|
- **Hashing**: MurmurHash3 for fast non-cryptographic hashing
|
|
|
|
### Data Serialization
|
|
- **JSON**: Fast JSON encoding/decoding
|
|
- **YAML**: Human-readable configuration format
|
|
- **Pickle**: Python object serialization
|
|
|
|
---
|
|
|
|
## Main Classes
|
|
|
|
### FileUtils
|
|
|
|
|
|
**Methods:**
|
|
|
|
| Method | Description | Algorithm |
|
|
|--------|-------------|-----------|
|
|
| `list_files(directory, pattern)` | List files matching pattern | Recursive directory traversal |
|
|
| `read_file(path)` | Read file contents | Buffered I/O |
|
|
| `write_file(path, content)` | Write to file | Atomic write with temp file |
|
|
| `hash_file(path, algorithm)` | Calculate file hash | Streaming hash calculation |
|
|
| `compress_file(path, format)` | Compress file | GZIP/LZMA compression |
|
|
|
|
**Example:**
|
|
|
|
```python
|
|
from semantica.utils import FileUtils
|
|
|
|
# List files
|
|
files = FileUtils.list_files(
|
|
directory="documents/",
|
|
pattern="*.pdf",
|
|
recursive=True
|
|
)
|
|
|
|
# Hash file
|
|
file_hash = FileUtils.hash_file("document.pdf", algorithm="sha256")
|
|
print(f"SHA256: {file_hash}")
|
|
|
|
# Compress file
|
|
FileUtils.compress_file("large_file.json", format="gzip")
|
|
```
|
|
|
|
---
|
|
|
|
### TextUtils
|
|
|
|
|
|
**Methods:**
|
|
|
|
| Method | Description | Algorithm |
|
|
|--------|-------------|-----------|
|
|
| `clean_text(text)` | Clean and normalize text | Regex-based cleaning |
|
|
| `tokenize(text, method)` | Tokenize text | Whitespace/punctuation splitting |
|
|
| `calculate_similarity(text1, text2)` | Calculate text similarity | Jaccard or Cosine similarity |
|
|
| `hash_text(text)` | Hash text string | MurmurHash3 |
|
|
| `remove_stopwords(text, language)` | Remove stopwords | Dictionary-based filtering |
|
|
|
|
**Example:**
|
|
|
|
```python
|
|
from semantica.utils import TextUtils
|
|
|
|
# Clean text
|
|
text = " Hello, World! \n\n "
|
|
cleaned = TextUtils.clean_text(text)
|
|
print(cleaned) # "Hello, World!"
|
|
|
|
# Calculate similarity
|
|
similarity = TextUtils.calculate_similarity(
|
|
"machine learning",
|
|
"deep learning",
|
|
method="jaccard"
|
|
)
|
|
print(f"Similarity: {similarity:.2f}")
|
|
```
|
|
|
|
---
|
|
|
|
### DataUtils
|
|
|
|
|
|
**Methods:**
|
|
|
|
| Method | Description | Algorithm |
|
|
|--------|-------------|-----------|
|
|
| `load_json(path)` | Load JSON file | JSON parsing |
|
|
| `save_json(data, path)` | Save to JSON | JSON serialization |
|
|
| `load_yaml(path)` | Load YAML file | YAML parsing |
|
|
| `save_yaml(data, path)` | Save to YAML | YAML serialization |
|
|
| `validate_schema(data, schema)` | Validate against schema | JSON Schema validation |
|
|
|
|
**Example:**
|
|
|
|
```python
|
|
from semantica.utils import DataUtils
|
|
|
|
# Load/save JSON
|
|
data = DataUtils.load_json("config.json")
|
|
DataUtils.save_json(data, "output.json", pretty=True)
|
|
|
|
# Load/save YAML
|
|
config = DataUtils.load_yaml("config.yaml")
|
|
DataUtils.save_yaml(config, "output.yaml")
|
|
|
|
# Validate schema
|
|
is_valid = DataUtils.validate_schema(
|
|
data=data,
|
|
schema={"type": "object", "properties": {...}}
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### LoggerUtils
|
|
|
|
|
|
**Methods:**
|
|
|
|
| Method | Description | Algorithm |
|
|
|--------|-------------|-----------|
|
|
| `get_logger(name)` | Get logger instance | Logger factory |
|
|
| `configure_logging(level, format)` | Configure logging | Handler setup |
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
```yaml
|
|
# config.yaml - Utils Configuration
|
|
|
|
utils:
|
|
logging:
|
|
level: INFO # DEBUG, INFO, WARNING, ERROR
|
|
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
|
handlers:
|
|
- console
|
|
- file
|
|
|
|
file:
|
|
default_encoding: utf-8
|
|
buffer_size: 8192
|
|
|
|
text:
|
|
default_language: en
|
|
remove_stopwords: true
|
|
```
|
|
|
|
---
|
|
|
|
## See Also
|
|
|
|
- [Core Module](core.md) - Framework orchestration
|
|
- [Ingest Module](ingest.md) - Data ingestion
|
|
- [Parse Module](parse.md) - Document parsing
|