semantica/docs/reference/utils.md

Utils Module

Utility functions and helper classes for common operations including file handling, text processing, and data manipulation.

Overview

• File Operations: File I/O, path manipulation, compression
• Text Utilities: String manipulation, encoding, hashing
• Data Utilities: JSON/YAML handling, serialization
• Logging: Structured logging with multiple handlers
• Validation: Schema validation, type checking

---

Algorithms Used

File Operations

• Hashing: MD5, SHA256 for file integrity checking
• Compression: GZIP, LZMA algorithms for file compression
• Chunking: Fixed-size or sliding window chunking for large files

Text Processing

• Tokenization: Whitespace and punctuation-based tokenization
• Similarity: Jaccard similarity, Cosine similarity
• Hashing: MurmurHash3 for fast non-cryptographic hashing

Data Serialization

• JSON: Fast JSON encoding/decoding
• YAML: Human-readable configuration format
• Pickle: Python object serialization

---

Main Classes

FileUtils

Methods:

Method	Description	Algorithm
list_files(directory, pattern)	List files matching pattern	Recursive directory traversal
read_file(path)	Read file contents	Buffered I/O
write_file(path, content)	Write to file	Atomic write with temp file
hash_file(path, algorithm)	Calculate file hash	Streaming hash calculation
compress_file(path, format)	Compress file	GZIP/LZMA compression

Example:

from semantica.utils import FileUtils

# List files
files = FileUtils.list_files(
    directory="documents/",
    pattern="*.pdf",
    recursive=True
)

# Hash file
file_hash = FileUtils.hash_file("document.pdf", algorithm="sha256")
print(f"SHA256: {file_hash}")

# Compress file
FileUtils.compress_file("large_file.json", format="gzip")

---

TextUtils

Methods:

Method	Description	Algorithm
clean_text(text)	Clean and normalize text	Regex-based cleaning
tokenize(text, method)	Tokenize text	Whitespace/punctuation splitting
calculate_similarity(text1, text2)	Calculate text similarity	Jaccard or Cosine similarity
hash_text(text)	Hash text string	MurmurHash3
remove_stopwords(text, language)	Remove stopwords	Dictionary-based filtering

Example:

from semantica.utils import TextUtils

# Clean text
text = "  Hello,   World!  \n\n  "
cleaned = TextUtils.clean_text(text)
print(cleaned)  # "Hello, World!"

# Calculate similarity
similarity = TextUtils.calculate_similarity(
    "machine learning",
    "deep learning",
    method="jaccard"
)
print(f"Similarity: {similarity:.2f}")

---

DataUtils

Methods:

Method	Description	Algorithm
load_json(path)	Load JSON file	JSON parsing
save_json(data, path)	Save to JSON	JSON serialization
load_yaml(path)	Load YAML file	YAML parsing
save_yaml(data, path)	Save to YAML	YAML serialization
validate_schema(data, schema)	Validate against schema	JSON Schema validation

Example:

from semantica.utils import DataUtils

# Load/save JSON
data = DataUtils.load_json("config.json")
DataUtils.save_json(data, "output.json", pretty=True)

# Load/save YAML
config = DataUtils.load_yaml("config.yaml")
DataUtils.save_yaml(config, "output.yaml")

# Validate schema
is_valid = DataUtils.validate_schema(
    data=data,
    schema={"type": "object", "properties": {...}}
)

---

LoggerUtils

Methods:

Method	Description	Algorithm
get_logger(name)	Get logger instance	Logger factory
configure_logging(level, format)	Configure logging	Handler setup

---

Configuration

# config.yaml - Utils Configuration

utils:
  logging:
    level: INFO  # DEBUG, INFO, WARNING, ERROR
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
    handlers:
      - console
      - file
      
  file:
    default_encoding: utf-8
    buffer_size: 8192
    
  text:
    default_language: en
    remove_stopwords: true

---

See Also

• Core Module - Framework orchestration
• Ingest Module - Data ingestion
• Parse Module - Document parsing