diff --git a/README.md b/README.md index 5693a66f..ab59edcd 100644 --- a/README.md +++ b/README.md @@ -4,51 +4,22 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![PyPI version](https://badge.fury.io/py/semanticore.svg)](https://badge.fury.io/py/semanticore) [![Downloads](https://pepy.tech/badge/semanticore)](https://pepy.tech/project/semanticore) -[![Tests](https://github.com/yourusername/semanticore/workflows/Tests/badge.svg)](https://github.com/yourusername/semanticore/actions) -**Transform Unstructured Data into Intelligent Semantic Layers for AI Systems** +**Transform unstructured data into structured semantic layers for LLMs, Agents, RAG systems, and Knowledge Graphs.** -SemantiCore is an open-source toolkit that transforms raw, unstructured data into semantic knowledge representations including ontologies, knowledge graphs, and context-aware embeddings. Built for developers creating AI agents, RAG systems, and intelligent applications that need to understand meaning, not just text. +SemantiCore bridges the gap between raw unstructured data and intelligent AI systems by providing a comprehensive toolkit for semantic extraction, schema generation, and knowledge representation. --- -## ๐Ÿš€ Core Features Overview +## ๐ŸŒŸ Why SemantiCore? -### ๐Ÿง  **Semantic Processing** -- **Multi-layer Understanding**: Lexical, syntactic, semantic, and pragmatic analysis -- **Entity & Relationship Extraction**: Named entities, relationships, and complex event detection -- **Context Preservation**: Maintain semantic context across document boundaries -- **Domain Adaptation**: Specialized processing for cybersecurity, finance, healthcare, research +Modern AI systems require structured, semantically rich data to perform effectively. SemantiCore solves the fundamental challenge of converting messy, unstructured information into clean, schema-compliant semantic layers that power: -### ๐ŸŽฏ **LLM Optimization** -- **Context Engineering**: Intelligent context compression and enhancement for LLMs -- **Prompt Optimization**: Semantic-aware prompt engineering and optimization -- **Memory Management**: Episodic, semantic, and procedural memory systems -- **Multi-Model Support**: OpenAI, Anthropic, Google Gemini, Hugging Face, local models - -### ๐Ÿ•ธ๏ธ **Knowledge Graphs** -- **Automated Construction**: Build knowledge graphs from unstructured data -- **Graph Databases**: Neo4j, KuzuDB, ArangoDB, Amazon Neptune integration -- **Semantic Reasoning**: Inductive, deductive, and abductive reasoning capabilities -- **Temporal Modeling**: Time-aware relationships and evolution tracking - -### ๐Ÿ“Š **Vector & Embeddings** -- **Contextual Embeddings**: Semantic embeddings with preserved context -- **Vector Stores**: Pinecone, Milvus, Weaviate, Chroma, FAISS integration -- **Hybrid Search**: Combine semantic and keyword search strategies -- **Embedding Models**: OpenAI, Cohere, Sentence Transformers, custom models - -### ๐Ÿ”— **Ontology Generation** -- **Automated Ontology Creation**: Generate OWL/RDF ontologies from data -- **Schema Evolution**: Dynamic schema adaptation and versioning -- **Standard Compliance**: Schema.org, FIBO, domain-specific ontologies -- **Multi-format Export**: OWL, RDF, JSON-LD, Turtle formats - -### ๐Ÿค– **Agent Integration** -- **Semantic Routing**: Intelligent request routing based on semantic understanding -- **Agent Orchestration**: Coordinate multiple AI agents with shared semantic context -- **Framework Integration**: LangChain, LlamaIndex, CrewAI, AutoGen compatibility -- **Real-time Processing**: Stream processing for live data semantic analysis +- **๐Ÿค– Intelligent Agents** - With type-safe, validated input/output schemas +- **๐Ÿ” RAG Systems** - Enhanced with semantic chunking and enriched metadata +- **๐Ÿ•ธ๏ธ Knowledge Graphs** - Automatically extracted entities, relations, and triples +- **๐Ÿ› ๏ธ LLM Tools** - Wrapped with semantic contracts for reliable operation +- **๐Ÿ“Š Data Pipelines** - Consistent, validated data flows across your stack --- @@ -57,685 +28,545 @@ SemantiCore is an open-source toolkit that transforms raw, unstructured data int ### Installation ```bash -# Basic installation +# Install via pip pip install semanticore -# Install with all integrations +# Or install with all dependencies pip install "semanticore[all]" -# Install specific providers -pip install "semanticore[openai,anthropic,neo4j,pinecone]" - -# Available extras: openai, anthropic, google, huggingface, neo4j, kuzu, -# pinecone, milvus, weaviate, chroma, langchain, llamaindex, crewai +# Development installation +git clone https://github.com/yourusername/semanticore.git +cd semanticore +pip install -e ".[dev]" ``` -### 30-Second Demo +### Basic Usage ```python from semanticore import SemantiCore -# Initialize with your preferred providers -core = SemantiCore( - llm_provider="openai", - embedding_model="text-embedding-3-large", - vector_store="pinecone", - graph_db="neo4j" -) +# Initialize the core engine +core = SemantiCore() -# Transform unstructured text into semantic knowledge +# Extract semantic information from text text = """ -Tesla reported Q4 2024 earnings with $25.2B revenue, a 15% increase year-over-year. -CEO Elon Musk highlighted the success of the Model Y and expansion in the Chinese market. -The company plans to launch three new models in 2025, including the long-awaited Cybertruck. +OpenAI released GPT-4 in March 2023, which significantly improved +reasoning capabilities over GPT-3.5. The model was trained using +reinforcement learning from human feedback (RLHF). """ -# Extract semantic information -result = core.extract_semantics(text) +# One-line semantic extraction +result = core.extract(text) -print("Entities:", result.entities) -# [Entity(name="Tesla", type="ORGANIZATION"), Entity(name="Elon Musk", type="PERSON")] - -print("Relationships:", result.relationships) -# [Relation(subject="Tesla", predicate="reported", object="Q4 2024 earnings")] - -print("Events:", result.events) -# [Event(type="EARNINGS_REPORT", date="Q4 2024", amount="$25.2B")] - -# Generate knowledge graph -knowledge_graph = core.build_knowledge_graph(text) -print("Graph nodes:", len(knowledge_graph.nodes)) -print("Graph edges:", len(knowledge_graph.edges)) +print(result.entities) # [Entity(name="OpenAI", type="ORGANIZATION"), ...] +print(result.relations) # [Relation(subject="OpenAI", predicate="released", object="GPT-4"), ...] +print(result.schema) # Auto-generated Pydantic schema +print(result.metadata) # Enriched contextual information ``` --- -## ๐Ÿ”ง Integration Examples +## ๐Ÿงฉ Core Features -### ๐Ÿค– LLM Provider Integration +### ๐Ÿง  Semantic Extraction Engine + +Advanced NLP pipeline that extracts meaningful structure from unstructured data: ```python -from semanticore.llm import LLMProvider +from semanticore.extract import EntityExtractor, RelationExtractor, TopicClassifier -# OpenAI Integration -openai_provider = LLMProvider( - provider="openai", - model="gpt-4-turbo", - api_key="your-openai-key" +# Named Entity Recognition with custom models +extractor = EntityExtractor( + model="en_core_web_trf", # spaCy model + custom_labels=["MALWARE", "THREAT_ACTOR", "VULNERABILITY"] ) -# Anthropic Integration -anthropic_provider = LLMProvider( - provider="anthropic", - model="claude-3-opus-20240229", - api_key="your-anthropic-key" -) +entities = extractor.extract("APT29 used FrostBite malware against critical infrastructure") -# Google Gemini Integration -gemini_provider = LLMProvider( - provider="google", - model="gemini-pro", - api_key="your-google-key" -) +# Relation and Triple Extraction +rel_extractor = RelationExtractor(llm_provider="openai") +relations = rel_extractor.extract_relations(text, entities) -# Hugging Face Integration -hf_provider = LLMProvider( - provider="huggingface", - model="mistralai/Mistral-7B-Instruct-v0.1", - api_key="your-hf-key" -) - -# Local Model Integration -local_provider = LLMProvider( - provider="local", - model_path="/path/to/model", - device="cuda" -) - -# Use with SemantiCore -core = SemantiCore(llm_provider=openai_provider) +# Topic Classification and Categorization +classifier = TopicClassifier() +topics = classifier.classify(text, categories=["cybersecurity", "technology", "politics"]) ``` -### ๐Ÿ•ธ๏ธ Knowledge Graph Database Integration +### ๐Ÿงฑ Dynamic Schema Generation + +Automatically generate type-safe schemas from extracted data: ```python -from semanticore.graph import GraphDatabase +from semanticore.schema import SchemaGenerator, validate_data -# Neo4j Integration -neo4j_db = GraphDatabase( - provider="neo4j", +# Generate Pydantic models from extracted entities +generator = SchemaGenerator() +schema = generator.from_entities(entities) + +# Export to various formats +schema.to_pydantic() # Python Pydantic model +schema.to_json_schema() # JSON Schema +schema.to_yaml() # YAML Schema +schema.to_typescript() # TypeScript interfaces + +# Validate new data against generated schema +is_valid = validate_data(new_data, schema) +``` + +### ๐Ÿ”Œ Universal Connectors + +Seamlessly connect to any data source: + +```python +from semanticore.connectors import FileConnector, WebConnector, APIConnector + +# File processing (PDF, DOCX, CSV, JSON, Markdown) +file_conn = FileConnector() +documents = file_conn.load("./documents/*.pdf") +semantic_docs = core.process_documents(documents) + +# Web scraping and RSS feeds +web_conn = WebConnector() +pages = web_conn.scrape_urls(["https://example.com/news"]) +web_semantics = core.extract_from_web(pages) + +# REST API integration +api_conn = APIConnector(base_url="https://api.example.com") +api_data = api_conn.fetch("/endpoints") +structured_data = core.structure_api_response(api_data) +``` + +### ๐Ÿงช Validation & Quality Assurance + +Ensure data quality and consistency across your pipeline: + +```python +from semanticore.validation import SchemaValidator, ConsistencyChecker, QualityMetrics + +# Schema validation +validator = SchemaValidator(schema) +validation_result = validator.validate(data) + +if not validation_result.is_valid: + print(f"Validation errors: {validation_result.errors}") + +# Consistency checking across multiple extractions +checker = ConsistencyChecker() +consistency_score = checker.check_consistency([result1, result2, result3]) + +# Quality metrics and confidence scoring +metrics = QualityMetrics() +quality_report = metrics.assess(extraction_result) +print(f"Extraction confidence: {quality_report.confidence}") +``` + +### ๐Ÿ“ Intelligent Chunking & Embedding + +RAG-optimized document processing with semantic awareness: + +```python +from semanticore.vectorizer import SemanticChunker, EmbeddingEngine + +# Semantic-aware chunking +chunker = SemanticChunker( + chunk_size=512, + overlap=50, + respect_boundaries=True, # Don't split entities/relations + add_metadata=True +) + +chunks = chunker.chunk_document(document, semantic_info=result) + +# Multi-modal embedding support +embedder = EmbeddingEngine( + provider="sentence-transformers", # or "openai", "huggingface" + model="all-MiniLM-L6-v2" +) + +embedded_chunks = embedder.embed_chunks(chunks) + +# Direct vector database integration +from semanticore.vector_stores import FAISSStore, PineconeStore + +store = FAISSStore() +store.add_embeddings(embedded_chunks) +``` + +### ๐Ÿ“š Knowledge Graph Export + +Transform extracted semantics into graph databases: + +```python +from semanticore.kg import Neo4jExporter, RDFExporter, KuzuExporter + +# Neo4j export with Cypher generation +neo4j_exporter = Neo4jExporter( uri="bolt://localhost:7687", - username="neo4j", + user="neo4j", password="password" ) -# KuzuDB Integration (Embedded Graph Database) -kuzu_db = GraphDatabase( - provider="kuzu", - database_path="/path/to/kuzu/db" -) +# Create nodes and relationships +neo4j_exporter.export_entities(entities) +neo4j_exporter.export_relations(relations) -# ArangoDB Integration -arango_db = GraphDatabase( - provider="arangodb", - host="localhost", - port=8529, - username="root", - password="password" -) +# RDF triple export +rdf_exporter = RDFExporter(format="turtle") +triples = rdf_exporter.to_triples(entities, relations) -# Amazon Neptune Integration -neptune_db = GraphDatabase( - provider="neptune", - endpoint="your-neptune-endpoint.amazonaws.com", - port=8182, - region="us-east-1" -) +# Query the generated knowledge graph +from semanticore.kg.query import GraphQuerier -# Build knowledge graph -from semanticore import SemantiCore - -core = SemantiCore(graph_db=neo4j_db) -documents = ["doc1.txt", "doc2.txt", "doc3.txt"] - -# Automatically extract entities and relationships, build graph -knowledge_graph = core.build_knowledge_graph_from_documents(documents) -print(f"Created graph with {knowledge_graph.node_count} nodes and {knowledge_graph.edge_count} edges") +querier = GraphQuerier(neo4j_exporter) +results = querier.cypher("MATCH (n:ORGANIZATION)-[r:RELEASED]->(m:PRODUCT) RETURN n, r, m") ``` -### ๐Ÿ“Š Vector Store Integration +### ๐Ÿ“ก Semantic Routing + +Intelligently route queries and tasks to appropriate handlers: ```python -from semanticore.vector import VectorStore +from semanticore.routing import SemanticRouter, IntentClassifier -# Pinecone Integration -pinecone_store = VectorStore( - provider="pinecone", - api_key="your-pinecone-key", - environment="us-west1-gcp", - index_name="semanticore-index" +# Set up routing rules +router = SemanticRouter() + +# Intent-based routing +router.add_intent_route("question_answering", qa_agent) +router.add_intent_route("data_extraction", extraction_pipeline) +router.add_intent_route("summarization", summary_agent) + +# Keyword and pattern-based routing +router.add_keyword_route(["threat", "malware", "vulnerability"], security_agent) +router.add_pattern_route(r"CVE-\d{4}-\d+", vulnerability_lookup) + +# LLM-powered semantic routing +router.add_semantic_route( + description="Handle complex analytical queries about financial data", + handler=financial_analysis_agent, + examples=["What's the trend in quarterly revenue?", "Analyze the risk factors"] ) -# Milvus Integration -milvus_store = VectorStore( - provider="milvus", - host="localhost", - port=19530, - collection_name="semantic_embeddings" -) - -# Weaviate Integration -weaviate_store = VectorStore( - provider="weaviate", - url="http://localhost:8080", - class_name="SemanticChunk" -) - -# Chroma Integration -chroma_store = VectorStore( - provider="chroma", - persist_directory="/path/to/chroma/db", - collection_name="documents" -) - -# FAISS Integration (Local) -faiss_store = VectorStore( - provider="faiss", - index_path="/path/to/faiss/index", - dimension=1536 -) - -# Use with SemantiCore for RAG -core = SemantiCore( - vector_store=pinecone_store, - embedding_model="text-embedding-3-large" -) - -# Semantic chunking and embedding -chunks = core.semantic_chunk_documents(documents) -embeddings = core.embed_chunks(chunks) -vector_store.store_embeddings(chunks, embeddings) - -# Semantic search -query = "What are the latest AI developments?" -results = core.semantic_search(query, top_k=5) -``` - -### ๐Ÿ”— Framework Integration - -```python -# LangChain Integration -from semanticore.integrations.langchain import SemanticChain -from langchain.chains import ConversationalRetrievalChain - -semantic_chain = SemanticChain( - semanticore_instance=core, - retriever_type="semantic", - context_engineering=True -) - -langchain_chain = ConversationalRetrievalChain( - retriever=semantic_chain.as_retriever(), - memory=semantic_chain.get_memory(), - return_source_documents=True -) - -# LlamaIndex Integration -from semanticore.integrations.llamaindex import SemanticIndex -from llama_index import VectorStoreIndex - -semantic_index = SemanticIndex( - semanticore_instance=core, - enable_semantic_routing=True -) - -llama_index = VectorStoreIndex.from_vector_store( - semantic_index.get_vector_store() -) - -# CrewAI Integration -from semanticore.integrations.crewai import SemanticCrew -from crewai import Agent, Task, Crew - -# Create semantic-aware agents -researcher = Agent( - role='Research Analyst', - goal='Analyze semantic patterns in data', - backstory='Expert in semantic data analysis', - semantic_memory=core.get_semantic_memory() -) - -writer = Agent( - role='Content Writer', - goal='Create semantic-rich content', - backstory='Specialist in semantic content creation', - semantic_memory=core.get_semantic_memory() -) - -# Create semantic crew -semantic_crew = SemanticCrew( - agents=[researcher, writer], - semantic_coordination=True, - knowledge_sharing=True -) +# Route incoming requests +query = "What are the latest cybersecurity threats targeting healthcare?" +handler = router.route(query) +response = handler.process(query) ``` --- -## ๐ŸŽฏ Advanced Features +## ๐ŸŽฏ Use Cases & Examples -### ๐Ÿง  Multi-Domain Semantic Processing +### ๐Ÿ” Cybersecurity Threat Intelligence ```python -from semanticore.domains import CybersecurityProcessor, FinanceProcessor, HealthcareProcessor +from semanticore.domains.cyber import ThreatIntelExtractor -# Cybersecurity semantic processing -cyber_processor = CybersecurityProcessor( - threat_intelligence_feeds=["misp", "stix"], - ontology="cybersecurity.owl", - enable_threat_hunting=True -) - -# Process security incidents -incident_report = """ -APT29 exploited CVE-2024-1234 in Microsoft Exchange to deploy Cobalt Strike. -The attack used spear-phishing emails with malicious attachments. +# Specialized cybersecurity extraction +threat_extractor = ThreatIntelExtractor() +threat_report = """ +APT29 (Cozy Bear) launched a sophisticated spear-phishing campaign +targeting US government agencies using a previously unknown malware +variant called FrostBite. The attack exploited CVE-2024-1234 in +Microsoft Exchange servers. """ -cyber_analysis = cyber_processor.analyze(incident_report) -print("Threat Actors:", cyber_analysis.threat_actors) -print("Vulnerabilities:", cyber_analysis.vulnerabilities) -print("Attack Techniques:", cyber_analysis.mitre_techniques) +intel = threat_extractor.extract(threat_report) +print(intel.threat_actors) # ["APT29", "Cozy Bear"] +print(intel.malware) # ["FrostBite"] +print(intel.vulnerabilities) # ["CVE-2024-1234"] +print(intel.attack_patterns) # ["spear-phishing", "server exploitation"] -# Financial semantic processing -finance_processor = FinanceProcessor( - market_data_sources=["yahoo", "alpha_vantage"], - ontology="finance.owl", - enable_sentiment_analysis=True -) - -# Healthcare semantic processing -health_processor = HealthcareProcessor( - medical_ontologies=["snomed", "icd10"], - enable_drug_interaction_detection=True -) +# Export to STIX format for threat intelligence platforms +stix_bundle = intel.to_stix() ``` -### ๐ŸŽฏ Context Engineering for RAG +### ๐Ÿงฌ Biomedical Research Assistant ```python -from semanticore.context import ContextEngineer +from semanticore.domains.biomedical import BiomedicalExtractor -# Advanced context engineering -context_engineer = ContextEngineer( - max_context_length=128000, - compression_strategy="semantic_preservation", - relevance_scoring=True -) +bio_extractor = BiomedicalExtractor() +research_text = """ +The study investigated the efficacy of remdesivir in treating COVID-19 +patients. Results showed a 31% reduction in recovery time compared to +placebo (p<0.001). Side effects included nausea in 12% of patients. +""" -# Optimize context for specific queries -query = "How can we improve cloud security against APT attacks?" -documents = load_security_documents() +bio_data = bio_extractor.extract(research_text) +print(bio_data.drugs) # ["remdesivir"] +print(bio_data.conditions) # ["COVID-19"] +print(bio_data.outcomes) # ["31% reduction in recovery time"] +print(bio_data.side_effects) # ["nausea"] -# Intelligent context compression -optimized_context = context_engineer.optimize_context( - query=query, - documents=documents, - preserve_entities=True, - maintain_relationships=True, - compression_ratio=0.3 # 70% reduction while preserving meaning -) - -print(f"Context compressed from {len(documents)} to {len(optimized_context)} tokens") -print(f"Semantic preservation: {context_engineer.preservation_score:.2%}") +# Generate structured clinical data +clinical_schema = bio_data.to_clinical_schema() ``` -### ๐Ÿ”„ Real-time Semantic Processing +### ๐Ÿ“Š Financial Document Analysis ```python -from semanticore.streaming import SemanticStreamProcessor +from semanticore.domains.finance import FinancialExtractor -# Real-time semantic processing -stream_processor = SemanticStreamProcessor( - input_streams=["kafka://events", "websocket://feeds"], - processing_pipeline=[ - "entity_extraction", - "relationship_detection", - "ontology_mapping", - "knowledge_graph_update" - ], - batch_size=100, - processing_interval="5s" -) +fin_extractor = FinancialExtractor() +earnings_report = """ +Q4 2024 revenue increased 15% YoY to $2.3B, driven by strong performance +in the cloud computing segment. Operating margin improved to 23.5% from +21.2% in the prior year. The company announced a $1B share buyback program. +""" -# Process streaming data -async for semantic_event in stream_processor.process(): - if semantic_event.confidence > 0.8: - # Update knowledge graph - core.update_knowledge_graph(semantic_event) - - # Trigger alerts if needed - if semantic_event.importance == "critical": - await alert_system.send_alert(semantic_event) -``` +financial_data = fin_extractor.extract(earnings_report) +print(financial_data.metrics) # {"revenue": "$2.3B", "margin": "23.5%"} +print(financial_data.periods) # ["Q4 2024"] +print(financial_data.events) # ["$1B share buyback program"] -### ๐Ÿ”€ Semantic Routing & Orchestration - -```python -from semanticore.routing import SemanticRouter - -# Multi-dimensional semantic routing -router = SemanticRouter( - routing_dimensions=["intent", "domain", "complexity", "urgency"], - agents={ - "security_analyst": SecurityAgent(), - "data_scientist": DataScienceAgent(), - "business_analyst": BusinessAgent() - } -) - -# Route queries to appropriate agents -query = "Analyze the security implications of our latest data breach" -routed_agent = router.route_query(query) -response = routed_agent.process(query) +# Export to financial analysis tools +financial_json = financial_data.to_standardized_json() ``` --- -## ๐Ÿ—๏ธ Architecture & Deployment +## ๐Ÿ”ง Advanced Configuration -### ๐Ÿข Enterprise Architecture +### Custom Model Integration ```python -from semanticore.enterprise import SemanticEnterprise +from semanticore.models import CustomLLMProvider -# Enterprise-grade deployment -enterprise = SemanticEnterprise( - deployment_mode="distributed", - scaling_strategy="auto", - monitoring_enabled=True, - security_features=[ - "encryption_at_rest", - "encryption_in_transit", - "access_control", - "audit_logging" - ] -) +# Integrate your own models +class MyCustomLLM(CustomLLMProvider): + def __init__(self, model_path): + self.model = load_model(model_path) + + def extract_entities(self, text): + return self.model.predict(text) -# Multi-tenant configuration -enterprise.configure_tenants({ - "healthcare_org": { - "compliance": ["hipaa", "gdpr"], - "ontology": "healthcare.owl", - "data_classification": "sensitive" - }, - "finance_org": { - "compliance": ["sox", "pci_dss"], - "ontology": "finance.owl", - "data_classification": "confidential" - } -}) +# Use custom model in SemantiCore +core = SemantiCore(llm_provider=MyCustomLLM("./my_model")) ``` -### โ˜๏ธ Cloud Deployment - -```yaml -# docker-compose.yml -version: '3.8' -services: - semanticore: - image: semanticore:latest - environment: - - SEMANTICORE_MODE=production - - OPENAI_API_KEY=${OPENAI_API_KEY} - - NEO4J_URI=${NEO4J_URI} - - PINECONE_API_KEY=${PINECONE_API_KEY} - ports: - - "8000:8000" - volumes: - - ./ontologies:/app/ontologies - - ./models:/app/models - depends_on: - - neo4j - - redis - - neo4j: - image: neo4j:latest - environment: - - NEO4J_AUTH=neo4j/password - ports: - - "7474:7474" - - "7687:7687" - - redis: - image: redis:alpine - ports: - - "6379:6379" -``` - -### ๐Ÿš€ Kubernetes Deployment - -```yaml -# k8s/deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: semanticore -spec: - replicas: 3 - selector: - matchLabels: - app: semanticore - template: - metadata: - labels: - app: semanticore - spec: - containers: - - name: semanticore - image: semanticore:latest - ports: - - containerPort: 8000 - env: - - name: SEMANTICORE_MODE - value: "production" - - name: DISTRIBUTED_PROCESSING - value: "true" - resources: - requests: - memory: "2Gi" - cpu: "1" - limits: - memory: "4Gi" - cpu: "2" -``` - ---- - -## ๐ŸŽ“ Examples & Use Cases - -### ๐Ÿ” Cybersecurity Intelligence +### Pipeline Customization ```python -# Threat intelligence analysis -threat_data = """ -New malware family 'StealthBot' discovered targeting financial institutions. -Uses advanced evasion techniques and communicates with C2 servers via encrypted channels. -Initial infection vector appears to be phishing emails with malicious PDF attachments. -""" +from semanticore.pipeline import Pipeline, Step -# Extract threat intelligence -threat_analysis = core.extract_threat_intelligence(threat_data) -print("Malware Family:", threat_analysis.malware_families) -print("Attack Vectors:", threat_analysis.attack_vectors) -print("Indicators:", threat_analysis.iocs) +# Build custom processing pipeline +pipeline = Pipeline([ + Step("preprocess", text_cleaner), + Step("extract_entities", entity_extractor), + Step("extract_relations", relation_extractor), + Step("enrich_metadata", metadata_enricher), + Step("validate", schema_validator), + Step("export", knowledge_graph_exporter) +]) -# Update threat knowledge graph -core.update_threat_landscape(threat_analysis) +# Process data through pipeline +results = pipeline.run(input_data) ``` -### ๐Ÿ“Š Financial Analysis +### Configuration Management ```python -# Market sentiment analysis -financial_news = """ -Tesla's Q4 earnings beat expectations with record deliveries. -Stock surged 12% in after-hours trading as investors responded positively -to the company's guidance for 2025 production targets. -""" - -# Extract financial insights -financial_analysis = core.extract_financial_semantics(financial_news) -print("Companies:", financial_analysis.companies) -print("Financial Metrics:", financial_analysis.metrics) -print("Sentiment:", financial_analysis.sentiment) -print("Market Impact:", financial_analysis.market_impact) -``` - -### ๐Ÿงฌ Research Intelligence - -```python -# Scientific literature analysis -research_paper = """ -Our study demonstrates that CRISPR-Cas9 gene editing can effectively -target oncogenes in pancreatic cancer cells, showing 85% reduction -in tumor growth in mouse models. -""" - -# Extract research insights -research_analysis = core.extract_research_semantics(research_paper) -print("Techniques:", research_analysis.techniques) -print("Findings:", research_analysis.findings) -print("Entities:", research_analysis.biological_entities) -print("Relationships:", research_analysis.causal_relationships) -``` - ---- - -## ๐Ÿ› ๏ธ Configuration - -### โš™๏ธ Configuration File - -```yaml # semanticore.yaml -llm: - provider: "openai" - model: "gpt-4-turbo" - api_key: "$OPENAI_API_KEY" - temperature: 0.1 - max_tokens: 4000 - -embeddings: - provider: "openai" - model: "text-embedding-3-large" - dimensions: 1536 - -vector_store: - provider: "pinecone" - api_key: "$PINECONE_API_KEY" - environment: "us-west1-gcp" - index_name: "semanticore" - -graph_database: - provider: "neo4j" - uri: "bolt://localhost:7687" - username: "neo4j" - password: "$NEO4J_PASSWORD" - -processing: - semantic_layers: ["lexical", "syntactic", "semantic", "pragmatic"] - enable_coreference_resolution: true - enable_temporal_reasoning: true - enable_causal_reasoning: true - -ontology: +extractors: + entity: + model: "en_core_web_trf" + confidence_threshold: 0.8 + relation: + llm_provider: "openai" + model: "gpt-4" + +schema: auto_generate: true - formats: ["owl", "rdf", "json-ld"] - validation: true - versioning: true -``` + validation_level: "strict" + +export: + formats: ["json", "rdf", "cypher"] + knowledge_graph: + provider: "neo4j" + batch_size: 1000 -### ๐Ÿ”ง Environment Variables - -```bash -# Core configuration -export SEMANTICORE_MODE=production -export SEMANTICORE_LOG_LEVEL=info - -# LLM providers -export OPENAI_API_KEY=your_openai_key -export ANTHROPIC_API_KEY=your_anthropic_key -export GOOGLE_API_KEY=your_google_key - -# Vector stores -export PINECONE_API_KEY=your_pinecone_key -export MILVUS_HOST=localhost -export MILVUS_PORT=19530 - -# Graph databases -export NEO4J_URI=bolt://localhost:7687 -export NEO4J_USERNAME=neo4j -export NEO4J_PASSWORD=your_password - -# Optional: Enable specific features -export ENABLE_REAL_TIME_PROCESSING=true -export ENABLE_SEMANTIC_CACHING=true -export ENABLE_DISTRIBUTED_PROCESSING=true +# Load configuration +from semanticore.config import load_config +config = load_config("semanticore.yaml") +core = SemantiCore(config=config) ``` --- -## ๐Ÿ“š Documentation & Resources +## ๐Ÿ—๏ธ Architecture -- **๐Ÿ“– [Full Documentation](https://docs.semanticore.ai)** -- **๐Ÿš€ [Quick Start Guide](https://docs.semanticore.ai/quickstart)** -- **๐Ÿ—๏ธ [Architecture Overview](https://docs.semanticore.ai/architecture)** -- **๐Ÿ”ง [API Reference](https://api.semanticore.ai)** -- **๐Ÿ’ก [Examples Repository](https://github.com/semanticore/examples)** -- **๐ŸŒ [Community Forum](https://community.semanticore.ai)** -- **๐Ÿ“บ [Video Tutorials](https://youtube.com/semanticore)** +SemantiCore follows a modular, extensible architecture: + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ SemantiCore Engine โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Connectors โ”‚ Extractors โ”‚ Schema โ”‚ Exporters โ”‚ +โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ +โ”‚ โ”‚ File โ”‚ โ”‚ โ”‚ NER โ”‚ โ”‚ โ”‚ Pydanticโ”‚ โ”‚ โ”‚ Neo4j โ”‚ โ”‚ +โ”‚ โ”‚ Web โ”‚ โ”‚ โ”‚ Relationsโ”‚ โ”‚ โ”‚ JSON โ”‚ โ”‚ โ”‚ RDF โ”‚ โ”‚ +โ”‚ โ”‚ API โ”‚ โ”‚ โ”‚ Topics โ”‚ โ”‚ โ”‚ YAML โ”‚ โ”‚ โ”‚ Vector โ”‚ โ”‚ +โ”‚ โ”‚ DB โ”‚ โ”‚ โ”‚ LLM โ”‚ โ”‚ โ”‚ TS โ”‚ โ”‚ โ”‚ DB โ”‚ โ”‚ +โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Validation & Quality Assurance โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ Semantic Routing โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +--- + +## ๐Ÿ“‹ Requirements + +- **Python**: 3.8+ +- **Core Dependencies**: spaCy, transformers, pydantic, networkx +- **Optional Dependencies**: + - **LLM Providers**: openai, anthropic, huggingface-hub + - **Vector Databases**: faiss-cpu, pinecone-client, weaviate-client + - **Graph Databases**: neo4j, rdflib, kuzudb + - **Document Processing**: PyMuPDF, python-docx, openpyxl + +--- + +## ๐Ÿ›ฃ๏ธ Roadmap + +### ๐Ÿš€ Version 1.0 (Current) +- โœ… Core semantic extraction engine +- โœ… Schema generation and validation +- โœ… Basic connectors (file, web, API) +- โœ… Neo4j and RDF export +- โœ… Vector database integration + +### ๐Ÿ”ฎ Version 1.1 (Q3 2025) +- ๐Ÿ”„ **Web-based visual schema editor** +- ๐Ÿ”„ **Real-time streaming support** (Kafka, MQTT, WebSockets) +- ๐Ÿ”„ **Advanced semantic routing** with learning capabilities +- ๐Ÿ”„ **Multi-modal support** (images, audio, video) + +### ๐Ÿ”ฎ Version 1.2 (Q4 2025) +- โณ **Domain-specific modules** (Legal, Healthcare, Finance) +- โณ **Graph reasoning engine** with inference capabilities +- โณ **Distributed processing** support +- โณ **Model fine-tuning** utilities + +### ๐Ÿ”ฎ Version 2.0 (2026) +- โณ **Custom DSL** for semantic pipeline definition +- โณ **AutoML** for extraction model optimization +- โณ **Federated learning** across distributed deployments +- โณ **Enterprise management** console + +--- ## ๐Ÿค Contributing -We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details. +We welcome contributions from the community! Here's how you can help: -### ๐Ÿ›ฃ๏ธ Development Roadmap +### ๐Ÿ› Report Issues +Found a bug or have a feature request? [Open an issue](https://github.com/yourusername/semanticore/issues) on GitHub. -**v1.0 (Current)** -- โœ… Core semantic processing engine -- โœ… Multi-LLM integration (OpenAI, Anthropic, Google) -- โœ… Knowledge graph construction (Neo4j, KuzuDB) -- โœ… Vector store integration (Pinecone, Milvus, Weaviate) -- โœ… Ontology generation and management +### ๐Ÿ’ป Contribute Code +1. Fork the repository +2. Create a feature branch: `git checkout -b feature/amazing-feature` +3. Make your changes and add tests +4. Run the test suite: `pytest` +5. Commit your changes: `git commit -m 'Add amazing feature'` +6. Push to the branch: `git push origin feature/amazing-feature` +7. Open a Pull Request -**v1.1 (Next)** -- ๐Ÿ”„ Multimodal processing (images, audio, video) -- ๐Ÿ”„ Advanced reasoning capabilities -- ๐Ÿ”„ Real-time streaming processing -- ๐Ÿ”„ Enhanced enterprise features +### ๐Ÿ“– Improve Documentation +Help us improve our documentation by: +- Fixing typos and clarifying explanations +- Adding new examples and use cases +- Creating tutorials and guides +- Translating documentation -**v1.2 (Future)** -- ๐Ÿ”„ Federated learning capabilities -- ๐Ÿ”„ Quantum-inspired semantic processing -- ๐Ÿ”„ Advanced causal reasoning -- ๐Ÿ”„ Autonomous semantic agents - -## ๐Ÿ“„ License - -SemantiCore is released under the MIT License. See [LICENSE](LICENSE) for details. - -## ๐Ÿ™ Acknowledgments - -- Built with โค๏ธ by the open-source community -- Inspired by the latest advances in semantic AI and knowledge representation -- Powered by cutting-edge LLM and embedding technologies +### ๐Ÿงช Testing +Help us maintain quality by: +- Writing unit tests for new features +- Testing on different platforms and Python versions +- Performance testing and optimization +- Integration testing with external services --- -**Ready to transform your unstructured data into intelligent semantic knowledge?** +## ๐Ÿ“š Documentation + +- **๐Ÿ“– [Full Documentation](https://semanticore.readthedocs.io/)** +- **๐Ÿš€ [Quick Start Guide](https://semanticore.readthedocs.io/quickstart/)** +- **๐Ÿ“‹ [API Reference](https://semanticore.readthedocs.io/api/)** +- **๐Ÿ’ก [Examples & Tutorials](https://semanticore.readthedocs.io/examples/)** +- **๐Ÿ”ง [Configuration Guide](https://semanticore.readthedocs.io/configuration/)** + +--- + +## ๐Ÿ† Community & Support + +- **๐Ÿ’ฌ [Discord Community](https://discord.gg/semanticore)** - Chat with users and developers +- **๐Ÿ“ง [Mailing List](https://groups.google.com/g/semanticore)** - Stay updated with announcements +- **๐Ÿฆ [Twitter](https://twitter.com/semanticore)** - Follow us for updates +- **๐Ÿ“บ [YouTube Channel](https://youtube.com/c/semanticore)** - Tutorials and demos +- **โ“ [Stack Overflow](https://stackoverflow.com/questions/tagged/semanticore)** - Get help with specific issues + +--- + +## ๐Ÿ“„ License + +SemantiCore is released under the **MIT License**. See the [LICENSE](LICENSE) file for details. + +``` +MIT License + +Copyright (c) 2025 SemantiCore Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +--- + +## ๐Ÿ™ Acknowledgments + +SemantiCore is built on the shoulders of giants. We thank the communities behind: + +- **๐Ÿค— Hugging Face** - For democratizing NLP and ML +- **๐ŸŒถ๏ธ spaCy** - For industrial-strength NLP +- **๐Ÿ”— Neo4j** - For graph database excellence +- **๐Ÿ Python** - For being an amazing ecosystem +- **๐Ÿง  OpenAI & Anthropic** - For advancing AI capabilities + +--- + +## ๐Ÿ“Š Project Stats + +![GitHub stars](https://img.shields.io/github/stars/yourusername/semanticore?style=social) +![GitHub forks](https://img.shields.io/github/forks/yourusername/semanticore?style=social) +![GitHub issues](https://img.shields.io/github/issues/yourusername/semanticore) +![GitHub pull requests](https://img.shields.io/github/issues-pr/yourusername/semanticore) +![PyPI downloads](https://img.shields.io/pypi/dm/semanticore) + +--- + +**Ready to transform your unstructured data into intelligent, semantic knowledge?** ```bash pip install semanticore ``` -**Get started in 30 seconds โ†’** [Quick Start Guide](https://docs.semanticore.ai/quickstart) +**[Get Started Now โ†’](https://semanticore.readthedocs.io/quickstart/)**