mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Remove trading notebooks and supply chain risk management notebook
- Deleted cookbook/use_cases/trading/01_Risk_Assessment.ipynb - Deleted cookbook/use_cases/trading/02_News_Sentiment_Analysis.ipynb - Deleted cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb - Removed empty trading directory - Updated documentation to reflect 14 cookbooks (down from 15) - Removed all references from README.md, docs/cookbook.md, docs/use-cases.md, docs/index.md, and STRATEGIES_SUMMARY.md
This commit is contained in:
KaifAhmad1
@@ -252,7 +252,7 @@ python -c "import semantica; print(semantica.__version__)"
|
||||
- [**Entity Extraction**](cookbook/introduction/05_Entity_Extraction.ipynb) - NER & Relationships
|
||||
- [**Building Knowledge Graphs**](cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) - Graph Construction
|
||||
|
||||
### Industry Use Cases (15 Cookbooks)
|
||||
### Industry Use Cases (14 Cookbooks)
|
||||
|
||||
**Domain-Specific Cookbooks** showcasing real-world applications with real data sources, advanced chunking strategies, temporal KGs, GraphRAG, and comprehensive Semantica module integration:
|
||||
|
||||
@@ -276,16 +276,11 @@ python -c "import semantica; print(semantica.__version__)"
|
||||
- [**Criminal Network Analysis**](cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) - OSINT RSS, deduplication, network centrality, graph analytics
|
||||
- [**Intelligence Analysis Orchestrator Worker**](cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) - Pipeline orchestrator, multi-source integration, conflict detection
|
||||
|
||||
#### Trading
|
||||
- [**Risk Assessment**](cookbook/use_cases/trading/01_Risk_Assessment.ipynb) - Yahoo Finance API, GraphRAG, entity-aware chunking, portfolio risk modeling
|
||||
- [**News Sentiment Analysis**](cookbook/use_cases/trading/02_News_Sentiment_Analysis.ipynb) - Financial RSS, semantic transformer chunking, enhanced GraphRAG, sentiment extraction
|
||||
|
||||
#### Renewable Energy
|
||||
- [**Energy Market Analysis**](cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) - Energy RSS, EIA API, temporal KGs, TemporalPatternDetector, trend prediction
|
||||
|
||||
#### Supply Chain
|
||||
- [**Supply Chain Data Integration**](cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) - Logistics RSS, deduplication, relationship mapping
|
||||
- [**Supply Chain Risk Management**](cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb) - Supply chain RSS, conflict detection, dependency analysis
|
||||
|
||||
> **Note:** Once published to PyPI, you'll be able to install with `pip install semantica`
|
||||
|
||||
@@ -683,7 +678,7 @@ print(f"Answer: {result.answer} | Nodes: {kg.node_count}, Edges: {kg.edge_count}
|
||||
|
||||
**Biomedical** — Drug discovery, medical literature analysis.
|
||||
|
||||
[**Explore Use Case Examples**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases) — See real-world implementations in finance, biomedical, cybersecurity, trading, and more. **15 comprehensive domain-specific cookbooks** with real data sources, advanced chunking strategies, temporal KGs, GraphRAG, and full Semantica module integration.
|
||||
[**Explore Use Case Examples**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases) — See real-world implementations in finance, biomedical, cybersecurity, and more. **14 comprehensive domain-specific cookbooks** with real data sources, advanced chunking strategies, temporal KGs, GraphRAG, and full Semantica module integration.
|
||||
|
||||
## 🔬 Advanced Features
|
||||
|
||||
|
||||
@@ -16,9 +16,6 @@
|
||||
| **Blockchain** |
|
||||
| `01_DeFi_Protocol_Intelligence` | `DuplicateDetector` (group) | `keep_last` | `relationship` | `voting` |
|
||||
| `02_Transaction_Network_Analysis` | `ClusterBuilder` (hierarchical) | `keep_most_complete` | `temporal` | `most_recent` |
|
||||
| **Trading** |
|
||||
| `01_Risk_Assessment` | `ClusterBuilder` (graph_based) | `merge_all` | `logical` | `highest_confidence` |
|
||||
| `02_News_Sentiment_Analysis` | `DuplicateDetector` (batch) | `keep_most_complete` | `type` | `voting` |
|
||||
| **Intelligence** |
|
||||
| `01_Criminal_Network_Analysis` | `EntityResolver` (fuzzy) | - | `value` | `credibility_weighted` |
|
||||
| `02_Intelligence_Analysis_Orchestrator_Worker` | `DuplicateDetector` (batch) | `merge_all` | `entity` | `voting` |
|
||||
@@ -47,10 +44,6 @@
|
||||
- **DeFi Protocols**: Group method for related protocols; keep_last for latest protocol info
|
||||
- **Transaction Networks**: Hierarchical clustering for nested groups; temporal for time-sensitive data
|
||||
|
||||
### Trading
|
||||
- **Risk Assessment**: Graph-based clustering for interconnected risks; logical conflict detection for risk rules
|
||||
- **News Sentiment**: Batch for large news datasets; voting for sentiment classification conflicts
|
||||
|
||||
### Intelligence
|
||||
- **Criminal Networks**: Fuzzy matching for intelligence data; credibility weighting for intelligence sources
|
||||
- **Intelligence Analysis**: Batch for multi-source integration; merge_all to combine all intelligence sources
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,815 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb)\n",
|
||||
"\n",
|
||||
"# Supply Chain Risk Management - Dependency Analysis & Risk Detection\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates **supply chain risk management** using Semantica with focus on **dependency analysis**, **risk pattern detection**, and **conflict resolution**. The pipeline detects risks in the supply chain by analyzing dependencies, external feeds, and resolving conflicts using reasoning and graph analytics.\n",
|
||||
"\n",
|
||||
"### Key Features\n",
|
||||
"\n",
|
||||
"- **Dependency Analysis**: Analyzes supply chain dependencies using graph reasoning\n",
|
||||
"- **Risk Pattern Detection**: Detects risk patterns in the supply chain using reasoning\n",
|
||||
"- **Conflict Detection**: Detects and resolves conflicts in risk data from multiple sources\n",
|
||||
"- **Risk Impact Analysis**: Analyzes risk impact using graph analytics\n",
|
||||
"- **Temporal Risk Tracking**: Tracks risk evolution over time\n",
|
||||
"- **External Feed Correlation**: Correlates external threat feeds with supply chain data\n",
|
||||
"\n",
|
||||
"### Learning Objectives\n",
|
||||
"\n",
|
||||
"- Understand how to detect and resolve conflicts in multi-source risk data\n",
|
||||
"- Learn to analyze supply chain dependencies using reasoning\n",
|
||||
"- Master risk pattern detection using graph reasoning\n",
|
||||
"- Explore risk impact analysis using graph analytics\n",
|
||||
"- Practice temporal risk tracking and evolution analysis\n",
|
||||
"- Analyze supply chain risks and mitigation strategies\n",
|
||||
"\n",
|
||||
"### Pipeline Flow\n",
|
||||
"\n",
|
||||
"```mermaid\n",
|
||||
"graph TD\n",
|
||||
" A[Multi-Source Risk Ingestion] --> B[Document Parsing]\n",
|
||||
" B --> C[Text Processing]\n",
|
||||
" C --> D[Entity Extraction]\n",
|
||||
" D --> E[Relationship Extraction]\n",
|
||||
" E --> F[Deduplication]\n",
|
||||
" F --> G[Conflict Detection]\n",
|
||||
" G --> H[KG Construction]\n",
|
||||
" H --> I[Embedding Generation]\n",
|
||||
" I --> J[Vector Store]\n",
|
||||
" H --> K[Dependency Analysis]\n",
|
||||
" H --> L[Risk Pattern Detection]\n",
|
||||
" H --> M[Risk Impact Analysis]\n",
|
||||
" H --> N[Temporal Risk Queries]\n",
|
||||
" J --> O[GraphRAG Queries]\n",
|
||||
" K --> P[Visualization]\n",
|
||||
" L --> P\n",
|
||||
" M --> P\n",
|
||||
" H --> Q[Export]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Configuration & Setup\n",
|
||||
"\n",
|
||||
"Configure API keys and set up constants for the supply chain risk management pipeline.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key-here\")\n",
|
||||
"\n",
|
||||
"# Configuration constants\n",
|
||||
"EMBEDDING_DIMENSION = 384\n",
|
||||
"EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
|
||||
"CHUNK_SIZE = 1000\n",
|
||||
"CHUNK_OVERLAP = 200\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Multi-Source Risk Data Ingestion\n",
|
||||
"\n",
|
||||
"Ingest supply chain risk data from multiple sources including risk RSS feeds, external threat feeds, and disruption APIs.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"os.makedirs(\"data\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"documents = []\n",
|
||||
"\n",
|
||||
"# Ingest from supply chain risk RSS feeds\n",
|
||||
"risk_feeds = [\n",
|
||||
" \"https://www.scrm.com/rss\",\n",
|
||||
" \"https://www.riskmanagement.com/rss\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for feed_url in risk_feeds:\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" feed_ingestor = FeedIngestor()\n",
|
||||
" feed_docs = feed_ingestor.ingest(feed_url, method=\"rss\")\n",
|
||||
" documents.extend(feed_docs)\n",
|
||||
" except Exception:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"# Example: Web ingestion from weather/disruption APIs (commented - requires API keys)\n",
|
||||
"# web_ingestor = WebIngestor()\n",
|
||||
"# weather_docs = web_ingestor.ingest(\"https://api.weather.com/disruptions\", method=\"api\")\n",
|
||||
"\n",
|
||||
"# Fallback: Sample risk data\n",
|
||||
"if not documents:\n",
|
||||
" risk_data = \"\"\"\n",
|
||||
" Supplier A depends on raw materials from Region R1 (high risk region).\n",
|
||||
" Disruption in Region R1 impacts Supplier A, causing supply chain risk.\n",
|
||||
" External feed: Weather alert in Region R1 may disrupt logistics.\n",
|
||||
" Risk mitigation: Identify alternative suppliers in Region R2.\n",
|
||||
" Supplier B depends on Region R1, creating dependency risk.\n",
|
||||
" Impact: High risk of supply chain disruption if Region R1 fails.\n",
|
||||
" \"\"\"\n",
|
||||
" with open(\"data/supply_chain_risks.txt\", \"w\", encoding=\"utf-8\") as f:\n",
|
||||
" f.write(risk_data)\n",
|
||||
" file_ingestor = FileIngestor()\n",
|
||||
" documents = file_ingestor.ingest(\"data/supply_chain_risks.txt\")\n",
|
||||
"\n",
|
||||
"print(f\"Ingested {len(documents)} documents from risk sources\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.parse import DocumentParser\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"parser = DocumentParser()\n",
|
||||
"\n",
|
||||
"parsed_documents = []\n",
|
||||
"for doc in documents:\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" parsed = parser.parse(\n",
|
||||
" doc.content if hasattr(doc, 'content') else str(doc),\n",
|
||||
" format=\"auto\"\n",
|
||||
" )\n",
|
||||
" parsed_documents.append(parsed)\n",
|
||||
" except Exception:\n",
|
||||
" parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n",
|
||||
"\n",
|
||||
"print(f\"Parsed {len(parsed_documents)} documents\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Text Processing\n",
|
||||
"\n",
|
||||
"Normalize risk data and split documents using relation-aware chunking to preserve dependency relationships.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.normalize import TextNormalizer\n",
|
||||
"from semantica.split import TextSplitter\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"normalizer = TextNormalizer()\n",
|
||||
"normalized_docs = []\n",
|
||||
"\n",
|
||||
"for doc in parsed_documents:\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" normalized = normalizer.normalize(\n",
|
||||
" doc if isinstance(doc, str) else str(doc),\n",
|
||||
" clean_html=True,\n",
|
||||
" normalize_entities=True,\n",
|
||||
" normalize_numbers=True,\n",
|
||||
" remove_extra_whitespace=True\n",
|
||||
" )\n",
|
||||
" normalized_docs.append(normalized)\n",
|
||||
" except Exception:\n",
|
||||
" normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n",
|
||||
"\n",
|
||||
"# Use relation-aware chunking to preserve dependency relationships\n",
|
||||
"relation_splitter = TextSplitter(\n",
|
||||
" method=\"relation_aware\",\n",
|
||||
" chunk_size=CHUNK_SIZE,\n",
|
||||
" chunk_overlap=CHUNK_OVERLAP\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"chunked_docs = []\n",
|
||||
"for doc_text in normalized_docs:\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" chunks = relation_splitter.split(doc_text)\n",
|
||||
" chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
|
||||
" except Exception:\n",
|
||||
" chunked_docs.append(doc_text)\n",
|
||||
"\n",
|
||||
"print(f\"Processed {len(chunked_docs)} relation-aware chunks\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Entity Extraction\n",
|
||||
"\n",
|
||||
"Extract supply chain risk entities including dependencies, risks, disruptions, impacts, mitigations, and regions.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NERExtractor\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"extractor = NERExtractor(\n",
|
||||
" provider=\"groq\",\n",
|
||||
" model=\"llama-3.1-8b-instant\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"entity_types = [\n",
|
||||
" \"Dependency\", \"Risk\", \"Disruption\", \"Impact\", \"Mitigation\", \"Region\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"all_entities = []\n",
|
||||
"for chunk in chunked_docs[:10]: # Limit for demo\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" entities = extractor.extract(\n",
|
||||
" chunk,\n",
|
||||
" entity_types=entity_types\n",
|
||||
" )\n",
|
||||
" all_entities.extend(entities)\n",
|
||||
" except Exception:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(all_entities)} entities\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Relationship Extraction\n",
|
||||
"\n",
|
||||
"Extract risk relationships including depends_on, causes, impacts, mitigates, and located_in.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import RelationExtractor\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"relation_extractor = RelationExtractor(\n",
|
||||
" provider=\"groq\",\n",
|
||||
" model=\"llama-3.1-8b-instant\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"relation_types = [\n",
|
||||
" \"depends_on\", \"causes\", \"impacts\",\n",
|
||||
" \"mitigates\", \"located_in\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"all_relationships = []\n",
|
||||
"for chunk in chunked_docs[:10]: # Limit for demo\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" relationships = relation_extractor.extract(\n",
|
||||
" chunk,\n",
|
||||
" relation_types=relation_types\n",
|
||||
" )\n",
|
||||
" all_relationships.extend(relationships)\n",
|
||||
" except Exception:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(all_relationships)} relationships\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Deduplication\n",
|
||||
"\n",
|
||||
"Deduplicate risk entities to ensure accurate risk analysis.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.deduplication import DuplicateDetector\n",
|
||||
"\n",
|
||||
"detector = DuplicateDetector()\n",
|
||||
"\n",
|
||||
"# Deduplicate entities\n",
|
||||
"risks = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n",
|
||||
"regions = [e for e in all_entities if e.get(\"type\") == \"Region\"]\n",
|
||||
"\n",
|
||||
"risk_duplicates = detector.detect_duplicates(risks, threshold=0.9)\n",
|
||||
"region_duplicates = detector.detect_duplicates(regions, threshold=0.85)\n",
|
||||
"\n",
|
||||
"deduplicated_risks = detector.resolve_duplicates(risks, risk_duplicates)\n",
|
||||
"deduplicated_regions = detector.resolve_duplicates(regions, region_duplicates)\n",
|
||||
"\n",
|
||||
"# Update entities list\n",
|
||||
"all_entities = [e for e in all_entities if e.get(\"type\") not in [\"Risk\", \"Region\"]]\n",
|
||||
"all_entities.extend(deduplicated_risks)\n",
|
||||
"all_entities.extend(deduplicated_regions)\n",
|
||||
"\n",
|
||||
"print(f\"Deduplicated: {len(risks)} -> {len(deduplicated_risks)} risks\")\n",
|
||||
"print(f\"Deduplicated: {len(regions)} -> {len(deduplicated_regions)} regions\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Conflict Detection\n",
|
||||
"\n",
|
||||
"Detect conflicts in risk data from multiple sources. This is unique to this notebook and critical for ensuring data quality in risk management.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.conflicts import ConflictDetector\n",
|
||||
"\n",
|
||||
"conflict_detector = ConflictDetector()\n",
|
||||
"\n",
|
||||
"# Detect conflicts in risk data\n",
|
||||
"conflicts = conflict_detector.detect_conflicts(\n",
|
||||
" entities=all_entities,\n",
|
||||
" relationships=all_relationships\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f\"Detected {len(conflicts)} conflicts in risk data\")\n",
|
||||
"\n",
|
||||
"# Resolve conflicts using highest confidence strategy\n",
|
||||
"if conflicts:\n",
|
||||
" resolved = conflict_detector.resolve_conflicts(\n",
|
||||
" conflicts,\n",
|
||||
" strategy=\"highest_confidence\"\n",
|
||||
" )\n",
|
||||
" print(f\"Resolved {len(resolved)} conflicts\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Knowledge Graph Construction\n",
|
||||
"\n",
|
||||
"Build a knowledge graph from risk entities and relationships to enable dependency and risk analysis.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import GraphBuilder\n",
|
||||
"\n",
|
||||
"builder = GraphBuilder()\n",
|
||||
"\n",
|
||||
"kg = builder.build(\n",
|
||||
" entities=all_entities,\n",
|
||||
" relationships=all_relationships\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f\"Built KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Embedding Generation & Vector Store\n",
|
||||
"\n",
|
||||
"Generate embeddings for risk documents and store them in a vector database for semantic search.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.embeddings import EmbeddingGenerator\n",
|
||||
"from semantica.vector_store import VectorStore\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"embedding_gen = EmbeddingGenerator(\n",
|
||||
" model_name=EMBEDDING_MODEL,\n",
|
||||
" dimension=EMBEDDING_DIMENSION\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Generate embeddings for chunks\n",
|
||||
"embeddings = []\n",
|
||||
"for chunk in chunked_docs[:20]: # Limit for demo\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" embedding = embedding_gen.generate(chunk)\n",
|
||||
" embeddings.append(embedding)\n",
|
||||
" except Exception:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"# Create vector store\n",
|
||||
"vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
|
||||
"\n",
|
||||
"# Add embeddings to vector store\n",
|
||||
"for i, (chunk, embedding) in enumerate(zip(chunked_docs[:20], embeddings)):\n",
|
||||
" try:\n",
|
||||
" vector_store.add(\n",
|
||||
" id=str(i),\n",
|
||||
" embedding=embedding,\n",
|
||||
" metadata={\"text\": chunk[:100]} # Store first 100 chars\n",
|
||||
" )\n",
|
||||
" except Exception:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Dependency Analysis\n",
|
||||
"\n",
|
||||
"Analyze supply chain dependencies using reasoning to identify dependency patterns. This is unique to this notebook and critical for risk management.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"reasoner = Reasoner(kg)\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" # Add rules for dependency analysis\n",
|
||||
" rules = [\n",
|
||||
" \"IF Supplier depends_on Region AND Region has Risk THEN Dependency creates_risk\",\n",
|
||||
" \"IF Dependency depends_on Region AND Region has Disruption THEN Dependency causes_impact\",\n",
|
||||
" \"IF Supplier depends_on Dependency AND Dependency has Risk THEN Supplier has_risk\"\n",
|
||||
" ]\n",
|
||||
" \n",
|
||||
" for rule in rules:\n",
|
||||
" reasoner.add_rule(rule)\n",
|
||||
" \n",
|
||||
" # Find dependency patterns\n",
|
||||
" dependency_patterns = reasoner.find_patterns(pattern_type=\"dependency\")\n",
|
||||
" print(f\"Detected {len(dependency_patterns)} dependency patterns\")\n",
|
||||
" \n",
|
||||
" # Infer dependency risks\n",
|
||||
" inferred_dependencies = reasoner.infer_facts()\n",
|
||||
" print(f\"Inferred {len(inferred_dependencies)} dependency relationships\")\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Dependency analysis completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Risk Pattern Detection\n",
|
||||
"\n",
|
||||
"Detect risk patterns in the supply chain using reasoning. This is unique to this notebook and enables proactive risk identification.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.reasoning import Reasoner\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" # Add rules for risk pattern detection\n",
|
||||
" risk_rules = [\n",
|
||||
" \"IF Region has Disruption AND Supplier depends_on Region THEN Risk impacts Supplier\",\n",
|
||||
" \"IF Disruption causes Impact AND Impact affects Supplier THEN Risk requires Mitigation\",\n",
|
||||
" \"IF Risk located_in Region AND Region has Disruption THEN Risk severity increases\"\n",
|
||||
" ]\n",
|
||||
" \n",
|
||||
" for rule in risk_rules:\n",
|
||||
" reasoner.add_rule(rule)\n",
|
||||
" \n",
|
||||
" # Find risk patterns\n",
|
||||
" risk_patterns = reasoner.find_patterns(pattern_type=\"risk\")\n",
|
||||
" print(f\"Detected {len(risk_patterns)} risk patterns\")\n",
|
||||
" \n",
|
||||
" # Identify high-risk dependencies\n",
|
||||
" high_risk = [e for e in all_entities if e.get(\"type\") == \"Risk\" and \"high\" in str(e).lower()]\n",
|
||||
" print(f\"Identified {len(high_risk)} high-risk items\")\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Risk pattern detection completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Risk Impact Analysis\n",
|
||||
"\n",
|
||||
"Analyze risk impact using graph analytics. This is unique to this notebook and helps assess the severity of supply chain risks.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import GraphAnalyzer\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"graph_analyzer = GraphAnalyzer(kg)\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" # Analyze graph structure for risk impact\n",
|
||||
" stats = graph_analyzer.get_statistics()\n",
|
||||
" print(f\"Graph statistics: {stats.get('num_nodes', 0)} nodes, {stats.get('num_edges', 0)} edges\")\n",
|
||||
" \n",
|
||||
" # Find paths between risks and impacts\n",
|
||||
" if all_entities:\n",
|
||||
" risk_entities = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n",
|
||||
" impact_entities = [e for e in all_entities if e.get(\"type\") == \"Impact\"]\n",
|
||||
" if risk_entities and impact_entities:\n",
|
||||
" source = risk_entities[0].get(\"name\", \"\")\n",
|
||||
" target = impact_entities[0].get(\"name\", \"\") if impact_entities else \"\"\n",
|
||||
" if source and target:\n",
|
||||
" impact_paths = graph_analyzer.find_paths(source=source, target=target, max_length=3)\n",
|
||||
" print(f\"Found {len(impact_paths)} paths between risk and impact\")\n",
|
||||
" \n",
|
||||
" # Analyze connectivity for risk propagation\n",
|
||||
" impacts = [e for e in all_entities if e.get(\"type\") == \"Impact\"]\n",
|
||||
" print(f\"Analyzed impact for {len(impacts)} risk impacts\")\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Risk impact analysis completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Temporal Risk Queries\n",
|
||||
"\n",
|
||||
"Query the knowledge graph to track risk evolution over time and analyze temporal risk patterns.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import TemporalGraphQuery\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"temporal_query = TemporalGraphQuery(kg)\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" # Query risk evolution over time\n",
|
||||
" if all_entities:\n",
|
||||
" risk_entities = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n",
|
||||
" if risk_entities:\n",
|
||||
" risk_id = risk_entities[0].get(\"name\", \"\")\n",
|
||||
" if risk_id:\n",
|
||||
" history = temporal_query.query_temporal_paths(\n",
|
||||
" source=risk_id,\n",
|
||||
" time_range=(None, None)\n",
|
||||
" )\n",
|
||||
" print(f\"Retrieved temporal history for risk: {risk_id}\")\n",
|
||||
" \n",
|
||||
" # Query evolution of risks over time\n",
|
||||
" evolution = temporal_query.query_evolution(\n",
|
||||
" entity_type=\"Risk\",\n",
|
||||
" time_granularity=\"day\"\n",
|
||||
" )\n",
|
||||
" print(f\"Analyzed risk evolution over time\")\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Temporal risk queries completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## GraphRAG Queries\n",
|
||||
"\n",
|
||||
"Use hybrid retrieval combining vector search and graph traversal to answer complex risk management questions.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.context import AgentContext\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"agent_context = AgentContext(\n",
|
||||
" vector_store=vector_store,\n",
|
||||
" knowledge_graph=kg\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"queries = [\n",
|
||||
" \"What are the high-risk dependencies in the supply chain?\",\n",
|
||||
" \"Which regions have supply chain disruptions?\",\n",
|
||||
" \"What mitigation strategies are available for Region R1 risks?\",\n",
|
||||
" \"What impacts do disruptions in Region R1 have on suppliers?\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for query in queries:\n",
|
||||
" try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" results = agent_context.query(\n",
|
||||
" query=query,\n",
|
||||
" top_k=5\n",
|
||||
" )\n",
|
||||
" print(f\"Query: {query}\")\n",
|
||||
" print(f\"Found {len(results.get('results', []))} relevant results\")\n",
|
||||
" except Exception:\n",
|
||||
" pass\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Visualization\n",
|
||||
"\n",
|
||||
"Visualize the supply chain risk knowledge graph to explore dependencies, risks, and mitigation strategies.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"visualizer = KGVisualizer()\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" visualizer.visualize(\n",
|
||||
" kg,\n",
|
||||
" output_path=\"supply_chain_risk_kg.html\",\n",
|
||||
" layout=\"force_directed\"\n",
|
||||
" )\n",
|
||||
" print(\"Knowledge graph visualization saved to supply_chain_risk_kg.html\")\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Visualization completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Export\n",
|
||||
"\n",
|
||||
"Export the knowledge graph in multiple formats for risk management reports and further analysis.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.export import GraphExporter\n",
|
||||
"from contextlib import redirect_stderr\n",
|
||||
"from io import StringIO\n",
|
||||
"\n",
|
||||
"exporter = GraphExporter()\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" with redirect_stderr(StringIO()):\n",
|
||||
" # Export as JSON\n",
|
||||
" exporter.export(kg, format=\"json\", output_path=\"supply_chain_risk_kg.json\")\n",
|
||||
" \n",
|
||||
" # Export as GraphML\n",
|
||||
" exporter.export(kg, format=\"graphml\", output_path=\"supply_chain_risk_kg.graphml\")\n",
|
||||
" \n",
|
||||
" # Export as CSV (for risk management reports)\n",
|
||||
" exporter.export(kg, format=\"csv\", output_path=\"supply_chain_risk_kg.csv\")\n",
|
||||
" \n",
|
||||
" print(\"Exported knowledge graph in JSON, GraphML, and CSV formats\")\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Export completed\")\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
|
||||
Supplier A provides Product X to Warehouse W1 located in City C1.
|
||||
Supplier B provides Product Y to Warehouse W2 located in City C2.
|
||||
Route R1 connects Warehouse W1 to Distribution Center D1.
|
||||
Route R2 connects Warehouse W2 to Distribution Center D2.
|
||||
Logistics: Product X shipped via Route R1 from W1 to D1.
|
||||
Logistics: Product Y shipped via Route R2 from W2 to D2.
|
||||
Warehouse W1 manages inventory for Product X.
|
||||
Distribution Center D1 serves Region R1.
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
|
||||
http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
|
||||
|
||||
<key id="type" for="node" attr.name="type" attr.type="string"/>
|
||||
<key id="confidence" for="node" attr.name="confidence" attr.type="double"/>
|
||||
|
||||
<graph id="G" edgedefault="directed">
|
||||
|
||||
|
||||
</graph>
|
||||
</graphml>
|
||||
@@ -0,0 +1,838 @@
|
||||
{
|
||||
"entities": [
|
||||
{
|
||||
"id": "Maersk",
|
||||
"name": "Maersk",
|
||||
"type": "PRODUCT",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Mediterranean",
|
||||
"name": "Mediterranean",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Mothership",
|
||||
"name": "Mothership",
|
||||
"type": "PRODUCT",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "CapEx",
|
||||
"name": "CapEx",
|
||||
"type": "PRODUCT",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Escort",
|
||||
"name": "Escort",
|
||||
"type": "PRODUCT",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Europe",
|
||||
"name": "Europe",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Entity(text='the East Coast'",
|
||||
"name": "Entity(text='the East Coast'",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "the East Coast",
|
||||
"name": "the East Coast",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Asia",
|
||||
"name": "Asia",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Zebra",
|
||||
"name": "Zebra",
|
||||
"type": "PRODUCT",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "IntraMove",
|
||||
"name": "IntraMove",
|
||||
"type": "PRODUCT",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Latin America",
|
||||
"name": "Latin America",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "Pacific-Norfolk Southern'",
|
||||
"name": "Pacific-Norfolk Southern'",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "North America",
|
||||
"name": "North America",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"id": "end_char=13714",
|
||||
"name": "end_char=13714",
|
||||
"type": "LOC",
|
||||
"confidence": 1.0,
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source": "Peterson",
|
||||
"target": "841",
|
||||
"type": "explain",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Toys'",
|
||||
"target": "end_char=5999",
|
||||
"type": "entity(text='85",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Toys'",
|
||||
"target": "six",
|
||||
"type": "entity(text='85",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Maersk",
|
||||
"target": "May 2022",
|
||||
"type": "get",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "J.B. Hunt'",
|
||||
"target": "Entity(text='more than 65'",
|
||||
"type": "Entity(text='more",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "advcl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "One",
|
||||
"target": "LMS",
|
||||
"type": "go",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Bloomberg News",
|
||||
"target": "2011",
|
||||
"type": "rank",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": ">Cold War",
|
||||
"target": "March",
|
||||
"type": "introduce",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "acl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "one",
|
||||
"target": "1095",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "3.3%",
|
||||
"target": "988",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "3.3%",
|
||||
"target": "September",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "U.S.",
|
||||
"target": "South Korea",
|
||||
"type": "provide",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "U.S.",
|
||||
"target": "Japan",
|
||||
"type": "provide",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "U.S.",
|
||||
"target": "South Korea",
|
||||
"type": "have",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "end_char=1538",
|
||||
"target": "Entity(text='Rob Jackson'",
|
||||
"type": "Entity(text='Rob",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "end_char=1538",
|
||||
"target": "Rob Jackson'",
|
||||
"type": "Entity(text='Rob",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Raymond Corp.",
|
||||
"target": "early 2026",
|
||||
"type": "introduce",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Raymond Corp.",
|
||||
"target": "1659",
|
||||
"type": "introduce",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Entity(text='Rob Jackson'",
|
||||
"target": "two-week'",
|
||||
"type": "Entity(text='Rob",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Entity(text='Rob Jackson'",
|
||||
"target": "'90% to 95%",
|
||||
"type": "Entity(text='Rob",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the U.S. Department of Labor",
|
||||
"target": "the State of Arkansas",
|
||||
"type": "award",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the U.S. Department of Labor",
|
||||
"target": "Arkansas",
|
||||
"type": "select",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ccomp -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Union Pacific Railroad'",
|
||||
"target": "'43'",
|
||||
"type": "entity(text='$85",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "advcl -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Union Pacific Railroad'",
|
||||
"target": "six",
|
||||
"type": "entity(text='$85",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "advcl -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Union Pacific Railroad'",
|
||||
"target": "end_char=6379",
|
||||
"type": "entity(text='$85",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "advcl -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Union Pacific Railroad'",
|
||||
"target": "'60,000",
|
||||
"type": "entity(text='$85",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "advcl -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "52.1%",
|
||||
"target": "the International Emergency Economic Powers Act",
|
||||
"type": "pay",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "conj -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "50.0%",
|
||||
"target": "Section 301",
|
||||
"type": "pay",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "conj -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Entity(text='the National Association of Manufacturers",
|
||||
"target": "2025",
|
||||
"type": "end_char=491",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Flavio Batista",
|
||||
"target": "href=\"https://gaports.com",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Flavio Batista",
|
||||
"target": "Ports",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Zebra",
|
||||
"target": "Fetch",
|
||||
"type": "purchase",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "relcl -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Sharma",
|
||||
"target": "three",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the White House",
|
||||
"target": "$73 million",
|
||||
"type": "withhold",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "conj -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "California complied.</p><p>Those",
|
||||
"target": "November 12",
|
||||
"type": "happen",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "Transportation",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "$73 million",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "DOT",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "Minnesota",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "up to $30.4 million",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "California",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Todd Spencer",
|
||||
"target": "$160 million",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "DOT",
|
||||
"target": "Salesforce",
|
||||
"type": "use",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Wallingford",
|
||||
"target": "President & CEO",
|
||||
"type": "serve",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Wallingford",
|
||||
"target": "President & CEO",
|
||||
"type": "serve",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Wallingford",
|
||||
"target": "Swisslog Americas",
|
||||
"type": "serve",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Kenco MHE Solutions",
|
||||
"target": "Wallingford",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Kenco MHE Solutions",
|
||||
"target": "Americas'",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Kenco MHE Solutions",
|
||||
"target": "Honeywell",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Kenco MHE Solutions",
|
||||
"target": "Entity(text='Kenco President &'",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Kenco MHE Solutions",
|
||||
"target": "Entity(text='Kenco President &'",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> conj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "the Kenco MHE Solutions",
|
||||
"target": "Kenco President &'",
|
||||
"type": "be",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> attr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Morrill Worcester",
|
||||
"target": "Arlington National Cemetery",
|
||||
"type": "deliver",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "relcl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Worcester",
|
||||
"target": "millions",
|
||||
"type": "deliver",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "acl -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "two",
|
||||
"target": "Indonesia.</span></p><p>CMA CGM",
|
||||
"type": "prepare",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "pcomp -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Christine Cabau Woehrel",
|
||||
"target": "four",
|
||||
"type": "say",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> dobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "State of Supply Chain Sustainability Report”-produced",
|
||||
"target": "the Massachusetts Institute of Technology’s Sustainable Supply Chain Lab",
|
||||
"type": "report”-produce",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "acl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "State of Supply Chain Sustainability Report”-produced",
|
||||
"target": "the past year",
|
||||
"type": "change",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Chain Management Professionals'",
|
||||
"target": "Entity(text='more than 1,200",
|
||||
"type": "Entity(text='more",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "parataxis -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Teradyne Robotics",
|
||||
"target": "Detroit",
|
||||
"type": "open",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Natrion",
|
||||
"target": "2018",
|
||||
"type": "found",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "conj -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "KICKER",
|
||||
"target": "Averitt",
|
||||
"type": "partner",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "acl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Federal Reserve",
|
||||
"target": "IKEA",
|
||||
"type": "move",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ccomp -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Annual Rail/Intermodal Roundtable",
|
||||
"target": "U.S.",
|
||||
"type": "examine",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "KICKER",
|
||||
"target": "Averitt",
|
||||
"type": "partner",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "acl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "The Missing Link in B2B Growth Is Hiding in Your Order Management Preparing",
|
||||
"target": "2026",
|
||||
"type": "hide",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "advcl -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "November",
|
||||
"target": "172",
|
||||
"type": "refrigerate",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "November",
|
||||
"target": "172",
|
||||
"type": "refrigerate",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "Q&A",
|
||||
"target": "Greenwich",
|
||||
"type": "interview",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "AI",
|
||||
"target": "2026",
|
||||
"type": "reshape",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ccomp -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "PMI",
|
||||
"target": "52.6",
|
||||
"type": "report",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "ROOT -> ... -> pobj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "UP",
|
||||
"target": "NS",
|
||||
"type": "acquire",
|
||||
"confidence": 0.8,
|
||||
"metadata": {
|
||||
"extraction_method": "dependency",
|
||||
"dependency_path": "relcl -> ... -> dobj"
|
||||
}
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"num_entities": 15,
|
||||
"num_relationships": 72,
|
||||
"temporal_enabled": false,
|
||||
"timestamp": "2025-12-30T23:12:22.191303",
|
||||
"entity_resolution_applied": false
|
||||
}
|
||||
}
|
||||
@@ -1,636 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/01_Risk_Assessment.ipynb)\n",
|
||||
"\n",
|
||||
"# Risk Assessment - Graph Analytics & Portfolio Risk Modeling\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates **portfolio risk assessment** using Semantica with focus on **graph-based analytics**, **portfolio risk modeling**, **market simulation**, and **dependency analysis**. The pipeline assesses portfolio risk using graph-based analytics and market simulations.\n",
|
||||
"\n",
|
||||
"### Key Features\n",
|
||||
"\n",
|
||||
"- **Graph-Based Analytics**: Uses graph analytics for portfolio risk analysis\n",
|
||||
"- **Portfolio Risk Modeling**: Models portfolio relationships and dependencies\n",
|
||||
"- **Market Simulation**: Simulates market scenarios using graph data\n",
|
||||
"- **Dependency Analysis**: Analyzes dependencies between portfolio components\n",
|
||||
"- **Risk Modeling**: Emphasizes graph analytics, reasoning, and risk modeling\n",
|
||||
"\n",
|
||||
"### Pipeline Architecture\n",
|
||||
"\n",
|
||||
"1. **Phase 0**: Setup & Configuration\n",
|
||||
"2. **Phase 1**: Portfolio Data Ingestion\n",
|
||||
"3. **Phase 2**: Entity Extraction (Price, Signal, Pattern, Indicator, Strategy)\n",
|
||||
"4. **Phase 3**: Financial Knowledge Graph Construction\n",
|
||||
"5. **Phase 4**: Graph Analytics (Dependencies, Relationships)\n",
|
||||
"6. **Phase 5**: Portfolio Risk Modeling\n",
|
||||
"7. **Phase 6**: Market Simulation\n",
|
||||
"8. **Phase 7**: Visualization & Risk Reporting\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Installation\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install -qU semantica networkx matplotlib plotly pandas groq\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 0: Setup & Configuration\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from semantica.core import Semantica, ConfigManager\n",
|
||||
"from semantica.kg import GraphAnalytics\n",
|
||||
"from semantica.reasoning import GraphReasoner\n",
|
||||
"from semantica.graph_store import GraphStore\n",
|
||||
"\n",
|
||||
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key\")\n",
|
||||
"\n",
|
||||
"config_dict = {\n",
|
||||
" \"project_name\": \"Risk_Assessment\",\n",
|
||||
" \"extraction\": {\"provider\": \"groq\", \"model\": \"llama-3.1-8b-instant\"},\n",
|
||||
" \"knowledge_graph\": {\"backend\": \"networkx\"}\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"config = ConfigManager().load_from_dict(config_dict)\n",
|
||||
"core = Semantica(config=config)\n",
|
||||
"print(\"Configured for risk assessment with graph analytics focus\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 1: Portfolio Data Ingestion\n",
|
||||
"\n",
|
||||
"Ingest portfolio data from databases, structured documents, and version-controlled configurations.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import DBIngestor, RepoIngestor, FileIngestor\n",
|
||||
"from semantica.parse import DocumentParser\n",
|
||||
"from semantica.normalize import TextNormalizer\n",
|
||||
"from semantica.split import TextSplitter\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"os.makedirs(\"data\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"documents = []\n",
|
||||
"\n",
|
||||
"# Option 1: Ingest from database (structured portfolio data)\n",
|
||||
"db_connection = \"sqlite:///data/portfolio.db\"\n",
|
||||
"try:\n",
|
||||
" db_ingestor = DBIngestor()\n",
|
||||
" # Create sample portfolio database structure\n",
|
||||
" import sqlite3\n",
|
||||
" conn = sqlite3.connect(\"data/portfolio.db\")\n",
|
||||
" cursor = conn.cursor()\n",
|
||||
" cursor.execute(\"\"\"\n",
|
||||
" CREATE TABLE IF NOT EXISTS holdings (\n",
|
||||
" id INTEGER PRIMARY KEY,\n",
|
||||
" symbol TEXT,\n",
|
||||
" allocation REAL,\n",
|
||||
" sector TEXT,\n",
|
||||
" risk_level TEXT\n",
|
||||
" )\n",
|
||||
" \"\"\")\n",
|
||||
" cursor.execute(\"\"\"\n",
|
||||
" INSERT OR REPLACE INTO holdings VALUES\n",
|
||||
" (1, 'AAPL', 0.30, 'Technology', 'Medium'),\n",
|
||||
" (2, 'MSFT', 0.25, 'Technology', 'Low'),\n",
|
||||
" (3, 'GOOGL', 0.20, 'Technology', 'Medium'),\n",
|
||||
" (4, 'BND', 0.25, 'Bonds', 'Low')\n",
|
||||
" \"\"\")\n",
|
||||
" conn.commit()\n",
|
||||
" conn.close()\n",
|
||||
" \n",
|
||||
" db_data = db_ingestor.ingest_database(db_connection, method=\"sqlite\")\n",
|
||||
" if db_data and \"data\" in db_data:\n",
|
||||
" for table_data in db_data[\"data\"]:\n",
|
||||
" documents.append(str(table_data))\n",
|
||||
" print(f\"Ingested portfolio data from database\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Database ingestion failed: {e}\")\n",
|
||||
"\n",
|
||||
"# Option 2: Parse structured portfolio documents (CSV/PDF)\n",
|
||||
"try:\n",
|
||||
" portfolio_csv = \"\"\"symbol,allocation,sector,risk_level\n",
|
||||
"AAPL,0.30,Technology,Medium\n",
|
||||
"MSFT,0.25,Technology,Low\n",
|
||||
"GOOGL,0.20,Technology,Medium\n",
|
||||
"BND,0.25,Bonds,Low\"\"\"\n",
|
||||
" with open(\"data/portfolio.csv\", \"w\") as f:\n",
|
||||
" f.write(portfolio_csv)\n",
|
||||
" \n",
|
||||
" doc_parser = DocumentParser()\n",
|
||||
" parsed_docs = doc_parser.parse(\"data/portfolio.csv\", file_type=\"csv\")\n",
|
||||
" if parsed_docs:\n",
|
||||
" documents.append(str(parsed_docs))\n",
|
||||
" print(f\"Parsed portfolio document\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Document parsing failed: {e}\")\n",
|
||||
"\n",
|
||||
"# Option 3: Ingest from version-controlled portfolio configs\n",
|
||||
"try:\n",
|
||||
" portfolio_config = \"\"\"\n",
|
||||
" Portfolio Configuration:\n",
|
||||
" - AAPL: 30% allocation, Tech sector\n",
|
||||
" - MSFT: 25% allocation, Tech sector \n",
|
||||
" - GOOGL: 20% allocation, Tech sector\n",
|
||||
" - BND: 25% allocation, Bonds\n",
|
||||
" Risk: High tech concentration (75%)\n",
|
||||
" \"\"\"\n",
|
||||
" os.makedirs(\"data/portfolio_config\", exist_ok=True)\n",
|
||||
" with open(\"data/portfolio_config/config.txt\", \"w\") as f:\n",
|
||||
" f.write(portfolio_config)\n",
|
||||
" repo_docs = FileIngestor().ingest(\"data/portfolio_config\")\n",
|
||||
" documents.extend(repo_docs)\n",
|
||||
" print(f\"Ingested portfolio configuration from repository structure\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Repository ingestion failed: {e}\")\n",
|
||||
"\n",
|
||||
"# Fallback: Sample data\n",
|
||||
"if not documents:\n",
|
||||
" portfolio_data = \"\"\"\n",
|
||||
" Portfolio contains: Stock AAPL (30%), Stock MSFT (25%), Stock GOOGL (20%), Bond BND (25%).\n",
|
||||
" AAPL price correlates with tech sector performance.\n",
|
||||
" MSFT depends on cloud services market growth.\n",
|
||||
" Portfolio risk: High concentration in tech sector (75%).\n",
|
||||
" Dependency: Tech sector downturn impacts 75% of portfolio.\n",
|
||||
" \"\"\"\n",
|
||||
" with open(\"data/portfolio.txt\", \"w\") as f:\n",
|
||||
" f.write(portfolio_data)\n",
|
||||
" documents = FileIngestor().ingest(\"data/portfolio.txt\")\n",
|
||||
" print(f\"Ingested {len(documents)} documents from sample data\")\n",
|
||||
"\n",
|
||||
"# Normalize financial data\n",
|
||||
"normalizer = TextNormalizer()\n",
|
||||
"normalized_documents = []\n",
|
||||
"for doc in documents:\n",
|
||||
" doc_content = doc.content if hasattr(doc, 'content') else str(doc)\n",
|
||||
" normalized_text = normalizer.normalize(\n",
|
||||
" doc_content,\n",
|
||||
" clean_html=True,\n",
|
||||
" normalize_entities=True,\n",
|
||||
" normalize_numbers=True,\n",
|
||||
" remove_extra_whitespace=True\n",
|
||||
" )\n",
|
||||
" normalized_documents.append(normalized_text)\n",
|
||||
"\n",
|
||||
"print(f\"Normalized {len(normalized_documents)} documents\")\n",
|
||||
"\n",
|
||||
"# Use token-based or recursive chunking for structured portfolio data\n",
|
||||
"splitter = TextSplitter(method=\"token\", chunk_size=1000, chunk_overlap=200)\n",
|
||||
"\n",
|
||||
"chunked_docs = []\n",
|
||||
"for doc_text in normalized_documents:\n",
|
||||
" chunks = splitter.split(doc_text)\n",
|
||||
" chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
|
||||
"\n",
|
||||
"print(f\"Created {len(chunked_docs)} token-based chunks\")\n",
|
||||
"print(\"Phase 1 complete: Portfolio data ingested from multiple structured sources\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector\n",
|
||||
"\n",
|
||||
"# Extract entities using LLM-based NER\n",
|
||||
"ner_extractor = NERExtractor(method=\"llm\", provider=\"groq\", llm_model=\"llama-3.1-8b-instant\")\n",
|
||||
"entities = []\n",
|
||||
"for doc_text in chunked_docs:\n",
|
||||
" extracted = ner_extractor.extract_entities(doc_text)\n",
|
||||
" entities.extend(extracted)\n",
|
||||
"\n",
|
||||
"# Filter for financial entity types\n",
|
||||
"financial_entities = [\n",
|
||||
" e for e in entities \n",
|
||||
" if e.type in [\"Price\", \"Signal\", \"Pattern\", \"Indicator\", \"Strategy\"] or\n",
|
||||
" any(keyword in e.text.lower() for keyword in [\"stock\", \"bond\", \"portfolio\", \"risk\", \"sector\"])\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Extract relationships\n",
|
||||
"relation_extractor = RelationExtractor(method=\"llm\", provider=\"groq\", llm_model=\"llama-3.1-8b-instant\")\n",
|
||||
"relationships = []\n",
|
||||
"for doc_text in chunked_docs:\n",
|
||||
" extracted_rels = relation_extractor.extract_relations(doc_text, entities=financial_entities)\n",
|
||||
" relationships.extend(extracted_rels)\n",
|
||||
"\n",
|
||||
"# Extract market events\n",
|
||||
"event_detector = EventDetector()\n",
|
||||
"events = []\n",
|
||||
"for doc_text in chunked_docs:\n",
|
||||
" extracted_events = event_detector.detect_events(doc_text)\n",
|
||||
" events.extend(extracted_events)\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(financial_entities)} financial entities\")\n",
|
||||
"print(f\"Extracted {len(relationships)} relationships\")\n",
|
||||
"print(f\"Extracted {len(events)} market events\")\n",
|
||||
"print(\"Phase 2 complete: Entity extraction with NER, relations, and events\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 3: Financial Knowledge Graph Construction\n",
|
||||
"\n",
|
||||
"Build financial knowledge graph using GraphStore (Neo4j) and generate domain ontology.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ontology import OntologyGenerator\n",
|
||||
"from semantica.context import AgentContext\n",
|
||||
"from semantica.vector_store import VectorStore\n",
|
||||
"\n",
|
||||
"# Build knowledge graph using core pipeline\n",
|
||||
"entity_dicts = [\n",
|
||||
" {\"text\": e.text, \"type\": e.type, \"start\": e.start, \"end\": e.end, \"confidence\": getattr(e, 'confidence', 0.9)}\n",
|
||||
" for e in financial_entities\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"relationship_dicts = [\n",
|
||||
" {\n",
|
||||
" \"subject\": rel.subject.text if hasattr(rel, 'subject') else str(rel.subject),\n",
|
||||
" \"predicate\": rel.predicate if hasattr(rel, 'predicate') else \"related_to\",\n",
|
||||
" \"object\": rel.object.text if hasattr(rel, 'object') else str(rel.object)\n",
|
||||
" }\n",
|
||||
" for rel in relationships\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"result = core.build_knowledge_base(\n",
|
||||
" sources=chunked_docs,\n",
|
||||
" custom_entity_types=[\"Price\", \"Signal\", \"Pattern\", \"Indicator\", \"Strategy\"],\n",
|
||||
" graph=True,\n",
|
||||
" embeddings=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"kg = result[\"knowledge_graph\"]\n",
|
||||
"print(f\"Built portfolio KG with {len(kg.get('entities', []))} entities\")\n",
|
||||
"\n",
|
||||
"# Generate financial domain ontology\n",
|
||||
"ontology_generator = OntologyGenerator(base_uri=\"https://semantica.dev/ontology/finance/\")\n",
|
||||
"ontology = ontology_generator.generate_ontology({\n",
|
||||
" \"entities\": entity_dicts,\n",
|
||||
" \"relationships\": relationship_dicts\n",
|
||||
"})\n",
|
||||
"\n",
|
||||
"print(f\"Generated ontology with {len(ontology.get('classes', []))} classes\")\n",
|
||||
"print(f\"Generated {len(ontology.get('properties', []))} properties\")\n",
|
||||
"\n",
|
||||
"# Setup GraphStore for persistent storage (Neo4j backend)\n",
|
||||
"# Note: For demo, we'll use networkx but show GraphStore usage\n",
|
||||
"try:\n",
|
||||
" graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
|
||||
" # In production, you would store the graph here\n",
|
||||
" # graph_store.create_node(labels=[\"Portfolio\"], properties={\"name\": \"Risk_Assessment\"})\n",
|
||||
" print(\"GraphStore configured for Neo4j (connection skipped in demo)\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"GraphStore connection skipped (Neo4j not available): {e}\")\n",
|
||||
"\n",
|
||||
"# Setup GraphRAG for portfolio analysis\n",
|
||||
"vector_store = VectorStore(backend=\"faiss\", dimension=384)\n",
|
||||
"if result.get(\"embeddings\"):\n",
|
||||
" vector_store.store_vectors(\n",
|
||||
" vectors=result[\"embeddings\"][\"vectors\"],\n",
|
||||
" metadata=result[\"embeddings\"][\"metadata\"]\n",
|
||||
" )\n",
|
||||
"context = AgentContext(vector_store=vector_store, knowledge_graph=kg)\n",
|
||||
"\n",
|
||||
"print(\"Phase 3 complete: Financial KG constructed with ontology and GraphStore\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 4: Graph Analytics\n",
|
||||
"\n",
|
||||
"Perform comprehensive graph analytics: centrality analysis, community detection, and path analysis.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Perform comprehensive graph analytics\n",
|
||||
"analytics = GraphAnalytics(kg)\n",
|
||||
"\n",
|
||||
"# Multiple centrality methods\n",
|
||||
"betweenness_centrality = analytics.calculate_centrality(method=\"betweenness\")\n",
|
||||
"degree_centrality = analytics.calculate_centrality(method=\"degree\")\n",
|
||||
"closeness_centrality = analytics.calculate_centrality(method=\"closeness\")\n",
|
||||
"eigenvector_centrality = analytics.calculate_centrality(method=\"eigenvector\")\n",
|
||||
"\n",
|
||||
"print(f\"Betweenness centrality: {len(betweenness_centrality)} nodes analyzed\")\n",
|
||||
"print(f\"Degree centrality: {len(degree_centrality)} nodes analyzed\")\n",
|
||||
"print(f\"Closeness centrality: {len(closeness_centrality)} nodes analyzed\")\n",
|
||||
"print(f\"Eigenvector centrality: {len(eigenvector_centrality)} nodes analyzed\")\n",
|
||||
"\n",
|
||||
"# Community detection for portfolio clusters\n",
|
||||
"try:\n",
|
||||
" communities = analytics.detect_communities(method=\"louvain\")\n",
|
||||
" print(f\"Detected {len(communities)} communities/clusters\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Community detection: {e}\")\n",
|
||||
"\n",
|
||||
"# Path analysis for dependency chains\n",
|
||||
"try:\n",
|
||||
" paths = analytics.find_paths(source=\"AAPL\", target=\"MSFT\", max_length=3)\n",
|
||||
" print(f\"Found {len(paths)} paths between portfolio components\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Path analysis: {e}\")\n",
|
||||
"\n",
|
||||
"# Use reasoning for dependency analysis\n",
|
||||
"reasoner = GraphReasoner(kg)\n",
|
||||
"dependencies = reasoner.find_patterns(pattern_type=\"dependency\")\n",
|
||||
"risk_patterns = reasoner.find_patterns(pattern_type=\"risk\")\n",
|
||||
"\n",
|
||||
"print(f\"Dependency analysis: {len(dependencies)} portfolio dependencies identified\")\n",
|
||||
"print(f\"Risk modeling: {len(risk_patterns)} risk patterns detected\")\n",
|
||||
"print(\"Phase 4 complete: Comprehensive graph analytics performed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Deduplication\n",
|
||||
"\n",
|
||||
"Resolve duplicate entities using graph-based clustering for risk assessment.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.deduplication import ClusterBuilder, EntityMerger\n",
|
||||
"from semantica.semantic_extract import Entity\n",
|
||||
"\n",
|
||||
"# Convert Entity objects to dictionaries\n",
|
||||
"print(f\"Converting {len(financial_entities)} entities to dictionaries...\")\n",
|
||||
"entity_dicts = [{\"name\": e.text, \"type\": e.label, \"start_char\": e.start_char, \"end_char\": e.end_char, \"confidence\": e.confidence} for e in financial_entities]\n",
|
||||
"\n",
|
||||
"# Use graph_based clustering for risk assessment (identifies interconnected risks)\n",
|
||||
"# merge_all strategy combines all information from related risk entities\n",
|
||||
"cluster_builder = ClusterBuilder(method=\"graph_based\", similarity_threshold=0.85)\n",
|
||||
"\n",
|
||||
"print(f\"Building clusters for {len(entity_dicts)} entities using graph-based method...\")\n",
|
||||
"clusters = cluster_builder.build_clusters(entity_dicts)\n",
|
||||
"\n",
|
||||
"print(f\"Detected {len(clusters)} clusters\")\n",
|
||||
"print(f\"Merging entities within clusters using merge_all strategy...\")\n",
|
||||
"merger = EntityMerger()\n",
|
||||
"merged_entities_dicts = []\n",
|
||||
"for cluster in clusters:\n",
|
||||
" if len(cluster) > 1:\n",
|
||||
" # Merge entities within each cluster\n",
|
||||
" merge_operations = merger.merge_duplicates(cluster, strategy=\"merge_all\", threshold=0.85)\n",
|
||||
" if merge_operations:\n",
|
||||
" merged_entities_dicts.extend([op.merged_entity for op in merge_operations])\n",
|
||||
" else:\n",
|
||||
" merged_entities_dicts.extend(cluster)\n",
|
||||
"\n",
|
||||
"# Convert back to Entity objects\n",
|
||||
"print(f\"Converting {len(merged_entities_dicts)} merged entities back to Entity objects...\")\n",
|
||||
"merged_entities = [\n",
|
||||
" Entity(text=e[\"name\"], label=e[\"type\"], start_char=e.get(\"start_char\", 0), end_char=e.get(\"end_char\", 0), confidence=e.get(\"confidence\", 1.0))\n",
|
||||
" for e in merged_entities_dicts\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"financial_entities = merged_entities\n",
|
||||
"print(f\"Deduplicated to {len(merged_entities)} unique entities\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 5: Portfolio Risk Modeling\n",
|
||||
"\n",
|
||||
"Detect conflicts, apply risk inference rules, and perform temporal graph analysis.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.conflicts import ConflictDetector, ConflictResolver\n",
|
||||
"\n",
|
||||
"# Use logical conflict detection for portfolio risk rules\n",
|
||||
"# highest_confidence strategy prioritizes the most confident risk assessment\n",
|
||||
"conflict_detector = ConflictDetector()\n",
|
||||
"conflict_resolver = ConflictResolver()\n",
|
||||
"\n",
|
||||
"print(f\"Detecting logical conflicts in {len(financial_entities)} entities and relationships...\")\n",
|
||||
"conflicts = conflict_detector.detect_conflicts(\n",
|
||||
" entities=financial_entities,\n",
|
||||
" relationships=all_relationships,\n",
|
||||
" method=\"logical\" # Detect logical conflicts (e.g., conflicting risk indicators)\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f\"Detected {len(conflicts)} logical conflicts\")\n",
|
||||
"\n",
|
||||
"if conflicts:\n",
|
||||
" print(f\"Resolving conflicts using highest_confidence strategy...\")\n",
|
||||
" resolved = conflict_resolver.resolve_conflicts(\n",
|
||||
" conflicts,\n",
|
||||
" strategy=\"highest_confidence\" # Prioritize most confident risk assessment\n",
|
||||
" )\n",
|
||||
" print(f\"Resolved {len(resolved)} conflicts\")\n",
|
||||
"else:\n",
|
||||
" print(\"No conflicts detected\")\n",
|
||||
"\n",
|
||||
"# Use reasoning for risk inference rules\n",
|
||||
"risk_rules = [\n",
|
||||
" \"IF sector_concentration > 0.7 THEN high_risk\",\n",
|
||||
" \"IF tech_sector_allocation > 0.5 AND bond_allocation < 0.3 THEN medium_risk\",\n",
|
||||
" \"IF portfolio_diversity < 0.3 THEN high_risk\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for rule in risk_rules:\n",
|
||||
" reasoner.add_rule(rule)\n",
|
||||
"\n",
|
||||
"# Infer risk levels\n",
|
||||
"inferred_risks = reasoner.infer_facts(kg)\n",
|
||||
"print(f\"Inferred {len(inferred_risks)} risk facts from rules\")\n",
|
||||
"\n",
|
||||
"# Temporal graph analysis (if temporal data available)\n",
|
||||
"try:\n",
|
||||
" temporal_kg = core.build_knowledge_base(\n",
|
||||
" sources=chunked_docs,\n",
|
||||
" custom_entity_types=[\"Price\", \"Signal\", \"Pattern\"],\n",
|
||||
" graph=True,\n",
|
||||
" temporal=True\n",
|
||||
" )\n",
|
||||
" print(\"Temporal knowledge graph constructed for time-series analysis\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Temporal analysis: {e}\")\n",
|
||||
"\n",
|
||||
"print(\"Phase 5 complete: Risk modeling with conflict detection and inference\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 6: Market Simulation\n",
|
||||
"\n",
|
||||
"Export risk data, generate reports, and prepare data for external analysis.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.export import ExportManager, ReportGenerator\n",
|
||||
"\n",
|
||||
"# Export to multiple formats\n",
|
||||
"export_manager = ExportManager()\n",
|
||||
"\n",
|
||||
"# Export to JSON\n",
|
||||
"export_manager.export_knowledge_graph(kg, \"data/portfolio_risk.json\", format=\"json\")\n",
|
||||
"print(\"Exported portfolio KG to JSON\")\n",
|
||||
"\n",
|
||||
"# Export to CSV\n",
|
||||
"export_manager.export_knowledge_graph(kg, \"data/portfolio_risk.csv\", format=\"csv\")\n",
|
||||
"print(\"Exported portfolio KG to CSV\")\n",
|
||||
"\n",
|
||||
"# Export to GraphML for external analysis\n",
|
||||
"export_manager.export_knowledge_graph(kg, \"data/portfolio_risk.graphml\", format=\"graphml\")\n",
|
||||
"print(\"Exported portfolio KG to GraphML\")\n",
|
||||
"\n",
|
||||
"# Export to RDF\n",
|
||||
"export_manager.export_knowledge_graph(kg, \"data/portfolio_risk.ttl\", format=\"rdf\", rdf_format=\"turtle\")\n",
|
||||
"print(\"Exported portfolio KG to RDF (Turtle)\")\n",
|
||||
"\n",
|
||||
"# Generate risk report\n",
|
||||
"report_generator = ReportGenerator()\n",
|
||||
"risk_report = report_generator.generate_report(\n",
|
||||
" kg,\n",
|
||||
" report_type=\"risk_assessment\",\n",
|
||||
" output_path=\"data/risk_report.html\",\n",
|
||||
" format=\"html\"\n",
|
||||
")\n",
|
||||
"print(\"Generated HTML risk assessment report\")\n",
|
||||
"\n",
|
||||
"# Market simulation data preparation\n",
|
||||
"simulation_data = {\n",
|
||||
" \"entities\": len(kg.get(\"entities\", [])),\n",
|
||||
" \"relationships\": len(kg.get(\"relationships\", [])),\n",
|
||||
" \"risk_patterns\": len(risk_patterns),\n",
|
||||
" \"conflicts\": len(conflicts),\n",
|
||||
" \"communities\": len(communities) if 'communities' in locals() else 0\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"print(f\"\\nMarket Simulation Data Summary:\")\n",
|
||||
"for key, value in simulation_data.items():\n",
|
||||
" print(f\" {key}: {value}\")\n",
|
||||
"\n",
|
||||
"print(\"Phase 6 complete: Market simulation data exported and reports generated\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 7: Visualization & Risk Reporting\n",
|
||||
"\n",
|
||||
"Visualize the knowledge graph and export to multiple formats.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"\n",
|
||||
"# Visualize knowledge graph\n",
|
||||
"visualizer = KGVisualizer()\n",
|
||||
"visualizer.visualize(kg, output_path=\"portfolio_risk_kg.html\")\n",
|
||||
"\n",
|
||||
"print(\"Risk assessment analysis complete\")\n",
|
||||
"print(\"Emphasizes: Graph analytics, portfolio risk modeling, market simulation, dependency analysis\")\n",
|
||||
"print(\"\\nGenerated outputs:\")\n",
|
||||
"print(\" - portfolio_risk_kg.html (visualization)\")\n",
|
||||
"print(\" - data/portfolio_risk.json (JSON export)\")\n",
|
||||
"print(\" - data/portfolio_risk.csv (CSV export)\")\n",
|
||||
"print(\" - data/portfolio_risk.graphml (GraphML export)\")\n",
|
||||
"print(\" - data/portfolio_risk.ttl (RDF export)\")\n",
|
||||
"print(\" - data/risk_report.html (risk report)\")\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -1,668 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/02_News_Sentiment_Analysis.ipynb)\n",
|
||||
"\n",
|
||||
"# News Sentiment Analysis - Semantic Extraction & Correlation\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates **news sentiment analysis** using Semantica with focus on **sentiment extraction**, **correlation analysis**, **financial KG**, and **price movement prediction**. The pipeline correlates news sentiment with price movements using financial knowledge graphs.\n",
|
||||
"\n",
|
||||
"### Key Features\n",
|
||||
"\n",
|
||||
"- **Sentiment Extraction**: Extracts sentiment from financial news articles\n",
|
||||
"- **Correlation Analysis**: Correlates sentiment with price movements\n",
|
||||
"- **Financial Knowledge Graph**: Builds financial KGs for correlation analysis\n",
|
||||
"- **Price Movement Prediction**: Predicts price movements based on sentiment\n",
|
||||
"- **Semantic Extraction**: Emphasizes semantic extraction and correlation analysis\n",
|
||||
"\n",
|
||||
"### Pipeline Architecture\n",
|
||||
"\n",
|
||||
"1. **Phase 0**: Setup & Configuration\n",
|
||||
"2. **Phase 1**: Financial News Ingestion\n",
|
||||
"3. **Phase 2**: Sentiment Extraction\n",
|
||||
"4. **Phase 3**: Financial Entity Extraction (Article, Sentiment, Price, Correlation, Trend)\n",
|
||||
"5. **Phase 4**: Financial Knowledge Graph Construction\n",
|
||||
"6. **Phase 5**: Sentiment-Price Correlation Analysis\n",
|
||||
"7. **Phase 6**: Price Movement Prediction\n",
|
||||
"8. **Phase 7**: Visualization & Export\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Installation\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install -qU semantica networkx matplotlib plotly pandas groq\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 0: Setup & Configuration\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from semantica.core import Semantica, ConfigManager\n",
|
||||
"from semantica.semantic_extract import SentimentAnalyzer\n",
|
||||
"from semantica.triplet_store import TripletStore\n",
|
||||
"\n",
|
||||
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key\")\n",
|
||||
"\n",
|
||||
"config_dict = {\n",
|
||||
" \"project_name\": \"News_Sentiment_Analysis\",\n",
|
||||
" \"extraction\": {\"provider\": \"groq\", \"model\": \"llama-3.1-8b-instant\"},\n",
|
||||
" \"knowledge_graph\": {\"backend\": \"networkx\"}\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"config = ConfigManager().load_from_dict(config_dict)\n",
|
||||
"core = Semantica(config=config)\n",
|
||||
"sentiment_analyzer = SentimentAnalyzer()\n",
|
||||
"print(\"Configured for news sentiment analysis with semantic extraction focus\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 1: Financial News Ingestion\n",
|
||||
"\n",
|
||||
"Ingest financial news from streams, web sources, and structured APIs.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import StreamIngestor, FileIngestor\n",
|
||||
"from semantica.parse import WebParser, StructuredDataParser\n",
|
||||
"from semantica.normalize import TextNormalizer\n",
|
||||
"from semantica.split import TextSplitter\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"os.makedirs(\"data\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"documents = []\n",
|
||||
"\n",
|
||||
"# Option 1: Ingest from real-time news streams (simulated)\n",
|
||||
"try:\n",
|
||||
" stream_ingestor = StreamIngestor()\n",
|
||||
" # Simulate stream ingestion with sample data\n",
|
||||
" stream_config = {\n",
|
||||
" \"stream_type\": \"kafka\",\n",
|
||||
" \"topics\": [\"financial_news\"],\n",
|
||||
" \"bootstrap_servers\": \"localhost:9092\"\n",
|
||||
" }\n",
|
||||
" # In production, this would connect to actual stream\n",
|
||||
" print(\"Stream ingestor configured (connection skipped in demo)\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Stream ingestion setup: {e}\")\n",
|
||||
"\n",
|
||||
"# Option 2: Parse HTML news articles\n",
|
||||
"try:\n",
|
||||
" web_parser = WebParser()\n",
|
||||
" # Simulate web parsing with sample HTML\n",
|
||||
" sample_html = \"\"\"\n",
|
||||
" <html>\n",
|
||||
" <head><title>Financial News</title></head>\n",
|
||||
" <body>\n",
|
||||
" <article>\n",
|
||||
" <h1>AAPL Earnings Beat Expectations</h1>\n",
|
||||
" <p>Apple Inc. reported strong quarterly earnings, exceeding analyst expectations.</p>\n",
|
||||
" </article>\n",
|
||||
" </body>\n",
|
||||
" </html>\n",
|
||||
" \"\"\"\n",
|
||||
" with open(\"data/sample_news.html\", \"w\") as f:\n",
|
||||
" f.write(sample_html)\n",
|
||||
" parsed_html = web_parser.parse_html(\"data/sample_news.html\")\n",
|
||||
" if parsed_html:\n",
|
||||
" documents.append(str(parsed_html))\n",
|
||||
" print(\"Parsed HTML news article\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Web parsing failed: {e}\")\n",
|
||||
"\n",
|
||||
"# Option 3: Parse structured JSON news APIs\n",
|
||||
"try:\n",
|
||||
" structured_parser = StructuredDataParser()\n",
|
||||
" news_json = {\n",
|
||||
" \"articles\": [\n",
|
||||
" {\n",
|
||||
" \"title\": \"MSFT Faces Regulatory Concerns\",\n",
|
||||
" \"content\": \"Microsoft faces new regulatory challenges in cloud services market.\",\n",
|
||||
" \"date\": \"2024-01-02\",\n",
|
||||
" \"sentiment\": \"negative\"\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"title\": \"GOOGL Announces New Product\",\n",
|
||||
" \"content\": \"Google announces innovative new product line.\",\n",
|
||||
" \"date\": \"2024-01-03\",\n",
|
||||
" \"sentiment\": \"positive\"\n",
|
||||
" }\n",
|
||||
" ]\n",
|
||||
" }\n",
|
||||
" import json\n",
|
||||
" with open(\"data/news.json\", \"w\") as f:\n",
|
||||
" json.dump(news_json, f)\n",
|
||||
" parsed_json = structured_parser.parse_json(\"data/news.json\")\n",
|
||||
" if parsed_json:\n",
|
||||
" documents.append(str(parsed_json))\n",
|
||||
" print(\"Parsed structured JSON news data\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Structured data parsing failed: {e}\")\n",
|
||||
"\n",
|
||||
"# Fallback: Sample data\n",
|
||||
"if not documents:\n",
|
||||
" news_data = \"\"\"\n",
|
||||
" 2024-01-01: Positive news about AAPL earnings beat expectations. Stock price rose 5%.\n",
|
||||
" 2024-01-02: Negative sentiment: MSFT faces regulatory concerns. Stock price dropped 3%.\n",
|
||||
" 2024-01-03: Mixed sentiment: GOOGL announces new product. Stock price stable.\n",
|
||||
" Correlation: Positive news correlates with price increases. Negative news correlates with price drops.\n",
|
||||
" \"\"\"\n",
|
||||
" with open(\"data/financial_news.txt\", \"w\") as f:\n",
|
||||
" f.write(news_data)\n",
|
||||
" documents = FileIngestor().ingest(\"data/financial_news.txt\")\n",
|
||||
" print(f\"Ingested {len(documents)} documents from sample data\")\n",
|
||||
"\n",
|
||||
"# Normalize sentiment data\n",
|
||||
"normalizer = TextNormalizer()\n",
|
||||
"normalized_documents = []\n",
|
||||
"for doc in documents:\n",
|
||||
" doc_content = doc.content if hasattr(doc, 'content') else str(doc)\n",
|
||||
" normalized_text = normalizer.normalize(\n",
|
||||
" doc_content,\n",
|
||||
" clean_html=True,\n",
|
||||
" normalize_entities=True,\n",
|
||||
" remove_extra_whitespace=True\n",
|
||||
" )\n",
|
||||
" normalized_documents.append(normalized_text)\n",
|
||||
"\n",
|
||||
"print(f\"Normalized {len(normalized_documents)} documents\")\n",
|
||||
"\n",
|
||||
"# Use hierarchical or topic-based chunking for semantic article chunking\n",
|
||||
"splitter = TextSplitter(method=\"hierarchical\", chunk_size=1000, chunk_overlap=200)\n",
|
||||
"# Alternative: topic_based chunking\n",
|
||||
"# splitter = TextSplitter(method=\"topic_based\", chunk_size=1000)\n",
|
||||
"\n",
|
||||
"chunked_docs = []\n",
|
||||
"for doc_text in normalized_documents:\n",
|
||||
" chunks = splitter.split(doc_text)\n",
|
||||
" chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
|
||||
"\n",
|
||||
"print(f\"Created {len(chunked_docs)} hierarchical chunks\")\n",
|
||||
"print(\"Phase 1 complete: Financial news ingested from multiple sources\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Extract sentiment from news articles\n",
|
||||
"sentiments = []\n",
|
||||
"for doc_text in chunked_docs:\n",
|
||||
" sentiment_result = sentiment_analyzer.analyze_sentiment(doc_text)\n",
|
||||
" if isinstance(sentiment_result, list):\n",
|
||||
" sentiments.extend(sentiment_result)\n",
|
||||
" else:\n",
|
||||
" sentiments.append(sentiment_result)\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(sentiments)} sentiment scores\")\n",
|
||||
"\n",
|
||||
"# Extract sentiment per entity (if entities available)\n",
|
||||
"entity_sentiments = {}\n",
|
||||
"for i, doc_text in enumerate(chunked_docs):\n",
|
||||
" sentiment = sentiment_analyzer.analyze_sentiment(doc_text)\n",
|
||||
" # Store sentiment with document index\n",
|
||||
" entity_sentiments[f\"doc_{i}\"] = sentiment\n",
|
||||
"\n",
|
||||
"# Emotion detection (if supported)\n",
|
||||
"try:\n",
|
||||
" emotions = []\n",
|
||||
" for doc_text in chunked_docs:\n",
|
||||
" # SentimentAnalyzer may support emotion detection\n",
|
||||
" emotion_result = sentiment_analyzer.analyze_sentiment(doc_text)\n",
|
||||
" emotions.append(emotion_result)\n",
|
||||
" print(f\"Detected emotions for {len(emotions)} documents\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"Emotion detection: {e}\")\n",
|
||||
"\n",
|
||||
"print(\"Phase 2 complete: Sentiment and emotion extraction performed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 3: Financial Entity Extraction\n",
|
||||
"\n",
|
||||
"Extract financial entities using spaCy-based NER and relation extraction.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
|
||||
"\n",
|
||||
"# Extract entities using spaCy-based NER (instead of LLM)\n",
|
||||
"ner_extractor = NERExtractor(method=\"ml\", model=\"en_core_web_sm\")\n",
|
||||
"entities = []\n",
|
||||
"for doc_text in chunked_docs:\n",
|
||||
" extracted = ner_extractor.extract_entities(doc_text)\n",
|
||||
" entities.extend(extracted)\n",
|
||||
"\n",
|
||||
"# Filter for financial entity types\n",
|
||||
"financial_entities = [\n",
|
||||
" e for e in entities \n",
|
||||
" if e.type in [\"Article\", \"Sentiment\", \"Price\", \"Correlation\", \"Trend\"] or\n",
|
||||
" any(keyword in e.text.lower() for keyword in [\"stock\", \"price\", \"earnings\", \"news\", \"sentiment\"])\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Extract sentiment-price relationships\n",
|
||||
"relation_extractor = RelationExtractor(method=\"dependency\")\n",
|
||||
"relationships = []\n",
|
||||
"for doc_text in chunked_docs:\n",
|
||||
" extracted_rels = relation_extractor.extract_relations(doc_text, entities=financial_entities)\n",
|
||||
" relationships.extend(extracted_rels)\n",
|
||||
"\n",
|
||||
"# Build financial knowledge graph with sentiment\n",
|
||||
"result = core.build_knowledge_base(\n",
|
||||
" sources=chunked_docs,\n",
|
||||
" custom_entity_types=[\"Article\", \"Sentiment\", \"Price\", \"Correlation\", \"Trend\"],\n",
|
||||
" graph=True,\n",
|
||||
" embeddings=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"kg = result[\"knowledge_graph\"]\n",
|
||||
"print(f\"Built financial KG with {len(kg.get('entities', []))} entities\")\n",
|
||||
"print(f\"Extracted {len(financial_entities)} financial entities\")\n",
|
||||
"print(f\"Extracted {len(relationships)} sentiment-price relationships\")\n",
|
||||
"print(\"Phase 3 complete: Financial entity extraction with spaCy NER\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 4: Financial Knowledge Graph Construction\n",
|
||||
"\n",
|
||||
"Build temporal knowledge graph using TripletStore (RDF) and generate financial news ontology.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ontology import OntologyGenerator\n",
|
||||
"from semantica.context import AgentContext\n",
|
||||
"from semantica.vector_store import VectorStore\n",
|
||||
"\n",
|
||||
"# Build temporal knowledge graph for time-series sentiment\n",
|
||||
"temporal_result = core.build_knowledge_base(\n",
|
||||
" sources=chunked_docs,\n",
|
||||
" custom_entity_types=[\"Article\", \"Sentiment\", \"Price\", \"Correlation\", \"Trend\"],\n",
|
||||
" graph=True,\n",
|
||||
" embeddings=True,\n",
|
||||
" temporal=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"temporal_kg = temporal_result[\"knowledge_graph\"]\n",
|
||||
"print(f\"Built temporal KG with {len(temporal_kg.get('entities', []))} entities\")\n",
|
||||
"\n",
|
||||
"# Generate financial news ontology\n",
|
||||
"entity_dicts = [\n",
|
||||
" {\"text\": e.text, \"type\": e.type, \"start\": e.start, \"end\": e.end}\n",
|
||||
" for e in financial_entities\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"relationship_dicts = [\n",
|
||||
" {\n",
|
||||
" \"subject\": rel.subject.text if hasattr(rel, 'subject') else str(rel.subject),\n",
|
||||
" \"predicate\": rel.predicate if hasattr(rel, 'predicate') else \"correlates_with\",\n",
|
||||
" \"object\": rel.object.text if hasattr(rel, 'object') else str(rel.object)\n",
|
||||
" }\n",
|
||||
" for rel in relationships\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"ontology_generator = OntologyGenerator(base_uri=\"https://semantica.dev/ontology/finance/news/\")\n",
|
||||
"ontology = ontology_generator.generate_ontology({\n",
|
||||
" \"entities\": entity_dicts,\n",
|
||||
" \"relationships\": relationship_dicts\n",
|
||||
"})\n",
|
||||
"\n",
|
||||
"print(f\"Generated ontology with {len(ontology.get('classes', []))} classes\")\n",
|
||||
"print(f\"Generated {len(ontology.get('properties', []))} properties\")\n",
|
||||
"\n",
|
||||
"# Setup TripletStore for RDF storage\n",
|
||||
"try:\n",
|
||||
" triplet_store = TripletStore(backend=\"rdflib\")\n",
|
||||
" # Store triplets in RDF format\n",
|
||||
" for rel in relationship_dicts:\n",
|
||||
" triplet_store.add_triplet(\n",
|
||||
" subject=rel[\"subject\"],\n",
|
||||
" predicate=rel[\"predicate\"],\n",
|
||||
" object=rel[\"object\"]\n",
|
||||
" )\n",
|
||||
" print(f\"Stored {len(relationship_dicts)} triplets in RDF store\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"TripletStore setup: {e}\")\n",
|
||||
"\n",
|
||||
"# Setup enhanced GraphRAG\n",
|
||||
"vector_store = VectorStore(backend=\"faiss\", dimension=384)\n",
|
||||
"if result.get(\"embeddings\"):\n",
|
||||
" vector_store.store_vectors(\n",
|
||||
" vectors=result[\"embeddings\"][\"vectors\"],\n",
|
||||
" metadata=result[\"embeddings\"][\"metadata\"]\n",
|
||||
" )\n",
|
||||
"context = AgentContext(vector_store=vector_store, knowledge_graph=kg)\n",
|
||||
"\n",
|
||||
"print(\"Phase 4 complete: Temporal KG constructed with TripletStore and ontology\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.reasoning import GraphReasoner\n",
|
||||
"from semantica.kg import GraphAnalytics\n",
|
||||
"\n",
|
||||
"# Use GraphReasoner with correlation rules\n",
|
||||
"reasoner = GraphReasoner(kg)\n",
|
||||
"correlations = reasoner.find_correlations(\n",
|
||||
" source_types=[\"Sentiment\"],\n",
|
||||
" target_types=[\"Price\"]\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Add correlation rules\n",
|
||||
"correlation_rules = [\n",
|
||||
" \"IF sentiment = positive THEN price_change > 0\",\n",
|
||||
" \"IF sentiment = negative THEN price_change < 0\",\n",
|
||||
" \"IF sentiment_score > 0.7 THEN strong_positive_correlation\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for rule in correlation_rules:\n",
|
||||
" reasoner.add_rule(rule)\n",
|
||||
"\n",
|
||||
"# Identify sentiment-price correlations\n",
|
||||
"sentiment_price_rels = [\n",
|
||||
" r for r in kg.get(\"relationships\", []) \n",
|
||||
" if \"correlate\" in str(r.get(\"predicate\", \"\")).lower() or\n",
|
||||
" \"sentiment\" in str(r.get(\"predicate\", \"\")).lower()\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Statistical correlation analysis\n",
|
||||
"import statistics\n",
|
||||
"sentiment_scores = [s.get(\"score\", 0) if isinstance(s, dict) else 0.5 for s in sentiments[:10]]\n",
|
||||
"if sentiment_scores:\n",
|
||||
" avg_sentiment = statistics.mean(sentiment_scores)\n",
|
||||
" print(f\"Average sentiment score: {avg_sentiment:.2f}\")\n",
|
||||
"\n",
|
||||
"# Use GraphAnalytics for pattern detection\n",
|
||||
"analytics = GraphAnalytics(kg)\n",
|
||||
"patterns = analytics.detect_patterns(pattern_type=\"correlation\")\n",
|
||||
"print(f\"Detected {len(patterns)} correlation patterns\")\n",
|
||||
"\n",
|
||||
"print(f\"Correlation analysis: {len(correlations)} sentiment-price correlations found\")\n",
|
||||
"print(f\"Sentiment-price relationships: {len(sentiment_price_rels)} relationships identified\")\n",
|
||||
"print(\"Phase 5 complete: Sentiment-price correlation analysis performed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 6: Price Movement Prediction\n",
|
||||
"\n",
|
||||
"Deduplicate news articles, use enhanced GraphRAG queries, and export predictions.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.deduplication import DuplicateDetector, EntityMerger\n",
|
||||
"from semantica.deduplication.methods import detect_duplicates\n",
|
||||
"from semantica.semantic_extract import Entity\n",
|
||||
"from semantica.export import ExportManager\n",
|
||||
"\n",
|
||||
"# Convert Entity objects to dictionaries\n",
|
||||
"print(f\"Converting {len(financial_entities)} entities to dictionaries...\")\n",
|
||||
"entity_dicts = [{\"name\": e.text, \"type\": e.label, \"start_char\": e.start_char, \"end_char\": e.end_char, \"confidence\": e.confidence} for e in financial_entities]\n"
|
||||
"\n",
|
||||
"# Use batch method for news sentiment analysis (efficient for large datasets)\n",
|
||||
"# keep_most_complete strategy preserves all news article details\n",
|
||||
"print(f\"Detecting duplicates in {len(entity_dicts)} entities using batch method...\")\n",
|
||||
"duplicates = detect_duplicates(entity_dicts, method=\"batch\", similarity_threshold=0.85)\n",
|
||||
"\n",
|
||||
"print(f\"Detected {len(duplicates)} duplicate candidates\")\n",
|
||||
"print(f\"Merging duplicates using keep_most_complete strategy...\")\n",
|
||||
"merger = EntityMerger()\n",
|
||||
"merge_operations = merger.merge_duplicates(entity_dicts, strategy=\"keep_most_complete\", threshold=0.85)\n",
|
||||
"\n",
|
||||
"# Extract merged entities from merge operations\n",
|
||||
"if merge_operations:\n",
|
||||
" resolved_entities = [op.merged_entity for op in merge_operations]\n",
|
||||
" merged_ids = set()\n",
|
||||
" for op in merge_operations:\n",
|
||||
" for source in op.source_entities:\n",
|
||||
" merged_ids.add(source.get(\"id\") or source.get(\"name\"))\n",
|
||||
" for entity in entity_dicts:\n",
|
||||
" entity_id = entity.get(\"id\") or entity.get(\"name\")\n",
|
||||
" if entity_id not in merged_ids:\n",
|
||||
" resolved_entities.append(entity)\n",
|
||||
"else:\n",
|
||||
" resolved_entities = entity_dicts\n",
|
||||
"\n",
|
||||
"# Convert back to Entity objects\n",
|
||||
"print(f\"Converting {len(resolved_entities)} resolved entities back to Entity objects...\")\n",
|
||||
"merged_entities = [\n",
|
||||
" Entity(text=e[\"name\"], label=e[\"type\"], start_char=e.get(\"start_char\", 0), end_char=e.get(\"end_char\", 0), confidence=e.get(\"confidence\", 1.0))\n",
|
||||
" for e in resolved_entities\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"financial_entities = merged_entities\n",
|
||||
"print(f\"Deduplicated {len(entity_dicts)} entities to {len(merged_entities)} unique entities\")\n",
|
||||
"\n",
|
||||
"# Use AgentContext for enhanced GraphRAG queries\n",
|
||||
"query_results = context.query(\n",
|
||||
" \"What is the correlation between positive sentiment and stock price increases?\",\n",
|
||||
" top_k=5\n",
|
||||
")\n",
|
||||
"print(f\"GraphRAG query returned {len(query_results) if query_results else 0} results\")\n",
|
||||
"\n",
|
||||
"# Price movement prediction based on sentiment\n",
|
||||
"predictions = []\n",
|
||||
"for i, sentiment in enumerate(sentiments[:5]):\n",
|
||||
" if isinstance(sentiment, dict):\n",
|
||||
" score = sentiment.get(\"score\", 0)\n",
|
||||
" else:\n",
|
||||
" score = 0.5\n",
|
||||
" \n",
|
||||
" if score > 0.6:\n",
|
||||
" prediction = \"price_increase\"\n",
|
||||
" elif score < 0.4:\n",
|
||||
" prediction = \"price_decrease\"\n",
|
||||
" else:\n",
|
||||
" prediction = \"price_stable\"\n",
|
||||
" \n",
|
||||
" predictions.append({\n",
|
||||
" \"article_id\": i,\n",
|
||||
" \"sentiment_score\": score,\n",
|
||||
" \"predicted_movement\": prediction\n",
|
||||
" })\n",
|
||||
"\n",
|
||||
"print(f\"Generated {len(predictions)} price movement predictions\")\n",
|
||||
"\n",
|
||||
"# Export predictions\n",
|
||||
"export_manager = ExportManager()\n",
|
||||
"export_manager.export_knowledge_graph(\n",
|
||||
" {\"predictions\": predictions},\n",
|
||||
" \"data/price_predictions.json\",\n",
|
||||
" format=\"json\"\n",
|
||||
")\n",
|
||||
"print(\"Exported price predictions to JSON\")\n",
|
||||
"\n",
|
||||
"print(\"Phase 6 complete: Price movement prediction with deduplication\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conflict Detection\n",
|
||||
"\n",
|
||||
"Detect and resolve conflicts in sentiment classifications.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.conflicts import ConflictDetector, ConflictResolver\n",
|
||||
"\n",
|
||||
"# Use type conflict detection for conflicting sentiment classifications\n",
|
||||
"# voting strategy aggregates multiple sentiment analysis sources\n",
|
||||
"conflict_detector = ConflictDetector()\n",
|
||||
"conflict_resolver = ConflictResolver()\n",
|
||||
"\n",
|
||||
"print(f\"Detecting type conflicts in {len(financial_entities)} entities...\")\n",
|
||||
"conflicts = conflict_detector.detect_conflicts(\n",
|
||||
" entities=financial_entities,\n",
|
||||
" relationships=all_relationships,\n",
|
||||
" method=\"type\" # Detect conflicts in entity types/classifications (e.g., sentiment)\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f\"Detected {len(conflicts)} type conflicts\")\n",
|
||||
"\n",
|
||||
"if conflicts:\n",
|
||||
" print(f\"Resolving conflicts using voting strategy...\")\n",
|
||||
" resolved = conflict_resolver.resolve_conflicts(\n",
|
||||
" conflicts,\n",
|
||||
" strategy=\"voting\" # Majority vote from multiple sentiment sources\n",
|
||||
" )\n",
|
||||
" print(f\"Resolved {len(resolved)} conflicts\")\n",
|
||||
"else:\n",
|
||||
" print(\"No conflicts detected\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Phase 7: Visualization & Export\n",
|
||||
"\n",
|
||||
"Visualize the knowledge graph and export to RDF (Turtle), JSON-LD, and HTML reports.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"from semantica.export import ReportGenerator\n",
|
||||
"\n",
|
||||
"# Visualize knowledge graph\n",
|
||||
"visualizer = KGVisualizer()\n",
|
||||
"visualizer.visualize(kg, output_path=\"news_sentiment_kg.html\")\n",
|
||||
"\n",
|
||||
"# Export to RDF (Turtle format)\n",
|
||||
"export_manager = ExportManager()\n",
|
||||
"export_manager.export_knowledge_graph(\n",
|
||||
" kg,\n",
|
||||
" \"data/news_sentiment.ttl\",\n",
|
||||
" format=\"rdf\",\n",
|
||||
" rdf_format=\"turtle\"\n",
|
||||
")\n",
|
||||
"print(\"Exported KG to RDF (Turtle format)\")\n",
|
||||
"\n",
|
||||
"# Export to JSON-LD\n",
|
||||
"export_manager.export_knowledge_graph(\n",
|
||||
" kg,\n",
|
||||
" \"data/news_sentiment.jsonld\",\n",
|
||||
" format=\"json-ld\"\n",
|
||||
")\n",
|
||||
"print(\"Exported KG to JSON-LD\")\n",
|
||||
"\n",
|
||||
"# Generate HTML report\n",
|
||||
"report_generator = ReportGenerator()\n",
|
||||
"report = report_generator.generate_report(\n",
|
||||
" kg,\n",
|
||||
" report_type=\"sentiment_analysis\",\n",
|
||||
" output_path=\"data/sentiment_report.html\",\n",
|
||||
" format=\"html\"\n",
|
||||
")\n",
|
||||
"print(\"Generated HTML sentiment analysis report\")\n",
|
||||
"\n",
|
||||
"print(\"News sentiment analysis complete\")\n",
|
||||
"print(\"Emphasizes: Sentiment extraction, correlation analysis, financial KG, price movement prediction\")\n",
|
||||
"print(\"\\nGenerated outputs:\")\n",
|
||||
"print(\" - news_sentiment_kg.html (visualization)\")\n",
|
||||
"print(\" - data/news_sentiment.ttl (RDF Turtle export)\")\n",
|
||||
"print(\" - data/news_sentiment.jsonld (JSON-LD export)\")\n",
|
||||
"print(\" - data/sentiment_report.html (HTML report)\")\n",
|
||||
"print(\" - data/price_predictions.json (price predictions)\")\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -463,32 +463,6 @@ Real-world examples and end-to-end applications across various industries.
|
||||
|
||||
</div>
|
||||
|
||||
### Trading
|
||||
|
||||
<div class="grid cards" markdown>
|
||||
|
||||
- :material-shield-check: **Risk Assessment**
|
||||
---
|
||||
Assessing trading risks using knowledge graphs, GraphRAG, entity-aware chunking, and portfolio risk modeling with Yahoo Finance API and RSS feeds.
|
||||
|
||||
**Topics**: Risk Management, Portfolio Analysis, GraphRAG, Dependency Analysis
|
||||
|
||||
**Difficulty**: Advanced
|
||||
|
||||
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/01_Risk_Assessment.ipynb)
|
||||
|
||||
- :material-newspaper-variant: **News Sentiment Analysis**
|
||||
---
|
||||
Analyzing news sentiment for trading signals and market predictions using financial RSS feeds, semantic transformer chunking, enhanced GraphRAG, and sentiment extraction.
|
||||
|
||||
**Topics**: Sentiment Analysis, Trading Signals, GraphRAG, Correlation Analysis
|
||||
|
||||
**Difficulty**: Intermediate
|
||||
|
||||
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/02_News_Sentiment_Analysis.ipynb)
|
||||
|
||||
</div>
|
||||
|
||||
### Renewable Energy
|
||||
|
||||
<div class="grid cards" markdown>
|
||||
@@ -519,16 +493,6 @@ Real-world examples and end-to-end applications across various industries.
|
||||
|
||||
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)
|
||||
|
||||
- :material-alert-octagon: **Supply Chain Risk Management**
|
||||
---
|
||||
Managing and mitigating supply chain risks using knowledge graphs with supply chain RSS feeds, conflict detection, and dependency analysis.
|
||||
|
||||
**Topics**: Risk Management, Supply Chain Resilience, Conflict Resolution, Dependency Analysis
|
||||
|
||||
**Difficulty**: Advanced
|
||||
|
||||
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
+2
-2
@@ -307,7 +307,7 @@ Power GraphRAG applications with:
|
||||
|
||||
- :material-chef-hat: **Cookbook**
|
||||
---
|
||||
Interactive tutorials, real-world examples, and **15 domain-specific cookbooks**.
|
||||
Interactive tutorials, real-world examples, and **14 domain-specific cookbooks**.
|
||||
|
||||
[:arrow_right: Explore Cookbook](cookbook.md)
|
||||
|
||||
@@ -443,7 +443,7 @@ print(result.confidence) # 0.98
|
||||
|
||||
- [Getting Started Guide](getting-started.md) - Your first knowledge graph in 5 minutes
|
||||
- [Core Concepts](concepts.md) - Deep dive into knowledge graphs and ontologies
|
||||
- [Cookbook](cookbook.md) - Real-world examples and **15 domain-specific cookbooks**
|
||||
- [Cookbook](cookbook.md) - Real-world examples and **14 domain-specific cookbooks**
|
||||
- [API Reference](reference/core.md) - Complete technical documentation
|
||||
|
||||
---
|
||||
|
||||
@@ -129,14 +129,6 @@ ontology = ontology_gen.generate_from_graph(kg)
|
||||
|
||||
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb)
|
||||
|
||||
- :material-chart-line: **Algorithmic Trading Signals**
|
||||
---
|
||||
Generate alpha by connecting disparate data points.
|
||||
|
||||
**Goal**: Build a graph of companies, supply chains, and global events to identify non-obvious trading signals.
|
||||
|
||||
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/01_Risk_Assessment.ipynb)
|
||||
|
||||
- :material-bitcoin: **Blockchain Analytics**
|
||||
---
|
||||
Trace funds and identify illicit activity.
|
||||
|
||||
Reference in New Issue
Block a user