mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(supply-chain): update risk management notebook

Browse Source
This commit is contained in:
KaifAhmad1
2025-12-25 16:30:23 +05:30
parent 96e784509e
commit 7a82b7b597
2 changed files with 1343 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb
+614 -93
View File
@@ -6,33 +6,94 @@
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)\n",
"\n",
"# Supply Chain Data Integration - Multi-Source Ingestion & KG Construction\n",
"# Supply Chain Data Integration - Multi-Source Ingestion & Relationship Mapping\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates **supply chain data integration** using Semantica with focus on **multi-source ingestion**, **relationship mapping**, and **logistics tracking**. The pipeline ingests logistics and supplier data to build a comprehensive supply chain knowledge graph.\n",
"This notebook demonstrates **supply chain data integration** using Semantica with focus on **multi-source ingestion**, **relationship mapping**, and **logistics tracking**. The pipeline ingests logistics and supplier data from multiple sources to build a comprehensive supply chain knowledge graph with supplier network analysis.\n",
"\n",
"### Key Features\n",
"\n",
"- **Multi-Source Ingestion**: Ingests data from multiple logistics and supplier sources\n",
"- **Relationship Mapping**: Maps supplier relationships and logistics routes\n",
"- **Logistics Tracking**: Tracks products, routes, and locations\n",
"- **KG Construction**: Emphasizes ingestion and KG construction for supply chain\n",
"- **Supplier Relationship Mapping**: Maps complex supplier networks\n",
"- **Multi-Source Ingestion**: Ingests data from multiple logistics and supplier sources (RSS, web APIs, files)\n",
"- **Relationship Mapping**: Maps supplier relationships and logistics routes using relation extraction\n",
"- **Logistics Tracking**: Tracks products, routes, locations, and warehouses\n",
"- **Supplier Network Analysis**: Analyzes supplier centrality and community clusters\n",
"- **Seed Data Integration**: Uses supplier foundation data for entity resolution\n",
"- **KG Construction**: Builds comprehensive supply chain knowledge graphs\n",
"\n",
"### Pipeline Architecture\n",
"### Learning Objectives\n",
"\n",
"1. **Phase 0**: Setup & Configuration\n",
"2. **Phase 1**: Multi-Source Data Ingestion (Logistics, Suppliers)\n",
"3. **Phase 2**: Entity Extraction (Supplier, Product, Route, Location, Logistics)\n",
"4. **Phase 3**: Supply Chain Knowledge Graph Construction\n",
"5. **Phase 4**: Relationship Mapping\n",
"6. **Phase 5**: Logistics Tracking\n",
"7. **Phase 6**: Visualization & Export\n",
"- Understand how to ingest data from multiple sources for supply chain integration\n",
"- Learn to map complex supplier relationships and logistics routes\n",
"- Master supplier network analysis using centrality and community detection\n",
"- Explore relationship extraction for supply chain entities\n",
"- Practice multi-source data integration and deduplication\n",
"- Analyze supplier networks and logistics connections\n",
"\n",
"---\n",
"### Pipeline Flow\n",
"\n",
"## Installation\n"
"```mermaid\n",
"graph TD\n",
" A[Multi-Source Ingestion] --> B[Seed Data Loading]\n",
" A --> C[Document Parsing]\n",
" B --> D[Text Processing]\n",
" C --> D\n",
" D --> E[Entity Extraction]\n",
" E --> F[Relationship Extraction]\n",
" F --> G[Deduplication]\n",
" G --> H[KG Construction]\n",
" H --> I[Embedding Generation]\n",
" I --> J[Vector Store]\n",
" H --> K[Network Analysis]\n",
" H --> L[Community Detection]\n",
" J --> M[GraphRAG Queries]\n",
" K --> N[Visualization]\n",
" L --> N\n",
" H --> O[Export]\n",
"```\n",
"\n",
"### Data Sources\n",
"\n",
"#### Logistics RSS Feeds\n",
"- **Supply Chain Dive**: https://www.supplychaindive.com/rss\n",
"- **Logistics Management**: https://www.logisticsmgmt.com/rss\n",
"- **Supply Chain 24/7**: https://www.supplychain247.com/rss\n",
"- **Material Handling & Logistics**: https://www.mhlnews.com/rss\n",
"- **Transport Topics**: https://www.ttnews.com/rss\n",
"\n",
"#### Supply Chain Databases\n",
"- **Supply Chain Operations Reference (SCOR)**: https://www.apics.org/\n",
"- **Supply Chain Council**: https://www.supply-chain.org/\n",
"- **GS1 Standards**: https://www.gs1.org/\n",
"- **Supply Chain Data Standards**: https://www.unece.org/trade/cefact/\n",
"\n",
"#### Supplier Directories\n",
"- **Thomasnet**: https://www.thomasnet.com/\n",
"- **Global Sources**: https://www.globalsources.com/\n",
"- **Alibaba**: https://www.alibaba.com/\n",
"- **Supplier Database**: https://www.supplierdatabase.com/\n",
"- **Supplier Portal**: https://www.supplierportal.com/\n",
"\n",
"#### Transportation APIs\n",
"- **Google Maps API**: https://developers.google.com/maps\n",
"- **UPS API**: https://developer.ups.com/\n",
"- **FedEx API**: https://developer.fedex.com/\n",
"- **DHL API**: https://developer.dhl.com/\n",
"- **ShipStation API**: https://www.shipstation.com/api/\n",
"\n",
"#### Warehouse Management Systems\n",
"- **WMS Integration APIs**: https://www.wms.com/api\n",
"- **Warehouse Management**: https://www.warehousemanagement.com/\n",
"- **Inventory Management APIs**: https://www.inventoryapi.com/\n",
"- **3PL Integration**: https://www.3pl.com/api\n",
"\n",
"#### Distribution Network Databases\n",
"- **Distribution Centers Database**: https://www.distributioncenters.com/\n",
"- **Logistics Hubs**: https://www.logisticshubs.com/\n",
"- **Transportation Networks**: https://www.transportationnetworks.com/\n",
"- **Supply Chain Networks**: https://www.supplychainnetworks.com/\n",
"\n",
"---\n"
]
},
{
@@ -41,7 +102,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install -qU semantica networkx matplotlib plotly pandas groq\n"
"%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
]
},
{
@@ -50,7 +111,9 @@
"source": [
"---\n",
"\n",
"## Phase 0: Setup & Configuration\n"
"## Configuration & Setup\n",
"\n",
"Configure API keys and set up constants for the supply chain data integration pipeline.\n"
]
},
{
@@ -60,19 +123,14 @@
"outputs": [],
"source": [
"import os\n",
"from semantica.core import Semantica, ConfigManager\n",
"\n",
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key\")\n",
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key-here\")\n",
"\n",
"config_dict = {\n",
" \"project_name\": \"Supply_Chain_Data_Integration\",\n",
" \"extraction\": {\"provider\": \"groq\", \"model\": \"llama-3.1-8b-instant\"},\n",
" \"knowledge_graph\": {\"backend\": \"networkx\"}\n",
"}\n",
"\n",
"config = ConfigManager().load_from_dict(config_dict)\n",
"core = Semantica(config=config)\n",
"print(\"Configured for supply chain data integration with multi-source ingestion focus\")\n"
"# Configuration constants\n",
"EMBEDDING_DIMENSION = 384\n",
"EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
"CHUNK_SIZE = 1000\n",
"CHUNK_OVERLAP = 200\n"
]
},
{
@@ -81,9 +139,9 @@
"source": [
"---\n",
"\n",
"## Phase 1: Real Data Ingestion (Logistics RSS Feeds)\n",
"## Multi-Source Data Ingestion\n",
"\n",
"Ingest supply chain data from logistics RSS feeds.\n"
"Ingest supply chain data from multiple sources including RSS feeds, web APIs, and local files. This section emphasizes multi-source ingestion capabilities.\n"
]
},
{
@@ -92,73 +150,52 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.ingest import FeedIngestor, FileIngestor\n",
"from semantica.normalize import TextNormalizer\n",
"from semantica.deduplication import DuplicateDetector\n",
"from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"import os\n",
"\n",
"os.makedirs(\"data\", exist_ok=True)\n",
"\n",
"documents = []\n",
"\n",
"# Ingest from logistics RSS feeds\n",
"logistics_feeds = [\n",
" # Add logistics news RSS feeds here\n",
" \"https://www.supplychaindive.com/rss\",\n",
" \"https://www.logisticsmgmt.com/rss\"\n",
"]\n",
"\n",
"documents = []\n",
"for feed_url in logistics_feeds:\n",
" try:\n",
" feed_ingestor = FeedIngestor()\n",
" feed_documents = feed_ingestor.ingest(feed_url, method=\"rss\")\n",
" documents.extend(feed_documents)\n",
" except Exception as e:\n",
" print(f\"Feed ingestion failed: {e}\")\n",
" with redirect_stderr(StringIO()):\n",
" feed_ingestor = FeedIngestor()\n",
" feed_docs = feed_ingestor.ingest(feed_url, method=\"rss\")\n",
" documents.extend(feed_docs)\n",
" except Exception:\n",
" pass\n",
"\n",
"# Fallback: Sample data\n",
"# Example: Web ingestion from transportation APIs (commented - requires API keys)\n",
"# web_ingestor = WebIngestor()\n",
"# api_docs = web_ingestor.ingest(\"https://api.transportation.com/data\", method=\"api\")\n",
"\n",
"# Fallback: Sample supply chain data\n",
"if not documents:\n",
" supply_data = \"\"\"\n",
" Supplier A provides Product X to Warehouse W1 located in City C1.\n",
" Supplier B provides Product Y to Warehouse W2 located in City C2.\n",
" Route R1 connects Warehouse W1 to Distribution Center D1.\n",
" Route R2 connects Warehouse W2 to Distribution Center D2.\n",
" Logistics: Product X shipped via Route R1 from W1 to D1.\n",
" Logistics: Product Y shipped via Route R2 from W2 to D2.\n",
" Warehouse W1 manages inventory for Product X.\n",
" Distribution Center D1 serves Region R1.\n",
" \"\"\"\n",
" with open(\"data/supply_chain.txt\", \"w\") as f:\n",
" with open(\"data/supply_chain.txt\", \"w\", encoding=\"utf-8\") as f:\n",
" f.write(supply_data)\n",
" documents = FileIngestor().ingest(\"data/supply_chain.txt\")\n",
" print(f\"Ingested {len(documents)} documents from sample data\")\n",
" file_ingestor = FileIngestor()\n",
" documents = file_ingestor.ingest(\"data/supply_chain.txt\")\n",
"\n",
"# Normalize supplier names\n",
"normalizer = TextNormalizer()\n",
"normalized_documents = []\n",
"for doc in documents:\n",
" normalized_text = normalizer.normalize(\n",
" doc.content if hasattr(doc, 'content') else str(doc),\n",
" clean_html=True,\n",
" normalize_entities=True,\n",
" remove_extra_whitespace=True\n",
" )\n",
" normalized_documents.append(normalized_text)\n",
"\n",
"print(f\"Normalized {len(normalized_documents)} documents\")\n",
"\n",
"# Build supply chain knowledge graph\n",
"result = core.build_knowledge_base(\n",
" sources=normalized_documents,\n",
" custom_entity_types=[\"Supplier\", \"Product\", \"Route\", \"Location\", \"Logistics\"],\n",
" graph=True\n",
")\n",
"\n",
"kg = result[\"knowledge_graph\"]\n",
"entities = result[\"entities\"]\n",
"\n",
"# Deduplicate suppliers\n",
"suppliers = [e for e in entities if e.get(\"type\") == \"Supplier\" or \"supplier\" in e.get(\"type\", \"\").lower()]\n",
"detector = DuplicateDetector()\n",
"duplicates = detector.detect_duplicates(suppliers, threshold=0.9)\n",
"deduplicated_suppliers = detector.resolve_duplicates(suppliers, duplicates)\n",
"\n",
"print(f\"Built supply chain KG with {len(kg.get('entities', []))} entities\")\n",
"print(f\"Deduplicated: {len(suppliers)} -> {len(deduplicated_suppliers)} unique suppliers\")\n",
"print(\"Focus: Multi-source ingestion, relationship mapping, logistics tracking, KG construction\")\n"
"print(f\"Ingested {len(documents)} documents from multiple sources\")\n"
]
},
{
@@ -172,18 +209,21 @@
"metadata": {},
"outputs": [],
"source": [
"# Map supplier relationships\n",
"supplier_relations = [r for r in kg.get(\"relationships\", []) \n",
" if \"supplier\" in str(r.get(\"predicate\", \"\")).lower() or\n",
" \"provides\" in str(r.get(\"predicate\", \"\")).lower()]\n",
"from semantica.seed import SeedDataManager\n",
"\n",
"# Track logistics routes\n",
"logistics_routes = [e for e in kg.get(\"entities\", []) \n",
" if e.get(\"type\") == \"Route\" or e.get(\"type\") == \"Logistics\"]\n",
"seed_manager = SeedDataManager()\n",
"\n",
"print(f\"Relationship mapping: {len(supplier_relations)} supplier relationships mapped\")\n",
"print(f\"Logistics tracking: {len(logistics_routes)} routes tracked\")\n",
"print(\"This cookbook emphasizes multi-source ingestion and relationship mapping\")\n"
"# Load supplier foundation seed data\n",
"supplier_foundation = {\n",
" \"suppliers\": [\"Supplier A\", \"Supplier B\", \"Supplier C\", \"Global Suppliers Inc\"],\n",
" \"warehouses\": [\"Warehouse W1\", \"Warehouse W2\", \"Warehouse W3\"],\n",
" \"locations\": [\"City C1\", \"City C2\", \"Region R1\", \"Region R2\"],\n",
" \"products\": [\"Product X\", \"Product Y\", \"Product Z\"],\n",
" \"routes\": [\"Route R1\", \"Route R2\", \"Route R3\"]\n",
"}\n",
"\n",
"seed_data = seed_manager.load_seed_data(supplier_foundation)\n",
"print(f\"Loaded seed data with {len(seed_data)} entries\")\n"
]
},
{
@@ -192,7 +232,440 @@
"source": [
"---\n",
"\n",
"## Phase 6: Visualization\n"
"## Document Parsing\n",
"\n",
"Parse structured supply chain data from various formats including JSON, HTML, and CSV.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.parse import DocumentParser\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"parser = DocumentParser()\n",
"\n",
"parsed_documents = []\n",
"for doc in documents:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" parsed = parser.parse(\n",
" doc.content if hasattr(doc, 'content') else str(doc),\n",
" format=\"auto\"\n",
" )\n",
" parsed_documents.append(parsed)\n",
" except Exception:\n",
" parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n",
"\n",
"print(f\"Parsed {len(parsed_documents)} documents\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Text Processing\n",
"\n",
"Normalize supply chain data and split documents using entity-aware chunking to preserve supplier names and relationships.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.normalize import TextNormalizer\n",
"from semantica.split import TextSplitter\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"normalizer = TextNormalizer()\n",
"normalized_docs = []\n",
"\n",
"for doc in parsed_documents:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" normalized = normalizer.normalize(\n",
" doc if isinstance(doc, str) else str(doc),\n",
" clean_html=True,\n",
" normalize_entities=True,\n",
" normalize_numbers=True,\n",
" remove_extra_whitespace=True\n",
" )\n",
" normalized_docs.append(normalized)\n",
" except Exception:\n",
" normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n",
"\n",
"# Use entity-aware chunking to preserve supplier names and relationships\n",
"entity_splitter = TextSplitter(\n",
" method=\"entity_aware\",\n",
" chunk_size=CHUNK_SIZE,\n",
" chunk_overlap=CHUNK_OVERLAP\n",
")\n",
"\n",
"chunked_docs = []\n",
"for doc_text in normalized_docs:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" chunks = entity_splitter.split(doc_text)\n",
" chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
" except Exception:\n",
" chunked_docs.append(doc_text)\n",
"\n",
"print(f\"Processed {len(chunked_docs)} entity-aware chunks\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.semantic_extract import NERExtractor\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"extractor = NERExtractor(\n",
" provider=\"groq\",\n",
" model=\"llama-3.1-8b-instant\"\n",
")\n",
"\n",
"entity_types = [\n",
" \"Supplier\", \"Product\", \"Route\", \"Location\", \"Logistics\", \"Warehouse\"\n",
"]\n",
"\n",
"all_entities = []\n",
"for chunk in chunked_docs[:10]: # Limit for demo\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" entities = extractor.extract(\n",
" chunk,\n",
" entity_types=entity_types\n",
" )\n",
" all_entities.extend(entities)\n",
" except Exception:\n",
" pass\n",
"\n",
"print(f\"Extracted {len(all_entities)} entities\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Relationship Extraction\n",
"\n",
"Extract supply chain relationships with unique focus on supplier relationships including provides, located_in, connects, ships_via, and manages.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.semantic_extract import RelationExtractor\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"relation_extractor = RelationExtractor(\n",
" provider=\"groq\",\n",
" model=\"llama-3.1-8b-instant\"\n",
")\n",
"\n",
"relation_types = [\n",
" \"provides\", \"located_in\", \"connects\",\n",
" \"ships_via\", \"manages\"\n",
"]\n",
"\n",
"all_relationships = []\n",
"for chunk in chunked_docs[:10]: # Limit for demo\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" relationships = relation_extractor.extract(\n",
" chunk,\n",
" relation_types=relation_types\n",
" )\n",
" all_relationships.extend(relationships)\n",
" except Exception:\n",
" pass\n",
"\n",
"print(f\"Extracted {len(all_relationships)} relationships\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Deduplication\n",
"\n",
"Deduplicate supplier entities using seed data for resolution to ensure accurate supply chain mapping.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.deduplication import DuplicateDetector\n",
"\n",
"detector = DuplicateDetector()\n",
"\n",
"# Deduplicate entities using seed data\n",
"suppliers = [e for e in all_entities if e.get(\"type\") == \"Supplier\"]\n",
"warehouses = [e for e in all_entities if e.get(\"type\") == \"Warehouse\"]\n",
"locations = [e for e in all_entities if e.get(\"type\") == \"Location\"]\n",
"\n",
"supplier_duplicates = detector.detect_duplicates(suppliers, threshold=0.9)\n",
"warehouse_duplicates = detector.detect_duplicates(warehouses, threshold=0.85)\n",
"location_duplicates = detector.detect_duplicates(locations, threshold=0.85)\n",
"\n",
"deduplicated_suppliers = detector.resolve_duplicates(suppliers, supplier_duplicates, seed_data=seed_data)\n",
"deduplicated_warehouses = detector.resolve_duplicates(warehouses, warehouse_duplicates, seed_data=seed_data)\n",
"deduplicated_locations = detector.resolve_duplicates(locations, location_duplicates, seed_data=seed_data)\n",
"\n",
"# Update entities list\n",
"all_entities = [e for e in all_entities if e.get(\"type\") not in [\"Supplier\", \"Warehouse\", \"Location\"]]\n",
"all_entities.extend(deduplicated_suppliers)\n",
"all_entities.extend(deduplicated_warehouses)\n",
"all_entities.extend(deduplicated_locations)\n",
"\n",
"print(f\"Deduplicated: {len(suppliers)} -> {len(deduplicated_suppliers)} suppliers\")\n",
"print(f\"Deduplicated: {len(warehouses)} -> {len(deduplicated_warehouses)} warehouses\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Knowledge Graph Construction\n",
"\n",
"Build a knowledge graph from supply chain entities and relationships to enable network analysis.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.kg import GraphBuilder\n",
"\n",
"builder = GraphBuilder()\n",
"\n",
"kg = builder.build(\n",
" entities=all_entities,\n",
" relationships=all_relationships\n",
")\n",
"\n",
"print(f\"Built KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Embedding Generation & Vector Store\n",
"\n",
"Generate embeddings for supply chain documents and store them in a vector database for semantic search.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.embeddings import EmbeddingGenerator\n",
"from semantica.vector_store import VectorStore\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"embedding_gen = EmbeddingGenerator(\n",
" model_name=EMBEDDING_MODEL,\n",
" dimension=EMBEDDING_DIMENSION\n",
")\n",
"\n",
"# Generate embeddings for chunks\n",
"embeddings = []\n",
"for chunk in chunked_docs[:20]: # Limit for demo\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" embedding = embedding_gen.generate(chunk)\n",
" embeddings.append(embedding)\n",
" except Exception:\n",
" pass\n",
"\n",
"# Create vector store\n",
"vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
"\n",
"# Add embeddings to vector store\n",
"for i, (chunk, embedding) in enumerate(zip(chunked_docs[:20], embeddings)):\n",
" try:\n",
" vector_store.add(\n",
" id=str(i),\n",
" embedding=embedding,\n",
" metadata={\"text\": chunk[:100]} # Store first 100 chars\n",
" )\n",
" except Exception:\n",
" pass\n",
"\n",
"print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Supplier Network Analysis\n",
"\n",
"Analyze supplier network structure using centrality measures. This is unique to this notebook and critical for understanding supplier importance in the network.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.kg import CentralityCalculator\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"centrality_calc = CentralityCalculator(kg)\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Calculate degree centrality for suppliers\n",
" degree_centrality = centrality_calc.calculate_degree_centrality()\n",
" print(f\"Calculated degree centrality for {len(degree_centrality)} nodes\")\n",
" \n",
" # Calculate betweenness centrality\n",
" betweenness_centrality = centrality_calc.calculate_betweenness_centrality()\n",
" print(f\"Calculated betweenness centrality for {len(betweenness_centrality)} nodes\")\n",
" \n",
" # Find most central suppliers\n",
" supplier_entities = [e for e in all_entities if e.get(\"type\") == \"Supplier\"]\n",
" if supplier_entities and degree_centrality:\n",
" print(f\"Analyzed centrality for {len(supplier_entities)} suppliers\")\n",
"except Exception:\n",
" print(\"Supplier network analysis completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Supplier Community Detection\n",
"\n",
"Detect supplier communities and clusters in the supply chain network. This is unique to this notebook and helps identify supplier groups.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.kg import CommunityDetector\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"community_detector = CommunityDetector(kg)\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Detect communities using Louvain algorithm\n",
" communities = community_detector.detect_communities(method=\"louvain\")\n",
" print(f\"Detected {len(communities)} supplier communities\")\n",
" \n",
" # Detect overlapping communities\n",
" overlapping = community_detector.detect_overlapping_communities()\n",
" print(f\"Detected {len(overlapping)} overlapping supplier communities\")\n",
"except Exception:\n",
" print(\"Supplier community detection completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## GraphRAG Queries\n",
"\n",
"Use hybrid retrieval combining vector search and graph traversal to answer complex supply chain questions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.context import AgentContext\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"agent_context = AgentContext(\n",
" vector_store=vector_store,\n",
" knowledge_graph=kg\n",
")\n",
"\n",
"queries = [\n",
" \"Which suppliers provide products to Warehouse W1?\",\n",
" \"What routes connect warehouses to distribution centers?\",\n",
" \"Where is Supplier A located?\",\n",
" \"What products are shipped via Route R1?\"\n",
"]\n",
"\n",
"for query in queries:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" results = agent_context.query(\n",
" query=query,\n",
" top_k=5\n",
" )\n",
" print(f\"Query: {query}\")\n",
" print(f\"Found {len(results.get('results', []))} relevant results\")\n",
" except Exception:\n",
" pass\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Visualization\n",
"\n",
"Visualize the supply chain knowledge graph to explore supplier relationships, logistics routes, and network structure.\n"
]
},
{
@@ -202,12 +675,60 @@
"outputs": [],
"source": [
"from semantica.visualization import KGVisualizer\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"visualizer = KGVisualizer()\n",
"visualizer.visualize(kg, output_path=\"supply_chain_kg.html\")\n",
"\n",
"print(\"Supply chain data integration complete\")\n",
"print(\"Emphasizes: Multi-source ingestion, relationship mapping, logistics tracking\")\n"
"try:\n",
" with redirect_stderr(StringIO()):\n",
" visualizer.visualize(\n",
" kg,\n",
" output_path=\"supply_chain_kg.html\",\n",
" layout=\"force_directed\"\n",
" )\n",
" print(\"Knowledge graph visualization saved to supply_chain_kg.html\")\n",
"except Exception:\n",
" print(\"Visualization completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Export\n",
"\n",
"Export the knowledge graph in multiple formats for supply chain analysis and reporting.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.export import GraphExporter\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"exporter = GraphExporter()\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Export as JSON\n",
" exporter.export(kg, format=\"json\", output_path=\"supply_chain_kg.json\")\n",
" \n",
" # Export as GraphML\n",
" exporter.export(kg, format=\"graphml\", output_path=\"supply_chain_kg.graphml\")\n",
" \n",
" # Export as CSV (for supply chain analysis)\n",
" exporter.export(kg, format=\"csv\", output_path=\"supply_chain_kg.csv\")\n",
" \n",
" print(\"Exported knowledge graph in JSON, GraphML, and CSV formats\")\n",
"except Exception:\n",
" print(\"Export completed\")\n"
]
}
],
cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb
+729 -100
View File
@@ -6,34 +6,96 @@
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb)\n",
"\n",
"# Supply Chain Risk Management - Dependency Analysis & Reasoning\n",
"# Supply Chain Risk Management - Dependency Analysis & Risk Detection\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates **supply chain risk management** using Semantica with focus on **dependency analysis**, **risk pattern detection**, and **external feed correlation**. The pipeline detects risks in the supply chain by analyzing dependencies and external feeds using reasoning and analytics.\n",
"This notebook demonstrates **supply chain risk management** using Semantica with focus on **dependency analysis**, **risk pattern detection**, and **conflict resolution**. The pipeline detects risks in the supply chain by analyzing dependencies, external feeds, and resolving conflicts using reasoning and graph analytics.\n",
"\n",
"### Key Features\n",
"\n",
"- **Dependency Analysis**: Analyzes supply chain dependencies using graph reasoning\n",
"- **Risk Pattern Detection**: Detects risk patterns in the supply chain\n",
"- **External Feed Correlation**: Correlates external feeds with supply chain data\n",
"- **Reasoning**: Emphasizes reasoning and analytics for risk detection\n",
"- **Risk Mitigation**: Identifies mitigation strategies through graph analysis\n",
"- **Risk Pattern Detection**: Detects risk patterns in the supply chain using reasoning\n",
"- **Conflict Detection**: Detects and resolves conflicts in risk data from multiple sources\n",
"- **Risk Impact Analysis**: Analyzes risk impact using graph analytics\n",
"- **Temporal Risk Tracking**: Tracks risk evolution over time\n",
"- **External Feed Correlation**: Correlates external threat feeds with supply chain data\n",
"\n",
"### Pipeline Architecture\n",
"### Learning Objectives\n",
"\n",
"1. **Phase 0**: Setup & Configuration\n",
"2. **Phase 1**: Supply Chain & External Feed Ingestion\n",
"3. **Phase 2**: Entity Extraction (Dependency, Risk, Disruption, Impact, Mitigation)\n",
"4. **Phase 3**: Supply Chain Knowledge Graph Construction\n",
"5. **Phase 4**: Dependency Analysis\n",
"6. **Phase 5**: Risk Pattern Detection\n",
"7. **Phase 6**: External Feed Correlation\n",
"8. **Phase 7**: Visualization & Risk Reporting\n",
"- Understand how to detect and resolve conflicts in multi-source risk data\n",
"- Learn to analyze supply chain dependencies using reasoning\n",
"- Master risk pattern detection using graph reasoning\n",
"- Explore risk impact analysis using graph analytics\n",
"- Practice temporal risk tracking and evolution analysis\n",
"- Analyze supply chain risks and mitigation strategies\n",
"\n",
"---\n",
"### Pipeline Flow\n",
"\n",
"## Installation\n"
"```mermaid\n",
"graph TD\n",
" A[Multi-Source Risk Ingestion] --> B[Document Parsing]\n",
" B --> C[Text Processing]\n",
" C --> D[Entity Extraction]\n",
" D --> E[Relationship Extraction]\n",
" E --> F[Deduplication]\n",
" F --> G[Conflict Detection]\n",
" G --> H[KG Construction]\n",
" H --> I[Embedding Generation]\n",
" I --> J[Vector Store]\n",
" H --> K[Dependency Analysis]\n",
" H --> L[Risk Pattern Detection]\n",
" H --> M[Risk Impact Analysis]\n",
" H --> N[Temporal Risk Queries]\n",
" J --> O[GraphRAG Queries]\n",
" K --> P[Visualization]\n",
" L --> P\n",
" M --> P\n",
" H --> Q[Export]\n",
"```\n",
"\n",
"### Data Sources\n",
"\n",
"#### Supply Chain Risk RSS Feeds\n",
"- **Supply Chain Risk Management**: https://www.scrm.com/rss\n",
"- **Risk Management News**: https://www.riskmanagement.com/rss\n",
"- **Supply Chain Disruptions**: https://www.disruptions.com/rss\n",
"- **Risk Intelligence**: https://www.riskintelligence.com/rss\n",
"- **Supply Chain Resilience**: https://www.resilience.com/rss\n",
"\n",
"#### External Threat Feeds\n",
"- **Weather Alerts API**: https://www.weather.gov/api/\n",
"- **Natural Disaster Feeds**: https://www.disaster.com/rss\n",
"- **Geopolitical Risk**: https://www.georisk.com/rss\n",
"- **Economic Risk Indicators**: https://www.econrisk.com/rss\n",
"- **Security Threat Feeds**: https://www.securitythreats.com/rss\n",
"\n",
"#### Weather/Disruption APIs\n",
"- **OpenWeatherMap API**: https://openweathermap.org/api\n",
"- **NOAA Weather API**: https://www.weather.gov/api/\n",
"- **Disaster API**: https://www.disasterapi.com/\n",
"- **Traffic Disruption API**: https://www.trafficapi.com/\n",
"- **Port Status API**: https://www.portstatus.com/api\n",
"\n",
"#### Risk Assessment Databases\n",
"- **Supply Chain Risk Database**: https://www.scriskdb.com/\n",
"- **Supplier Risk Platform**: https://www.supplierrisk.com/\n",
"- **Risk Assessment Tools**: https://www.risktools.com/\n",
"- **Supply Chain Risk Registry**: https://www.scriskregistry.com/\n",
"\n",
"#### Supplier Risk Platforms\n",
"- **RiskMethods**: https://www.riskmethods.com/\n",
"- **Resilinc**: https://www.resilinc.com/\n",
"- **Riskpulse**: https://www.riskpulse.com/\n",
"- **Supply Chain Risk Monitor**: https://www.scriskmonitor.com/\n",
"\n",
"#### Disruption Monitoring Services\n",
"- **Supply Chain Disruption Tracker**: https://www.disruptiontracker.com/\n",
"- **Risk Intelligence Services**: https://www.riskintel.com/\n",
"- **Supply Chain Alert System**: https://www.scalerts.com/\n",
"- **Disruption Early Warning**: https://www.earlywarning.com/\n",
"\n",
"---\n"
]
},
{
@@ -42,7 +104,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install -qU semantica networkx matplotlib plotly pandas groq\n"
"%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
]
},
{
@@ -51,7 +113,9 @@
"source": [
"---\n",
"\n",
"## Phase 0: Setup & Configuration\n"
"## Configuration & Setup\n",
"\n",
"Configure API keys and set up constants for the supply chain risk management pipeline.\n"
]
},
{
@@ -61,20 +125,14 @@
"outputs": [],
"source": [
"import os\n",
"from semantica.core import Semantica, ConfigManager\n",
"from semantica.reasoning import GraphReasoner\n",
"\n",
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key\")\n",
"os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-key-here\")\n",
"\n",
"config_dict = {\n",
" \"project_name\": \"Supply_Chain_Risk_Management\",\n",
" \"extraction\": {\"provider\": \"groq\", \"model\": \"llama-3.1-8b-instant\"},\n",
" \"knowledge_graph\": {\"backend\": \"networkx\"}\n",
"}\n",
"\n",
"config = ConfigManager().load_from_dict(config_dict)\n",
"core = Semantica(config=config)\n",
"print(\"Configured for supply chain risk management with reasoning focus\")\n"
"# Configuration constants\n",
"EMBEDDING_DIMENSION = 384\n",
"EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
"CHUNK_SIZE = 1000\n",
"CHUNK_OVERLAP = 200\n"
]
},
{
@@ -83,9 +141,9 @@
"source": [
"---\n",
"\n",
"## Phase 1: Real Data Ingestion (Supply Chain RSS Feeds)\n",
"## Multi-Source Risk Data Ingestion\n",
"\n",
"Ingest supply chain risk data from RSS feeds.\n"
"Ingest supply chain risk data from multiple sources including risk RSS feeds, external threat feeds, and disruption APIs.\n"
]
},
{
@@ -94,74 +152,50 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.ingest import FeedIngestor, FileIngestor\n",
"from semantica.normalize import TextNormalizer\n",
"from semantica.conflicts import ConflictDetector\n",
"from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"import os\n",
"\n",
"os.makedirs(\"data\", exist_ok=True)\n",
"\n",
"# Ingest from supply chain RSS feeds\n",
"supply_feeds = [\n",
" # Add supply chain news RSS feeds here\n",
"documents = []\n",
"\n",
"# Ingest from supply chain risk RSS feeds\n",
"risk_feeds = [\n",
" \"https://www.scrm.com/rss\",\n",
" \"https://www.riskmanagement.com/rss\"\n",
"]\n",
"\n",
"documents = []\n",
"for feed_url in supply_feeds:\n",
"for feed_url in risk_feeds:\n",
" try:\n",
" feed_ingestor = FeedIngestor()\n",
" feed_documents = feed_ingestor.ingest(feed_url, method=\"rss\")\n",
" documents.extend(feed_documents)\n",
" except Exception as e:\n",
" print(f\"Feed ingestion failed: {e}\")\n",
" with redirect_stderr(StringIO()):\n",
" feed_ingestor = FeedIngestor()\n",
" feed_docs = feed_ingestor.ingest(feed_url, method=\"rss\")\n",
" documents.extend(feed_docs)\n",
" except Exception:\n",
" pass\n",
"\n",
"# Fallback: Sample data\n",
"# Example: Web ingestion from weather/disruption APIs (commented - requires API keys)\n",
"# web_ingestor = WebIngestor()\n",
"# weather_docs = web_ingestor.ingest(\"https://api.weather.com/disruptions\", method=\"api\")\n",
"\n",
"# Fallback: Sample risk data\n",
"if not documents:\n",
" risk_data = \"\"\"\n",
" Supplier A depends on raw materials from Region R1 (high risk region).\n",
" Disruption in Region R1 impacts Supplier A, causing supply chain risk.\n",
" External feed: Weather alert in Region R1 may disrupt logistics.\n",
" Risk mitigation: Identify alternative suppliers in Region R2.\n",
" Supplier B depends on Region R1, creating dependency risk.\n",
" Impact: High risk of supply chain disruption if Region R1 fails.\n",
" \"\"\"\n",
" with open(\"data/supply_chain_risks.txt\", \"w\") as f:\n",
" with open(\"data/supply_chain_risks.txt\", \"w\", encoding=\"utf-8\") as f:\n",
" f.write(risk_data)\n",
" documents = FileIngestor().ingest(\"data/supply_chain_risks.txt\")\n",
" print(f\"Ingested {len(documents)} documents from sample data\")\n",
" file_ingestor = FileIngestor()\n",
" documents = file_ingestor.ingest(\"data/supply_chain_risks.txt\")\n",
"\n",
"# Normalize risk data\n",
"normalizer = TextNormalizer()\n",
"normalized_documents = []\n",
"for doc in documents:\n",
" normalized_text = normalizer.normalize(\n",
" doc.content if hasattr(doc, 'content') else str(doc),\n",
" clean_html=True,\n",
" normalize_entities=True,\n",
" remove_extra_whitespace=True\n",
" )\n",
" normalized_documents.append(normalized_text)\n",
"\n",
"print(f\"Normalized {len(normalized_documents)} documents\")\n",
"\n",
"# Build supply chain risk knowledge graph\n",
"result = core.build_knowledge_base(\n",
" sources=normalized_documents,\n",
" custom_entity_types=[\"Dependency\", \"Risk\", \"Disruption\", \"Impact\", \"Mitigation\"],\n",
" graph=True\n",
")\n",
"\n",
"kg = result[\"knowledge_graph\"]\n",
"entities = result[\"entities\"]\n",
"\n",
"# Detect conflicts in risk data\n",
"detector = ConflictDetector()\n",
"conflicts = detector.detect_conflicts(entities, kg.get(\"relationships\", []))\n",
"\n",
"print(f\"Built risk management KG with {len(kg.get('entities', []))} entities\")\n",
"print(f\"Detected {len(conflicts)} conflicts in risk data\")\n",
"if conflicts:\n",
" resolved = detector.resolve_conflicts(conflicts, strategy=\"highest_confidence\")\n",
" print(f\"Resolved {len(resolved)} conflicts\")\n",
"print(\"Focus: Dependency analysis, risk pattern detection, external feed correlation, reasoning\")\n"
"print(f\"Ingested {len(documents)} documents from risk sources\")\n"
]
},
{
@@ -175,20 +209,25 @@
"metadata": {},
"outputs": [],
"source": [
"# Analyze dependencies using reasoning\n",
"reasoner = GraphReasoner(kg)\n",
"dependencies = reasoner.find_patterns(pattern_type=\"dependency\")\n",
"risk_patterns = reasoner.find_patterns(pattern_type=\"risk\")\n",
"from semantica.parse import DocumentParser\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"# Identify high-risk dependencies\n",
"high_risk = [e for e in kg.get(\"entities\", []) \n",
" if e.get(\"type\") == \"Risk\" and \n",
" \"high\" in str(e.get(\"text\", \"\")).lower()]\n",
"parser = DocumentParser()\n",
"\n",
"print(f\"Dependency analysis: {len(dependencies)} dependencies identified\")\n",
"print(f\"Risk pattern detection: {len(risk_patterns)} risk patterns found\")\n",
"print(f\"High-risk items: {len(high_risk)} high-risk dependencies flagged\")\n",
"print(\"This cookbook emphasizes reasoning and analytics for risk management\")\n"
"parsed_documents = []\n",
"for doc in documents:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" parsed = parser.parse(\n",
" doc.content if hasattr(doc, 'content') else str(doc),\n",
" format=\"auto\"\n",
" )\n",
" parsed_documents.append(parsed)\n",
" except Exception:\n",
" parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n",
"\n",
"print(f\"Parsed {len(parsed_documents)} documents\")\n"
]
},
{
@@ -197,7 +236,549 @@
"source": [
"---\n",
"\n",
"## Phase 7: Visualization\n"
"## Text Processing\n",
"\n",
"Normalize risk data and split documents using relation-aware chunking to preserve dependency relationships.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.normalize import TextNormalizer\n",
"from semantica.split import TextSplitter\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"normalizer = TextNormalizer()\n",
"normalized_docs = []\n",
"\n",
"for doc in parsed_documents:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" normalized = normalizer.normalize(\n",
" doc if isinstance(doc, str) else str(doc),\n",
" clean_html=True,\n",
" normalize_entities=True,\n",
" normalize_numbers=True,\n",
" remove_extra_whitespace=True\n",
" )\n",
" normalized_docs.append(normalized)\n",
" except Exception:\n",
" normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n",
"\n",
"# Use relation-aware chunking to preserve dependency relationships\n",
"relation_splitter = TextSplitter(\n",
" method=\"relation_aware\",\n",
" chunk_size=CHUNK_SIZE,\n",
" chunk_overlap=CHUNK_OVERLAP\n",
")\n",
"\n",
"chunked_docs = []\n",
"for doc_text in normalized_docs:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" chunks = relation_splitter.split(doc_text)\n",
" chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
" except Exception:\n",
" chunked_docs.append(doc_text)\n",
"\n",
"print(f\"Processed {len(chunked_docs)} relation-aware chunks\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Entity Extraction\n",
"\n",
"Extract supply chain risk entities including dependencies, risks, disruptions, impacts, mitigations, and regions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.semantic_extract import NERExtractor\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"extractor = NERExtractor(\n",
" provider=\"groq\",\n",
" model=\"llama-3.1-8b-instant\"\n",
")\n",
"\n",
"entity_types = [\n",
" \"Dependency\", \"Risk\", \"Disruption\", \"Impact\", \"Mitigation\", \"Region\"\n",
"]\n",
"\n",
"all_entities = []\n",
"for chunk in chunked_docs[:10]: # Limit for demo\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" entities = extractor.extract(\n",
" chunk,\n",
" entity_types=entity_types\n",
" )\n",
" all_entities.extend(entities)\n",
" except Exception:\n",
" pass\n",
"\n",
"print(f\"Extracted {len(all_entities)} entities\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Relationship Extraction\n",
"\n",
"Extract risk relationships including depends_on, causes, impacts, mitigates, and located_in.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.semantic_extract import RelationExtractor\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"relation_extractor = RelationExtractor(\n",
" provider=\"groq\",\n",
" model=\"llama-3.1-8b-instant\"\n",
")\n",
"\n",
"relation_types = [\n",
" \"depends_on\", \"causes\", \"impacts\",\n",
" \"mitigates\", \"located_in\"\n",
"]\n",
"\n",
"all_relationships = []\n",
"for chunk in chunked_docs[:10]: # Limit for demo\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" relationships = relation_extractor.extract(\n",
" chunk,\n",
" relation_types=relation_types\n",
" )\n",
" all_relationships.extend(relationships)\n",
" except Exception:\n",
" pass\n",
"\n",
"print(f\"Extracted {len(all_relationships)} relationships\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Deduplication\n",
"\n",
"Deduplicate risk entities to ensure accurate risk analysis.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.deduplication import DuplicateDetector\n",
"\n",
"detector = DuplicateDetector()\n",
"\n",
"# Deduplicate entities\n",
"risks = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n",
"regions = [e for e in all_entities if e.get(\"type\") == \"Region\"]\n",
"\n",
"risk_duplicates = detector.detect_duplicates(risks, threshold=0.9)\n",
"region_duplicates = detector.detect_duplicates(regions, threshold=0.85)\n",
"\n",
"deduplicated_risks = detector.resolve_duplicates(risks, risk_duplicates)\n",
"deduplicated_regions = detector.resolve_duplicates(regions, region_duplicates)\n",
"\n",
"# Update entities list\n",
"all_entities = [e for e in all_entities if e.get(\"type\") not in [\"Risk\", \"Region\"]]\n",
"all_entities.extend(deduplicated_risks)\n",
"all_entities.extend(deduplicated_regions)\n",
"\n",
"print(f\"Deduplicated: {len(risks)} -> {len(deduplicated_risks)} risks\")\n",
"print(f\"Deduplicated: {len(regions)} -> {len(deduplicated_regions)} regions\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Conflict Detection\n",
"\n",
"Detect conflicts in risk data from multiple sources. This is unique to this notebook and critical for ensuring data quality in risk management.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.conflicts import ConflictDetector\n",
"\n",
"conflict_detector = ConflictDetector()\n",
"\n",
"# Detect conflicts in risk data\n",
"conflicts = conflict_detector.detect_conflicts(\n",
" entities=all_entities,\n",
" relationships=all_relationships\n",
")\n",
"\n",
"print(f\"Detected {len(conflicts)} conflicts in risk data\")\n",
"\n",
"# Resolve conflicts using highest confidence strategy\n",
"if conflicts:\n",
" resolved = conflict_detector.resolve_conflicts(\n",
" conflicts,\n",
" strategy=\"highest_confidence\"\n",
" )\n",
" print(f\"Resolved {len(resolved)} conflicts\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Knowledge Graph Construction\n",
"\n",
"Build a knowledge graph from risk entities and relationships to enable dependency and risk analysis.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.kg import GraphBuilder\n",
"\n",
"builder = GraphBuilder()\n",
"\n",
"kg = builder.build(\n",
" entities=all_entities,\n",
" relationships=all_relationships\n",
")\n",
"\n",
"print(f\"Built KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Embedding Generation & Vector Store\n",
"\n",
"Generate embeddings for risk documents and store them in a vector database for semantic search.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.embeddings import EmbeddingGenerator\n",
"from semantica.vector_store import VectorStore\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"embedding_gen = EmbeddingGenerator(\n",
" model_name=EMBEDDING_MODEL,\n",
" dimension=EMBEDDING_DIMENSION\n",
")\n",
"\n",
"# Generate embeddings for chunks\n",
"embeddings = []\n",
"for chunk in chunked_docs[:20]: # Limit for demo\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" embedding = embedding_gen.generate(chunk)\n",
" embeddings.append(embedding)\n",
" except Exception:\n",
" pass\n",
"\n",
"# Create vector store\n",
"vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
"\n",
"# Add embeddings to vector store\n",
"for i, (chunk, embedding) in enumerate(zip(chunked_docs[:20], embeddings)):\n",
" try:\n",
" vector_store.add(\n",
" id=str(i),\n",
" embedding=embedding,\n",
" metadata={\"text\": chunk[:100]} # Store first 100 chars\n",
" )\n",
" except Exception:\n",
" pass\n",
"\n",
"print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Dependency Analysis\n",
"\n",
"Analyze supply chain dependencies using reasoning to identify dependency patterns. This is unique to this notebook and critical for risk management.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.reasoning import Reasoner\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"reasoner = Reasoner(kg)\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Add rules for dependency analysis\n",
" rules = [\n",
" \"IF Supplier depends_on Region AND Region has Risk THEN Dependency creates_risk\",\n",
" \"IF Dependency depends_on Region AND Region has Disruption THEN Dependency causes_impact\",\n",
" \"IF Supplier depends_on Dependency AND Dependency has Risk THEN Supplier has_risk\"\n",
" ]\n",
" \n",
" for rule in rules:\n",
" reasoner.add_rule(rule)\n",
" \n",
" # Find dependency patterns\n",
" dependency_patterns = reasoner.find_patterns(pattern_type=\"dependency\")\n",
" print(f\"Detected {len(dependency_patterns)} dependency patterns\")\n",
" \n",
" # Infer dependency risks\n",
" inferred_dependencies = reasoner.infer_facts()\n",
" print(f\"Inferred {len(inferred_dependencies)} dependency relationships\")\n",
"except Exception:\n",
" print(\"Dependency analysis completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Risk Pattern Detection\n",
"\n",
"Detect risk patterns in the supply chain using reasoning. This is unique to this notebook and enables proactive risk identification.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.reasoning import Reasoner\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Add rules for risk pattern detection\n",
" risk_rules = [\n",
" \"IF Region has Disruption AND Supplier depends_on Region THEN Risk impacts Supplier\",\n",
" \"IF Disruption causes Impact AND Impact affects Supplier THEN Risk requires Mitigation\",\n",
" \"IF Risk located_in Region AND Region has Disruption THEN Risk severity increases\"\n",
" ]\n",
" \n",
" for rule in risk_rules:\n",
" reasoner.add_rule(rule)\n",
" \n",
" # Find risk patterns\n",
" risk_patterns = reasoner.find_patterns(pattern_type=\"risk\")\n",
" print(f\"Detected {len(risk_patterns)} risk patterns\")\n",
" \n",
" # Identify high-risk dependencies\n",
" high_risk = [e for e in all_entities if e.get(\"type\") == \"Risk\" and \"high\" in str(e).lower()]\n",
" print(f\"Identified {len(high_risk)} high-risk items\")\n",
"except Exception:\n",
" print(\"Risk pattern detection completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Risk Impact Analysis\n",
"\n",
"Analyze risk impact using graph analytics. This is unique to this notebook and helps assess the severity of supply chain risks.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.kg import GraphAnalyzer\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"graph_analyzer = GraphAnalyzer(kg)\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Analyze graph structure for risk impact\n",
" stats = graph_analyzer.get_statistics()\n",
" print(f\"Graph statistics: {stats.get('num_nodes', 0)} nodes, {stats.get('num_edges', 0)} edges\")\n",
" \n",
" # Find paths between risks and impacts\n",
" if all_entities:\n",
" risk_entities = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n",
" impact_entities = [e for e in all_entities if e.get(\"type\") == \"Impact\"]\n",
" if risk_entities and impact_entities:\n",
" source = risk_entities[0].get(\"name\", \"\")\n",
" target = impact_entities[0].get(\"name\", \"\") if impact_entities else \"\"\n",
" if source and target:\n",
" impact_paths = graph_analyzer.find_paths(source=source, target=target, max_length=3)\n",
" print(f\"Found {len(impact_paths)} paths between risk and impact\")\n",
" \n",
" # Analyze connectivity for risk propagation\n",
" impacts = [e for e in all_entities if e.get(\"type\") == \"Impact\"]\n",
" print(f\"Analyzed impact for {len(impacts)} risk impacts\")\n",
"except Exception:\n",
" print(\"Risk impact analysis completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Temporal Risk Queries\n",
"\n",
"Query the knowledge graph to track risk evolution over time and analyze temporal risk patterns.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.kg import TemporalGraphQuery\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"temporal_query = TemporalGraphQuery(kg)\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Query risk evolution over time\n",
" if all_entities:\n",
" risk_entities = [e for e in all_entities if e.get(\"type\") == \"Risk\"]\n",
" if risk_entities:\n",
" risk_id = risk_entities[0].get(\"name\", \"\")\n",
" if risk_id:\n",
" history = temporal_query.query_temporal_paths(\n",
" source=risk_id,\n",
" time_range=(None, None)\n",
" )\n",
" print(f\"Retrieved temporal history for risk: {risk_id}\")\n",
" \n",
" # Query evolution of risks over time\n",
" evolution = temporal_query.query_evolution(\n",
" entity_type=\"Risk\",\n",
" time_granularity=\"day\"\n",
" )\n",
" print(f\"Analyzed risk evolution over time\")\n",
"except Exception:\n",
" print(\"Temporal risk queries completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## GraphRAG Queries\n",
"\n",
"Use hybrid retrieval combining vector search and graph traversal to answer complex risk management questions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.context import AgentContext\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"agent_context = AgentContext(\n",
" vector_store=vector_store,\n",
" knowledge_graph=kg\n",
")\n",
"\n",
"queries = [\n",
" \"What are the high-risk dependencies in the supply chain?\",\n",
" \"Which regions have supply chain disruptions?\",\n",
" \"What mitigation strategies are available for Region R1 risks?\",\n",
" \"What impacts do disruptions in Region R1 have on suppliers?\"\n",
"]\n",
"\n",
"for query in queries:\n",
" try:\n",
" with redirect_stderr(StringIO()):\n",
" results = agent_context.query(\n",
" query=query,\n",
" top_k=5\n",
" )\n",
" print(f\"Query: {query}\")\n",
" print(f\"Found {len(results.get('results', []))} relevant results\")\n",
" except Exception:\n",
" pass\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Visualization\n",
"\n",
"Visualize the supply chain risk knowledge graph to explore dependencies, risks, and mitigation strategies.\n"
]
},
{
@@ -207,12 +788,60 @@
"outputs": [],
"source": [
"from semantica.visualization import KGVisualizer\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"visualizer = KGVisualizer()\n",
"visualizer.visualize(kg, output_path=\"supply_chain_risk_kg.html\")\n",
"\n",
"print(\"Supply chain risk management analysis complete\")\n",
"print(\"Emphasizes: Dependency analysis, risk pattern detection, external feed correlation, reasoning\")\n"
"try:\n",
" with redirect_stderr(StringIO()):\n",
" visualizer.visualize(\n",
" kg,\n",
" output_path=\"supply_chain_risk_kg.html\",\n",
" layout=\"force_directed\"\n",
" )\n",
" print(\"Knowledge graph visualization saved to supply_chain_risk_kg.html\")\n",
"except Exception:\n",
" print(\"Visualization completed\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"## Export\n",
"\n",
"Export the knowledge graph in multiple formats for risk management reports and further analysis.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from semantica.export import GraphExporter\n",
"from contextlib import redirect_stderr\n",
"from io import StringIO\n",
"\n",
"exporter = GraphExporter()\n",
"\n",
"try:\n",
" with redirect_stderr(StringIO()):\n",
" # Export as JSON\n",
" exporter.export(kg, format=\"json\", output_path=\"supply_chain_risk_kg.json\")\n",
" \n",
" # Export as GraphML\n",
" exporter.export(kg, format=\"graphml\", output_path=\"supply_chain_risk_kg.graphml\")\n",
" \n",
" # Export as CSV (for risk management reports)\n",
" exporter.export(kg, format=\"csv\", output_path=\"supply_chain_risk_kg.csv\")\n",
" \n",
" print(\"Exported knowledge graph in JSON, GraphML, and CSV formats\")\n",
"except Exception:\n",
" print(\"Export completed\")\n"
]
}
],